Published on
 // 16 min read

Getting started with RHACS Policy-as-Code

Authors

Red Hat Advanced Cluster Security for Kubernetes (RHACS) 4.6.0 landed in early December. The release included a number of improvements, including support for CSAF-VEX vulnerability data formats, scanner support for Red Hat Enterprise Linux CoreOS (the operating system that underpins OpenShift), and compliance reporting.

This release also included one of the mostly requested features for RHACS - managing security policy-as-code! This allows you to manage all of the securty policies for RHACS declaratively and take advantage of GitOps reconciliation.

Why is this so widely requested? I think there's a number of reasons:

  • DevSecOps workflows:'DevSecOps' is now the de factor standard for building and deploying applications. Policy-as-code makes DevSecOps possible, allowing deveolopers and security teams to collaborate on security policy updates, as well as bringing development tools and skills to security teams (managing code commits, deploying policy via CI/CD pipelines, etc)

  • Consistency and Automation: Using a security policy-as-code approach ensures that policies are applied consistently across environments. Automation through CI/CD pipelines means reduces the risk of human error or inconsistent policy enforcement.

  • Version Control and Auditing: Security policies stored as code can be versioned and tracked in the same way as application code. This provides a clear audit trail, allowing teams to see who made changes, what changes were made, and why. This also aids compliance efforts, as historical versions of policies can be reviewed when needed.

  • Scalability: As organisations grow, managing security policies manually becomes increasingly difficult. With security policies as code, scaling to handle large numbers of environments or resources becomes much easier. The same policies can be applied across multiple accounts, regions, or clusters without duplication of effort.

  • Improved collaboration: Security-as-code encourages cross-functional collaboration. Developers, security teams, and operations can all work from the same framework and tools, making security a shared responsibility. The clear separation of policy definition and implementation also improves communication between teams and fosters a more proactive security culture.

  • Faster changes: With security policies defined in code, updates can be pushed quickly and reliably through automated pipelines. This leads to faster security policy changes, making it easier to adapt to new security threats or compliance requirements without slowing down development cycles.

Policy-as-code approach

Technically, it's always been possible to manage RHACS policies as code using automation and Git Webhooks. One implementation of this could be using Ansible:

  • A change to a security policy is checked into Github
  • Github webhooks are configured to issue notifications to Ansible Automation Platform when changes are made
  • Ansible Automation Platform starts an Ansible playbook that clones a Git repository holding security policies
  • The Ansible playbook then uses the RHACS RESTful API to update security policy, using the latest policies obtained from the Git repo

This works well, and you can see a video of it here in action. Neil Carpenter initially published an implementation of this Ansible / RHACS workflow, and I've uploaded a derivative here: https://github.com/shaneboulden/ansible-acs-policy-demo

But, this isn't a 'standard' approach to GitOps, and there's no reconciliation available. ArgoCD monitors for changes to deployed artifacts or those stored in code, and then 'reconciles' the latest code-base with deployed artifacts. This implementation is missing this critical control-loop.

Instead, the Red Hat Advanced Cluster Security for Kubernetes (RHACS) native policy-as-code implementation is designed to work with ArgoCD. It uses Kubernetes Custom Resource objects (CRs) to represent security policies, and store these in a Git repository. When changes are made to either the code-base or running config, ArgoCD performs reconciliation, using the RHACS Operator to make changes to running Central instance. You can see a diagram showing this process here:

Getting started

Let's get started with Red Hat Advanced Cluster Security for Kubernetes (RHACS) and security policy-as-code. Make sure you have a RHACS Central instance 4.6+ deployed via the operator, and have enrolled at least one cluster. This can also be the local cluster, as shown here (note the internal cluster address for RHACS Central):

operator-install
operator-central-cr
operator-scs-cr
rhacs-dashboard

Now, let's create a new policy. The format for RHACS Policy custom resources is documented here, and shown below:

kind: SecurityPolicy
apiVersion: config.stackrox.io/v1alpha1
metadata:
  name: short-name
spec:
  policyName: A longer form name
# ...

To see the fields available to construct security policy, you can use oc explain securitypolicy.spec:

$ oc explain securitypolicy.spec
GROUP:      config.stackrox.io
KIND:       SecurityPolicy
VERSION:    v1alpha1

FIELD: spec <Object>

DESCRIPTION:
    SecurityPolicySpec defines the desired state of SecurityPolicy
    
FIELDS:
  categories	<[]string> -required-
    Categories is a list of categories that this policy falls under.  Category
    names must already exist in Central.

  criteriaLocked	<boolean>
    Read-only field. If true, the policy's criteria fields are rendered
    read-only.

  description	<string>
    Description is a free-form text description of this policy.

  disabled	<boolean>
    Disabled toggles whether or not this policy will be executing and actively
    firing alerts.

  enforcementActions	<[]string>
    Enforcement lists the enforcement actions to take when a violation from this
    policy is identified.  Possible value are UNSET_ENFORCEMENT,
    SCALE_TO_ZERO_ENFORCEMENT, UNSATISFIABLE_NODE_CONSTRAINT_ENFORCEMENT,
    KILL_POD_ENFORCEMENT, FAIL_BUILD_ENFORCEMENT, FAIL_KUBE_REQUEST_ENFORCEMENT,
    FAIL_DEPLOYMENT_CREATE_ENFORCEMENT, and. FAIL_DEPLOYMENT_UPDATE_ENFORCEMENT.
...

I'm going to create a simple policy that checks for netcat inside container images. I think this is important, because netcat could easily be used by an attacker with access to a pod to access services, and try and move laterally. netcat could also be used with a living off the land

PS. If you're interested in 'living off the land' techniques I ran a webinar in November 2024, which is now available on-demand at redhat.com

A great strategy to minimise this risk is simply removing netcat from container images. Here's a SecurityPolicy object that checks for netcat in images, and ensures we can flag this in CI/CD pipelines, or using the RHACS admission controller.

kind: SecurityPolicy
apiVersion: config.stackrox.io/v1alpha1
metadata:
  name: netcat-in-image
spec:
  policyName: Netcat in Image
  categories:
  - Package Management
  description: 'This policy checks for container images containing netcat'
  disabled: false
  remediation: Use the base image package manager to remove "nmap-ncat"
  lifecycleStages:
  - BUILD
  - DEPLOY
  policySections:
  - sectionName: Rule 1
    policyGroups:
    - fieldName: Image Component
      booleanOperator: OR
      negate: false
      values:
      - value: nmap-ncat=
  rationale: Netcat potentially allows attackers to move laterally, and access external
    services from a running pod
  severity: HIGH_SEVERITY

I've committed this to a repo available here. You can see the policy in the policies directory.

repo

The first step in automating this policy with GitOps is adding the repository to OpenShift GitOps. You can find the route for the OpenShift GitOps UI in the openshift-gitops namespace:

$ oc get routes -n openshift-gitops
NAME                      HOST/PORT                                                                        PATH   SERVICES                  PORT    TERMINATION          WILDCARD
openshift-gitops-server   openshift-gitops-server-openshift-gitops.apps.cluster1.example.com          openshift-gitops-server   https   reencrypt/Redirect   None

Once you've accessed the OpenShift GitOps UI, select Settings and then Repositories.

gitops-repo
gitops-repo1
gitops-repo2

Now that the repo is connected, we can create an ArgoCD application using the repo:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: rhacs-policies
  namespace: openshift-gitops
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    namespace: acs-central
    server:  https://kubernetes.default.svc
  project: default
  source:
    path: policies
    repoURL: https://github.com/shaneboulden/stackrox-policy-as-code.git
    targetRevision: HEAD
  syncPolicy:
    automated: {}

You can check that the application is created and synchronised via the ArgoCD dashboard:

gitops-app

You may see this error:

securitypolicies.config.stackrox.io is forbidden: User "system:serviceaccount:openshift-gitops:openshift-gitops-argocd-application-controller" cannot create resource "securitypolicies" in API group "config.stackrox.io" in the namespace "acs-central"

This a well-documented issue with permissions for the ArgoCD service account, and the recommended solution is to grant the ArgoCD service account the admin role in the target namespace.

oc adm policy add-role-to-user admin system:serviceaccount:openshift-gitops:openshift-gitops-argocd-application-controller -n acs-central

Observe that the ArgoCD application is sync'd, and the policy is now created in the cluster:

gitops-policy11
gitops-policy12
$ oc get securitypolicies -n acs-central
NAME              AGE
netcat-in-image   22h

Finally, you can check that the policy is now created in the RHACS instance. You'll note that the policy is listed as externally managed, and if you try and edit it, you'll get a warning that any changes will be overwritten at the next sync.

gitops-policy1
gitops-policy2
gitops-policy3

Testing out the new policy

Now that we have a new policy, let's test it out! I'm going to run an image check via the roxctl CLI, and check that an image I know is vulnerable fails this policy.

$ roxctl -e "central-acs-central.apps.cluster1.sandbox1177.opentlc.com:443" --ins
ecure-skip-tls-verify=true image check -i quay.io/smileyfritz/chat-client:latest

Policy check results for image: quay.io/smileyfritz/chat-client:latest
(TOTAL: 1, LOW: 0, MEDIUM: 0, HIGH: 1, CRITICAL: 0)

+-----------------+----------+--------------+--------------------------------+--------------------------------+--------------------------------+
|     POLICY      | SEVERITY | BREAKS BUILD |          DESCRIPTION           |           VIOLATION            |          REMEDIATION           |
+-----------------+----------+--------------+--------------------------------+--------------------------------+--------------------------------+
| Netcat in Image |   HIGH   |      -       |     This policy checks for     |   - Image includes component   |   Use the base image package   |
|                 |          |              |  container images containing   |      'nmap-ncat' (version      | manager to remove "nmap-ncat"  |
|                 |          |              |             netcat             |         2:7.92-1.el8)          |                                |
+-----------------+----------+--------------+--------------------------------+--------------------------------+--------------------------------+

Great! This policy that we've created and managed via OpenShift GitOps correctly identifies netcat in Red Hat-based container images.

Updating the policy

Our policy works, currently it isn't 'enforcing'. If you check the output above, the 'BREAKS BUILD' field is a '-' - meaning that this policy won't fail any pipelines. Let's add some enforcement actions to the policy, via GitOps.

Firstly, I want to understand the enforcement actions available. Let's run oc explain securitypolicy.spec again to see the actions available:

$ oc explain securitypolicy.spec
GROUP:      config.stackrox.io
KIND:       SecurityPolicy
VERSION:    v1alpha1

FIELD: spec <Object>

DESCRIPTION:
    SecurityPolicySpec defines the desired state of SecurityPolicy

FIELDS:
...
  enforcementActions    <[]string>
    Enforcement lists the enforcement actions to take when a violation from this
    policy is identified.  Possible value are UNSET_ENFORCEMENT,
    SCALE_TO_ZERO_ENFORCEMENT, UNSATISFIABLE_NODE_CONSTRAINT_ENFORCEMENT,
    KILL_POD_ENFORCEMENT, FAIL_BUILD_ENFORCEMENT, FAIL_KUBE_REQUEST_ENFORCEMENT,
    FAIL_DEPLOYMENT_CREATE_ENFORCEMENT, and. FAIL_DEPLOYMENT_UPDATE_ENFORCEMENT.

There's a few here that I particularly care about:

  • SCALE_TO_ZERO_ENFORCEMENT: This will ensure that pods that are not blocked by the admission controller (e.g. on secure clusters with contactImageScanners set to DoNotScanInline) are still scaled down.

  • FAIL_BUILD_ENFORCEMENT: This will ensure that CI/CD pipelines that include roxctl image check will fail (i.e. return a non-zero return code from roxctl)

  • FAIL_DEPLOYMENT_CREATE_ENFORCEMENT: This will ensure that the admission controller will block deployment creation on secured clusters with contactImageScanners set to ScanIfMissing.

  • FAIL_DEPLOYMENT_UPDATE_ENFORCEMENT: Similarly, any updates to deployments that reference vulnerable images will fail.

Here's our SecurityPolicy with the enforcement actions added:

kind: SecurityPolicy
apiVersion: config.stackrox.io/v1alpha1
metadata:
  name: netcat-in-image
spec:
  policyName: Netcat in Image
  categories:
  - Package Management
  description: 'This policy checks for container images containing netcat'
  disabled: false
  enforcementActions:
    - SCALE_TO_ZERO_ENFORCEMENT
    - FAIL_BUILD_ENFORCEMENT
    - FAIL_DEPLOYMENT_CREATE_ENFORCEMENT
    - FAIL_DEPLOYMENT_UPDATE_ENFORCEMENT
  remediation: Use the base image package manager to remove "nmap-ncat"
  lifecycleStages:
  - BUILD
  - DEPLOY
  policySections:
  - sectionName: Rule 1
    policyGroups:
    - fieldName: Image Component
      booleanOperator: OR
      negate: false
      values:
      - value: nmap-ncat=
  rationale: Netcat potentially allows attackers to move laterally, and access external
    services from a running pod
  severity: HIGH_SEVERITY

I've commited these changes and sync'd the ArgoCD application, and can now see that the SecurityPolicy object is updated:

enforcement1

If I now take a look at the policy in Red Hat Advanced Cluster Security for Kubernetes (RHACS), I can see that the 'enforcement' toggles for 'Build' and 'Deploy' are now enabled:

enforcement2

Let's try out this policy again:

$ roxctl -e "central-acs-central.apps.cluster1.sandbox1177.opentlc.com:443" --insecure-skip-tls-verify=true image check -i quay.io/smileyfritz/chat-client:latest
Policy check results for image: quay.io/smileyfritz/chat-client:latest
(TOTAL: 1, LOW: 0, MEDIUM: 0, HIGH: 1, CRITICAL: 0)

+-----------------+----------+--------------+--------------------------------+--------------------------------+--------------------------------+
|     POLICY      | SEVERITY | BREAKS BUILD |          DESCRIPTION           |           VIOLATION            |          REMEDIATION           |
+-----------------+----------+--------------+--------------------------------+--------------------------------+--------------------------------+
| Netcat in Image |   HIGH   |      X       |     This policy checks for     |   - Image includes component   |   Use the base image package   |
|                 |          |              |  container images containing   |      'nmap-ncat' (version      | manager to remove "nmap-ncat"  |
|                 |          |              |             netcat             |         2:7.92-1.el8)          |                                |
+-----------------+----------+--------------+--------------------------------+--------------------------------+--------------------------------+

Great! Our policy now breaks the build, ensuring that any pipelines referencing images containing netcat will fail at the roxctl image check.

Lowering the barrier to GitOps adoption

GitOps provides scale, consistency and enables collaboration, though it can be challenging to initially adopt these practices and processes.

I'm going to explore this more in another article, but one thing that may help initially is a script to convert Red Hat Advanced Cluster Security for Kubernetes (RHACS) exported policies to SecurityPolicy objects that can be managed with GitOps / ArgoCD.

I've added the export2gitops.py here, which does just this. You can use this utility to convert exported policies, e.g.

$ python export2gitops.py -i sample-policy.json -o netcat-in-image.yaml
✅ YAML output saved to netcat-in-image.yaml

$ cat netcat-in-image.yaml
kind: SecurityPolicy
apiVersion: config.stackrox.io/v1alpha1
metadata:
  name: netcat-in-image
spec:
  policyName: Netcat in Image
  categories:
  - Package Management
  description: 'This policy checks for container images containing netcat '
  disabled: false
  remediation: Use the base image package manager to remove "nmap-ncat"
  lifecycleStages:
  - BUILD
  - DEPLOY
  policySections:
  - sectionName: Rule 1
    policyGroups:
    - fieldName: Image Component
      booleanOperator: OR
      negate: false
      values:
      - value: nmap-ncat=
  rationale: Netcat potentially allows attackers to move laterally, and access external
    services from a running pod
  severity: HIGH_SEVERITY

Wrap up

In this article I've looked at managing security policy-as-code, using OpenShift GitOps and Red Hat Advanced Cluster Security for Kubernetes (RHACS). I looked at how to create new policies via code, and see this reflected in RHACS via ArgoCD workflows.

I also briefly looked at how we can start to lower the barrier to GitOps adoption, noting this can be a complex workflow if you're unfamiliar with Git, and GitOps processes. I'll explore this more in another article.

Discuss on Twitter • View on GitHub