GTFOBins and StackRox

A couple of months ago I wrote an article on StackRox and another open source projcet, GTFOBins. I looked at how you can create a StackRox policy from the GTFOBins source, and use the roxctl CLI to scan container images and identify GTFOBins components.

The catalyst for this article was a Microsoft threat intelligence report on cyber threat actors employing "Living off the Land" techniques, and I covered a high-level overview of strategies to mitigate these threats in another article.

This article builds on the last one, and looks at identifying GTFOBins process execution inside containers at runtime.

Kubernetes runtime security

Runtime security is often the main concern for organisations operating digital services on Kubernetes, and this was highlighted in the Red Hat State of Kubernetes Security reports over the last few years. The 'State of Kubernetes Security' is a report created by Red Hat that examines the security challenges organizations face when it comes to cloud-native development and how they address these challenges to protect their applications and IT environments. The report is based on surveys of more than 300 DevOps, engineering and security professionals, highlighting how companies are adopting containers and Kubernetes while still balancing the security of these environments.

Over the last few years we've seen a pretty consistent interest in runtime security. The 2022 State of Kubernetes security report called this out as the number one concern for organisations:

The 2023 State of Kubernetes security report also highlighted runtime security as a significant concern, with nearly half (49%) of organisations surveyed experiencing a security issue at runtime.

Often I find this is where organisations have the most concerns with cloud-native security. They'd usually push an agent down to protected servers, alerting on anomalous activity or indicators of compromise (IOCs) at runtime. But cloud-native workloads - which are usually running in containers - don't have agents. So how do we monitor workloads at runtime?

One approach that is increasingly being adopted for runtime observability for cloud-native workloads is eBPF. eBPF, or extended Berkeley Packet Filter, is a technology that empowers developers to enhance and introspect the inner workings of the Linux kernel in a safe and efficient manner. Originally designed for network packet filtering, eBPF has evolved into a versatile framework for programmable data plane processing. With eBPF, you can write and load custom programs into the kernel, enabling real-time monitoring, tracing, and dynamic network control without requiring kernel modifications.

eBPF has some really interesting applications for security. Using eBPF we can see and understand all system calls, including system calls from containers. We can also create a packet and socket-level view of network traffic flows, helping to create a baseline of network activity, including network traffic for container applications. This means that we can start to monitor container workloads at runtime, without needing to add additional libraries to the container image, or running an agent on the node.

There is still discussion about whether BPF/eBPF is the right tool for security monitoring, and there's a really good article from Brendan Gregg. Brendan highlights that, while there are some limitations to using observability tools like eBPF for security monitoring, there is also potential for an awesome eBPF security product. I think that StackRox is well on the way there, and it uses eBPF for runtime security monitoring of container workloads running on Kubernetes.

StackRox and runtime process execution

During installation StackRox pushes an eBPF program down to nodes to identify processes executing inside containers at runtime, and also identify network traffic flows inside containers. StackRox leverages the Falco libraries via a custom fork to support eBPF.

StackRox also contains a number of pre-built policies to identify anomalous activity at runtime. You can find the complete list of out-of-the-box policies here, which includes:

Detecting netcat execution: Netcat activity could indicate an attacker trying to discover additional services running within a cluster, or trying to move laterally.
Detecting binaries that can be used to manipulate network configuration and management: This could indicate an attacker trying to sniff passwords from the wire, or otherwise discover network traffic.
Detecting cryptocurrency mining binaries
Detecting shell as a subprocess of a Java application: It's pretty unusual for a Java process to spawn a shell, and this activity should be investigated.

GTFOBins component execution at runtime

Let's look at how we can identify "living off the land" activity at runtime for container workloads. StackRox contains a number of policies that can be used to identify anomalous runtime activity, but I want to create a new policy dedicated to detecting GTFOBins components.

In the last article I introduced a script that walks the GTFOBins source, and generates a StackRox policy that can detect components in container images. When a container is scanned (either by roxctl or in a CI/CD pipeline) the container image components are evaluated, and any GTFOBins components are detected and highlighted.

$ roxctl image check  -e "central-acs-central.apps.cluster1.example.com:443" --insecure-skip-tls-verify --image quay.io/smileyfritz/log4shell-app:v0.5
Policy check results for image: quay.io/smileyfritz/log4shell-app:v0.5
(TOTAL: 4, LOW: 4, MEDIUM: 0, HIGH: 0, CRITICAL: 0)
+--------------------------------+----------+--------------+--------------------------------+--------------------------------+--------------------------------+
|             POLICY             | SEVERITY | BREAKS BUILD |          DESCRIPTION           |           VIOLATION            |          REMEDIATION           |
+--------------------------------+----------+--------------+--------------------------------+--------------------------------+--------------------------------+
|        90-Day Image Age        |   LOW    |      -       |   Alert on deployments with    |     - Image was created at     |   Rebuild your image, push a   |
|                                |          |              |    images that haven't been    |   2022-02-16 03:35:10 (UTC)    | new minor version (with a new  |
|                                |          |              |       updated in 90 days       |                                |   immutable tag), and update   |
|                                |          |              |                                |                                |    your service to use it.     |
+--------------------------------+----------+--------------+--------------------------------+--------------------------------+--------------------------------+
|  Alpine Linux Package Manager  |   LOW    |      -       | Alert on deployments with the  |   - Image includes component   |      Run `apk --purge del      |
|         (apk) in Image         |          |              |  Alpine Linux package manager  |      'apk-tools' (version      | apk-tools` in the image build  |
|                                |          |              |         (apk) present          |           2.10.1-r0)           |   for production containers.   |
+--------------------------------+----------+--------------+--------------------------------+--------------------------------+--------------------------------+
|  Docker CIS 4.1: Ensure That   |   LOW    |      -       |   Containers should run as a   |    - Image has user 'root'     | Ensure that the Dockerfile for |
|  a User for the Container Has  |          |              |         non-root user          |                                |  each container switches from  |
|          Been Created          |          |              |                                |                                |         the root user          |
+--------------------------------+----------+--------------+--------------------------------+--------------------------------+--------------------------------+
|  GTFOBins component in image   |   LOW    |      -       |   Alert on deployments with    |   - Image includes component   |   Use your package manager's   |
|                                |          |              |  GTFOBins components present   | 'busybox' (version 1.28.4-r2)  |      "remove", "purge" or      |
|                                |          |              |                                |                                |   "erase" command to remove    |
|                                |          |              |                                |   - Image includes component   |  GTFOBins components from the  |
|                                |          |              |                                |   'bzip2' (version 1.0.6-r6)   |   image build for production   |
|                                |          |              |                                |                                |          containers.           |
|                                |          |              |                                |   - Image includes component   |                                |
|                                |          |              |                                |    'gcc' (version 6.4.0-r9)    |                                |
|                                |          |              |                                |                                |                                |
|                                |          |              |                                |   - Image includes component   |                                |
|                                |          |              |                                |  'socat' (version 1.7.3.2-r4)  |                                |
+--------------------------------+----------+--------------+--------------------------------+--------------------------------+--------------------------------+

I've extended this script to now also create a runtime policy for StackRox and GTFOBins components, as well as a deployment-time policy. The script is here:

import json
import os
import re

def get_gtfobins(root):
    root, _, files = next(os.walk(root))
    names = []
    for name in files:
        if not name.endswith('.md'):
            continue
        names.append(re.split("(.*)\.md",name)[1])
    return names
        
def build_policies(root):
    policies = []

    polmap = {'build': {}, 'runtime': {}}
    polmap['build']['name'] = "GTFOBins component in image"
    polmap['build']['description'] = "Alert on deployments with GTFOBins components present"
    polmap['build']['lifecycleStages'] = ["BUILD","DEPLOY"]
    polmap['build']['sortName'] = "GTFOBins component in image"
    polmap['build']['sortLifecycleStage'] = "BUILD,DEPLOY"
    polmap['build']['eventSource'] = "NOT_APPLICABLE"
    polmap['build']['fieldName'] = "Image Component"

    polmap['runtime']['name'] = "GTFOBins component execution detected"
    polmap['runtime']['description'] = "Detects contains running a GTFOBins component"
    polmap['runtime']['lifecycleStages'] = ["RUNTIME"]
    polmap['runtime']['sortName'] = "GTFOBins component execution"
    polmap['runtime']['sortLifecycleStage'] = "RUNTIME"
    polmap['runtime']['eventSource'] = "DEPLOYMENT_EVENT"
    polmap['runtime']['fieldName'] = "Process Name"

    for key in polmap:
        policy = {}
        policy['name'] = polmap[key]['name']
        policy['description'] = polmap[key]['description']
        policy['rationale'] = "Leaving GTFOBins components inside container images makes it easier for attackers to 'live off the land' inside a container environment"
        policy['remediation'] = "Use your package manager's \"remove\", \"purge\" or \"erase\" command to remove GTFOBins components from the image build for production containers."
        policy['disabled'] = False
        policy['categories'] = ["Security Best Practices"]
        policy['lifecycleStages'] = polmap[key]['lifecycleStages']
        policy['eventSource'] = polmap[key]['eventSource']
        policy['exclusions'] = []
        policy['scope'] = []
        policy['severity'] = "LOW_SEVERITY"
        policy['enforcementActions'] = []
        policy['notifiers'] = []
        policy['SORTName'] = polmap[key]['sortName']
        policy['SORTLifecycleStage'] = polmap[key]["sortLifecycleStage"]
        policy['SORTEnforcement'] = False
        policy['policyVersion'] = "1.1"
        section = {}
        section['sectionName'] = "Policy Section 1"
        section['policyGroups'] = []
        section['policyGroups'].append({})
        section['policyGroups'][0]['fieldName'] = polmap[key]["fieldName"]
        section['policyGroups'][0]['booleanOperator'] = "OR"
        section['policyGroups'][0]['negate'] = False
        section['policyGroups'][0]['values'] = []

        names = get_gtfobins(root)
        for name in names:
            if(key == "build"):
                section['policyGroups'][0]['values'].append({"value": '%s' % "".join([name,"="])})
            else:
                section['policyGroups'][0]['values'].append({"value": '%s' % name })

        policy['policySections'] = [section]
        policy['mitreAttackVectors'] = []

        # TA0002 - Execution. Many of the GTFOBins components enable execution
        # in various contexts.
        policy['mitreAttackVectors'].append({
            "tactic": "TA0002",
            "techniques": ["T1059.004"]
        })

        # TA0007 - Discovery. Binaries like dmidecode can be used to provide information
        # on the environment
        policy['mitreAttackVectors'].append({
            "tactic": "TA0007",
            "techniques": ["T1046"]
        }) 

        # TA0008 - Lateral movement. Binaries like ssh and netcat can be used 
        # to enable lateral movement.
        policy['mitreAttackVectors'].append({
            "tactic": "TA0008",
            "techniques": ["T1570"]
        })

        # TA0010 - Exfiltration. Binaries like tar, curl and tftp can be used to exfiltrate
        # data from environments.
        policy['mitreAttackVectors'].append({
            "tactic": "TA0010",
            "techniques": ["T1567"]
        })

        # TA0043 - Reconnaissance. 
        policy['mitreAttackVectors'].append({
            "tactic": "TA0043",
            "techniques": ["T1592"]
        })

        policy['criteriaLocked'] = False
        policy['mitreVectorsLocked'] = False
        policy['isDefault'] = False
        policies.append(policy)

    return policies

if __name__ == '__main__':
    policy = build_policies("_gtfobins/")
    print(json.dumps({"policies": policy }))

When the script is now run it generates two policies; one for GTFOBins components in container images, and one that detects GTFOBins processes executing in a container at runtime.

$ python scripts/generate-stackrox-policy.py | jq
{
  "policies": [
    {
      "name": "GTFOBins component in image",
      "description": "Alert on deployments with GTFOBins components present",
      "rationale": "Leaving GTFOBins components inside container images makes it easier for attackers to 'live off the land' inside a container environment",
      "remediation": "Use your package manager's \"remove\", \"purge\" or \"erase\" command to remove GTFOBins components from the image build for production containers.",
      "disabled": false,
      "categories": [
        "Security Best Practices"
      ],
      "lifecycleStages": [
        "BUILD",
        "DEPLOY"
      ],
...
(snip)
...
 {
      "name": "GTFOBins component execution detected",
      "description": "Detects contains running a GTFOBins component",
      "rationale": "Leaving GTFOBins components inside container images makes it easier for attackers to 'live off the land' inside a container environment",
      "remediation": "Use your package manager's \"remove\", \"purge\" or \"erase\" command to remove GTFOBins components from the image build for production containers.",
      "disabled": false,
      "categories": [
        "Security Best Practices"
      ],
      "lifecycleStages": [
        "RUNTIME"
      ],

Here's a closer look at these two policies:

GTFOBins component in image: This policy targets the deploy stage of the container application lifecycle. It means that when a new Kubernetes deployment is created StackRox will first validate whether the container image contains GTFOBins components, via the validting admission controller. If GTFOBins components are detected StackRox will block the deployment, and create a policy violation.
GTFOBins component execution detected: This policy targets the runtime stage of the container application lifecycle. This policy will report GTFOBins process executions detected at runtime, via the StackRox eBPF program.

Having both of these policies is important to protect container applications. Ideally we want all GTFOBins components to be caught at the deploy stage, and we can prevent these binaries ever being deployed to the platform inside a container workload. But there may be reasons we want to allow some binaries (particular applications need them, debugging, etc). In these instances, we want the monitoring provided by the runtime process to detect and report on GTFOBins execution at runtime.

You can create this new policy by simply running the script (python scripts/generate-stackrox-policy.py) and then uploading the policy into StackRox where they'll be available [I've detailed the process to import policies from JSON into StackRox in the last article].

Trying it out

Ok - let's try out this runtime policy!

Create a new workload that contains a known GTFOBins component. In this case I'm deploying a container image that I know is vulnerable to a critical CVE, and that also contains a socat binary.

oc new-app quay.io/smileyfritz/log4shell-app:v0.5

Once the workload is running, exec into the pod and run the socat command:

oc exec -n devops $(oc get pods -n devops | awk '{if (NR!=1) print $1}') -- /usr/bin/socat
2023/09/07 05:01:13 socat[22] E exactly 2 addresses required (there are 0); use option "-h" for help

Ok - the output from the container indicates we ran the socat command with no arguments. If the policy is working, you should see a new violation created in StackRox.

Success! We can now detect GTFOBins usage at runtime for container workloads.

Wrapping up

In this article I expanded on a previous write-up by looking at runtime detection of GTFOBins. If you want to take a closer look at eBPF and StackRox there's a great article here, and I'd highly recommend Brendan Gregg's excellent guide to learning eBPF Tracing.