Sigstore - Signature Sorcery

The first time that I tried out the Sigstore open source project I was completely blown away. I had a container image that I wanted to digitally sign, so I pulled down the cosign binary, generated a keypair, signed the container image, and pushed the signatures to quay.io.

The best part was that all of this happened in about 2 minutes! I remember thinking - "wow, that was easy."

The more I delved into the Sigstore project, the more I realised that this truly was signature sorcery. "Wait, there's also a transparency ledger which records signed metadata about my container image? And it's freely available??"

Sigstore background

Sigstore is an open source project, and now part of the Open Source Security Foundation (OpenSSF). It's an incredibly active community that includes Red Hat (Luke Hinds is a co-founder), Google, ChainGuard, GitHub, and many others.

Sigstore has rapidly become the standard for signing, verifying and protecting software for its ability to automate the process of digitally signing and validating software artifacts. It's seeing rapid adoption - all Kubernetes releases starting with 1.24 include cryptographically signed Sigstore certificates, giving users the ability to verify signatures and have greater confidence in the origin of each and every deployed Kubernetes binary, source code bundle and container image.

Importantly, Sigstore lowers the barrier for digital signing. It achieves this by making software signing and attestation more accessible for creators, without having to become experts in crytographic signing, and provides “public benefit” instances of its services that can be used by anyone. This means that open source projects - who might not have had access to signature infrastructure previously - can now sign releases and record metadata about how releases were created, allowing open source consumers to make more informed decisions on trust.

There's a few components that comprise Sigstore and make it easy to get up and running with signing releases.

Cosign

Cosign supports container signing, verification, and storage in an Open Container Initiative registry. Cosign doesn't just support containers though - it also supports software bill of materials (SBOM), WebAssembly (WASM) modules, Tekton bundles and more.

Rekor

Rekor provides an immutable tamper-resistant ledger of metadata generated within a software project's supply chain. This ledger is append-only and once entries are added they cannot be modified; a valid log can be cryptographically verified by any third-party.

Rekor ensures that organisations can make decisions on trust. The timestamps and contents of the ledger cannot be tampered or removed later, enabling organisations to validate log entries and software signatures.

Rekor has recently reached General Availability. The Sigstore community maintains the public-benefit Rekor instance and provides a 99.5% availability service level objective (SLO) and round-the-clock pager support, and uses semantic versioning rules for API stability.

Fulcio

Fulcio is a free-to-use certificate authority for issuing code signing certificates for an OpenID Connect (OIDC) identity, such as email address.

Fulcio only issues short-lived certificates that are valid for 10 minutes. This is a really good thing for the software trust model that Sigstore introduces. It means that we no longer have to diligently protect GPG keys used to sign containers - the Fulcio X509 certificates are only valid for 10 minutes, greatly minimising the risk that these keys could be exposed, and used to sign fraudulent releases. Fulcio has also recently reached General Availability.

Intro to Cosign, Rekor and Fulcio

Ok - let's get hands-on! You can follow the Sigstore docs to grab a copy of the cosign and rekor-cli binaries that you'll need for this section.

You can use Fulcio-issues certificates to sign containers by simply using cosign sign. I'm going to sign the image quay.io/smileyfritz/chat-client:latest. Note that you require access to the registry / repository for the image you want to sign, as the signature is pushed to the registry and stored as an OCI object.

cosign works best with digests, not tags. If you try to use a tag, like latest, with an image you'll be presented with a warning.

WARNING: Image reference quay.io/smileyfritz/chat-client:latest uses a tag, not a digest, to identify the image to sign.
    This can lead you to sign a different image than the intended one. Please use a
    digest (example.com/ubuntu@sha256:abc123...) rather than tag
    (example.com/ubuntu:latest) for the input to cosign. The ability to refer to
    images by tag will be removed in a future release.

The first thing we need is the digest for the image. You can grab this easily with crane

$ crane digest quay.io/smileyfritz/chat-client:latest
sha256:86c9473bdae1201f870c4a328b07bcba5e18a6b4e83c26669cf82a36d7b8cbd5

Now we can use cosign to create ephemeral keys and certificates, get them signed automatically by the Fulcio CA, and store these in the Rekor transparency log.

$ cosign sign quay.io/smileyfritz/chat-client@sha256:86c9473bdae1201f870c4a328b07bcba5e18a6b4e83c26669cf82a36d7b8cbd5

At this point a workflow kicks off which requests you to grant permission to have your information stored permanently in the Rekor transparency logs. If you accept, a browser window will open and you will be directed to a page that asks you to log in with Sigstore. You can authenticate with GitHub, Google, or Microsoft. Note that the email address that is tied to these credentials will be permanently visible in the Rekor transparency log. This makes it publicly visible that you are the one who signed the given artifact, and helps others trust the given artifact.

Generating ephemeral keys...
Retrieving signed certificate...

        Note that there may be personally identifiable information associated with this signed artifact.
        This may include the email address associated with the account with which you authenticate.
        This information will be used for signing this artifact and will be stored in public transparency logs and cannot be removed later.

Once you enter the verification code, cosign will sign the image and push the signature to the registry.

Successfully verified SCT...
tlog entry created with index: 18012395
Pushing signature to: quay.io/smileyfritz/chat-client

Great! The container is now signed. We can verify the signatures with cosign also.

$ cosign verify --certificate-identity shane.boulden@gmail.com --certificate-oidc-issuer https://github.com/login/oauth quay.io/smileyfritz/chat-client@sha256:86c9473bdae1201f870c4a328b07bcba5e18a6b4e83c26669cf82a36d7b8cbd5

Verification for quay.io/smileyfritz/chat-client@sha256:86c9473bdae1201f870c4a328b07bcba5e18a6b4e83c26669cf82a36d7b8cbd5 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The code-signing certificate was verified using trusted certificate authority certificates

We can also verify that the signature has been stored in the transparency log. You can see that the log index has been emitted for the previous signing action:

tlog entry created with index: 18012395

We can use the rekor-cli to have a look at the data stored on the transparency log.

$ rekor-cli get --log-index 18012395

LogID: c0d23d6ad406973f9559f3ba2d1ca01f84147d8ffc5b8445c224f98b9591801d
Index: 18012395
IntegratedTime: 2023-04-15T06:06:16Z
UUID: 24296fb24b8ad77ad6bd8d95e7e87347f2038be47a9a63238a72848d3d76429365091f2d120770d0
Body: {
  "HashedRekordObj": {
    "data": {
      "hash": {
        "algorithm": "sha256",
        "value": "49411c8fcbe824a82854879ad19240511b1556b61f7ae7f13371fed8460fbee9"
      }
...

We can string a couple of commands together to extract the certificate used to sign the container image.

$ rekor-cli get --log-index 18012395 --format json | jq -r '.Body.HashedRekordObj.signature.publicKey.content' | base64 -d | openssl x509 -noout -text

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            01:6d:49:45:8f:3e:e9:dd:10:05:b0:f5:f3:93:b9:5e:48:b8:aa:1a
        Signature Algorithm: ecdsa-with-SHA384
        Issuer: O = sigstore.dev, CN = sigstore-intermediate
        Validity
            Not Before: Apr 15 06:06:13 2023 GMT
            Not After : Apr 15 06:16:13 2023 GMT
        Subject: 
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
        ...
        ...
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature
            X509v3 Extended Key Usage: 
                Code Signing
            ...
            X509v3 Subject Alternative Name: critical
                email:shane.boulden@gmail.com
            1.3.6.1.4.1.57264.1.1: 
                https://github.com/login/oauth
          ...

This output indicates that the certificate was issued from the Sigstore Fulcio public benefit instance to me, based on my OIDC token that was issued from GitHub. It's been stored on the Rekor transparency ledger in perpetuity, so anyone can verify that I've signed this container image. Awesome!

And that's it! We've signed a container with cosign using certificates and keys signed by the Fulcio CA, and created entries in the Rekor transparency ledger recording the signing action. If you want to see some of the other signing workflows available, there's an excellent in-depth tutorial available at chainguard.dev.

Sigstore and OpenShift

OpenShift supports cosign and rekor in Technology Preview with OpenShift Pipelines and Tekton Chains. If you have a handy OpenShift environment laying around, there's a couple of simple steps to get up and running with Sigstore and start signing containers.

1. Install the OpenShift Pipelines operator. OpenShift Pipelines provides the Tekton Chains component in Technology Preview, which allows us to embed container signing inside Tekton pipelines.

2. Configure Tekton Chains. You can find a quick guide here. Make sure that you patch the Tekton Chains config to use in-toto attestation formats and store attestations in the Rekor transparency ledger:

$ oc patch configmap chains-config -n openshift-pipelines -p='{"data":{"artifacts.taskrun.format": "in-toto"}}'

$ oc patch configmap chains-config -n openshift-pipelines -p='{"data":{"artifacts.taskrun.storage": "oci"}}'

$ oc patch configmap chains-config -n openshift-pipelines -p='{"data":{"transparency.enabled": "true"}}'

3. Configure your Tekton pipeline to emit results. Specifically, Tekton Chains looks for Tekton results named IMAGE_DIGEST and IMAGE_URL to identify new container images that need to be signed. You can add these results to an existing pipeline using a Tekton task:

...
results:
  - name: IMAGE_DIGEST
    description: Digest of the image just built.
  - name: IMAGE_URL
    description: Name of the image just built.
...
- name: digest-to-results
    image: $(params.BUILDER_IMAGE)
    script: cat $(workspaces.source.path)/image-digest | tee /tekton/results/IMAGE_DIGEST

  - name: name-to-results
    image: $(params.BUILDER_IMAGE)
    script: echo $(params.IMAGE) | tee /tekton/results/IMAGE_URL

You can find the complete example at my GitHub here

That's really all there is. You don't need to explicitly add steps to your build pipeline to sign containers - the Tekton Chains controller will detect new releases from the IMAGE_URL and IMAGE_DIGEST results, use cosign to sign these containers with the key provided, and store the signatures as OCI manifest objects alongside your container images in the registry.

You can confirm this using crane:

$ crane ls quay.io/smileyfritz/chat-client
0.5
sha256-f41cc30a1507e3eb3117e0e31be7604fe484c48caf335cd1cdfe1c1991ada077.att
sha256-f41cc30a1507e3eb3117e0e31be7604fe484c48caf335cd1cdfe1c1991ada077.sig
v0.6
sha256-822be3ad91788b82d275ef8077132c09e4339c415823e286df79d23608a278fe.att
sha256-822be3ad91788b82d275ef8077132c09e4339c415823e286df79d23608a278fe.sig
latest
sha256-c446564d0876f0af0eba11a93352711cf836b7bc082e50f9151d5d9e90817d75.att
sha256-c446564d0876f0af0eba11a93352711cf836b7bc082e50f9151d5d9e90817d75.sig
...

Here you we can see tags (0.5, v0.6 and latest), as well as signatures and attestations created by Sigstore and Tekton Chains components stored within the container image repository.

Tekton Chains has recorded an Attestation type in the Rekor ledger. To access this we can first search for any transparency log entries using the image digest.

$ rekor-cli search --sha=86c9473bdae1201f870c4a328b07bcba5e18a6b4e83c26669cf82a36d7b8cbd5
Found matching entries (listed by UUID):
24296fb24b8ad77a7f9157e02f971e0c373e2ec65c2a72bc7642728bba4ca5a8cec931d2a0bb7d8e

Now that we have the transparency ledger entry UUID, we can have a look at the attestation.

$ rekor-cli get --uuid=24296fb24b8ad77a7f9157e02f971e0c373e2ec65c2a72bc7642728bba4ca5a8cec931d2a0bb7d8e --format json | jq -r .Attestation | jq
{
  "_type": "https://in-toto.io/Statement/v0.1",
  "predicateType": "https://slsa.dev/provenance/v0.2",
  "subject": [
    {
      "name": "quay.io/smileyfritz/chat-client",
      "digest": {
        "sha256": "86c9473bdae1201f870c4a328b07bcba5e18a6b4e83c26669cf82a36d7b8cbd5"
      }
    }
  ],
  "predicate": {
    "builder": {
      "id": "https://tekton.dev/chains/v2"
    },
    "buildType": "tekton.dev/v1beta1/TaskRun",
    "invocation": {
      "configSource": {},
      "parameters": {
        "BUILDER_IMAGE": "quay.io/buildah/stable:v1.17.0"
...

I've created a video here showing how to configure Tekton Chains and create a pipeline that signs container images, and also how Tekton Chains can use the public benefit Rekor instance to sign and store metadata about releases.

Wrapping up

This has been a short, practical guide to getting started signing container images with Sigstore and OpenShift Pipelines. In the next article, I'll look at verifying container signatures - using a Kubernetes admission controller to block unsigned and untrusted content, and only permit signed container images.