Ansible and AWS IAM Roles Anywhere

Want to hear a NotebookLM-generated podcast created from this article? Check it out here.

I had an incredibly interesting question last week, and it goes like this:

"We use Ansible to orchestrate workloads on AWS. In the past we've used AWS access tokens, but we want to move towards temporary / short-lived credentials for access to AWS resources."

"If our Ansible appliance was hosted on AWS we could simply use AWS STS. But Ansible is on-premises.

"What can we do?"

I think this is the perfect opportunity to introduce AWS IAM Roles Anywhere, and how this fundamentally changes the security architecture for organisations using AWS resources.

At the end of this article you'll have an understanding of:

• How AWS IAM Roles Anywhere help you use short-lived credentials to access AWS resources, from anywhere
• How to integrate Red Hat Identity Management (IDM) as a trust anchor for AWS IAM Roles Anywhere
• How to use Ansible with certificates issued by Red Hat IDM together with AWS IAM Roles Anywhere to access Route53 resources using short-lived credentials

[PS: In the last article I spoke about GPU-accelerated Windows Desktop sessions. You can see a sneak-peek of the next article on GPUs and OpenShift Virtualization in the video below]

Let's dive in!

AWS IAM Roles Anywhere

What is AWS IAM Roles Anywhere? Essentially it's a way for applications running outside of AWS to access temporary AWS credentials, and use those to access AWS services. This means that you can use the same IAM policies and roles for native AWS applications to access AWS resources, but now extending those capabilities to workload that might be on other cloud providers, or in your own datacentre.

This means that we no longer need access keys. Access keys are horrible - they stick around forever, and they are the first target for threat actors who manage to compromise a workload inside your account. Using IAM Roles Anywhere means that we don't ned to keep access keys around, and can instead use short-lived credentials.

IAM Roles Anywhere relies on trusted X.509 certificates to verify the identity of a workload running outside of AWS and establish trust. At a high-level it might sound like you've simply swapped long-lived access tokens for long-lived x509 certificates / keys. But, that's not really the case. X.509 certificates have huge advantages over access keys:

• Attributes that can be used to tie a certificate to a service
• Ability to revoke or renew certificates
• Standard APIs and access mechanisms for certificates (e.g. PKCS#11)
• Secure storage mechanisms (TPM, HSM, etc)

Ok, so we know that:

• IAM Roles Anywhere allows workloads outside of AWS to assume IAM roles, just like if they were running on AWS
• It uses x509 certificates for authentication and creates a trust anchor with a certificate authority (CA)

Using IAM Roles Anywhere relies on organisations managing trusted X.509 certificates, and having public key infrastructure (PKI) available to do this.

FreeIPA is a popular platform for PKI and certificate management in Linux environments, and is built-in to Red Hat Enterprise Linux (RHEL) via Red Hat Identity Management (IDM). IDM provides a complete Linux domain identity management solution, and can manage users, SSH keys, host-based access rules, and sudo rules. It can also issue and rotate certificates for services, hosts and users using the built-in dogtag certificate authority.

Using Red Hat IDM in this solution allows us to easily rotate certificates / keys, and allow Ansible to exchange these for short-lived credentials via IAM Roles Anywhere. Here's a diagram:

Getting started with Ansible, Red Hat Identity Management and AWS IAM Roles Anywhere

Enough talk - let's dive in to this solution and see how we can use AWS IAM Roles Anywhere with Red Hat Identity Management and Ansible.

I'm going to create a very simple Red Hat IDM (FreeIPA) server for this article. I already have DNS in my lab, and I don't want to manage that via IDM. I also don't need any replicas. This is straightforward with the ipa-server-install utility.

Firstly add the required firewall rules and enable the RHEL 9 BaseOS and AppStream repositories. There's no need to directly attach a subscription assuming you are using Red Hat Simple Content Access:

firewall-cmd --permanent --add-port={80/tcp,443/tcp,389/tcp,636/tcp,88/tcp,88/udp,464/tcp,464/udp,53/tcp,53/udp}

subscription-manager repos --enable=rhel-9-for-x86_64-baseos-rpms
subscription-manager repos --enable=rhel-9-for-x86_64-appstream-rpms

Now you can install the IDM server:

ipa-server-install --hostname=idm.rock.lab -n rock.lab -p <password> -a <password> -r ROCK.LAB

Once the install completes you should have a brand-new IDM server available:

We're primarily going to use the built-in dogtag CA services in this article, and you can check that CA has been created in the UI.

Creating an AWS IAM Roles Anywhere Trust Anchor and additional IAM roles

Now that we have an IDM server and a built-in dogtag CA created we can move to AWS configuration. You can navigate to the Roles Anywhere console here (changing your region accordingly).

You can see that AWS gives us a handy set of 'Setup steps' we need to step through to configure IAM Roles Anywhere:

• Create trust anchor (in our case, Red Hat IDM)
• Configure roles
• Use roles anywhere

Let's start by creating a trust anchor with Red Hat IDM. You can find the CA on the IDM server at /etc/ipa/ca.crt, and you can simply upload this to the portal.

Now that we have a trust anchor we can create (or modify) existing roles to use the Roles Anywhere trust anchor. I have an existing AWS IAM policy which allows services to lookup and modify Route53 records, and I use for certbot certificate creation. You can see it here:

{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Effect": "Allow",
           "Action": [
               "route53:ListHostedZones",
               "route53:GetChange"
           ],
           "Resource": [
               "*"
           ]
       },
       {
           "Effect": "Allow",
           "Action": [
               "route53:ChangeResourceRecordSets",
               "route53:ListResourceRecordSets"
           ],
           "Resource": [
               "arn:aws:route53:::hostedzone/<my-hosted-zone-id>"
           ]
       }
   ]
}

To use this with Roles Anywhere I just need to update a role, or create a new one. I've named this RolesAnywhere-Certbot. The important part of creating / editing this role is to update the Trust relationships and include the service rolesanywhere.amazonaws.com in the principal:

{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Sid": "",
           "Effect": "Allow",
           "Principal": {
               "Service": "rolesanywhere.amazonaws.com"
           },
           "Action": [
               "sts:AssumeRole",
               "sts:SetSourceIdentity",
               "sts:TagSession"
           ]
       }
   ]
}

The last step here in the AWS console is to create a profile, and ensure that permissions are enforced on the role session when one or more roles are assumed by your non AWS workload.

You can see here that I've attached the RolesAnywhere-Certbot role to the profile, and configured the session duration for 1 hour. You can also configure which X.509 certificate attributes are mapped into the session, though I'll just accept the defaults.

Enrolling IDM clients and requesting a certificate for IAM Roles Anywhere

Now that we have an IDM server available we can onboard our Ansible node as a client. Fortunately this is pretty straightforward - I can just install the ipa-client utility and point it at IDM.

# dnf install ipa-client -y
# ipa-client-install --mkhomedir
This program will set up IPA client.
Version 4.11.0
...

You'll get a few questions from the client during the config, and you can see examples of my responses below:

DNS discovery failed to determine your DNS domain
Provide the domain name of your IPA server (ex: example.com): rock.lab
Provide your IPA server name (ex: ipa.example.com): idm.rock.lab
The failure to use DNS to find your IPA server indicates that your resolv.conf file is not properly configured.
Autodiscovery of servers for failover cannot work with this configuration.
If you proceed with the installation, services will be configured to always access the discovered server for all operations and will not fail over to other servers in case of failure.
Proceed with fixed values and no DNS discovery? [no]: yes
Do you want to configure chrony with NTP server or pool address? [no]:
Client hostname: ansible.rock.lab
Realm: ROCK.LAB
DNS Domain: rock.lab
IPA Server: idm.rock.lab
BaseDN: dc=rock,dc=lab

Continue to configure the system with these values? [no]: yes

Once the client configuration completes you should be able to see your new host in the IDM UI.

Ok, now we can start creating certificates! There is one subtle but very important point I need to make at this point. The default expiry for IDM certificates is two years, but I want short-lived and regularly rotated certificates that aren't valid longer than 60 days.

You can configure this in IDM by creating a new certificate profile. Make sure that you have valid Kerberos credentials on the IDM server:

[idm ~]# kinit admin@ROCK.LAB

Now that you have valid credentials, you can extract the IPA service certificate profile:

ipa certprofile-show caIPAserviceCert --out=caIPAserviceCert.cfg
-----------------------------------------------------------
Profile configuration stored in file 'caIPAserviceCert.cfg'
-----------------------------------------------------------
  Profile ID: caIPAserviceCert
  Profile description: Standard profile for network services
  Store issued certificates: True

Let's copy this to a new file and make a few changes:

mv caIPAserviceCert.cfg caRolesAnywhere.cfg

< desc=This certificate profile is for enrolling server certificates with IPA-RA agent authentication.
---
> desc=This certificate profile is for providing certificates for use with AWS IAM Roles Anywhere
9c9
< name=IPA-RA Agent-Authenticated Server Certificate Enrollment
---
> name=AWS IAM Roles Anywhere certificate profile
38c38
< policyset.serverCertSet.2.constraint.params.range=740
---
> policyset.serverCertSet.2.constraint.params.range=70
41c41
< policyset.serverCertSet.2.default.params.range=731
---
> policyset.serverCertSet.2.default.params.range=61
112c112
< profileId=caIPAserviceCert
---
> profileId=caRolesAnywhere

Save and close the file and create a new IPA certificate profile.

# ipa certprofile-import --file=caRolesAnywhere.cfg --store=true --desc="AWS IAM Roles Anywhere profile"
Profile ID: caRolesAnywhere
----------------------------------
Imported profile "caRolesAnywhere"
----------------------------------
  Profile ID: caRolesAnywhere
  Profile description: AWS IAM Roles Anywhere profile
  Store issued certificates: True

Now we can simply request a certificate from our IDM-enrolled Ansible node and the expiry should be set to 60 days. The best part is that certmonger will automatically track this certificate, and renew it via IDM before it expires. How good is that??

# mkdir /etc/iam-anywhere
# ipa-getcert request -k /etc/iam-anywhere/private.key -f /etc/iam-anywhere/cert.crt  --profile caRolesAnywhere

# ipa-getcert list
Number of certificates and requests being tracked: 1.
Request ID '20240903054001':
        status: NEED_KEY_GEN_PERMS
        stuck: yes
        key pair storage: type=FILE,location='/etc/iam-anywhere/private.key'
        certificate: type=FILE,location='/etc/iam-anywhere/cert.crt'
        CA: IPA
        issuer:
        subject:
        issued: unknown
        expires: unknown
        profile: caRolesAnywhere
        pre-save command:
        post-save command:
        track: yes
        auto-renew: yes

Hmm - that doesn't look right. Let's just make sure that SELinux is happy.

# ausearch -m avc -ts today
----
time->Wed Aug 28 12:08:14 2024
type=PROCTITLE msg=audit(1724810894.358:385): proctitle=2F7573722F7362696E2F636572746D6F6E676572002D53002D70002F72756E2F636572746D6F6E6765722E706964002D6E002D6432
type=SYSCALL msg=audit(1724810894.358:385): arch=c000003e syscall=257 success=no exit=-13 a0=ffffff9c a1=5579e6da5c50 a2=c2 a3=180 items=0 ppid=30513 pid=30542 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="certmonger" exe="/usr/sbin/certmonger" subj=system_u:system_r:certmonger_t:s0 key=(null)
type=AVC msg=audit(1724810894.358:385): avc:  denied  { create } for  pid=30542 comm="certmonger" name="private.key" scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:etc_t:s0 tclass=file permissive=0

Looks like exactly the issue! certmonger doesn't like the SELinux labels on our new /etc/iam-anywhere directory. Let's update it and try again.

# semanage fcontext -a -t cert_t "/etc/iam-anywhere(/.*)?"
# restorecon -vvFR /etc/iam-anywhere/
Relabeled /etc/iam-anywhere from unconfined_u:object_r:etc_t:s0 to system_u:object_r:cert_t:s0

Let's resubmit the existing request to IDM and see if the certificate gets issued.

ipa-getcert list
Number of certificates and requests being tracked: 1.
Request ID '20240903054001':
        status: NEED_KEY_GEN_PERMS
        stuck: yes
        key pair storage: type=FILE,location='/etc/iam-anywhere/private.key'
        certificate: type=FILE,location='/etc/iam-anywhere/cert.crt'
        CA: IPA
        issuer:
        subject:
        issued: unknown
        expires: unknown
        pre-save command:
        post-save command:
        track: yes
        auto-renew: yes

# ipa-getcert resubmit -i '20240903054001'
Resubmitting "20240903054001" to "IPA".

# ipa-getcert list
Number of certificates and requests being tracked: 1.
Request ID '20240903054001':
        status: MONITORING
        stuck: no
        key pair storage: type=FILE,location='/etc/iam-anywhere/private.key'
        certificate: type=FILE,location='/etc/iam-anywhere/cert.crt'
        CA: IPA
        issuer: CN=Certificate Authority,O=ROCK.LAB
        subject: CN=ansible.rock.lab,O=ROCK.LAB
        issued: 2024-08-28 12:18:53 AEST
        expires: 2024-10-28 12:18:53 AEST
        dns: ansible.rock.lab
        principal name: host/ansible.rock.lab@ROCK.LAB
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        profile: caRolesAnywhere
        pre-save command:
        post-save command:
        track: yes
        auto-renew: yes

That looks a lot better. Now we can use this key/certificate to request short-lived credentials to access AWS services. You can also see this created in the IDM console:

Creating and running an Ansible playbook

There's been a lot of work to get to this point, but I think it's been worth it. Now we have:

• A Red Hat Identity Management (IDM) server created with a built-in dogtag certificate authority, that we can use to issue certificates to hosts and services

• Certificate profiles created that expire certificates issued by IDM after 60 days, and ensure that certmonger rotates these on enrolled hosts

• A trust anchor created for our on-premises Red Hat IDM server and AWS IAM Roles Anywhere

• IAM roles and profiles created

Creating the Ansible playbook is pretty simple - it just looks like this:

---
- name: List Route 53 DNS Records
  hosts: localhost
  gather_facts: no
  vars:
    hosted_zone_id: "YOUR_HOSTED_ZONE_ID"  # Replace with your Hosted Zone ID

  tasks:
    - name: List all DNS records in the hosted zone
      amazon.aws.route53_info:
        hosted_zone_id: "{{ hosted_zone_id }}"
        query: "record_sets"
      register: route53_records

    - name: Display DNS records
      debug:
        var: route53_records.resource_record_sets

The question is though - how do we use IAM Roles Anywhere in this playbook?

Fortunately AWS has already solved this using the AWS IAM Roles Anywhere Credential Helper. This is an open source utility that can take the X.509 certificates issued by IDM, and use the IAM Roles Anywhere CreateSession API to return temporary credentials we can use in an Ansible playbook.

You can grab a copy of the utility from the Github releases page:

Once you have a local copy of the aws_signing_helper you can try it out. Simply load up the CLI with the certificate / key issued by Red Hat IDM, and the ARNs for the trust anchor, role and profile you created earlier.

/usr/local/bin/aws_signing_helper credential-process \
  --certificate /etc/iam-anywhere/cert.crt --private-key /etc/iam-anywhere/private.key \
  --trust-anchor-arn <your-trust-anchor-arn> --role-arn <your-role-arn> \
  --profile-arn <your-profile-arn>

You should see that the aws_signing_helper CLI returns a set of temporary credentials (access key / secret access key), and expiration date of an hour:

{"Version":1,"AccessKeyId":"<temporary-access-key>","SecretAccessKey":"<temporary-secret-access-key>","Expiration":"2024-09-02T02:13:19Z"}

To use the AWS IAM Roles Anywhere credential helper in our Ansible playbook, we just need to configure the ~/.aws/config file with a credential_process line:

# cat ~/.aws/config
[default]
region=ap-southeast-2
credential_process = /usr/local/bin/aws_signing_helper credential-process \
  --certificate /etc/iam-anywhere/cert.crt --private-key /etc/iam-anywhere/private.key \
  --trust-anchor-arn <your-trust-anchor-arn> --role-arn <your-role-arn> \
  --profile-arn <your-profile-arn>

And running the Ansible playbook should now "just work":

# ansible-playbook route53.yaml

PLAY [List Route 53 DNS Records] ***************************************************************************************************************

TASK [List all DNS records in the hosted zone] *************************************************************************************************
ok: [localhost]

TASK [Display DNS records] *********************************************************************************************************************
ok: [localhost] => {
    "route53_records.resource_record_sets": [
        {
            "name": "blueradish.net.",
            "resource_records": [
                {
                    "value": "ns-1445.awsdns-52.org."
                },
                {
                    "value": "ns-1833.awsdns-37.co.uk."
                },
                {
                    "value": "ns-641.awsdns-16.net."
                },
                {
                    "value": "ns-300.awsdns-37.com."
                }
            ],
            "ttl": 172800,
            "type": "NS"

Wrapping up

In this article I looked at AWS IAM Roles Anywhere, and how you can integrate Ansible with a hybrid cloud identity model. This provides a great solution for organisations looking to ditch AWS access keys / secret access keys and move to temporary credentials across their hybrid cloud. I used Red Hat Identity Management (IDM) in this article to provide public key infrastructure (PKI), though you can use any certificate authority you'd like.

AWS IAM Roles Anywhere uses trusted X.509 digital certificates to identify workloads and exchange the certificate / key for temporary access credentials. Trusted X.509 certificates is a popular approach for workloads identities, and other cloud platforms are also exploring this, notably Google Workload Identity Federation which supports configuring workload identity federation with X.509 certificates in a pre-GA preview.

Workloads might also be able to obtain a SAML assertion or OpenID Connect Token, and I'll take a look at this approach in another article.

Happy automating!