As I discussed in Black Hat Preview: Automating Cloud Security Policy Compliance, you can combine Amazon S3 and IAM roles to securely provision configuration files (or any other files) and credentials to Amazon EC2 or VPC instances. Here are the details.
The problem to solve
One of the issues in automating infrastructure is securely distributing security credentials and configuration files to servers that start automatically, without human interaction. You don’t want to embed security credentials in the images that are the basis for these running instances (servers), and you need to ensure that only approved instances can access the credentials, without necessarily knowing anything about the instance.
The answer is to leverage the cloud infrastructure itself to identify and propagate the credentials. We can handle it all in the management plane without manually touching servers or embedding long-term credentials.
In our example we will use these to distribute an initial Chef configuration file and validation certificate.
Securely bootstrapping Chef clients
A Chef node is a server or workstation with the Chef agent running on it. This is the application that connects to the Chef server to accept configuration pushes and run scripts (recipes). There are four ways to install it on a cloud instance:
- Manually log into the instance and install the
chef-clientsoftware, then transfer over the server’s validation certificate and configuration file. Almost nobody does this in the cloud, outside of development and testing.
- Embed the client software and configuration files in the machine image for use at launch (instantiation). This is common but means you need to maintain your own images rather than using public ones.
- Remotely bootstrap the instance. The Chef management software (knife) can connect to the instance via
sshto configure everything automatically, but that requires its `ssh private key.
cloud-initor another installation script to install Chef onto a clean ‘base’ (unconfigured) instance, and IAM roles to allow the instance to connect to a repository such as S3 to download the initial configuration file and server certificate. This is what I will explain.
Configuring AWS IAM roles to distribute credentials
Amazon recently added a feature called IAM roles. Amazon Web Services (like some other public and private cloud platforms) supports granular identity management down to the object level. This is critical for proper segregation and isolation of cloud assets. AWS previously only supported users and groups, which are (and I’m simplifying) static collections of users, servers, and other objects in AWS.
Users and groups are great, but they provide users or servers with static security credentials such as an Access Key and Secret Key, X.509 certificate, or a username and password for web UI access.
IAM Roles are different. They are temporary credentials applied to AWS assets, such as a running instance. They include an Access Key and Secret Key provided to the object via an API call, and a token. You need all three to sign requests, and they are rotated (approximately every 24 hours in my experience).
So if someone steals the keys they won’t work without the token. If they also get a token it expires in a day.
In our example we will create an S3 bucket to hold our Chef configuration
client.rb configuration file and
validation.pem digital certificate. We will then switch over to AWS IAM to create a new role and assign it read privileges to S3. Then we will tweak the policy to only allow access to that bucket.
Finally we will launch an instance, assign the role, then log in and show the credentials. You wouldn’t do this in production, but it illustrates how roles work.
Step by step
I assume you have some knowledge of AWS here. If you want granular instructions, take our class. I also assume you have a Chef server set up in EC2 someplace, or use Hosted Chef. If you want to know how to do that, take the class. 🙂
- Log in and ensure your Chef server has its own Security Group.
- Create a new Security Group for your instances (or pick any group you already have). Our example is very locked down, which may not be appropriate for your environment.
- Open ports 4000, 4040, and 80 in the Chef Server security group from your instance security group. I haven’t had time to play with it, but we might be able to double down and allow access by role. I will test before Black Hat – it doesn’t take long, but I just got the idea. Return the favor and open 4000 and 4040 into the instance group from the server group.
Amazon S3 section of AWS Console
- Create a new bucket (e.g.,
cloudsec). Load a random file for testing later if you want. If you have a Chef server place
validation.pemhere – you will need these to complete our example.
- Create a new role called
ChefClient. You can do this all via API or by write the policy by hand, but we use the GUI.
- Select AWS Service Roles, then Amazon EC2. This grants the designated rights to EC2 assets with the assigned role. Continue.
- Select Select Policy Template and then Amazon S3 Read Only Access. Continue.
- After this you can name the policy, then adjust it to apply to only the single bucket – not your entire Amazon S3 account. Change the entry “Resource: *” to “Resource: arn:aws:s3:::your_bucket”. I also added a safety wildcard, so your policy should look like the screenshot below
- Launch a new instance. Ubuntu is a safe bet, and what we use to demonstrate the temporary credentials.
- On the Instance Details screen assign your IAM role. You also probably want to put it in the same availability zone as your Chef server, and later on into the right security group.
- Once everything is running, log into your instance.
wget -O -- -q 'http://169.254.169.254/latest/meta-data/iam/security-credentials/myrole', replacing ‘myrole’ with the name of your role (case sensitive). You should see your temporary AWS credentials, when they were issued, and when they expire.
You have now configured your environment to support transfer of the security credentials only to instances assigned the appropriate role (
ChefClient in my case). Your instance has temporary credentials that Amazon rotates for you, minimizing exposure. AWS also requires a token so the Access Key and Secret Key won’t work by themselves. Our next post will show how to use
cloud-init to install the Chef client software, download the configuration file and server certificate, and run a scripted install of Chef.
With this all configured, all you need to do is assign the role and embed the
cloud-init script, and instances will automatically (and securely) configure themselves and connect to the Chef server.