Would you like the ability to revert unapproved security group (firewall) changes in Amazon Web Services in 10 seconds, without external tools? That’s about 10-20 minutes faster than is typically possible with a SIEM or other external tools. If that got your attention, then read on…

If you follow me on Twitter, you might have noticed I went a bit nuts when Amazon Web Services announced their new CloudWatch events a couple weeks ago. I saw them as an incredibly powerful too for event driven security. I will post about the underlying concepts tomorrow, but right now I think it’s better to just show you how it works first. This entire thing took about 4 hours to put together, and it was my first time writing a Lambda function or using Python in 10 years.

This example configures an AWS account to automatically revert any Security Group (firewall) changes without human interaction, using nothing but native AWS capabilities. No security tools, no servers, nada. Just wiring together things already built into AWS. In my limited testing it’s effective in 10 seconds or less, and it’s only 100 lines of code – including comments. Yes, this post is much longer than the code to make it all work.

I will walk you through setting it up manually, but in production you would want to automate this configuration so you can manage it across multiple AWS accounts. That’s what we use Trinity for, and I’ll talk more about automating automation at the end of the post. Also, this is Amazon specific because no other providers yet expose the needed capabilities.

For background it might help to read the AWS CloudWatch events launch post. The short version is that you can instrument a large portion of AWS, and trigger actions based on a wide set of very granular events. Yes, this is an example of the kind of research we are focusing on as part of our cloud pivot.

This might look long, but if you follow my instructions you can set it all up in 10-15 minutes. Tops.

Prep Work: Turn on CloudTrail

If you use AWS you should have CloudTrail set up already; if not you need to activate it and feed the logs to CloudWatch using these instructions. This only takes a minute or two if you accept all the defaults.

Step 1: Configure IAM

To make life easier I put all my code up on the Securosis public GitHub repository. You don’t need to pull that code – you will copy and paste everything into your AWS console.

Your first step is to configure an IAM policy for your workflow; then create a role that Lambda can assume when running the code. Lambda is an AWS service that allows you to store and run code based on triggers. Lambda code runs in a container, but doesn’t require you to manage containers or servers for it. You load the code, and then it executes when triggered. You can build entirely serverless architectures with Lambda, which is useful if you want to eliminate most of your attack surface, but that’s a discussion for another day.

IAM in Amazon Web Services is how you manage who can do what in your account, including the capabilities of Amazon services themselves. It is ridiculously granular and powerful, and so the most critical security tool for protecting AWS accounts.

• Log into the AWS console. Got to the Identity and Access Management (IAM) dashboard.
• Click on Policies, then Create Policy.
• Choose Create Your Own Policy.
• Name it lambda_revert_security_group. Enter a description, then copy and paste my policy from GitHub. My policy allows the Lambda function to access CloudWatch logs, write to the log, view security group information, and revoke ingress or egress statements (but not create new ones). Damn, I love granular policies!
• 
• Once the policy is set you need to Create New Role. This is the role which the Lambda function will assume when it runs.
  • Name it lambda_revert_security_group, assign it an AWS Lambda role type, then attach the lambda_revert_security_group policy you just created.

That’s it for the IAM changes. Next you need to set up the Lambda function and the CloudWatch event.

Step 2: Create the Lambda function

First make sure you know which AWS region you are working in. I prefer us-west-2 (Oregon) for lab work because it is up to date and tends to support new capabilities early. us-east-1 is the granddaddy of regions, but my lab account has so much cruft after 6+ years that things don’t always work right for me there.

• Go to Lambda (under Compute on the main services page) and Create a Lambda function.
• Don’t pick a blueprint – hit the Skip button to advance to the next page.
• Name your function revertSecurityGroup. Enter a description, and pick Python for the runtime. Then paste my code into the main window. After that pick the lambda_revert_security_group IAM role the function will use. Then click Next, then Create function.

A few points on Lambda. You aren’t billed until the function triggers; then you are billed per request and runtime. Lambda is very good for quick tasks, but it does have a timeout (I think an hour these days), and the longer you run a function the less sense it makes compared to a dedicated server. I actually looked at migrating Trinity to Lambda because we could offload our workflows, but at that time it had a 5-minute timeout, and running hour-long workflows at scale would likely have killed us financially.

Now some notes on my code.

• The main function handler includes a bunch of conditional statements you can use to only trigger reverting security group changes based on things like who requested the change, which security group was changed, whether the security group is in a specified VPC, and whehter the security group has a particular tag. None of those lines will work for you, because they refer to specific identifiers in my account – you need to change them to work in your account.
• By default, the function will revert any security group change in your account. You need to cut and paste the line “revert_security_group(event)” into a conditional block to run only on matching conditions.
• The function only works for inbound rule changes. It is trivial to modify for egress rule changes, or to restrict both ingress and egress. The IAM policy we set will work for both – you just need to change the code.
• This only works for EC2-VPC. EC2-Classic works differently, and my code cannot parse the EC2-Classic API.
• The code pulls the event details, finds the changes (which could be multiple changes submitted at the same time), and reverses them.
• There may be ways around this. I ran through it over the weekend and tested multiple ways of making an EC2-VPC security group change, and my reversion always worked, but there might be a way I don’t know about that would change the log format enough that my code wouldn’t work. I plan to update it to work with EC2-Classic, but neither I nor Securosis ever uses that EC2-Classic, and we advise clients not to use it, so that is a low priority. If you find a hole, please drop me a line.
• This works for internal (security group to security group) changes as well as external or internal IP address based rules.

Step 3: Configure the CloudWatch Event trigger

CloudWatch is Amazon’s built-in logging service. You cannot turn it off because it is the tool AWS uses to monitor and manage the performance of your instances and services. CloudWatch Logs is a newer feature you can use to store various log streams, including CloudTrail, the service that records all API calls in your account (including internal AWS calls).

• Go to CloudWatch, then Events, then Create rule.
• In the Event selector > Select event source, pick AWS API call. This only works with CloudTrail turned on.
• Pick EC2 as the Service name. Then click Specific operation(s), then AuthorizeSecurityGroupIngress. You can also add egress if you want.
• For Targets, pick Add target then Lambda function, and then select the one you just created. If you have a notification function you could add it here to get a text message or email whenever it runs, or send an alert to your SIEM.
• Then name it. It’s active by default.

Now test it. Go into the console, make a security group change, wait about 10 seconds, and refresh the console. Your changes should be gone. You can also check the CloudWatch log to see what happened, the details of the API call, and how the function executed.

Automating for Scale

This might only take 10-15 minutes once you have the code and know the process, but imagine configuring all this on hundreds or thousands of accounts at a time – which is typical for a mid-size or large organization with many projects.

To scale this up you need to create a new account deployment package. That’s what we use Trinity for (okay, that’s what I’m currently coding into Trinity, for our internal use right now). The idea is that when you provision an account you hook into it and blast out all the configurations, settings, Lambda functions, etc. using automation code.

In last year’s Black Hat training we demonstrated that, with demo code to configure alerts on IAM changes via CloudTrail and CloudWatch. We plan to go into more detail in our new Advanced Cloud Security and Applied DevOps class this summer.

It isn’t really all that complicated. Once you spend time on your cloud platform of choice and learn some basic coding via the APIs, the rest is pretty easy. It’s just basic check-a-setting, make-a-change stuff – no complex math or crazy decision trees needed (for the most part).

This is seriously exciting stuff – we security professionals can now directly manage, monitor, and manipulate our infrastructure using the exact same tools as development and operations. The infrastructure itself can identify and fix configuration and other issues – including security issues – faster than a person or (most) external tools.

Try it out. It’s easy to get started, and with minimal work you can make my sample code work for a whole host of different situations beyond basic firewalling.