I recently wrote a series on automating cloud security configuration management by taking advantage of DevOps principles and properties of the cloud. Today I will build on that to show you how the management plane can make security easier than traditional infrastructure with a little ruby code.
This is another example of material covered in our Black Hat cloud security training class.
Abstraction enhances management
People tend to focus on multitenancy, but the cloud’s most interesting characteristics are abstraction and automation. Separating our infrastructure from the physical boxes and wires it runs on, and adding a management plane, gives us a degree of control that is difficult or impossible to obtain by physically tracing all those wires and walking around to the boxes.
Dev and ops guys really get this, but we in security haven’t all been keeping up – not that we are stupid, but we have different priorities.
That management plane enables us to do things such as instantly survey our environment and get details on every single server. This is an inherent feature of the cloud, because if you can’t find a server the cloud doesn’t know where it is – which would mean a) it effectively doesn’t exist, and b) you cannot be billed for it. There ain’t no Neo hiding away in AWS or OpenStack.
For security this is very useful. It makes it nearly impossible for an unmanaged system to hide in your official cloud (although someone can always hook something in somewhere else). It also enables near-instant control.
For example, quarantining a system is a snap. With a few clicks or command lines you can isolate something on the network, lock down management plane access, and lock out logical access. We can do all this on physical servers, but not as quickly or easily.
(I know I am skipping over various risks, but we have covered them before and they are fodder for future posts).
In today’s example I will show you how 40 lines of commented Ruby (just 23 lines without comments!) can scan your cloud and identify any unmanaged systems.
Finding unmanaged cloud servers with AWS, Chef, and Ruby
This examples is actually super simple. It is a short Ruby program that uses the Amazon Web Services API to list all running instances. Then it uses the Chef API to get a list of managed clients from your Chef server (or Hosted Chef). Compare the list, find any discrepancies, and profit.
This is only a basic proof of concept – I found seen far more complex and interesting management programs using the same principles, but none of them written by security professionals. So consider this a primer. (And keep in mind that I am no longer a programmer, but this only took a day to put together).
There are a couple constraints. I designed this for EC2, which limits the number of instances you can run. Nearly the same code would work for VPC, but while I run everything live in memory, there you would probably need a database to run this at scale. This was also built for quick testing, and in a real deployment you would want to enhance the security with SSL and better credential management. For example, you could designate a specific security account with IAM credentials for Amazon Web Services that only allows it to pull instance attributes but not initiate other actions.
You could even install this on an instance inside EC2 using IAM roles, as we discussed previously.
Lastly, I believe I discovered two different bugs in the Ridley gem, which is why I have to correlate on names instead of IP addresses – which would be more canonical. That cost me a couple hours of frustration.
Here is the code. To use it you need a few things:
- An access key and secret key for AWS with rights to list instances.
- A Chef server, and a client and private key file with rights to make API calls.
- The
aws-sdkandridleyRuby gems. - Network access to your Chef server.
Remember, all this can be adapted for other cloud platforms, depending on their API support.
# Securitysquirrel proof of concept by rmogull@securosis.com
# This is a simple demonstration that evaluates your EC2 environment and identifies instances not managed with Chef.
# It demonstrates rudimentary security automation by gluing AWS and Chef together using APIs.
# You must install the aws-sdk and ridley gems. ridley is a Ruby gem for direct Chef API access.
require "rubygems"
require "aws-sdk"
require "ridley"
# This is a PoC, so I hard-coded the credentials. Fill in your own, or adjust the
program to use a configuration file or environment variables. Don't forget to specify the region...
AWS.config(access_key_id: 'your-access-key', secret_access_key: 'your-secret-key', region: 'us-west-2')
# Fill in the ec2 class
ec2 = AWS.ec2 #=> AWS::EC2
ec2.client #=> AWS::EC2::Client
# Memoize is an AWS function to speed up collecting data by keeping the hash in local cache.
This line creates a list of EC2 private DNS names, which we will use to identify nodes in Chef.
instancelist = AWS.memoize { ec2.instances.map(&:private_dns_name) }
# Start a ridley connection to our Chef server. You will need to fill in your own credentials or
pull them from a configuration file or environment variables.
ridley = Ridley.new(
server_url: "http://your.chef.server",
client_name: "your-client-name",
client_key: "./client.pem",
ssl: { verify: false }
)
# Ridley has a bug, so we need to work on the node name, which in our case is the same as the EC2 private DNS name.
For some reason node.all doesn't pull IP addresses (it should) which we would prefer to use.
nodes = ridley.node.all
nodenames = nodes.map { |node| node.name }
# For every EC2 instance, see if there is a corresponding Chef node.
puts ""
puts ""
puts "Instance => managed?"
puts ""
instancelist.each do |thisinstance|
managed = nodenames.include?(thisinstance)
puts " #{thisinstance} #{managed} "
end
Where to go next
If you run the code above you should see output like this:
Instance => managed?
ip-172-xx-37-xxx.us-west-2.compute.internal true
ip-172-xx-37-xx.us-west-2.compute.internal true
ip-172-xx-35-xxx.us-west-2.compute.internal true
ip-172-3xx1-40-xxx.us-west-2.compute.internal false
That is the internal DNS name of the instance and whether or not it is managed by Chef. You could easily adapt this to pull different attributes, such as the instance ID.
No big deal, but think about where you could take it from here. Using the same framework, you could then:
- Identify the owner of the server.
- Quarantine it on the network.
- Transfer management within AWS to the security team.
- Snapshot it and shut it down.
- Bootstrap it into Chef automatically, if you have the private key for the server.
- Allow it to continue running but cut off external network connections.
- Automatically kick off a vulnerability assessment.
Can we do this with traditional infrastructure? Sure, by building a scanner and hoping we have the proper network access. Definitely not as easy as here, nor as automatic. This is what we call Software Defined Security.
The most interesting part of this is bridging the divide between DevOps and security. We in security are, for the most part, really behind on understanding today’s operational models. Te DevOps folks are implementing a hell of a lot of security without us. But they don’t understand security principles as well as we do, so they need our help.
I am learning a lot as I go through all this, and the more I learn, the more it confirms that the future of security will look very different than the way we practice it today, in very positive ways.
Comments