Research

I was reading Roger Grimes’ interview with an offensive cybersecurity operator, and one key quote really stood out: I wish we spent as much time defensively as we do offensively. We have these thousands and thousands of people in coordinate teams trying to exploit stuff. But we don’t have any large teams that I know of for defending ourselves. In the real world, armies spend as much time defending as they do preparing for attacks. We are pretty one-sided in the battle right now. As much as I enjoy playing offensive security guy once a year at Defcon, I find defense to be a much more interesting challenge. Unfortunately many in our community don’t consider it as ‘sexy’ as penetration testing or vulnerability research. We need to change that. Most of us started our exploration of technology as hackers. I am fully willing to admit I was fascinated by cracking systems, and engaged in activities as a kid that could land me in jail now. Nothing major – I always assumed it was much easier to catch hackers and phreaks than it really was. I mean seriously, it wouldn’t have been all that hard back then. It turns out no one was looking – who knew? That’s what I get for assessing national computer law enforcement capabilities based on repeated viewings of War Games. But breaking things is, in many ways, far less challenging than protecting them. I am sick and tired of seeing researchers and pen testers on various mailing lists brag about how easy it is to get into their clients’ systems. I suspect the ones who understand the complexity of defending complex environments with limited resources keep their mouths shut. Breakers, with very few exceptions, aren’t accountable. Outside of movies, there are no consequences if they fail. Not yet, at least. No guns to the head as you sit in front of 32 widescreen monitors with 8 keyboards spread out in front of you and a coked- megelomaniac watching you waste part of your 60-second window on a visualization so your code looks good for the cameras. Nope. Builders? Defenders? Our lives are nothing but accountability. We are the firefighters, doctors, cops, and engineers all wrapped into one. Without us who would keep the porn flowing? It is a far more complex challenge, with nowhere near enough disciples. Many of our smartest focus on offensive security for obvious economic reasons. If you are good there is more money, less accountability, and more freedom. Smart defenders, even if they come up with a groundbreaking idea, need time and resources to build it – which often means productizing it and dealing with idiotic investors and bureaucracies. There are far fewer opportunities for smart defenders to perform research leading to practical tools and techniques. The only thing that can change this is money. Sure, I’d love to lead a cultural revolution, but that is more my desire to send people to re-education camps than any inherent belief we will all suddenly focus on defense due to some higher calling. (I’m serious about the camps – I have some awesome ideas). We need some serious investment – and not in academic institutions who often fail to remember sh*t needs to work outside a lab. Breaking and offensive research are important. Doing them well is hard. But defending? That is a challenge. I suspect I will be talking about this at Defcon. But with more beer. On to the Summary: Webcasts, Podcasts, Outside Writing, and Conferences Adrian’s DR Post Why Database Assessment. Rich has another article at Macworld on security for switchers. Rich’s DR post: Security Needs More Designers. Mike’s article at Information Week. Dave Lewis on Disaster Recovery at CSO Online Favorite Securosis Posts Mike Rothman: Using Amazon IAM Roles to Distribute Security Credentials (for Chef). Holy crap. A blog post from an analyst with code and screen shots! OMG… See, some analyst have some kung fu after all. David Mortman: Rich’s first post on security automation. Rich: Continuous Security Monitoring: Classification. This is a good series. Other Securosis Posts The Endpoint Security Buyer’s Guide [New Series]. Living to fight another day…. Another Disclosure Debacle, with a Twist. Using cloud-init and s3cmd to Automatically Download Chef Credentials. Incite 7/10/2013: Selfies. Kudos: Microsoft’s App Store Security Policy. How Not to Handle a Malware Outbreak. RSA Acquires Aveksa. Multitenancy is the Least Interesting Security Property of Cloud Computing. Continuous Security Monitoring: Defining CSM. Calendar Bites Google Security in the Ass. Proactive WebAppSec. Why. Continuous. Security. Monitoring? [New Series]. New Paper: Quick Wins with Website Protection Services. Database Denial of Service: Attacks. OpenStack Security Guide Released. Favorite Outside Posts Mike Rothman: Proving the skeptics wrong. You can only achieve true success when you do things for the right reasons. Seth Godin reminds me that proving someone wrong isn’t one of them. At some point you run out of people to rail against… Adrian Lane: Data Leakage In A Google World. People forget that Google is a powerful tool and often finds data companies did not want exposed. It’s a tool to hack with, and yes, a tool to phish with. Chris Pepper: Solaris patching is broken because Oracle is dumb and irresponsible. David Mortman: Dear Speaker, I Loathe You. Sincerely, Your Event Planner. Funny. Rich: No, Hacker Really Does Mean Hacker. Yep. Get over it. Research Reports and Presentations Quick Wins with Website Protection Services. Email-based Threat Intelligence: To Catch a Phish. Network-based Threat Intelligence: Searching for the Smoking Gun. Understanding and Selecting a Key Management Solution. Building an Early Warning System. Implementing and Managing Patch and Configuration Management. Defending Against Denial of Service (DoS) Attacks. Securing Big Data: Security Recommendations for Hadoop and NoSQL Environments. Tokenization vs. Encryption: Options for Compliance. Pragmatic Key Management for Data Encryption. Top News and Posts Google releases fix to OEMs for Blue Security Android security hole. This is seriously ugly. How the US (probably) spied on European allies’ encrypted faxes. Researcher finds way to commandeer any Facebook account from his mobile phone. Crimelords: Stolen credit cards… keep ‘em. It’s all about banking logins now. DEF CON to Feds. Aeroplan Provides Proactive Customer Alerting A Black Hat,

@gepeto42 had a good post: Windows Azure SQL Database, formely known as SQL Azure, is Microsoft’s managed database platform in Azure. While it is based on Microsoft SQL Server, it has various limitations that can impact how you secure and manage it. It also has some features that can help improve security. Helpful details. I am really hunting for more real-world cloud security examples, so please keep them coming… Share:

APIs go through a software lifecycle, just like any other application. The purchaser of the API develops, tests, and manages code as before, but when they publish new versions the API gateway comes into play. The gateway is what implements operational polices for APIs – serving as a proxy to enforce security, application throttling, event logging, and routing of API requests. Exposing APIs and parameters, as the API owner grants access to developers, is a security risk in and of itself. Injection attacks, semantic attacks, and any other way for an attacker to manipulate API calls is fair game unless you filter requests. Today’s post will focus on implementation of security controls through the API gateway, and how the gateway protects the API. Exposing APIs What developers get access to is the first step in securing an API. Some API calls may not be suitable for developers – some features and functions are only appropriate for internal developers or specific partners. In other case some versions of an API call are out of date, or use of internal features has been deprecated but must be retained for limited backward compatibility. The API gateway determines what a developer gets access to, based on their credentials. The gateway helps developers discover what API calls are available to them – with all the associated documentation, sample scripts, and validation tools. But behind the scenes it also constricts what each developer can see. The gateway exposes new and updated calls to developers, and acts as a proxy layer to reduce the API attack surface. The gateway may expose different API interfaces to developers depending on which credentials they provide and the authorization mapping provided by the API owner. Most gateway providers actually help with the entire production lifecycle of deployment, update, deprecation, and deletion – all based on security and access control settings. URL whitelisting We define ‘what’ an application developer can access when we provision the API – URL whitelisting defines ‘how’ it can be used. It is called a ‘whitelist’ because anything that matches it is allowed; unmatching requests are dropped. API gateways filter incoming requests according to the rules you set, validating that incoming requests meet formatting requirements. This checking catches and stops some mistakes; the API gateway’s security prevents some mistakes from proceeding by preventing use of unauthorized requests. This may be used to restrict which capabilities are available to different groups of developers, as well as which features are accessible to external requests; the gateway also prevents direct access to back end services. Incoming API calls run through a series of filters, checking general correctness of request headers and API call format. Calls that are too long, have missing parameters, or otherwise clearly fail to meet the specification are filtered out. Most whitelists are implemented as series of filters, which allows the API owner to add checks as needed and tune how API calls are validated. The owner of the API can add or delete filters as desired. Each platform comes with its own pre-defined URL filters but most customers create and add their own. Parameter parsing (Injection attacks: XML attacks JSON attacks CSRF) Attackers target application parameters. This is a traditional way to bypass access control and gain unauthorized access to back-end resources. API gateways also provide capabilities to examine user-defined content. “Parameter parsing” is examination of user-supplied content for specified attack signatures – they may identify attacks or API misuse. Content inspection works much like a ‘blacklists’ to identify known malicious API usage. Tests typically include regular expression checks of headers and content for SQL injection and cross-site scripting. Parameters are checked sequentially, one rule at a time. Some platforms provide means to programmatically extend checking, altering both which checks are performed and how they are parsed, depending on the parameters of the API call. For example you might check the contents of an XML stream for both structure and to ensure that it does not contain binary code. API gateways typically provide packaged policies for content signature of know malicious parameters, but the owner API determines which policies are deployed. Our next post will offer a selection guide – with specific comments on deployment models, evaluation checklists, and key technology differentiators. Share:

As we discussed in Defining CSM, identifying your critical assets and monitoring them continuously is a key success factor for your security program – at least if you are interested in figuring out what’s been compromised. But reality says you can’t watch everything all the time, even with these new security big data analytical thingies. So the success of your security program hinges your ability to prioritize what to do. That was the main focus of our Vulnerability Management Evolution research last year. Prioritizing requires you to determine how different asset classes will be monitored. You need a consistent process to classify assets. To define this process let’s borrow liberally from Mike’s Pragmatic CSO methodology – identifying what’s important to your organization is the critical first step. So a critical step is to make sure you’ve got a clear idea about priorities and to get the senior management buy-in on what those priorities are. You don’t want to spend a lot of money protecting a system that has low perceived value to the business. That would be silly. – The Pragmatic CSO, p25. One of the hallmarks of a mature security program is having this elusive buy-in from all levels and areas of the organization. And that doesn’t happen by itself. Business System Focus When you talk to folks about their data leak prevention efforts, a big impediment to sustainable success for the initiative is the ongoing complexity of classification. It’s just overwhelming to try putting all your organization’s data into buckets and then to maintain those buckets over time. The same issues apply to classifying computing assets. Does this server fit into that bucket? What about that network security device? And that smartphone? Multiply that by a couple hundred thousand servers, endpoints, and users and you start to understand the challenges of classification. An approach that can be very helpful for overcoming this overwhelm is to think about your computing devices in terms of a business system. To understand what that means, let’s return to The Pragmatic CSO: The key to any security program is to make sure that the most critical business systems are protected. You are not concerned about specific desktops, servers or switches. You need only be focused on fully functioning business systems. Obviously every fully functioning system consists of many servers, switches, databases, storage, and applications. All of these components need to be protected to ensure the safety of the system. – The Pragmatic CSO, p23 This requires aligning specific devices to the business systems they serve. Then those devices inherit the criticality of the business system. Simple, right? Components such as SANs and perimeter security gateways are used by multiple business systems, so they need to be classified with the most critical business system they serve. By the way, you are doing this already if you have any regulatory oversight. You know those in-scope assets for your PCI assessment? You associated those devices with PCI-relevant systems with access to protected data. Thoey require protection in accordance with the PCI-DSS guidance. Those efforts have been based on what you need to do to understand your PCI (or other mandate) scope, and we are talking about extending that mentality across your entire environment. Limited Buckets To understand the difficulty of managing all these combinations, consider the inability of many organizations to implement role-based access control on their key enterprise applications. That was largely because something like a general ledger application has hundreds of roles, with each role involving multiple access rules. Each employee may have multiple roles, so RBAC required managing A * R * E entitlements. Good luck with that. We suggest limiting the number of buckets used to classify business systems. Maybe it’s 2. You know, the stuff where you will get fired if breached, and the stuff where you won’t. Or maybe it’s 3 or 5. It’s no more than that. We are talking about monitoring devices in this series, but you needs to implement and manage different security controls for each level. It’s the concept we called Vaulting a couple years ago, also commonly known as “security enclaves”. After identifying and classifying your business systems into a manageable number of buckets you can start to think about how to monitor each class of devices according to its criticality. Be sure to build in triggers and catalysts to revisit your classifications. For example, if a business system is opened to trading partners or you authorize a new device to access critical data. As long as you understand these classifications from a point in time and need to be updated periodically, this process works well. Later in this series we will talk about different levels of security monitoring, based on the data sources and access you have to devices and specific use case(s) you are trying to achieve. Employees Count Too We have been talking about business systems and associated computing devices used to support them, but we cannot forget the weakest link in pretty much every organization: employees. You need to classify employees just like business systems. Do they have access to stuff that would be bad if it’s breached, and how are they accessing it – mobile vs. desktop, remote vs. on-network, etc.? The reality is that you can place very limited trust in endpoint devices. We see a new story of this 0-day or that breach daily, compounded by idiotic action taken by some employee. It is no wonder no one trusts endpoints. And we have no issue with that stance. If that forces you to apply more discipline and tighter controls to devices it’s all good. There is definitely a different risk profile to a low-level employee operating on a device sitting on the corporate network, compared to your CFO accessing unannounced financials on an Android tablet from a cafe in China. Part of your CSM process must be classifying, protecting, and monitoring employee devices. Where legally appropriate, of course. Gaining Consensus Now that you have bought into this classification discipline, you need to make it reality. This is where the fun begins – it requires buy-in within the organization, which is

Our man Dave Lewis has a great post on CSO Online, When Disaster Comes Calling, about the importance of making sure your disaster recovery plan actually can help you when you have, uh, a disaster. Folks don’t always remember that sometimes success is living to fight another day. At one organization that I worked for the role of disaster recovery planning fell to an individual that had neither the interest nor the wherewithal to accomplish the task. This is a real problem for many companies and organizations. The fate of their operations can, at times, reside in the hands of someone who is disinclined to properly perform the task. Sounds like a recipe for failure to me. I would say the same goes for incident response. Far too many organizations just don’t put in the time, effort, or urgency to make sure they are prepared. Until they get religion – when their business is down or their darkest secrets show up on a forum in Eastern Europe. Or you can get a bit more proactive by asking some questions and making sure someone in your organization knows the answers. So what is the actionable take away to had from this post? Take some time to review your organizations disaster recovery plans. Are backups taken? Are they tested? Are they stored offsite? Does the disaster recovery plan even exist anywhere on paper? Has that plan been tested with the staff? No plan survives first contact with the “enemy” but, it is far better to be well trained and prepared than to be caught unawares. What Dave said. Share:

Last year we documented our thoughts on buying Endpoint Security Management offerings, which basically include patch, configuration, device control, and file integrity monitoring – increasingly bundled in suites to simplify management. We planned to dig into the evolution of endpoint security suites earlier this year but the fates intervened and we got pulled into other research initiatives. Which is just as well because these endpoint security and management offerings have consolidated more quickly than we anticipated, so it makes sense to treat all these functions within a consistent model. We are pleased to kick off a new series called the “Endpoint Security Buyer’s Guide,” where we will discuss all these functions, update some of our research from last year, and provide clear buying criteria for those of you looking at these solutions in the near future. As always we will tackle the topic from the perspective of an organization looking to buy and implement these solutions, and build the series using our Totally Transparent Research methodology. Before we get going we would like to thank our friends at Lumension for potentially licensing the content when the project is done. They have long supported of our research, which we certainly appreciate. The Ongoing Challenge of Securing Endpoints We have seen this movie before – in both the online and offline worlds. You have something and someone else wants to steal it. Or maybe your competitors want to beat you in the marketplace through less than savory tactics. Or you have devices that would be useful as part of a bot network. You are a target, regardless of how large or small your organization is, whether you like it or not. Many companies make the serious mistake of thinking it won’t happen to them. With search engines and other automated tools looking for common vulnerabilities everyone is a target. Humans, alas, remain gullible and flawed. Regardless of the training you provide, employees continue to click stuff, share information, and fall for simple social engineering attacks. So your endpoints remain some of the weakest links in your security defenses. Even worse for you, unsophisticated attacks on the endpoints remain viable, so your adversaries do not need serious security kung fu to beat your defenses. The industry has responded, but not quickly enough. There is an emerging movement to take endpoints out of play. Whether using isolation technologies at the operating system or application layer, draconian whitelisting approaches, or even virtualizing desktops, organizations no longer trust endpoints and have started building complimentary defenses in acknowledgement of that reality. But those technologies remain immature, so the problem of securing endpoints isn’t going away any time soon. Emerging Attack Vectors You cannot pick up a technology trade publication without seeing terms like “Advanced Malware” and “Targeted Attacks.” We generally just laugh at all the attacker hyperbole thrown around by the media. You need to know one simple thing: these so-called “advanced attackers” are only as advanced as they need to be. If you leave the front door open they don’t need to sneak in through the ventilation ducts. Many successful attacks today are caused by simple operational failures. Whether it’s an inability to patch in a timely fashion, or to maintain secure configurations, far too many people leave the proverbial doors open on their devices. Or attackers target users via sleight-of-hand and social engineering. Employees unknowingly open the doors for attackers – and enable data compromise. There is no use sugar-coating anything. Attacker capabilities improve much faster than defensive technologies, processes, and personnel. We were recently ruminating in the Securosis chat room that offensive security (attacking things) continues to be far sexier than defense. As long as that’s the case, defenders will be on the wrong side of the battle. Device Sprawl Remember the good old days, when devices consisted of DOS PCs and a few dumb terminals? The attack surface consisted of the floppy drive. Yeah, those days are gone. Now we have a variety of PC variants running numerous operating systems. Those PCs may be virtualized and they may connect in from anywhere in the world – including networks you do not control. Even better, many employees carry smartphones in their pockets, but ‘smartphones’ are really computers. Don’t forget tablet computers either – each with as much computing power as a 20-year-old mainframe. So any set of controls and processes you implement must be consistently enforced across the sprawl of all your devices. You need to make sure your malware defenses can support this diversity. Every attack starts with one compromised device. More devices means more complexity, a far greater attack surface, and a higher likelihood of something going wrong. Again, you need to execute on your endpoint security strategy flawlessly. But you already knew that. BYOD As uplifting as dealing with emerging attack vectors and device sprawl is, we are not done complicating things. It is not just endpoints you have to defend any more. Many organizations support employee-owned devices. Don’t forget about contractors and other business partners who may have authorized access to your networks and critical data stores, connecting with devices you don’t control. Most folks assume that BYOD (bring your own device) just means dealing with those pesky Android phones and iPads, but we know many finance folks itching to get all those PCs off the corporate books. That means you need to eventually support any variety of PC or Mac any employee wants to use. Of course the security controls you put in place need to be consistent, whether your organization or an employee owns a device. The big difference is granularity of management. If a corporate device is compromised you just nuke it from orbit, as they say. Well not literally, but you need to wipe the machine down to bare metal ensuring no vestiges of the malware remain. But what about those pictures of Grandma on an employee’s device? What about their personal email and address book? Blow those away and the uproar is likely to be much worse than just idling someone for a few hours while

I picked this one up from Slashdot (yes, I still read it sometimes): Following a blog post by security company Secunia, VideoLAN (vendor of popular VLC media player) president Jean-Baptiste Kempf accuses Secunia of lying in a blog post titled ‘More lies from Secunia.’ It seems that Secunia and Jean-Baptiste Kempf have different views on whether a vulnerability has been patched. Read the VideoLAN response. It has specifics on the bugs, response times, and patches. Seems like Secunia is at fault here, and while we often ding vendors for poor disclosure responses, researchers also have responsibilities. Share:

Many people focus (often wrongly) on the new risks of cloud computing, but I am far more interested in leveraging cloud computing to improve security. I am deep into creating the advanced material for our Cloud Security Fundamentals class at Black Hat and want to toss out one of the tidbits we will cover. This is a bit more than a sneak peek, so if you plan to attend the class, don’t read this or you might get bored. A couple parts of this process are useful for more than security, so I will break it up into a few shorter pieces, each as self-contained as possible. The process is very easy once you piece it together but I had a very hard time finding necessary instructions, and there are a few tricks that really racked my analyst brain. But this isn’t about SEO – I want to make it easier for future IT pros to find what they are looking for. Overview In this example we will automate hooking into cloud servers (‘instances’) and securely deploying a configuration management tool, including automated distribution of security credentials. Specifically, we will use Amazon EC2, S3, IAM, and OpsCode Chef; and configure them to handle completely unattended installation and configuration. This is designed to cover both manually launching instances and autoscaling. With very minor modification you can use this process for Amazon VPC. With more work you could also use it for different public and private cloud providers, but in those cases the weakest link will typically be IAM. There are a few ways you can bridge that gap if necessary – I don’t know them all, but I do know they exist. First, let’s define what I mean by cloud security policy compliance. That is a broad term, and in this case I am specifically referring to automating the process of hooking servers into a configuration management infrastructure and enforcing policies. By using a programmatic configuration management system like Chef we can enforce baseline security policies across the infrastructure and validate that they are in use. For example, you can enforce that all servers are properly hardened at the operating system level, with the latest patches, and that all applications are configured according to corporate standards. The overall process is: Bootstrap all new instances into the configuration management infrastructure. Push policies to the servers, including initial and update policies. Validate that policies deployed. Continuously scan the environment for rogue systems. Isolate, integrate, or remove the rogue systems. The example we will cover in the next few posts only covers the first couple steps in detail. I have the rest mapped out but may not get it all ready in time for Black Hat – first I need to dust off some programming skills, and I learned a long time ago never to promise a release date. All of this is insanely cool, and only the very basics of Software Defined Security. Here is specifically what we will cover: Using cloud-init to bootstrap new Amazon EC2 instances. Use Amazon IAM roles to provide Temporary rotating security credentials to the instance to access the initial configuration file and digital certificate for Chef. Automatic installation of Chef, using the provided credentials. Instances will use the configuration file and digital certificate to connect to a Chef server running in EC2. The Chef server is locked down to only accept connections from specified Security Groups. S3 is configured to only allow read access of the credentials from instances with the assigned IAM role. The tools in use and how to configure them manually. I will start with how IAM roles work and how to configure them, next how to lock down access using IAM and Security Groups, then how to build the cloud-init script with details on the command-line tools it installs and configures, and finally how it connects securely back up S3 for credentials. Okay, let’s roll up our sleeves and get started… Part 2: Using Amazon IAM Roles to Distribute Security Credentials (for Chef) Part 3: Using cloud-init and s3cmd to Automatically Download Chef Credentials Share:

As I discussed in Black Hat Preview: Automating Cloud Security Policy Compliance, you can combine Amazon S3 and IAM roles to securely provision configuration files (or any other files) and credentials to Amazon EC2 or VPC instances. Here are the details. The problem to solve One of the issues in automating infrastructure is securely distributing security credentials and configuration files to servers that start automatically, without human interaction. You don’t want to embed security credentials in the images that are the basis for these running instances (servers), and you need to ensure that only approved instances can access the credentials, without necessarily knowing anything about the instance. The answer is to leverage the cloud infrastructure itself to identify and propagate the credentials. We can handle it all in the management plane without manually touching servers or embedding long-term credentials. In our example we will use these to distribute an initial Chef configuration file and validation certificate. Securely bootstrapping Chef clients A Chef node is a server or workstation with the Chef agent running on it. This is the application that connects to the Chef server to accept configuration pushes and run scripts (recipes). There are four ways to install it on a cloud instance: Manually log into the instance and install the chef-client software, then transfer over the server’s validation certificate and configuration file. Almost nobody does this in the cloud, outside of development and testing. Embed the client software and configuration files in the machine image for use at launch (instantiation). This is common but means you need to maintain your own images rather than using public ones. Remotely bootstrap the instance. The Chef management software (knife) can connect to the instance via ssh to configure everything automatically, but that requires its `ssh private key. Use cloud-init or another installation script to install Chef onto a clean ‘base’ (unconfigured) instance, and IAM roles to allow the instance to connect to a repository such as S3 to download the initial configuration file and server certificate. This is what I will explain. Configuring AWS IAM roles to distribute credentials Amazon recently added a feature called IAM roles. Amazon Web Services (like some other public and private cloud platforms) supports granular identity management down to the object level. This is critical for proper segregation and isolation of cloud assets. AWS previously only supported users and groups, which are (and I’m simplifying) static collections of users, servers, and other objects in AWS. Users and groups are great, but they provide users or servers with static security credentials such as an Access Key and Secret Key, X.509 certificate, or a username and password for web UI access. IAM Roles are different. They are temporary credentials applied to AWS assets, such as a running instance. They include an Access Key and Secret Key provided to the object via an API call, and a token. You need all three to sign requests, and they are rotated (approximately every 24 hours in my experience). So if someone steals the keys they won’t work without the token. If they also get a token it expires in a day. In our example we will create an S3 bucket to hold our Chef configuration client.rb configuration file and validation.pem digital certificate. We will then switch over to AWS IAM to create a new role and assign it read privileges to S3. Then we will tweak the policy to only allow access to that bucket. Finally we will launch an instance, assign the role, then log in and show the credentials. You wouldn’t do this in production, but it illustrates how roles work. Step by step I assume you have some knowledge of AWS here. If you want granular instructions, take our class. I also assume you have a Chef server set up in EC2 someplace, or use Hosted Chef. If you want to know how to do that, take the class. 🙂 AWS Console Log in and ensure your Chef server has its own Security Group. Create a new Security Group for your instances (or pick any group you already have). Our example is very locked down, which may not be appropriate for your environment. Open ports 4000, 4040, and 80 in the Chef Server security group from your instance security group. I haven’t had time to play with it, but we might be able to double down and allow access by role. I will test before Black Hat – it doesn’t take long, but I just got the idea. Return the favor and open 4000 and 4040 into the instance group from the server group. Amazon S3 section of AWS Console Create a new bucket (e.g., cloudsec). Load a random file for testing later if you want. If you have a Chef server place client.rb and validation.pem here – you will need these to complete our example. IAM section Create a new role called ChefClient. You can do this all via API or by write the policy by hand, but we use the GUI. Select AWS Service Roles, then Amazon EC2. This grants the designated rights to EC2 assets with the assigned role. Continue. Select Select Policy Template and then Amazon S3 Read Only Access. Continue. After this you can name the policy, then adjust it to apply to only the single bucket – not your entire Amazon S3 account. Change the entry “Resource: *” to “Resource: arn:aws:s3:::your_bucket”. I also added a safety wildcard, so your policy should look like the screenshot below EC2 section Launch a new instance. Ubuntu is a safe bet, and what we use to demonstrate the temporary credentials. On the Instance Details screen assign your IAM role. You also probably want to put it in the same availability zone as your Chef server, and later on into the right security group. Instance Once everything is running, log into your instance. Type wget -O — -q ‘http://169.254.169.254/latest/meta-data/iam/security-credentials/myrole’, replacing ‘myrole’ with the name of your role (case sensitive). You should see your temporary AWS credentials, when they were issued, and when they expire. You have now configured your environment to support transfer of the security credentials only to instances assigned the appropriate role (ChefClient in my case). Your instance has temporary credentials that Amazon rotates for you, minimizing exposure. AWS also requires a token so the Access Key

Our last post described how to use Amazon EC2, S3, and IAM as a framework to securely and automatically download security policies and credentials. That’s the infrastructure side of the problem, and this post will show what you need to do to the instance to connect to this infrastructure, grab the credentials, install and configure Chef, and connect to the Chef server. The advantage of this structure is that you don’t need to embed credentials into your machine image, and you can use stock (generic) operating systems are on public clouds. In private clouds it is also useful because it reduces the number of machine images to maintain. These instructions can be modified to work in other cloud platforms, but your mileage will vary. They also require an operating system that supports cloud-init (Windows uses ec2config, which I know very little about, but also appears to support user data scripts). I will walk through the details of how this works, but you won’t use any of these steps manually. They are just explanation, to give you what you need to adapt this for other circumstances. Using cloud-init cloud-init is software for certain Linux variants that allows your cloud controller to pass scripts to new instances as they are launched from an image (bootstrapped). It was created by Canonical (the Ubuntu guys) and is very frequently packaged into Linux machine images (AMIs). ec2config offers similar functionality for Windows. Users pass the script to their instances, specifying the User Data field (for web interface) or argument (for command line). It is a bit of a pain because you don’t get any feedback – you need to debug from the system log – but it works well and allows tight control. Commands run as root before anyone can even log into the instance, so cloud-init is excellent for setting up secure configurations, loading ssh keys, and installing software. Note that cloud-init is a bootstrapping tool for configuring an instance the first time it runs – it is not a management tool because after launch you cannot access it any more. For an example see our full script at the bottom of this post. You can download and manipulate files easily with cloud-init, but unless you want to embed static credentials in your script there is an authentication issue. That’s where AWS IAM roles and S3 help, thanks to a very recent update to s3cmd. Configuring s3cmd to use IAM roles s3cmd is a command-line tool to access Amazon S3. Amazon S3 isn’t like a normal file share – it is only accessible through Amazon’s API. s3cmd provides access to S3 like a local directory, as well as administration of S3. It is available in most Linux repositories for packaged installation, but the bundled versions do not yet support IAM roles. Version 1.5 alpha 2 and later add role support, so that’s what we need to use. You can download the alpha 3 release, but if you are reading this post in the future I suggest checking for a more recent version on the main page, linked above. To install s3cmd just untar the file. If you aren’t using roles you now need to configure it with your credentials. But if you have assigned a role, s3cmd should work out of the box without a configuration file. Unfortunately I discovered a lot of weirdness once I tried to out it in a cloud-init script. The issue is that running it under cloud-init runs it as root, which changes s3cmd’s behavior a bit. I needed to create a stub configuration file without any credentials, then use a command-line argument to specify that file. Here is what the stub file looks like: [default] access_key = secret_key = security_token = Seriously, that’s it. Then you can use a command line such as: s3cmd –config /s3cmd-1.5.0-alpha3/s3cfg ls s3://cloudsec/ Where s3cfg is your custom configuration file (you can see the path there too). That’s all you need. s3cmd detects that it is running in role mode and pulls your IAM credentials if you don’t specify them in the configuration file. Scripted installation of the Chef client The Chef client is very easy to install automatically. The only tricky bit is the command-line arguments to skip the interactive part of the install; then you copy the configuration files where they are needed. The main instructions for package installation are in the Chef wiki. You can also use the omnibus installer, but packaged installation is better for automated scripting. The Chef instructions show you how to add the OpsCode repository to Ubuntu so you can “apt-get install”. The trick is to point the installer to your Chef server, using the following code instead of a straight “apt-get install chef-client”: echo “chef chef/chef_server_url string http://your-server-IP:4000” \ | sudo debconf-set-selections && sudo apt-get install chef -y –force-yes Then use s3cmd to download client.rb & validation.pem and place them into the proper locations. In our case this looks like: s3cmd –config /s3cmd-1.5.0-alpha3/s3cfg –force get s3://cloudsec/client.rb /etc/chef/client.rb s3cmd –config /s3cmd-1.5.0-alpha3/s3cfg –force get s3://cloudsec/validation.pem /etc/chef/validation.pem That’s it! Tying it all together The process is really easy once you set this up, and I went into a ton of extra detail. Here’s the overview: Set up your S3, Chef server, and IAM role as described in the previous post. Upload client.rb and validation.pem from your Chef server into your bucket. (Execute “knife client ./” to create them). Launch a new instance. Select the IAM Role you set up for Chef and your S3 bucket. Specify your customized cloud-init script, customized from the sample below, into the User Data field or command-line argument. You can also host the script as a file and load it from a central repository using the include file option. Execute chef-client. Profit. If it all worked you will see your new instance registered in Chef once the install scripts run. If you don’t see it check the System Log (via AWS – no need to log into the server) to see where you script failed. This is the script we will use for our training, which should be easy to adapt. #cloud-config apt_update: true #apt_upgrade: true packages: — curl fixroutingsilliness: – &fix_routing_silliness | public_ipv4=$(curl -s http://169.254.169.254/latest/meta-data/public-ipv4) ifconfig eth0:0 $public_ipv4 up configchef: — &configchef | echo “deb http://apt.opscode.com/ precise-0.10 main” | sudo tee /etc/apt/sources.list.d/opscode.list apt-get update curl http://apt.opscode.com/packages@opscode.com.gpg.key | sudo apt-key add – echo “chef chef/chef_server_url string http://ec2-54-218-102-48.us-west-2.compute.amazonaws.com:4000” | sudo debconf-set-selections && sudo apt-get install