Research

Update: Amazon published some details. Less than 10% of AWS systems are affected, and the vulnerability will be disclosed October 1st. As suspected this is about Xen – not the bash vulnerability. Yesterday I received notice that Amazon Web Services is force rebooting one of my instances. Then more emails started rolling in, and it looks like many (or all) of them will be rebooted during a single maintenance window. It has been a few years since this happened, and the reason ties into how AWS updates the servers your instances run on. We actually teach this in our cloud security training class, including how to architect your own cloud so you might not have to do the same thing – with, of course, many caveats. My initial assumption was application of a quiet security patch, and that looks dead on: From @ClipperChip via Matt Green on Twitter: Amazon rebooting all AWS instances (https://t.co/xg2XoXDdEe) + an undisclosed advisory on http://t.co/PdLqk8qXSE http://t.co/Fo1beT7xrN 🙂 And here is what looks like that vuln: XSA-108 | 2014-10-01 12:00 | none (yet) assigned | (Prereleased, but embargoed) How AWS updates servers Amazon uses a modified version of the Xen hypervisor. Our understanding of their architecture indicates they do not support live migration. Live migration, available under VMware as vMotion, allows you to move a running virtual machine from one physical host to another without shutting it down. When you build a cloud, host servers consist of (at least) a hypervisor with management and connectivity components. Sometimes, as with OpenStack, you even have a usable operating system. All these components need to be updated periodically. Some updates require rebooting the host server. To update the hypervisor you typically need to shut down the virtual machines (instances) running on top of it. There are two common ways to manage these updates to reduce downtime: Update a host without any virtual machines running on it, then live migrate instances from a vulnerable host to a patched one. Then update the vulnerable host once all its instances are running elsewhere. If you cannot live migrate, do the same thing by shutting down and restarting the instances. If you built your cloud properly you can set a rule in the controller to not launch instances on the vulnerable host while preparing to reboot. Then the simple act of shutting down and relaunching the instance will automatically migrate it to a patched host. In case you didn’t realize, every time you shut an instance down and start it again you likely move to a new host server. That is just normal cloud automation at work. When AWS has a large security patch like this they cannot rely on all customers conveniently relaunching during the desired window, so they need to take a maintenance window and do it for all affected users. Simple reboots generally do not trigger a host migration because a reboot doesn’t actually shutdown the entire instance – the virtual machine just executes the operating system shutdown and reboot procedures, but the instance is never destroyed or completely halted. Many people don’t architect resilient servers to handle reboots, which is the problem. Or the reboots require some manual testing. This is why I am a massive fan of DevOps – its techniques provide extra resiliency for situations like this – but that’s for another post. Our cloud security training covers this, and one critical security requirement when building a private (or public) cloud is to understand your patching requirements and their implications for instances. For example if you architect for live migration you can reduce required reboots, by accepting different implications and constraints. Share:

Updated: I made a mistake and gave Akamai credit. Stephane doesn’t work for them – I misread the post. Fixed. Critical update: Red Hat confirmed their patch is incomplete, and patched bash is still exploitable. The technical term is “cluster fuck”. Anything you patch now will need to be repatched later. For critical systems consider the workaround in their post. For everything else, wait until your vendors release complete patches. Earlier today details of a vulnerability in the UNIX/Linux/OS X tool bash, discovered by Stephane Chazelas, became public with a disclosure and patch by Red Hat. It is called Shellshock, and it might be worse than Heartbleed. Most of you reading this are likely extremely familiar with bash, but in case you aren’t it is the most popular command-line shell program in the UNIX world, installed on pretty much anything and everything. From Red Hat: Coming back to the topic, the vulnerability arises from the fact that you can create environment variables with specially-crafted values before calling the bash shell. These variables can contain code, which gets executed as soon as the shell is invoked. The name of these crafted variables does not matter, only their contents. You might be thinking that someone needs to log in before they can ever reach bash, so no big deal, right? Wrong. Access to bash is embedded in a ton of applications. From CGI scripts running on Apache web sites to all sorts of random applications. Here is the short explanation of why this is so bad, and why we will likely be dealing with it for years: bash is embedded and accessed in so many ways that we cannot fully understand its depth of use. Many systems you would never think of as having a command line use bash to run other programs. I have used it myself, a bunch, in programs I have written – and I barely code. We cannot possibly understand all the ways an attacker could interact with bash to exploit this vulnerability. As Rob Graham has discovered, this is likely wormable. That places it into Code Red/Nimbda territory. A workable bug that can exploit public web servers is scary. We don’t know for sure, Rob doesn’t know for sure, but it looks very very possible. Potential worms are like staring at the smoking volcano while the earthquakes stir your martini – they aren’t the sort of thing you can wait for definitive proof on before taking seriously. There are rumors the patch may be incomplete. There is already a Metasploit module. Gee, thanks guys… you couldn’t give us a day? I strongly suggest keeping up with Rob’s analysis. There is really only one option: patch. It isn’t a fancy patch, but fragile systems could still suffer downtime. And you may need to re-patch if the original patch turns out to be faulty, which is always terrible. I will patch my systems and keep my ears open for any updates. Don’t trust any security vendor who claims they can block this. Patching is the only way to fix the core problem, which likely includes multiple exploit vectors. I will give bonus points to anyone who finds a vendor using Shellshock in their marketing, which then turns out to have a vulnerable product. Any security product based on UNIX/Linux is potentially vulnerable, although not necessarily exploitable. I suspect the Microsoft Security Response Center is very much enjoying their quiet evening. Share:

I have a great job. The combination of extended coverage areas, coupled with business to tech, and everything in between, makes it so. In this week alone I have talked to customers about Agile development and process adjustments, technical details of how to deploy masking for Hadoop, how to choose between two SIEM vendors, and talked to a couple vendors about Oracle and SAP security. The breadth of stuff I am exposed to is awesome. People often ask me if I want to go back to being a CTO or offer me VP of Engineering positions, but I cannot imagine going back to just focusing on one platform. I don’t get my hands as dirty, but in some ways it is far more difficult to learn nuances of half a dozen competitive product areas than jus one. And what a great time to be neck deep in security … so long as I don’t drown in data. Learning about DevOps is fascinating. Talking to people who are pushing forward with continuous integration and deployment, and watching them break apart old dev/QA/IT cycles, provides a euphoric glimpse at what’s possible with Agile code development. Then I speak with more traditional firms, still deeply embedded in 24-month waterfall development. The long tail (and neck, and back) of their process feels like a cold bucket of reality – I wonder if a significant percentage of companies will ever be agile. When I contrast Why Security Automation is the Way Forward with mid-sized enterprises, I get another cold slap from reality. I speak with many firms who cannot get servers patched every other quarter. Security patches for open source will come faster than before, but organizational lag holds firm. It is clear that many firms have a decade-long transition to more agile processes in store, and some will never break down the cultural barriers between different teams within their companies. Gunnar’s recent To Kill A Flaw post is excellent. Too good, in fact – his post includes several points that demand their own blog entries. One of the key points Gunnar has been making lately, especially in light of the nude celebrity photo leaks, is that credentials are a “zero day” attack. You need to keep that in mind when designing identity and access management today. If a guessed password provides a clear way in, you need to be able to live with that kind of 0-day. That is why we see a push away from simple passwords toward identity tokens, time-limited access, and risk-based authorization on the back end. Not only is it harder to compromise credentials, the relative risk score moves from 10 to about 4 because the scope of damage is lessened. A family member who is a bit technically challenged asked me “Is the Bash Bug Bad?” “Bad. Bad-bad-bad!” I left it at that. I think I will use that answer for press as well. On to the Summary: Webcasts, Podcasts, Outside Writing, and Conferences Adrian quoted on Chip and Pin. Rich quoted in Denver Post on How to Protect Your Data in the Cloud Favorite Securosis Posts Mike Rothman: Secure Agile Development: Building a Security Tool Chain. The testing process is where the security gets real. Great series from Adrian and Rich. Adrian Lane: Why the bash vulnerability is such a big deal (updated). Excellent overview by Rich on the bash vulnerability hyped as ‘shellshock’. Other Securosis Posts Why Amazon is Rebooting Your Instances (Updated). Hindsight FTW. Summary: Run Free. Secure Agile Development: Process Adjustments. Incite 9/17/2014: Break the Cycle. Firestarter: Apple Pay. Fix Something. Favorite Outside Posts Mike Rothman: The Pirate Bay Operations: 21 Virtual Machines Are Used To Run The File-sharing Website. This cloud thing might not be a fad. This is how you take an international network of stuff and move it quickly… And the torrents are pleased. Adrian Lane: Can Static Analysis replace Code Reviews? The case for security … this shows why old-fashioned manual scans cannot be fully replaced by static analysis. It also shows the need to train developers on what type of flaws to look for. Good post! Research Reports and Presentations Pragmatic WAF Management: Giving Web Apps a Fighting Chance. The Security Pro’s Guide to Cloud File Storage and Collaboration. The 2015 Endpoint and Mobile Security Buyer’s Guide. Analysis of the 2014 Open Source Development and Application Security Survey. Defending Against Network-based Distributed Denial of Service Attacks. Reducing Attack Surface with Application Control. Leveraging Threat Intelligence in Security Monitoring. The Future of Security: The Trends and Technologies Transforming Security. Security Analytics with Big Data. Security Management 2.5: Replacing Your SIEM Yet? Top News and Posts Three critical changes to PCI DSS 3.0 Trustworthy Computing RSA Signature Forgery in NSS Data Masking Bundled with Cloudera, Hortonworks Apple releases iOS 8 with 56 security patches CloudFlare Introduces SSL Without Private Key SSL Issues with this Blog? by Branden Williams. Bash ‘shellshock’ bug is wormable Funds in Limbo After FBI Seizes Funds from Cyberheist. Not a new problem, just a new cause. Jimmy John’s Confirms Breach at 216 Stores Julian Sanchez on NSA reform. Home Depot’s Former Security Engineer Had a Legacy of Sabotage Ping Identity Scoops $35M To Authenticate Everywhere Blog Comment of the Week This week’s best comment goes to Andrew Hay, in response to Why the bash vulnerability is such a big deal (updated). As per a conversation I had with HD Moore, he loves to release the Metasploit modules as quickly as possible in an effort to eliminate pay-per-exploit companies from profiting off of a particular vuln. I kind of agree with him. Share: