Why Amazon is Rebooting Your Instances (Updated)

Update: Amazon published some details. Less than 10% of AWS systems are affected, and the vulnerability will be disclosed October 1st. As suspected this is about Xen – not the bash vulnerability. Yesterday I received notice that Amazon Web Services is force rebooting one of my instances. Then more emails started rolling in, and it looks like many (or all) of them will be rebooted during a single maintenance window. It has been a few years since this happened, and the reason ties into how AWS updates the servers your instances run on. We actually teach this in our cloud security training class, including how to architect your own cloud so you might not have to do the same thing – with, of course, many caveats. My initial assumption was application of a quiet security patch, and that looks dead on: From @ClipperChip via Matt Green on Twitter: Amazon rebooting all AWS instances ( + an undisclosed advisory on 🙂 And here is what looks like that vuln: XSA-108 | 2014-10-01 12:00 | none (yet) assigned | (Prereleased, but embargoed) How AWS updates servers Amazon uses a modified version of the Xen hypervisor. Our understanding of their architecture indicates they do not support live migration. Live migration, available under VMware as vMotion, allows you to move a running virtual machine from one physical host to another without shutting it down. When you build a cloud, host servers consist of (at least) a hypervisor with management and connectivity components. Sometimes, as with OpenStack, you even have a usable operating system. All these components need to be updated periodically. Some updates require rebooting the host server. To update the hypervisor you typically need to shut down the virtual machines (instances) running on top of it. There are two common ways to manage these updates to reduce downtime: Update a host without any virtual machines running on it, then live migrate instances from a vulnerable host to a patched one. Then update the vulnerable host once all its instances are running elsewhere. If you cannot live migrate, do the same thing by shutting down and restarting the instances. If you built your cloud properly you can set a rule in the controller to not launch instances on the vulnerable host while preparing to reboot. Then the simple act of shutting down and relaunching the instance will automatically migrate it to a patched host. In case you didn’t realize, every time you shut an instance down and start it again you likely move to a new host server. That is just normal cloud automation at work. When AWS has a large security patch like this they cannot rely on all customers conveniently relaunching during the desired window, so they need to take a maintenance window and do it for all affected users. Simple reboots generally do not trigger a host migration because a reboot doesn’t actually shutdown the entire instance – the virtual machine just executes the operating system shutdown and reboot procedures, but the instance is never destroyed or completely halted. Many people don’t architect resilient servers to handle reboots, which is the problem. Or the reboots require some manual testing. This is why I am a massive fan of DevOps – its techniques provide extra resiliency for situations like this – but that’s for another post. Our cloud security training covers this, and one critical security requirement when building a private (or public) cloud is to understand your patching requirements and their implications for instances. For example if you architect for live migration you can reduce required reboots, by accepting different implications and constraints. Share:

Read Post

Why the bash vulnerability is such a big deal (updated)

Updated: I made a mistake and gave Akamai credit. Stephane doesn’t work for them – I misread the post. Fixed. Critical update: Red Hat confirmed their patch is incomplete, and patched bash is still exploitable. The technical term is “cluster fuck”. Anything you patch now will need to be repatched later. For critical systems consider the workaround in their post. For everything else, wait until your vendors release complete patches. Earlier today details of a vulnerability in the UNIX/Linux/OS X tool bash, discovered by Stephane Chazelas, became public with a disclosure and patch by Red Hat. It is called Shellshock, and it might be worse than Heartbleed. Most of you reading this are likely extremely familiar with bash, but in case you aren’t it is the most popular command-line shell program in the UNIX world, installed on pretty much anything and everything. From Red Hat: Coming back to the topic, the vulnerability arises from the fact that you can create environment variables with specially-crafted values before calling the bash shell. These variables can contain code, which gets executed as soon as the shell is invoked. The name of these crafted variables does not matter, only their contents. You might be thinking that someone needs to log in before they can ever reach bash, so no big deal, right? Wrong. Access to bash is embedded in a ton of applications. From CGI scripts running on Apache web sites to all sorts of random applications. Here is the short explanation of why this is so bad, and why we will likely be dealing with it for years: bash is embedded and accessed in so many ways that we cannot fully understand its depth of use. Many systems you would never think of as having a command line use bash to run other programs. I have used it myself, a bunch, in programs I have written – and I barely code. We cannot possibly understand all the ways an attacker could interact with bash to exploit this vulnerability. As Rob Graham has discovered, this is likely wormable. That places it into Code Red/Nimbda territory. A workable bug that can exploit public web servers is scary. We don’t know for sure, Rob doesn’t know for sure, but it looks very very possible. Potential worms are like staring at the smoking volcano while the earthquakes stir your martini – they aren’t the sort of thing you can wait for definitive proof on before taking seriously. There are rumors the patch may be incomplete. There is already a Metasploit module. Gee, thanks guys… you couldn’t give us a day? I strongly suggest keeping up with Rob’s analysis. There is really only one option: patch. It isn’t a fancy patch, but fragile systems could still suffer downtime. And you may need to re-patch if the original patch turns out to be faulty, which is always terrible. I will patch my systems and keep my ears open for any updates. Don’t trust any security vendor who claims they can block this. Patching is the only way to fix the core problem, which likely includes multiple exploit vectors. I will give bonus points to anyone who finds a vendor using Shellshock in their marketing, which then turns out to have a vulnerable product. Any security product based on UNIX/Linux is potentially vulnerable, although not necessarily exploitable. I suspect the Microsoft Security Response Center is very much enjoying their quiet evening. Share:

Read Post

Friday Summary: September 26, 2014

I have a great job. The combination of extended coverage areas, coupled with business to tech, and everything in between, makes it so. In this week alone I have talked to customers about Agile development and process adjustments, technical details of how to deploy masking for Hadoop, how to choose between two SIEM vendors, and talked to a couple vendors about Oracle and SAP security. The breadth of stuff I am exposed to is awesome. People often ask me if I want to go back to being a CTO or offer me VP of Engineering positions, but I cannot imagine going back to just focusing on one platform. I don’t get my hands as dirty, but in some ways it is far more difficult to learn nuances of half a dozen competitive product areas than jus one. And what a great time to be neck deep in security … so long as I don’t drown in data. Learning about DevOps is fascinating. Talking to people who are pushing forward with continuous integration and deployment, and watching them break apart old dev/QA/IT cycles, provides a euphoric glimpse at what’s possible with Agile code development. Then I speak with more traditional firms, still deeply embedded in 24-month waterfall development. The long tail (and neck, and back) of their process feels like a cold bucket of reality – I wonder if a significant percentage of companies will ever be agile. When I contrast Why Security Automation is the Way Forward with mid-sized enterprises, I get another cold slap from reality. I speak with many firms who cannot get servers patched every other quarter. Security patches for open source will come faster than before, but organizational lag holds firm. It is clear that many firms have a decade-long transition to more agile processes in store, and some will never break down the cultural barriers between different teams within their companies. Gunnar’s recent To Kill A Flaw post is excellent. Too good, in fact – his post includes several points that demand their own blog entries. One of the key points Gunnar has been making lately, especially in light of the nude celebrity photo leaks, is that credentials are a “zero day” attack. You need to keep that in mind when designing identity and access management today. If a guessed password provides a clear way in, you need to be able to live with that kind of 0-day. That is why we see a push away from simple passwords toward identity tokens, time-limited access, and risk-based authorization on the back end. Not only is it harder to compromise credentials, the relative risk score moves from 10 to about 4 because the scope of damage is lessened. A family member who is a bit technically challenged asked me “Is the Bash Bug Bad?” “Bad. Bad-bad-bad!” I left it at that. I think I will use that answer for press as well. On to the Summary: Webcasts, Podcasts, Outside Writing, and Conferences Adrian quoted on Chip and Pin. Rich quoted in Denver Post on How to Protect Your Data in the Cloud Favorite Securosis Posts Mike Rothman: Secure Agile Development: Building a Security Tool Chain. The testing process is where the security gets real. Great series from Adrian and Rich. Adrian Lane: Why the bash vulnerability is such a big deal (updated). Excellent overview by Rich on the bash vulnerability hyped as ‘shellshock’. Other Securosis Posts Why Amazon is Rebooting Your Instances (Updated). Hindsight FTW. Summary: Run Free. Secure Agile Development: Process Adjustments. Incite 9/17/2014: Break the Cycle. Firestarter: Apple Pay. Fix Something. Favorite Outside Posts Mike Rothman: The Pirate Bay Operations: 21 Virtual Machines Are Used To Run The File-sharing Website. This cloud thing might not be a fad. This is how you take an international network of stuff and move it quickly… And the torrents are pleased. Adrian Lane: Can Static Analysis replace Code Reviews? The case for security … this shows why old-fashioned manual scans cannot be fully replaced by static analysis. It also shows the need to train developers on what type of flaws to look for. Good post! Research Reports and Presentations Pragmatic WAF Management: Giving Web Apps a Fighting Chance. The Security Pro’s Guide to Cloud File Storage and Collaboration. The 2015 Endpoint and Mobile Security Buyer’s Guide. Analysis of the 2014 Open Source Development and Application Security Survey. Defending Against Network-based Distributed Denial of Service Attacks. Reducing Attack Surface with Application Control. Leveraging Threat Intelligence in Security Monitoring. The Future of Security: The Trends and Technologies Transforming Security. Security Analytics with Big Data. Security Management 2.5: Replacing Your SIEM Yet? Top News and Posts Three critical changes to PCI DSS 3.0 Trustworthy Computing RSA Signature Forgery in NSS Data Masking Bundled with Cloudera, Hortonworks Apple releases iOS 8 with 56 security patches CloudFlare Introduces SSL Without Private Key SSL Issues with this Blog? by Branden Williams. Bash ‘shellshock’ bug is wormable Funds in Limbo After FBI Seizes Funds from Cyberheist. Not a new problem, just a new cause. Jimmy John’s Confirms Breach at 216 Stores Julian Sanchez on NSA reform. Home Depot’s Former Security Engineer Had a Legacy of Sabotage Ping Identity Scoops $35M To Authenticate Everywhere Blog Comment of the Week This week’s best comment goes to Andrew Hay, in response to Why the bash vulnerability is such a big deal (updated). As per a conversation I had with HD Moore, he loves to release the Metasploit modules as quickly as possible in an effort to eliminate pay-per-exploit companies from profiting off of a particular vuln. I kind of agree with him. Share:

Read Post

Totally Transparent Research is the embodiment of how we work at Securosis. It’s our core operating philosophy, our research policy, and a specific process. We initially developed it to help maintain objectivity while producing licensed research, but its benefits extend to all aspects of our business.

Going beyond Open Source Research, and a far cry from the traditional syndicated research model, we think it’s the best way to produce independent, objective, quality research.

Here’s how it works:

  • Content is developed ‘live’ on the blog. Primary research is generally released in pieces, as a series of posts, so we can digest and integrate feedback, making the end results much stronger than traditional “ivory tower” research.
  • Comments are enabled for posts. All comments are kept except for spam, personal insults of a clearly inflammatory nature, and completely off-topic content that distracts from the discussion. We welcome comments critical of the work, even if somewhat insulting to the authors. Really.
  • Anyone can comment, and no registration is required. Vendors or consultants with a relevant product or offering must properly identify themselves. While their comments won’t be deleted, the writer/moderator will “call out”, identify, and possibly ridicule vendors who fail to do so.
  • Vendors considering licensing the content are welcome to provide feedback, but it must be posted in the comments - just like everyone else. There is no back channel influence on the research findings or posts.
    Analysts must reply to comments and defend the research position, or agree to modify the content.
  • At the end of the post series, the analyst compiles the posts into a paper, presentation, or other delivery vehicle. Public comments/input factors into the research, where appropriate.
  • If the research is distributed as a paper, significant commenters/contributors are acknowledged in the opening of the report. If they did not post their real names, handles used for comments are listed. Commenters do not retain any rights to the report, but their contributions will be recognized.
  • All primary research will be released under a Creative Commons license. The current license is Non-Commercial, Attribution. The analyst, at their discretion, may add a Derivative Works or Share Alike condition.
  • Securosis primary research does not discuss specific vendors or specific products/offerings, unless used to provide context, contrast or to make a point (which is very very rare).
    Although quotes from published primary research (and published primary research only) may be used in press releases, said quotes may never mention a specific vendor, even if the vendor is mentioned in the source report. Securosis must approve any quote to appear in any vendor marketing collateral.
  • Final primary research will be posted on the blog with open comments.
  • Research will be updated periodically to reflect market realities, based on the discretion of the primary analyst. Updated research will be dated and given a version number.
    For research that cannot be developed using this model, such as complex principles or models that are unsuited for a series of blog posts, the content will be chunked up and posted at or before release of the paper to solicit public feedback, and provide an open venue for comments and criticisms.
  • In rare cases Securosis may write papers outside of the primary research agenda, but only if the end result can be non-biased and valuable to the user community to supplement industry-wide efforts or advances. A “Radically Transparent Research” process will be followed in developing these papers, where absolutely all materials are public at all stages of development, including communications (email, call notes).
    Only the free primary research released on our site can be licensed. We will not accept licensing fees on research we charge users to access.
  • All licensed research will be clearly labeled with the licensees. No licensed research will be released without indicating the sources of licensing fees. Again, there will be no back channel influence. We’re open and transparent about our revenue sources.

In essence, we develop all of our research out in the open, and not only seek public comments, but keep those comments indefinitely as a record of the research creation process. If you believe we are biased or not doing our homework, you can call us out on it and it will be there in the record. Our philosophy involves cracking open the research process, and using our readers to eliminate bias and enhance the quality of the work.

On the back end, here’s how we handle this approach with licensees:

  • Licensees may propose paper topics. The topic may be accepted if it is consistent with the Securosis research agenda and goals, but only if it can be covered without bias and will be valuable to the end user community.
  • Analysts produce research according to their own research agendas, and may offer licensing under the same objectivity requirements.
  • The potential licensee will be provided an outline of our research positions and the potential research product so they can determine if it is likely to meet their objectives.
  • Once the licensee agrees, development of the primary research content begins, following the Totally Transparent Research process as outlined above. At this point, there is no money exchanged.
  • Upon completion of the paper, the licensee will receive a release candidate to determine whether the final result still meets their needs.
  • If the content does not meet their needs, the licensee is not required to pay, and the research will be released without licensing or with alternate licensees.
  • Licensees may host and reuse the content for the length of the license (typically one year). This includes placing the content behind a registration process, posting on white paper networks, or translation into other languages. The research will always be hosted at Securosis for free without registration.

Here is the language we currently place in our research project agreements:

Content will be created independently of LICENSEE with no obligations for payment. Once content is complete, LICENSEE will have a 3 day review period to determine if the content meets corporate objectives. If the content is unsuitable, LICENSEE will not be obligated for any payment and Securosis is free to distribute the whitepaper without branding or with alternate licensees, and will not complete any associated webcasts for the declining LICENSEE. Content licensing, webcasts and payment are contingent on the content being acceptable to LICENSEE. This maintains objectivity while limiting the risk to LICENSEE. Securosis maintains all rights to the content and to include Securosis branding in addition to any licensee branding.

Even this process itself is open to criticism. If you have questions or comments, you can email us or comment on the blog.