Cloud Forensics 101

Last week I wrote up my near epic fail on Amazon Web Services where I ‘let’ someone launch a bunch of Litecoin mining instances in my account. Since then I received some questions on my forensics process, so I figure this is a good time to write up the process in more detail. Specifically, how to take a snapshot and use it for forensic analysis. I won’t cover all the steps at the AWS account layer – this post focuses on what you should do for a specific instance, not your entire management plane. Metadata The first step, which I skipped, is to collect all the metadata associated with the instance. There is an easy way, a hard way (walk through the web UI and take notes manually), and the way I’m building into my nifty tool for all this that I will release at RSA (or sooner, if you know where to look). The best way is to use the AWS command line tools for your operating system. Then run the command aws ec2 describe-instances –instance-ids i-5203422c (inserting your instance ID). Note that you need to follow the instructions linked above to properly configure the tool and your credentials. I suggest piping the output to a file (e.g., aws ec2 describe-instances –instance-ids i-5203422c > forensic-metadata.log) for later examination. You should also get the console output, which is stored by AWS for a short period on boot/reboot/termination. aws ec2 get-console-output –instance-id i-5203422c. This might include a bit more information if the attacker mucked with logs inside the instance, but won’t be useful for a hacked instance because it is only a boot log. This is a good reason to use a tool that collects instance logs outside AWS. That is the basics of the metadata for an instance. Those two pieces collect the most important bits. The best option would be CloudTrail logs, but that is fodder for a future post. Instance Forensics Now on to the instance itself. While you might log into it and poke around, I focused on classical storage forensics. There are four steps: Take a snapshot of all storage volumes. Launch an instance to examine the volumes. Attach the volumes. Perform your investigation. If you want to test any of this, feel free to use the snapshot of the hacked instance that was running in my account (well, one of 10 instances). The snapshot ID you will need is snap-ccd3e9c6. Snapshot the storage volumes I will show all this using the web interface, but you can also manage all of it using the command line or API (which is how I now do it, but that code wasn’t ready when I had my incident). There is a slightly shorter way to do this in the web UI by going straight to volumes, but that way is easier to botch, so I will show the long way and you can figure out the shorter alternative yourself. Click Instances in your EC2 management console, then check the instance to examine. Look at the details on the bottom, click the Block Devices, then each entry. Pull the Volume ID for every attached volume. Switch to Volumes and then snapshot each volume you identified in the steps above. Label each snapshot so you remember it. I suggest date and time, “Forensics”, and perhaps the instance ID. You can also add a name to your instance, then skip direct to Volumes and search for volumes attached to it. Remember, once you take a snapshot, it is read-only – you can create as many copies as you like to work on without destroying the original. When you create a volume from an instance it doesn’t overwrite the snapshot, it gets another copy injected into storage. Snapshots don’t capture volatile memory, so if you need RAM you need to either play with the instance itself or create a new image from that instance and launch it – perhaps the memory will provide more clues. That is a different process for another day. Launch a forensics instance Launch the operating system of your choice, in the same region as your snapshot. Load it with whatever tools you want. I did just a basic analysis by poking around. Attach the storage volumes Go to Snapshots in the management console. Click the one you want, right-click, and then “Create Volume from Snapshot”. Make sure you choose the same Availability Zone as your forensics instance. Seriously, make sure you choose the same Availability Zone as your instance. People always mess this up. (By ‘people’, I of course mean ‘I’). Go back to Volumes. Select the new volume when it is ready, and right click/attach. Select your forensics instance. (Mine is stopped in the screenshot – ignore that). Set a mount point you will remember. Perform your investigation Create a mount point for the new storage volumes, which are effectively external hard drives. For example, sudo mkdir /forensics. Mount the new drive, e.g., sudo mount /dev/xvdf1 /forensics. Amazon may change the device mapping when you attach the drive (technically your operating system does that, not AWS, and you get a warning when you attach). Remember, use sudo bash (or the appropriate equivalent for your OS) if you want to peek into a user account in the attached volume. And that’s it. Remember you can mess with the volume all you want, then attach a new one from the snapshot again for another pristine copy. If you need a legal trail my process probably isn’t rigorous enough, but there should be enough here that you can easily adapt. Again, try it with my snapshot if you want some practice on something with interesting data inside. And after RSA check back for a tool which automates nearly all of this. Share:

Read Post

Security Management 2.5: Selection Process

With vendor evaluations in hand, you are ready to make your decision, right? The answer is both yes and no. We know the importance of this decision – you are here because your first attempt at this project wasn’t as successful as it needed to be. After the vendor evaluation process you are in a position to distinguish innovative technologies from pigs with fresh lipstick. But now you need to see which of the vendors is actually the best fit for you! Successful decision-making on SIEM replacement goes beyond vendor evaluation – it entails evaluating yourself too. It is important to differentiate between the two because you cannot make a decision without taking a long hard look at yourself, your team, and your company. This is an area where many projects fail, so let’s break the decision down to ensure you can make a good recommendation and feel comfortable with it – from both internal and external perspectives. But remember the selection of the ‘right’ vendor may come down to more than matching needs against capabilities. The output of our Security Management 2.5 process is not really a decision – it’s more of a recommendation. The final decision will likely be made in the executive suite. That’s why we focused so much on gathering data (quantitative where possible) – you will need to defend your recommendation until the purchase order is signed. And probably afterwards. Defensible Position We won’t mince words. This decision generally isn’t about objective or technical facts – especially since most of you reading this have an incumbent in play, typically part of a big company with important relationships with heavies inside your shop. This could get political, or the decision might be entirely financial, so you need your ducks in a row and a compelling argument for any change. And even then you might not be able to push through a full replacement. In that case the answer might be to supplement. In this scenario you still aggregate information with the existing platform, but then you feed it to the new platform for analysis, reporting, forensics, etc. across the enterprise. Given the economic cost of running both, this is unacceptable for some organizations, but if your hands are tied on replacement, this kind of creative approach is worth considering. But that is still only the external part of the decision process. In many cases the (perceived) failure of your existing SIEM may be self-inflicted. So we also need to evaluate and explain the causes of the failure, with assurance that you can avoid those issues this time. If not your successor will be in the same boat in another 2-3 years. So before you put your neck on the chopping block and advocate for change (if that is what you decide), do some deep internal analysis as well. Looking in the mirror First, let’s make sure you really re-examined the existing platform in terms of the original goals. Did your original goals adequately map your needs at the time, or was there stuff you did not anticipate? How have your goals changed over time? Be honest! Do not let ego get in the way of doing what’s right, and take a hard and fresh look at the decision to ensure you don’t repeat previous mistakes. Did you kick off this process because you were pissed at the original vendor? Or because they got bought and seemed to forget about the platform? Do you know what it will take to get the incumbent where it needs to be – and whether that is even possible? Is it about throwing professional services at the issues? Is there a fundamental technology problem? Did you assess the issues critically the first time around? If it was a skills issue, have you successfully addressed it? Can your folks build and maintain the platform moving forward? Are you looking at a managed service to take that concern off the table? If it was a resource problem, do you now have enough staff for proper care and feeding? Yes, the new generation of platforms requires less expertise to keep operational, but don’t be naive – no matter what any sales rep says, you cannot simply set and forget them. Whatever you pick will require expertise to deploy, manage, tune, and analyze reports. These platforms are not self-aware – by a long shot. Remember, there are no right or wrong answers here, but the truth (and your commitment) will become clear when you need to sell something to management. Some of you may worry that management will see the need for replacement as “your fault” for choosing the incumbent, so make sure you have answers to these questions and that you aren’t falling into a self-delusional trap. You need your story straight and your motivations clear. Have a straightforward and honest assessment of what is going right and wrong, so you are not caught off guard when asked to justify changes and new expenses. Setting Expectations Revisiting requirements provides insight into what you need the security management platform to do. Remember, not everything is Priority #1, so pick your top three must-have items and prioritize the requirements. You can prioritize specific use cases (compliance, security, forensics, operations), and have a pretty good feeling about whether the new platform or incumbent will meet your expectations. If you love some new features of the challenger(s), will your organization leverage them? Firing off alerts faster won’t help if your team takes a week to investigate each issue, or cannot keep up with the increased demand. The new platform’s ability to look at application and database traffic doesn’t matter if your developers won’t help you understand normal behavior to build the rule set. Fancy network flow analysis can be a productivity sink if your DNS and directory infrastructure is a mess and you can’t reliably map an IP to user ID. Does your existing product have too many features? Yes, some organizations simply cannot take advantage of (or

Read Post

Reducing Attack Surface with Application Control: The Double-Edged Sword [New Series]

The problems of protecting endpoints are pretty well understood. As we described in The 2014 Guide to Endpoint Security, you have stuff (private data and/or intellectual property) that others want. On the other hand, you have employees who need to do their jobs and require access to said private data and/or intellectual property. Those employees have sensitive data on their devices, so you need to protect their endpoints. It’s not like this is anything new. Protecting endpoints has been a focus of security professionals since, well, always – with decidedly unimpressive results. Why is protecting endpoints so hard? It can’t be a matter of effort, right? Billions have been spent on research to identify better ways to protect these devices. Organizations have spent tens of billions on endpoint security products and services. Yet, every minute more devices are compromised, more data is stolen, and security folks keep having to answer senior management, regulators, and ultimately customers as to why this keeps happening. The lack of demonstrable progress comes down to two intertwined causes. First, devices are built using software that has defects attackers can exploit. Nothing is perfect, especially not software, so every line of code presents attack surface. Second, employees can be fooled into taking action (such as installing software or clicking a link) that results in a successful attack. These two causes can’t really be separated. If the device isn’t vulnerable, then nothing an employee does should result in a successful attack. And likewise, if the employee doesn’t allow delivery of the attack/exploit code by clicking things, having vulnerable software is less of an issue. So if you can disrupt either causes your endpoints will be far better protected. Of course this is much easier said than done. In this new series, “Reducing Attack Surface with Application Control,” we will dig into the good and bad of application control (also known as application white listing) technology, talking about how AppControl can stop malware in its tracks and mitigate the risks of both vulnerable software and gullible users. We won’t shy away from addressing head-on the perception issues of endpoint lockdown, which cause many organizations to disregard the technology as infeasible in their environments. Finally, we will discuss use cases where AppControl makes a lot of sense and how it can favorably impact security posture, both reducing the attack surface of vulnerable devices and protecting users from themselves. Accelerating Attacker Innovation We mentioned the billions of dollars being spent on research to protect endpoint devices more effectively. It is legitimate to ask why these efforts haven’t really worked. It comes back to attackers innovating faster than defenders. And even if technology emerges to protect devices more effectively, it takes years for new technologies to become pervasive enough to blunt the impact of attackers across a broad market. The reactive nature of traditional malware defenses – in terms of finding an attack, profiling it, and developing a signature to block it on the device – makes existing mitigations too little too late. Attackers now randomly change what attacks look like using polymorphic malware, so looking for malware files cannot solve the problem. Additionally, attackers have new and increasingly sophisticated means to contact their command and control (C&C) systems and obscure data during exfiltration, making detection all the harder. Attackers also do a lot more testing now to make sure their attacks work before they use them. Endpoint security technologies can be bought for a very small investment, so attackers refine their malware to ensure it works against a majority of the defenses in use. This causes security professionals to look at different ways of breaking the kill chain, as we described in The CISO’s Guide to Advanced Attackers. You can do this a couple different ways: Impede Delivery: If the attacker cannot deliver the attack to a vulnerable device, the chain is broken. This involves effectively stopping tactics like phishing, either by blocking the email before it gets to an employee or training employees not to click things that would result in malware delivery. Stop Compromise: Even if the attack does reach a device, if it cannot execute and exploit the device, the chain is broken. This involves a different approach to protecting endpoints, and will be the main focus of this series. Block C&C: If the device is compromised, but cannot contact the command and control infrastructure to receive instructions and additional attack code, the impact of the attack is reduced. This requires the ability to analyze all outbound network traffic for C&C patterns, as well as watching for contact with networks with bad reputations. We discussed many of these tactics in our Network-based Threat Intelligence research. Block Exfiltration: The last defense is to stop the exfiltration of data from your environment. Whether via data leak prevention technology or some other means of content or egress filtering to detect protected content, if you can stop data from leaving your environment there is no loss. The earlier you break the kill chain, the better. But in the real world, you are best served by a multi-faceted approach encompassing all the options listed above. Now let’s dig into the Stop Compromise strategy for breaking the kill chain, which is really where application control fits into the security control hierarchy. Stop Code Execution. Stop Malware. The main focus of anti-virus and anti-malware technology since the beginning has been to stop malicious code from executing on a device, thus stopping compromise. What has been evolving is how the malware is detected, and what parts of devices software can access. There are currently a handful of approaches. Block the Bad: This is the traditional AV approach of matching malware signatures against code executing on the device. The problem is scale because there is so much bad that you cannot possible expect an endpoint to check for every attack since the beginning of time. Improve Heuristics: It is impossible to block all malware because it is constantly changing, so you need to focus on what

Read Post

Firestarter: Crisis Communications

Okay, we have content in this thing. We promise. But we can’t stop staring at our new title video sequence. I mean, just look at it! This week Rich, Mike, and Adrian discuss Target, Snapchat, RSA, and why no one can get crisis communications correct. Sorry we hit technical difficulties with the live Q&A Friday, but we think we have the kinks worked out (I’d blame Mike if I were inclined to point fingers). Our plan is to record Friday again – keep an eye on our Google+ page for the details. Share:

Read Post

Totally Transparent Research is the embodiment of how we work at Securosis. It’s our core operating philosophy, our research policy, and a specific process. We initially developed it to help maintain objectivity while producing licensed research, but its benefits extend to all aspects of our business.

Going beyond Open Source Research, and a far cry from the traditional syndicated research model, we think it’s the best way to produce independent, objective, quality research.

Here’s how it works:

  • Content is developed ‘live’ on the blog. Primary research is generally released in pieces, as a series of posts, so we can digest and integrate feedback, making the end results much stronger than traditional “ivory tower” research.
  • Comments are enabled for posts. All comments are kept except for spam, personal insults of a clearly inflammatory nature, and completely off-topic content that distracts from the discussion. We welcome comments critical of the work, even if somewhat insulting to the authors. Really.
  • Anyone can comment, and no registration is required. Vendors or consultants with a relevant product or offering must properly identify themselves. While their comments won’t be deleted, the writer/moderator will “call out”, identify, and possibly ridicule vendors who fail to do so.
  • Vendors considering licensing the content are welcome to provide feedback, but it must be posted in the comments - just like everyone else. There is no back channel influence on the research findings or posts.
    Analysts must reply to comments and defend the research position, or agree to modify the content.
  • At the end of the post series, the analyst compiles the posts into a paper, presentation, or other delivery vehicle. Public comments/input factors into the research, where appropriate.
  • If the research is distributed as a paper, significant commenters/contributors are acknowledged in the opening of the report. If they did not post their real names, handles used for comments are listed. Commenters do not retain any rights to the report, but their contributions will be recognized.
  • All primary research will be released under a Creative Commons license. The current license is Non-Commercial, Attribution. The analyst, at their discretion, may add a Derivative Works or Share Alike condition.
  • Securosis primary research does not discuss specific vendors or specific products/offerings, unless used to provide context, contrast or to make a point (which is very very rare).
    Although quotes from published primary research (and published primary research only) may be used in press releases, said quotes may never mention a specific vendor, even if the vendor is mentioned in the source report. Securosis must approve any quote to appear in any vendor marketing collateral.
  • Final primary research will be posted on the blog with open comments.
  • Research will be updated periodically to reflect market realities, based on the discretion of the primary analyst. Updated research will be dated and given a version number.
    For research that cannot be developed using this model, such as complex principles or models that are unsuited for a series of blog posts, the content will be chunked up and posted at or before release of the paper to solicit public feedback, and provide an open venue for comments and criticisms.
  • In rare cases Securosis may write papers outside of the primary research agenda, but only if the end result can be non-biased and valuable to the user community to supplement industry-wide efforts or advances. A “Radically Transparent Research” process will be followed in developing these papers, where absolutely all materials are public at all stages of development, including communications (email, call notes).
    Only the free primary research released on our site can be licensed. We will not accept licensing fees on research we charge users to access.
  • All licensed research will be clearly labeled with the licensees. No licensed research will be released without indicating the sources of licensing fees. Again, there will be no back channel influence. We’re open and transparent about our revenue sources.

In essence, we develop all of our research out in the open, and not only seek public comments, but keep those comments indefinitely as a record of the research creation process. If you believe we are biased or not doing our homework, you can call us out on it and it will be there in the record. Our philosophy involves cracking open the research process, and using our readers to eliminate bias and enhance the quality of the work.

On the back end, here’s how we handle this approach with licensees:

  • Licensees may propose paper topics. The topic may be accepted if it is consistent with the Securosis research agenda and goals, but only if it can be covered without bias and will be valuable to the end user community.
  • Analysts produce research according to their own research agendas, and may offer licensing under the same objectivity requirements.
  • The potential licensee will be provided an outline of our research positions and the potential research product so they can determine if it is likely to meet their objectives.
  • Once the licensee agrees, development of the primary research content begins, following the Totally Transparent Research process as outlined above. At this point, there is no money exchanged.
  • Upon completion of the paper, the licensee will receive a release candidate to determine whether the final result still meets their needs.
  • If the content does not meet their needs, the licensee is not required to pay, and the research will be released without licensing or with alternate licensees.
  • Licensees may host and reuse the content for the length of the license (typically one year). This includes placing the content behind a registration process, posting on white paper networks, or translation into other languages. The research will always be hosted at Securosis for free without registration.

Here is the language we currently place in our research project agreements:

Content will be created independently of LICENSEE with no obligations for payment. Once content is complete, LICENSEE will have a 3 day review period to determine if the content meets corporate objectives. If the content is unsuitable, LICENSEE will not be obligated for any payment and Securosis is free to distribute the whitepaper without branding or with alternate licensees, and will not complete any associated webcasts for the declining LICENSEE. Content licensing, webcasts and payment are contingent on the content being acceptable to LICENSEE. This maintains objectivity while limiting the risk to LICENSEE. Securosis maintains all rights to the content and to include Securosis branding in addition to any licensee branding.

Even this process itself is open to criticism. If you have questions or comments, you can email us or comment on the blog.