Applied Network Security Analysis: The Forensics Use Case

Most organizations don’t really learn about the limitations of event logs, until forensic investigators hold up their hands and explain they know what happened, but aren’t really sure how. Huh? How could that happen? It’s pretty simple: logs are a backward-looking indicator. They can help you piece together what happened, but you can only infer how. In a forensic investigation inferring anything is suboptimal. You want to know, especially given the needs to isolate the root cause of the attack and to establish remediations to ensure it doesn’t happen again. So we need to look at additional data sources to fill in gaps in what the logs tell you. Let’s take a look at a simplified scenario to illuminate the issues. We’ll look at the scenario both from the standpoint of a log-only analysis and then with a few other data sources added. For a more detailed incident response scenario, check out our React Faster and Better paper. The Forensic Limitations of Logs It’s the call you never want to get. The Special Agent on the other end of the line called to give you a heads-up: they found some of your customer data as part of another investigation into some cyber-crime activity that helps fund a domestic terrorist ring. Normally the Feds aren’t interested in giving you a heads-up until their investigation is done, but you have a relationship with this agent from your work together in the local InfraGard chapter. So he did you a huge favor. The first thing you need to do is figure out what was lost and how. To the logs! You aren’t sure how it happened, but you see some strange log records indicating changes on a application server in the DMZ. Given the nature of the data your agent friend passed along, you check the logs on the database server where that data resides as well. Interestingly enough, you find a gap in the logs on the database server, where your system collected no log records for a five-minute period a few days ago. You aren’t sure exactly what happened, but you know with reasonable certainty that something happened. And it probably wasn’t good. Now you work backwards and isolate the additional systems compromised as the attackers made their way through the infrastructure to reach their target. It’s pretty resource intensive, but by searching in the log manager you can isolate devices with gaps in their logs during the window you identified. The attackers were pretty effective, taking advantage of unpatched vulnerabilities (Damn, Ops!) and covering their tracks by turning off logging where necessary. At this point you know the attack path, and at least part of what was stolen, thanks to the FBI. Beyond that you are blind. So what can you do to make sure you aren’t similarly suprised somewhere down the line? You can set the logging system to alert if you don’t get any log records from critical assets in any 2-minute period. Again, this isn’t perfect and will result in a bunch more alerts, but at least you’ll know something is amiss before the FBI calls. With only log data you can identify what was attacked but probably not how the attack happened. Forensics Driven by Broader Data Let’s take a look at an alternative scenario with a few other data sources such as full network packet capture, network flow records, and configuration files. Of course it is still a bad day when you get the call from your pal the Special Agent. Of course Applied Network Security Analysis cannot magically make you omniscient, but how you investigate breaches changes. You still start with the logs on the perimeter server and identify the device that served as the attacker’s initial foothold. But you’ve implemented the Full Packet Capture Sandwich architecture described in the last post, so you are capturing the network traffic in your DMZ. You proceed to the network analysis console (using the full packet capture stream) and search all the traffic to and from the compromised server. Most sessions to that server are typical – standard application traffic. But you find some reconnaissance, and then something pretty strange: an executable injected into the server via faulty field validation on the web app (Damn, Developers!). Okay, this confirms the first point of exploit. Next we go to the target (keeping in mind what data was compromised) and do a similar analysis. Again, with our full packet capture sandwich in place, we captured traffic to/from the database server as well. As in the log-only scenario, we pinpoint the time period when logging was turned off, then perform a search in our analysis console to figure out what happened during that 5-minute period on that segment. Yep, a privileged account turned off logging on the database server and added an admin account to the database. Awesome. Using that account, the attacker dumped the database table and moved the data to a staging server elsewhere on your network. Now you know which data was taken, but how? You aren’t capturing all the traffic on your network (infeasible), so you have some blind spots, but with your additional data sources you are able to pinpoint the attack path. The NetFlow records coming from the compromised database server show the path to the staging server. The configuration records from the staging server indicate what executables were installed, which enabled the attacker to package and encrypt the payload for exfiltration. Further analysis of the NetFlow data shows the exfiltration, presumably to yet another staging server on another compromised network elsewhere. It’s not perfect, because you are figuring out what already happened. But now you can get back to your FBI buddy with a lot more information about what tactics the attacker used, and maybe even evidence that might be helpful in prosecution. Can’t Everyone Get Along? Clearly this is a simplified scenario that perfectly demonstrates the need to collect additional data sources to isolate the root cause and attack path of any

Read Post

Friday Summary: October 28, 2011

I really enjoyed Marco Arment’s I finally cracked it post, both because he captured the essence of Apple TV here and now, and because his views on media – as a consumer – are exactly in line with mine. Calling DVRs “a bad hack” is spot-on. I went through this process 7 years ago when I got rid of television. I could not accept a 5 minute American Idol segment in the middle of the 30 minute Fox ‘news’ broadcast. Nor the other 200 channels of crap surrounding the three channels I wanted. At the time people thought I was nuts, but now I run into people (okay – only a handful) who have pulled the plug on the broadcast media of cable and satellite. Most people are still frustrated with me when they say “Hey, did you see SuperJunk this weekend?” and I say “No, I don’t get television.” They mutter something like ‘Luddite’ and wonder off. Don’t get me wrong, I have a television. A very nice one in fact, but I have been calling it a ‘monitor’ for the last few years because it’s not attached to broadcast media. But not getting broadcast television does not make me a Luddite – quite to the contrary, I am waiting for the future. I am waiting for the day when I can get the rest of the content I want just as I get streaming Netflix today. And it’s not just the content, but the user experience as well. I don’t want to be boxed into some bizarre set of rules the content owners think I should follow. I don’t want half-baked DRM systems or advertising thrust at me – and believe me, this is what many of the other streaming boxes are trying to do. I don’t want to interact with a content provider because I am not interested – it was a bad idea proven foul a long time ago. Just let me watch what I want to watch when I want to watch it. Not so hard. But I wanted to comment on Marco’s point about Apple and their ability to be disruptive. My guess is that Apple TV will go fully a la carte: Show by show, game by game, movie by movie. But the major difference is we would get first run content, not just stuff from 2004. Somebody told me the other day that HBO stands for “Hey, Beastmaster’s On!”, which is how some of the streaming services and many of the movie channels feel. SOS/DD. The long tail of the legacy television market. The major gap in today’s streaming is first run programming. All I really want that I don’t have today is the Daily Show and… the National Football League (queue Monday Night Football soundtrack). And that’s the point where Mr. Arment’s analysis and mine diverge – the NFL. I agree that whatever Apple offers will likely be disruptive because the technology will simplify how we watch, rather than tiptoeing around legacy businesses and perverse contracts. But today there is only one game in town: the NFL. That’s why all those people pay $60 (in many cases it’s closer to $120) a month – to watch football. You placate kids with DVDs; you subscribe to cable for football! Just about every man I know, and 30% of the women, want to watch their NFL home team on Sunday. It’s the last remaining reason people still pay for cable or satellite in this economy. Make no mistake – the NFL is the 600 lb. gorilla of television. They currently hold sway over every cable and satellite network in the US. And the NFL makes a ridiculous amount of money because networks must pay princely sums for NFL games to be in the market. Which is why the distributors are so persnickety about not having NFL games on the Internet. Why else would they twist the arm of the federal government to shut down a guy relaying NFL games onto the Internet? (Thanks a ton for that one you a-holes – metropolitan areas broadcast over-the-air for free but it’s illegal to stream? WTF?) Nobody broadcasts live games over the Internet!?! Why not?!? The NFL could do it directly – they are already set up with “Game Pass” and “Game Rewind” – but likely can’t because fat network contracts prohibit it. Someone would need to spend the $$$ to get Internet distribution rights. Someone should, because there is huge demand, but there are only a handful of firms which could ante up a billion dollars to compete with DirecTV. But when this finally happens it will be seriously disruptive. Cable boxes will be (gleefully) dumped. Satellite providers will actually have competition, forcing them to alter their contacts and rates, and go back to delivering quality picture. ISPs will be pressured to actually deliver the bandwidth they claim to be selling. Consumers will get what they want at lower cost and with greater convenience. Networks will scramble to license the rest of their content to any streaming service provider they can, increasing content availability and pushing prices lower. If Apple wants to be disruptive, they will stream NFL games over the Internet on demand. If they can get rights to broadcast NFL for a reasonable price, they win. The company that gets the NFL for streaming wins. If Apple doesn’t, bet that Amazon will. On to the Summary: Webcasts, Podcasts, Outside Writing, and Conferences Rich quoted on SaaS security services. Adrian quoted in SearchSOA. Compliance Holds Up Los Angeles Google Apps Deployment. Mike plays master of the obvious. Ask the auditor before you commit to something that might be blocked by compliance. Duh! Favorite Securosis Posts Adrian Lane: A Kick-Ass Cloud Database Security Automation Example. And most IaaS cloud providers have the hooks to do most of this today. You can even script the removal of base database utilities you don’t want. Granted, you still have to set permissions on data and users, but the

Read Post

Totally Transparent Research is the embodiment of how we work at Securosis. It’s our core operating philosophy, our research policy, and a specific process. We initially developed it to help maintain objectivity while producing licensed research, but its benefits extend to all aspects of our business.

Going beyond Open Source Research, and a far cry from the traditional syndicated research model, we think it’s the best way to produce independent, objective, quality research.

Here’s how it works:

  • Content is developed ‘live’ on the blog. Primary research is generally released in pieces, as a series of posts, so we can digest and integrate feedback, making the end results much stronger than traditional “ivory tower” research.
  • Comments are enabled for posts. All comments are kept except for spam, personal insults of a clearly inflammatory nature, and completely off-topic content that distracts from the discussion. We welcome comments critical of the work, even if somewhat insulting to the authors. Really.
  • Anyone can comment, and no registration is required. Vendors or consultants with a relevant product or offering must properly identify themselves. While their comments won’t be deleted, the writer/moderator will “call out”, identify, and possibly ridicule vendors who fail to do so.
  • Vendors considering licensing the content are welcome to provide feedback, but it must be posted in the comments - just like everyone else. There is no back channel influence on the research findings or posts.
    Analysts must reply to comments and defend the research position, or agree to modify the content.
  • At the end of the post series, the analyst compiles the posts into a paper, presentation, or other delivery vehicle. Public comments/input factors into the research, where appropriate.
  • If the research is distributed as a paper, significant commenters/contributors are acknowledged in the opening of the report. If they did not post their real names, handles used for comments are listed. Commenters do not retain any rights to the report, but their contributions will be recognized.
  • All primary research will be released under a Creative Commons license. The current license is Non-Commercial, Attribution. The analyst, at their discretion, may add a Derivative Works or Share Alike condition.
  • Securosis primary research does not discuss specific vendors or specific products/offerings, unless used to provide context, contrast or to make a point (which is very very rare).
    Although quotes from published primary research (and published primary research only) may be used in press releases, said quotes may never mention a specific vendor, even if the vendor is mentioned in the source report. Securosis must approve any quote to appear in any vendor marketing collateral.
  • Final primary research will be posted on the blog with open comments.
  • Research will be updated periodically to reflect market realities, based on the discretion of the primary analyst. Updated research will be dated and given a version number.
    For research that cannot be developed using this model, such as complex principles or models that are unsuited for a series of blog posts, the content will be chunked up and posted at or before release of the paper to solicit public feedback, and provide an open venue for comments and criticisms.
  • In rare cases Securosis may write papers outside of the primary research agenda, but only if the end result can be non-biased and valuable to the user community to supplement industry-wide efforts or advances. A “Radically Transparent Research” process will be followed in developing these papers, where absolutely all materials are public at all stages of development, including communications (email, call notes).
    Only the free primary research released on our site can be licensed. We will not accept licensing fees on research we charge users to access.
  • All licensed research will be clearly labeled with the licensees. No licensed research will be released without indicating the sources of licensing fees. Again, there will be no back channel influence. We’re open and transparent about our revenue sources.

In essence, we develop all of our research out in the open, and not only seek public comments, but keep those comments indefinitely as a record of the research creation process. If you believe we are biased or not doing our homework, you can call us out on it and it will be there in the record. Our philosophy involves cracking open the research process, and using our readers to eliminate bias and enhance the quality of the work.

On the back end, here’s how we handle this approach with licensees:

  • Licensees may propose paper topics. The topic may be accepted if it is consistent with the Securosis research agenda and goals, but only if it can be covered without bias and will be valuable to the end user community.
  • Analysts produce research according to their own research agendas, and may offer licensing under the same objectivity requirements.
  • The potential licensee will be provided an outline of our research positions and the potential research product so they can determine if it is likely to meet their objectives.
  • Once the licensee agrees, development of the primary research content begins, following the Totally Transparent Research process as outlined above. At this point, there is no money exchanged.
  • Upon completion of the paper, the licensee will receive a release candidate to determine whether the final result still meets their needs.
  • If the content does not meet their needs, the licensee is not required to pay, and the research will be released without licensing or with alternate licensees.
  • Licensees may host and reuse the content for the length of the license (typically one year). This includes placing the content behind a registration process, posting on white paper networks, or translation into other languages. The research will always be hosted at Securosis for free without registration.

Here is the language we currently place in our research project agreements:

Content will be created independently of LICENSEE with no obligations for payment. Once content is complete, LICENSEE will have a 3 day review period to determine if the content meets corporate objectives. If the content is unsuitable, LICENSEE will not be obligated for any payment and Securosis is free to distribute the whitepaper without branding or with alternate licensees, and will not complete any associated webcasts for the declining LICENSEE. Content licensing, webcasts and payment are contingent on the content being acceptable to LICENSEE. This maintains objectivity while limiting the risk to LICENSEE. Securosis maintains all rights to the content and to include Securosis branding in addition to any licensee branding.

Even this process itself is open to criticism. If you have questions or comments, you can email us or comment on the blog.