Metrics

Analysis of Trustwave’s 2010 Breach Report

By Rich

Trustwave just released their latest breach (and penetration testing) report, and it's chock full of metrics goodness. Like the Verizon Data Breach Investigations Report, it's a summary of information based on their responses to real breaches, with a second section on results from their penetration tests.

The breach section is the best part, and I already wrote about one lesson in a quick post on DLP. Here are a few more nuggets that stood out:

1. It took an average of 156 days to detect a breach, and only 9% of victims detected the breach on their own -- the rest were discovered by outside groups, including law enforcement and a credit card companies.
2. Chip and PIN was called out for being more resilient against certain kinds of attacks, based on global analysis of breaches. Too bad we won't get it here in the US anytime soon.
3. One of the biggest sources of breaches was remote access applications for point of sale systems. Usually these are in place for third party support/management, and configured poorly.
4. Memory parsing (scanning memory to pull sensitive information as it's written to RAM) was a very common attack technique. I find this interesting, since certain versions of memory parsing attacks have virtualization implications... and thus cloud implications. This is something to keep an eye on, and an area I'm researching more.
5. As I mentioned in the post 5 minutes ago, encryption was only used once for exfiltration.

Now some suggestions to the SpiderLabs guys:

1. I realize it isn't politically popular, but it would totally rock if you, Verizon, and other response teams started using a standard base of metrics. You can always add your own stuff on top, but that would really help us perform external analysis across a wider base of data points. If you're interested, we'd totally be up for playing neutral third party and coordinating and publishing a recommended metrics base.
2. The pen testing section would be a lot more interesting if you released the metrics, as opposed to a "top 10" list of issues found. We don't know if number 1 was 10x more common than issue 10, or 100x more common.

It's great that we now have another data source, and I consider all these reports mandatory reading, and far more interesting than surveys.

–Rich

Creating a Standard for Data Breach Costs

By Rich

One thing that's really tweaked me over the years when evaluating data breaches is the complete lack of consistency in costs reporting. On one side we have reports and surveys coming up with "per record" costs, often without any transparency as to where the numbers came from. On the other side are those that try and look at lost share value, or directly reported losses from public companies in their financial statements, but I think we all know how inconsistent those numbers are as well.

Also, from what I can tell, in most of the "per record" surveys, the biggest chunk (by far) are fuzzy soft costs like "reputation damage". Not that there aren't any losses due to reputation damage, but I've never seen any sort of justified model that accurately measures those costs over time. Take TJX for example -- they grew sales after their breach.

So here's a modest proposal for how we could break out breach costs in a more consistent manner:

Per Incident (Hard Costs):

1. Incident investigation
2. Incident remediation/recovery
3. PR/media relations costs
4. Optional: Legal fees
5. Optional: Compliance violation penalties
6. Optional: Legal settlements

Per Record (Hard Costs):

1. Notification costs (list creation, printing, postal fees).
2. Optional: Customer response costs (help desk per-call costs).
3. Optional: Customer protection costs (fraud alerts, credit monitoring).

Per Incident

(Soft Costs... e.g., not always directly attributable to the incident): Trending is key here -- especially trends that predate the incident.

1. Customer Churn (% increase over trailing 6 month rate): 1 week, 1 month, 6 months, 12 months, n months.
2. Stock Hit (not sure of best metric here, maybe earnings per share): 1 week, 1 month, 6 months, 12 months, n months.
3. Revenue Impact (compared to trailing 12 months): 1 week, 1 month, 6 months, 12 months, n months.

I tried to break them out into hard and soft costs (hard being directly tied to the incident, soft being polluted by other factors). Also, I recognize that not every organization can measure every category for every incident.

Not that I expect everyone to magically adopt this for standard reporting, but until we transition to a mechanism like this we don't have any chance of really understanding breach costs.

–Rich

The CIS Consensus Metrics and Project Quant

By Rich

Just before release, the Center for Internet Security sent us a preview copy of the CIS Consensus Metrics. I'm a longtime fan of the Center, and, once I heard they were starting on this project, was looking forward to the results.

Overall I think they did a solid job on a difficult problem. Is it perfect? Is it complete? No, but it's a heck of a good start. There are a couple things that stand out:

• They do a great job of interconnecting different metrics, and showing you how you can leverage a single collected data attribute across multiple higher-level metrics. For example, a single "technology" (product/version) is used in multiple places, for multiple metrics. It's clear they've designed this to support a high degree of automation across multiple workflows, supporting technologies, and operational teams.
• I like how they break out data attributes from security metrics. Attributes are the feeder data sets we use to create the security metrics. I've seen other systems that intermix the data with the metrics, creating confusion.
• Their selected metrics are a reasonable starting point for characterizing a security program. They don't cover everything, but that makes it more likely you can collect them in the first place. They make it clear this is a start, with more metrics coming down the road.
• The metrics are broken out by business function -- this version covering incident management, vulnerability management, patch management, application security, configuration management, and financial.
• The metric descriptions are clear and concise, and show the logic behind them. This makes it easy to build your own moving forward.

There are a few things that could also be improved:

• The data attributes are exhaustive. Without automated tool support, they will be very difficult to collect.
• The document suggests prioritization, but doesn't provide any guidance. A companion paper would be nice.

This isn't a mind-bending document, and we've seen many of these metrics before, but not usually organized together, freely available, well documented, or from a respected third party. I highly recommend you go get a copy.

Now on to the CIS Consensus Metrics and Project Quant...

I've had some people asking me if Quant is dead thanks to the CIS metrics. While there's the tiniest bit of overlap, the two projects have different goals, and are totally complementary. The CIS metrics are focused on providing an overview for an entire security program, while Quant is focused on building a detailed operational metrics model for patch management. In terms of value, this should provide:

1. Detailed costs associated with each step of a patch management process, and a model to predict costs associated with operational changes.
2. Measurements of operational efficiency at each step of patch management to identify bottlenecks/inefficiencies and improve the process.
3. Overall efficiency metrics for the entire patch management process.

CIS and Quant overlap for the last goal, but not for the first two. If anything, Quant will be able to feed the CIS metrics. The CIS metrics for patch management include:

• Patch Policy Compliance
• Patch Management Coverage
• Mean Time to Patch

I highly suspect all of these will appear in Quant, but we plan on digging into much greater depth to help the operational folks directly measure and optimize their processes.

–Rich

Friday Summary - May 22, 2009

By Rich

Adrian has been out sick with the flu all week. He claims it's just the normal flu, but I swear he caught it from those bacon bits I saw him putting on his salad the other day. Either that, or he's still recovering from last week's Buffett outing. He also conveniently timed his recovery with his wife's birthday, which I consider to be entirely too suspicious for mere coincidence.

While Adrian was out, we passed a couple milestones with Project Quant. I think we've finally nailed a reasonable start to defining a patch management process, and I've drafted up a sample of our first survey. We could use some feedback on both of these if you have the time. Next week will be dedicated to breaking out all the patch management phases and mapping out specific sub-processes. Once we have those, we can start defining the individual metrics. I've taken a preliminary look at the Center for Internet Security's Consensus Metrics, and I don't see any conflicts (or too much overlap), which is nice.

When we look at security metrics we see that most fall into two broad categories. On one side are the fluffy (and thus crappy) risk/threat metrics we spend a lot of time debunking on this site. They are typically designed to feed into some sort of ROI model, and don't really have much to do with getting your job done. I'm not calling all risk/threat work crap, just the ones that like to put a pretty summary number at the end, usually with a dollar sign, but without any strong mathematical basis.

On the other side are broad metrics like the Consensus Metrics, designed to give you a good snapshot view of the overall management of your security program. These aren't bad, are often quite useful when used properly, and can give you a view of how you are doing at the macro level.

The one area where we haven't seen a lot of work in the security community is around operational metrics. These are deep dive, granular models, to measure operational efficiency in specific areas to help improve associated processes. That's what we're trying to do with Quant -- take one area of security, and build out metrics at a detailed enough level that they don't just give you a high level overview, but help identify specific bottlenecks and inefficiencies. These kinds of metrics are far too detailed to achieve the high-level goals of programs like the Consensus Metrics, but are far more effective at benchmarking and improving the processes they cover.

In my ideal world we would have a series of detailed metrics like Quant, feeding into overview models like the Consensus Metrics. We'll have our broad program benchmarks, as well as detailed models for individual operational areas. My personal goal is to use Quant to really nail one area of operational efficiency, then grow out into neighboring processes, each with its own model, until we map out as many areas as possible. Pick a spot, perfect it, move on.

And now for the week in review:

Webcasts, Podcasts, Outside Writing, and Conferences

• Martin and I cover a diverse collection of stories in Episode 151 of the Network Security Podcast
• I wrote up the OS X Java vulnerability for TidBITS.
• I was quoted at MacNewsWorld on the same issue.
• Another quote, this time in EWeek on "data for ransom" schemes.
• Dark Reading covered Project Quant in its post on the Center for Internet Security's Consensus Metrics

Favorite Securosis Posts

• Rich: The Pragmatic Data (Information-Centric) Security Cycle. I've been doing a lot of thinking on more practical approaches to security in general, and this is one of the first outcomes.
• Adrian: I've been feeling foul all week, and thus am going with the lighter side of security -- I Heart Creative Spam.

Favorite Outside Posts

• Adrian: Yes, Brownie himself is now considered a cybersecurity expert. Or not..
• Rich: Johnny Long, founder of Hackers for Charity, is taking a year off to help the impoverished in Africa. He's quit his job, and no one is paying for this. We just made a donation, and you should consider giving if you can.

Top News and Posts

• Good details on the IIS WebDAV vulnerability by Thierry Zoller.
• Hoff on the cloud and the Google outage.
• Imperva points us to highlights on practical recommendations from the FBI and Secret Service on reducing financial cybercrime.
• Oops -- the National Archives lost a drive with sensitive information from the Clinton administration. As usual, lax controls were the cause.
• Some solid advice on controlling yourself when you really want that tasty security job. You know, before you totally piss off the hiring manager.
• We bet you didn't know that Google Chrome was vulnerable to the exact same vulnerability as Safari in the Pwn2Own contest. That's because they both use WebKit.
• Adobe launches a Reader and Acrobat security initiative. New incident response, patch cycles, and secure development efforts. This is hopefully Adobe's equivalent to the Trustworthy Computing Initiative.

Blog Comment of the Week

This week's best comment was by Jim Heitela in response to Security Requirements for Electronic Medical Records:

Good suggestions. The other industry movement that really will amplify the need for healthcare organizations to get their security right is regional/national healthcare networks. A big portion of the healthcare IT $ in the Recovery Act are going towards establishing these networks, where the security of EPHI will only be as good as the weakest accessing node. Establishing adequate standards for partners in these networks will be pretty key. And, also thanks to changes that were started as a part of the Recovery Act, healthcare organizations are now being required to actually assess 3rd party risk for business associates, versus just getting them to sign a business associate agreement. Presumably this would be anyone in a RHIO/RHIN.

–Rich

Our Financial System is Under a Coordinated, Sophisticated Attack

By Rich

This is a great day for security researchers, and a bad day for anyone with a bank account.

First up is the release of the 2009 Verizon Data Breach Investigations Report. This is now officially my favorite breach metrics source, and it's chock full of incredibly valuable information. I love the report because it's not based on bullshit surveys, but on real incident investigations. The results are slowly spreading throughout the blogosphere, and we won't copy them all here, but a few highlights:

1. Verizon's team alone investigated cases that resulted in the loss of 285 million records. That's just them, never mind all the other incident response teams.
2. Most organizations do a crap job with security- this is backed up with a series of metrics on which security controls are in place and how incidents are discovered.
3. Essentially no organizations really complied with all the PCI requirements- but most get certified anyway.

Liquidmatrix has a solid summary of highlights, and I don't want to repeat their work. As they say,

Read pages 46-49 of the report and do what it says. Seriously. It’s the advice that I would give if you were paying me to be your CISO.

And we'll add some of our own advice soon.

Next is an article on organized cybercrime by Brian Krebs THAT YOU MUST GO READ NOW. (I realize it might seem like we have a love affair with Brian or something, but he's not nearly my type). Brian digs beyond the report, and his investigative journalism shows what many of us believe to be true- there is a concerted attack on our financial system that is sophisticated and organized, and based out of Eastern Europe.

I talked with Brain and he told me,

You know all those breaches last year? Most of them are a handful of groups.

Here are a couple great tidbits from the article:

For example, a single organized criminal group based in Eastern Europe is believed to have hacked Web sites and databases belonging to hundreds of banks, payment processors, prepaid card vendors and retailers over the last year. Most of the activity from this group occurred in the first five months of 2008. But some of that activity persisted throughout the year at specific targets, according to experts who helped law enforcement officials respond to the attacks, but asked not to be identified because they are not authorized to speak on the record.

...

One hacking group, which security experts say is based in Russia, attacked and infiltrated more than 300 companies -- mainly financial institutions -- in the United States and elsewhere, using a sophisticated Web-based exploitation service that the hackers accessed remotely. In an 18-page alert published to retail and banking partners in November, VISA described this hacker service in intricate detail, listing the names of the Web sites and malicious software used in the attack, as well as the Internet addresses of dozens of sites that were used to offload stolen data.

...

Steve Santorelli, director of investigations at Team Cymru, a small group of researchers who work to discover who is behind Internet crime, said the hackers behind the Heartland breach and the other break-ins mentioned in this story appear to have been aware of one another and unofficially divided up targets. "There seem, on the face of anecdotal observations, to be at least two main groups behind many of the major database compromises of recent years," Santorelli said. "Both groups appear to be giving each other a wide berth to not step on each others' toes."

Keep in mind that this isn't the same old news. We're not talking about the usual increase in attacks, but a sophistication and organizational level that developed materially in 2007-2008.

To top it all off, we have this article over at Wired on PIN cracking. This one also ties in to the Verizon report. Another quote:

"We're seeing entirely new attacks that a year ago were thought to be only academically possible," says Sartin. Verizon Business released a report Wednesday that examines trends in security breaches. "What we see now is people going right to the source ... and stealing the encrypted PIN blocks and using complex ways to un-encrypt the PIN blocks."

If you read more deeply, you learn that the bad guys haven't developed some quantum crypto, but are taking advantage of weak points in the system where the data is unencrypted, even if only in memory.

Really fascinating stuff, and I love that we're getting real information on real breaches.

–Rich

The Mozilla Metrics Project

By Rich

Ryan Naraine just posted an article over at ZDNet about a project I'm extremely excited to be involved with.

Just before RSA I was invited by Window Snyder over at Mozilla to work with them on a project to take a new look at software security metrics. Window has posted the details of the project over on the Mozilla security blog, and here's an excerpt:

Mozilla has been working with security researcher and analyst Rich Mogull for a few months now on a project to develop a metrics model to measure the relative security of Firefox over time. We are trying to develop a model that goes beyond simple bug counts and more accurately reflects both the effectiveness of secure development efforts, and the relative risk to users over time. Our goal in this first phase of the project is to build a baseline model we can evolve over time as we learn what works, and what does not. We do not think any model can define an absolute level of security, so we decided to take the approach of tracking metrics over time so we can track relative improvements (or declines), and identify any problem spots. This information will support the development of Mozilla projects including future versions of Firefox. ... Below is a summary of the project goals, and the xls of the model is posted at http://securosis.com/publications/MozillaProject2.xls. The same content as a set of .csvs is available here: http://securosis.com/publications/MozillaProject.zip This is a preliminary version and we are currently looking for feedback. The final version will be a far more descriptive document, but for now we are using a spreadsheet to refine the approach. Feel free to download it, rip it apart, and post your comments. This is an open project and process. Eventually we will release this to the community at large with the hope that other organizations can adapt it to their own needs.

Although I love my job, it's not often I get to develop original research like this with an organization like Mozilla. We really think we have the opportunity to contribute to the security and development communities in an impactful way.

If you'd like to contribute, please comment over at the Mozilla blog, or email me directly. I'd like to keep the conversation over there, rather than in comments here.

This is just the spreadsheet version (and a csv version); the final product will be more of a research note, describing the metrics, process, and so on.

I'm totally psyched about this.

–Rich