I’m in the air (literally) on the way to Metricon 6; so I’m thinking a lot about metrics, quantification, and the like. Of course most of the discussion at Metricon will focus on how practitioners can build metrics programs to make their security programs more efficient, maybe more effective, and certainly more substantiated (with data, as opposed to faith). Justifiably so – to mature the practice of security we need to quantify it better.

But I can’t pass up the opportunity to poke a bit at the type of quantification that comes from the vendor community. Surveys and analyses which always end up building a business case for security products and services. The latest masterpiece from the king of vendor-sponsored quantification, Larry Ponemon, is the 2nd annual cost of cyber-crime survey – sponsored by HP/ArcSight. To be clear, I’m not picking (too much) on Dr. Larry, but I wanted to put the data he presents in the report (PDF) in the proper context and talk briefly about how a typical end user should use reports like this.

First of all, Ponemon interviewed 50 end users to derive his data. It’s been a long time since I’ve done the math to determine statistical significance, but I can’t imagine that a sample size of 50 qualifies. When you look at some of the results, his findings are all over the map. The high level sound bites include a median annualized cost of $5.9 million from “cyber crime,” whatever that means. The range of annualized losses goes from $1.5 to $36.5 million. That’s a pretty wide range, eh?

His numbers are up fairly dramatically from last year, which plays into the story that things are bad and getting worse. Unsurprisingly, that’s good for generating FUD (Fear, Uncertainty, and Doubt). And that’s what we need to keep in mind about these surveys. Being right is less important than telling a good story, but we’ll get to that.

Let’s contrast that against Verizon Business’s 2011 DBIR, which used 761 data points from their own data, data from the US Secret Service, and additional data from Dutch law enforcement as a supplement. 761 vs 50. I’m no mathematician, but which data set sounds more robust and representative of the overall population to you?

Even better is one of Larry’s other findings, which I include in its entirety because it must be seen to be believed.

The most costly cyber crimes are those caused by malicious code, denial of service, stolen or hijacked devices and malicious insiders. These account for more than 90 percent of all cyber crime costs per organization on an annual basis. Mitigation of such attacks requires enabling technologies such as SIEM and enterprise GRC solutions.

Really? Mitigation of malicious code attacks requires SIEM and GRC? Maybe I’m splitting hairs here, but this kind of absolute statement make me nuts. The words matter. I understand the game. Ponemon needs to create some urgency for ArcSight’s prospects to justify the report, so throw a little love at SIEM and GRC. Rock on. Yeah, the cynic is in the house.

This statement is then justified by some data that says surveyed customers using SIEM lost on average 25% less than those without SIEM. Those folks with SIEM were able to detect faster and contain more effectively. Which is true in my experience. But only if the company makes a significant and ongoing investment. Right – to the tune of millions of dollars. I wonder if any of those 50 companies had, let’s say, a failed SIEM implementation? Were they counted in the SIEM bucket?

Again, let’s not confuse correctness of the data with the story you need to tell to do your job. That’s the value of these reports. They provide data, that is not your own, allowing you to tell a story internally. Lord knows our organizations want to see hard costs, showing real losses, to justify continued spending on security. This is the same message I deliver with our Data Breaches presentation. The data doesn’t matter – the story does.

A key skill for any management position is the ability to tell a story. In the security business, our stories must paint a picture of what can happen if the organization takes its eyes off the ball. If the money is spent elsewhere and the flanks are left unprotected. Understand that your VP of Sales is telling his/her story, about how further investment in sales is important. VPs of manufacturing tell stories about the need to upgrade equipment in the factories, and so on and so forth. So your story needs to be good.

Not all of us are graced with a breach to create instant urgency for continued security investment. Though if you believe Ponemon’s data, fewer and fewer escape unscathed each year. So you need to create your own story – preferably leveraging another organization’s pain rather than your own. In this case, the empirical correctness of the data isn’t important. It’s how the data allows you to make the points you need.

Share: