Thanks in large part to the data loss database, there’s recently been some great work on analyzing breaches. I’ve used it myself to produce some slick looking presentation graphs and call attention to the ever-growing data breach epidemic.

But there’s one problem. Not a little problem, but a big honking leviathan lurking in the deep with a malevolent gleam in its black eyes.

Breach notification statistics don’t tell us anything, at all, about fraud or the real state of data breaches.

The statistics we’re all using are culled from breach notifications- the public declarations made by organizations (or the press) after an incident occurs. All a notification says is that information was lost, stolen, or simply misplaced. Notifications are a tool to warn individuals that their information was exposed, and perhaps they should take some extra precautions to protect themselves. At least that’s what the regulations say, but the truth is they are mostly a tool to shame companies into following better security practices, while giving exposed customers an excuse to sue them.

But notifications don’t tell us a damn thing about how much fraud is out there, and which exposures result in losses.

(Okay- the one exception is that any notification results in losses for a business that goes through the process).

In other words, we don’t know which of the myriad of exposures we read about daily in the press result in damages to those whose records were lost. They are also self reported; and I know for a fact there are incidents where companies did not disclose because they didn’t think they’d get caught.

For example, based on the statistics nearly a third of all breach notifications are the result of lost laptops, computers, and portable media (around 85 million records, out of around 316 million total lost records). About 51 million of those records were the result of two incidents (the VA in the US, and HMRC in the UK). The resulting fraud?

Unknown. No idea. Zip. Nada. In all those cases, I don’t know of a single one where we can tie the fraud to the lost data.

In some cases we really can track back the fraud. TJX is a great example, and the losses may be in the many tens of millions of dollars. ChoicePoint is another example, with 800 cases of identity theft resulting from 163,000 violated records (a number that’s probably really around 500,000, but ChoicePoint limited the scope of their investigation).

What we need are fraud statistics, not self-reported breach notification statistics. We do the best with what we have, but according to the notification stats we should all be encrypting laptops before we secure our web applications, yet the few fraud statistics available support the contrary conclusion.

In other words, we do not have the metrics we need to make informed risk management decisions.

This also creates a self-fulfilling negative feedback loop. Notifications result in measurable losses to businesses, driving security spending to prevent incidents that cause notifications, which may not represent prioritized security/loss risks.

When you read these things, especially on the slides shoved down your throat by desperate vendors (it’s usually slide 2 or 3), ask yourself if each one is an exposure, or actual fraud.