Russell Thomas and a bunch of his friends recently posted a research paper called How Bad Is It? – A Branching Activity Model to Estimate the Impact of Information Security Breaches, which attempts to provide a structure for estimating the impact of a breach. This work is necessary – we have no benchmarks, or even consensus, about what breached organizations should even be counting.
This is an academic research paper, and to be honest I am not a big fan of academic papers. I have pretty bad TL;DR syndrome. But I did check out the introduction, and noted some interesting tidbits.
Empirical research on breach losses often use ad hoc taxonomies for “quantified” and “non-quantified” costs as part of surveys or interviews of subject matter experts. There is no theoretical basis for these taxonomies, which limits their generality and research significance.
Finally, several consulting firms publish survey-based studies. Most notable is the “Cost of a Data Breach” reports by Ponemon Institute (Ponemon Institute 2012). These survey results are widely publicized and widely quoted, even in policy discussions, but they have no foundation in theoretical or empirical academic research, and they have very serious methodological flaws (Thomas 2011a)
In summary, without some reliable and robust breach impact estimation methods, quantified information security will continue to be a “weak hypothesis” (Verendel 2009).
This is true. It warms my cockles (can I say that out loud?) that these guys are calling out survey monkeys like Ponemon because the industry seems set on using those numbers to justify what we do.
But I have to say I’m a little disappointed by Russell’s attempt to jump on the indicators of compromise bandwagon in his New School blog post on the paper. He unnecessarily concocts a meaningless description of this breach impact estimation model by mentioning Indicators of Impact. Huh? Total non-sequitur, though I do understand the desire to capitalize on the popularity and momentum of the phrase Indicators of XXX.
But let’s call this what it is. An attempt to build an academically rigorous model to estimate the cost of a breach, based upon factors that can be reasonably estimated and quantified. It would be nice to see this kind of stuff added to GRC platforms and the like, to enable us to track and estimate these costs. Ultimately I believe that as we mature as a profession we will need this kind of research to help define a common vernacular for estimating loss.
Photo credit: “Impact Hoodie Design for 2006” originally uploaded by Will Foster
Reader interactions
9 Replies to “Estimating Breach Impact”
Not sure why I should have to google all of your blog posts, and I doubt that anyone else would do that, either.
Still, it’s your blog, and you can say what you wish.
I have asked Russ to remove the language about Ponemon and Verizon from the WEIS paper.
We shall see what he does, and what I will do if he doesn’t:)
Happy Easter,
Patrick
@Patrick, in this specific post I only mention Ponemon, but if you look through my blogging history (ah, the magic of the Google) it’s pretty easy to see that I’ve gone after pretty every kind of survey (even our own) at one time or another.
I understand being a Ponemon fellow might have put you in a tight spot here, given the inflammatory language Russell used in the paper (which I conveniently excerpted). But that falls into the category of not my problem.
You guys sent the shot across the bow. In your paper. I just ran it a bit farther.
So, Mike, in your original blog, who else besides Ponemon do you lampoon by name? I seem to have missed that.
This is pretty funny. Both @Russell and @Patrick wrote longer responses than my initial comments. I guess the old adage, “I would have written less if I only had the time,” continues to hold.
As Russell said, it isn’t just Ponemon that I lampoon. It’s pretty much every survey. We’ve done surveys before and I think we do as good a job as can be done, but it’s still a very flawed mechanism to draw conclusions and provide guidance.
To be clear, I think the goal of standardizing vernacular and attributes to assess (and eventually) estimate impact is a GREAT thing. I just objected to the terminology. I don’t think these are “indicators” per se, given the attempt to parallel the indicators of compromise terminology.
I hope that more work is done in this area because it’s sorely needed in the industry, and healthy and spirited debate about terminology is a good thing.
And after all, if I didn’t have an opportunity to “be Mike” what fun would that be…
Mike.
Thanks for your comments, Patrick.
Regarding the phrase “survey monkeys”, I read that as “Mike being Mike” (God love him!) I know him to be an equal-opportunity nose tweaker, and, like nearly everyone else, I’ve been on the receiving end of one of his tweaks. I don’t take offense because I believe that Mike is just trying to wake people up and to provoke useful debate. I certainly don’t agree with him on all subjects, but I do believe the debates are helpful.
Regarding survey-based methods to estimate cost of data breaches, I agree that they have *some* value, especially at the early stages of a very immature field. They can reveal the *opinions* of survey respondents regarding aspects of breach severity and costs. But when it comes to measures and costs far removed from Security, such as lifetime value of a customer, I’m skeptical that the survey respondents are doing anything more than guessing.
And, yes, Ponemon is not alone, and is better than many others. Some are so bad that it’s a waste of time to even critique them, IMO.
But even if the survey methods and respondents were ideally chosen and executed, my primary objection regards the methods used to calculate cost components and then to combine them into the Cost per Breached Record measure. I’ve detailed my critique elsewhere (New School of Information Security Blog), so I won’t repeat them here.
Regarding Verizon and their DBIR, its very true that it falls way short of what is needed if InfoSec is to become like evidence-based medicine. For example, they haven’t yet published anything on breach cost or impact, and, yes, their analysis has been limited to the specific breach cases they have access to. Also, it’s hard for them to say anything about the population because they don’t sample randomly from the general population.
But I think they are the “gold standard” for the following reasons:
– they are building a database of information from actual breaches
– they are careful in the use of statistical analysis, and add narrative explanation and interpretation to help readers make sense of the data and avoid misunderstandings.
– VERIS framework for standardizing collection of breach information
– they have an expanding number of partnerships with international organizations that collect breach information, thereby greatly enlarging the pool of data.
This approach moves us closer to the ultimate goal (in my mind): widespread standardized disclosure of InfoSec metrics, including breach impacts. Even in that ideal world, there will be a place for survey methods, but it won’t be for estimating the cost of data breaches. Instead, surveys can be most useful to reveal perceptions, attitudes, preferences, and values of managers and professionals as they evaluate alternatives and make decisions.
Russ
To follow up –
I just realized that I have coined a new expression:
Indicator of Immaturity, or, as I will dub it henceforth, “IoI.”
My immature, educated guess is that the number of IoI’s to be found in the set of information security practices and practitioners approaches 100% of the members of the set.
Patrick
Good morning, Mike –
I am one of the co-authors of Russ’ paper.
I have long been interested in building models to help us quantify and understand the range of potential breach impacts – for this reason, and because Russ is a friend and colleague, I agreed to take part.
Everyone should understand that Russ has done almost all of the heavy lifting so far, and that, in the current iteration of the paper, none of the comments from the other co-authors have been incorporated.
By way of disclaimer, I am a friend of Larry Ponemon, and a Fellow at the Ponemon Institute.
Having said that, I have collegial relationships with lots of people and groups, including Russ, members of the Verizon DBIR team, and others.
So I don’t think that I am particularly biased – at least no more than anyone else.
I have commented to Russ, and will also say to you, that calling groups like Ponemon “survey monkeys” is unhelpful. There is a methodology in what Ponemon Institute does, backed by people with PhD’s in survey methods.
If you are going to call out Ponemon, then, in fairness, you have to call out a long, long list of others, including Symantec, Deloitte, PwC, and the World Economic Forum.
My background relevant to this discussion comes from 17 years of work in evidence-based medicine.
One of the things you learn to appreciate in that endeavor is that there are different types of evidence, ranging from anecdotes to case series to randomized controlled trials to meta-analyses of randomized controlled trials.
From my perspective, the Verizon DBIR – our current “gold” standard, falls near the bottom of valid evidence – anyone who had ever worked in evidence-based medicine would tell you the same thing.
This is not a knock on Verizon – they have worked hard to improve what they publish. But they can only worth with the evidence that they have.
Rather, it is an indicator of the immaturity of information security and the lack of experience of many information security practitioners.
Perhaps a study of case reports, as the Verizon report is, ranks slightly higher than a survey, but, they are still both pretty close to the bottom of the list, so I view the matter as something akin to the pot calling the kettle black.
I look to thought leaders like you, Mike, to have a more balanced view.
Perhaps I am hopelessly naïve in that hope.
Best wishes,
Patrick Florer
Risk Centric Security, Inc.
Fellow, Ponemon Institute
One more thing: “Indicators of Impact” is one component of our method, not a label for the method as a whole.
Also, people could use Indicators of Impact for purposes other that what we’ve proposed, e.g. to support other breach impact estimation methods, to help document case studies, or to create table top exercises for help Incident Response teams interface with the rest of the business, etc.
Russ
Thanks for the comments and encouragement, Mike.
I plead guilty to writing a long paper. We cover a lot of ground and need to substantiate it as much as possible, given the academic venue. Later on we’ll create papers and presentation materials that are shorter and more oriented to practitioners.
Regarding our use of the term “Indicators of Impact”, I plead guilty to piggy-backing on the popularity of “Indicators of Compromise”, but I claim that our use of the term is fully justified, and not just a gimmick.
First, as you’ll see in Appendix B, these are *indicators* in the sense of “signs” or “signals”. They are a subset of all the possible evidence that one might evaluate, but they are more narrowly defined than the general term “evidence”. You’ll notice that they are all defined as binary conditions with simple “yes/no” answers. Further they are defined so that they can be associated with one or more specific response/recovery activities. Therefore, if you see a particular set of Indicators of Impact, then you can infer a specific set of response/recovery activities. With activities identified, you should be able to estimate a branching activity model which links them together, and then use that model to estimate breach impact in terms of costs or other metrics.
Second, the use case is similar to Indicators of Compromise. Investigators accumulate Indicators during the course of an investigation, hopefully with some standard formats for collection, aggregation, and sharing. Indicators themselves don’t provide any answers. Instead, they are inputs to analysis processes that generate answers and estimates.
Finally, other terms didn’t seem to work. “Evidence” is too general. “Symptoms” invites too much subjective interpretation, esp. regarding “severity” of symptoms. Of course, neither “measures” nor “metrics” applies since these binary conditions don’t measure anything by themselves.
One other comment — like all signs, signals, and indicators, our Indicators of Impact will not be perfectly correlated with breach response/recovery activities. There will be false positives and spurious correlations. There is also a “base rate” for each of these, since they might exist in the absence of a breach event.
Stay tuned for our case studies. I think you and all practitioners will find them most useful to evaluate the proposed methods.
Russ