As some of you know, I’ve always been pretty critical of quantitative risk frameworks for information security, especially the Annualized Loss Expectancy (ALE) model taught in most of the infosec books. It isn’t that I think quantitative is bad, or that qualitative is always materially better, but I’m not a fan of funny math.
Let’s take ALE. The key to the model is that your annual predicted losses are the losses from a single event, times the annual rate of occurrence. This works well for some areas, such as shrinkage and laptop losses, but is worthless for most of information security. Why? Because we don’t have any way to measure the value of information assets.
Oh, sure, there are plenty of models out there that fake their way through this, but I’ve never seen one that is consistent, accurate, and measurable. The closest we get is Lindstrom’s Razor, which states that the value of an asset is at least as great as the cost of the defenses you place around it. (I consider that an implied or assumed value, which may bear no correlation to the real value).
I’m really only asking for one thing out of a valuation/loss model:
The losses predicted by a risk model before an incident should equal, within a reasonable tolerance, those experienced after an incident.
In other words, if you state that X asset has $Y value, when you experience a breach or incident involving X, you should experience $Y + (response costs) losses. I added, “within a reasonable tolerance” since I don’t think we need complete accuracy, but we should at least be in the ballpark. You’ll notice this also means we need a framework, process, and metrics to accurately measure losses after an incident.
If someone comes into my home and steals my TV, I know how much it costs to replace it. If they take a work of art, maybe there’s an insurance value or similar investment/replacement cost (likely based on what I paid for it). If they steal all my family photos? Priceless – since they are impossible to replace and I can’t put a dollar sign on their personal value. What if they come in and make a copy of my TV, but don’t steal it? Er… Umm… Ugh.
I don’t think this is an unreasonable position, but I have yet to see a risk framework with a value/loss model that meets this basic requirement for information assets.
Reader interactions
34 Replies to “FireStarter: The Only Value/Loss Metric That Matters”
Ben,
I have studied FAIR, OCTAVE, and whatever else I can get my hands on. Every framework has to have a loss/valuation component at some point. ALE is the simple example, but isn’t alone.
All of the concerns that have been raised about estimating impact are legitimate. Part of the problem with many approaches to-date, however, is that they’ve concentrated on asset value and not clearly differentiated that from asset liability. Another challenge is that we tend to do a poor job of categorizing how loss materializes.
What I’ve had success with in FAIR is to carve loss into two components — Primary and Secondary. Primary loss occurs directly as a result of an event (e.g., productivity loss due to an application being down, investigation costs, replacement costs, etc.), while Secondary loss occurs as a consequence of stakeholder reactions to the event (e.g., fines/judgments, reputation effects, the costs associated with managing both of those, etc.). I also sub-categorize losses as materializing in one or more of six forms (productivity, response, replacement, competitive advantage, fines/judgments, and reputation).
With the clarity provided by differentiating between the Primary and Secondary loss components, and the six forms of loss, I find it much easier to get good estimates from the business subject matter experts (e.g., Legal, Marketing, Operations, etc.). To make effective use of these estimates we use them as input to PERT distribution functions, which then become part of a Monte Carlo analysis.
Despite what some people might think, this is actually a very straightforward process, and simple spreadsheet tools remove the vast majority of the complexity. Besides results that stand up to scrutiny, another advantage is that a lot of the data you get from the business SME’s is reusable from analysis to analysis, which streamlines the process considerably.
You’ve rather lost me… the post starts out as a criticism of ALE (fine, easy target), but then concludes with “…I have yet to see a risk framework with a value/loss model that meets this basic requirement for information assets.” Rothman further adds on “But I
Well, whether you call it ALE or something else, the basic idea is that we want to predict what something is going to cost us. While I agree that the (current) concept of ALE is misguided, if you replace “guess the value of your data” with “insert actual cost” then it’s basically the same procedure you are going to follow to fulfill Rich’s requirement.
Because somewhere down the line, you are going to have to decide which risks to mitigate and which to ignore. That requires you make up your mind about what is going to happen to you and what is not, thus forcing some more or less accurate estimate of likelihood.
And estimating the worth of your assets is going to be just as tough after the fact as before.
In all likelihood, we must therefore settle for vague estimates for the foreseeable future. Preferably (in my book) without quantitative metrics.
Having said that, the world is not entirely without risk frameworks that move in the right direction. IIRC, Information Security Forum has a risk analysis methodology (FIRM) that bases it’s estimates for likelihood/probability on actual events. Ie. not “how often will this happen?”, but “how often has this happened?”.
If you could extend that to also as the same question for actual loss, we would be getting somewhere.
And whatever way you try to estimate your risk: the lack of real incident data is going to be Bump-In-the-Road #1.
To change that requires widespread industry information sharing, which I guess is sort of a “holy grail” of infosec.
And if we can achieve that, then I am not so sure that “traditional” risk analysis methods won’t work well after all.
>>>
We don
I don’t think anyone would argue the premise that predictions should meet reality. And I think there are two basic points here:
1) our predictive models for expected loss stink
2) feedback into those decisions (data gathered from breeches) stink too
Completely agree, and both are required for meaningful change, if I lay out a pretty good process:
Step 1 is we try some predictive process: if a tornado hits, we’ll lose X.
Step 2: we get a pulse on reality: when tornado hits, count the nickels and dimes, compare to X.
Step 3: revise original predictive process to deal with second tornado.
The gotcha of course is that a tornado may never come so it would be great if we could learn from neighbors down the road, or update the predictive process for tornados hitting by learning from a tape backup failure (for example).
I don’t disagree at all with what’s been said here, but I do think we’re doing the right thing by a) trying stuff b) getting feedback and c) talking about trying stuff. Even something as silly as the ALE process provides us a place to start comparing our theory with our reality.
I don’t think the problem lies in risk models trying to be predictive and place value on loss, but the problem is that there’s no feedback, no process for improvement, and in doing so make more claims to their accuracy than are not supportable by data.
@john,
I agree that we can’t put a dollar value on the loss of a tornado, but most of the risk models try to do exactly that, which was my point.
Agree that valuing the loss is challenging as well. You can paint a worst case scenario, but what about direct costs (replacement, clean-up, disclosure, etc.) vs. indirect costs (brand damage, etc.). How much do you model in there?
Again, no one knows the answer here and that’s the point of the post.
Mike.
John,
You’ve nailed the problem, and what inspired this post. We don’t have a way to value the vast majority of IP/data in any consistent fashion. It’s like every piece of data is a work of art or family photo.
> But what about those events/incidents, which cannot
> be modeled? Your proverbial black swans like a massive
> data breach or a weaponized zero day attack.
Putting a dollar value on the loss experienced by a zero-day attack is like putting a dollar value on the loss experienced by a tornado. You simply can’t do that, nor should you try. The thing you need to try to measure is the impact.
A tornado can do a lot of damage, but if it misses your building, your loss is zero. (Perhaps you can measure lost productivity during the tornado drill, but that’s not what we’re talking about here.) If the tornado takes your entire building out, now you can measure. Because you just lost building, and that has a real cost. Same for hardware, stolen laptop, compromised system you have to rebuild, etc. I think the real tough question here is “How much is your data worth?” And I wish I had a good model for that. Perhaps my business users can shed light on it…
Rich points out the difficulties of valuing assets for the purposes of an ALE-type of analysis. But I think getting close to the “annual rate of occurrence” is even harder than getting to asset value. Yes, there are some events (like a lost laptop) where we have plenty of data. Then we can model those out for both value and occurrence.
But what about those events/incidents, which cannot be modeled? Your proverbial black swans like a massive data breach or a weaponized zero day attack. Just as it is hard to estimate the value of the asset being impacted, it’s even harder to figure out when/if those kinds of events (massive potential loss, seemingly very small chance of occurring) will occur.
Which is another way of stating why we think ALE is crap. You don’t know the value of the asset, and you don’t know how often a certain event is going to happen. Hmmm. Seems like a risk analysis foundation built on quicksand to me.
But I’m sure the risk modelers out there will tell me Bayesian estimation factors all this in, eh?
Mike.