FireStarter: The Only Value/Loss Metric That Matters

Rich points out the difficulties of valuing assets for the purposes of an ALE-type of analysis. But I think getting close to the “annual rate of occurrence” is even harder than getting to asset value. Yes, there are some events (like a lost laptop) where we have plenty of data. Then we can model those out for both value and occurrence.

But what about those events/incidents, which cannot be modeled? Your proverbial black swans like a massive data breach or a weaponized zero day attack. Just as it is hard to estimate the value of the asset being impacted, it’s even harder to figure out when/if those kinds of events (massive potential loss, seemingly very small chance of occurring) will occur.

Which is another way of stating why we think ALE is crap. You don’t know the value of the asset, and you don’t know how often a certain event is going to happen. Hmmm. Seems like a risk analysis foundation built on quicksand to me.

But I’m sure the risk modelers out there will tell me Bayesian estimation factors all this in, eh?

Mike.

> But what about those events/incidents, which cannot
> be modeled? Your proverbial black swans like a massive
> data breach or a weaponized zero day attack.

Putting a dollar value on the loss experienced by a zero-day attack is like putting a dollar value on the loss experienced by a tornado. You simply can’t do that, nor should you try. The thing you need to try to measure is the impact.

A tornado can do a lot of damage, but if it misses your building, your loss is zero. (Perhaps you can measure lost productivity during the tornado drill, but that’s not what we’re talking about here.) If the tornado takes your entire building out, now you can measure. Because you just lost building, and that has a real cost. Same for hardware, stolen laptop, compromised system you have to rebuild, etc. I think the real tough question here is “How much is your data worth?” And I wish I had a good model for that. Perhaps my business users can shed light on it…

John,

You’ve nailed the problem, and what inspired this post. We don’t have a way to value the vast majority of IP/data in any consistent fashion. It’s like every piece of data is a work of art or family photo.

@john,
I agree that we can’t put a dollar value on the loss of a tornado, but most of the risk models try to do exactly that, which was my point.

Agree that valuing the loss is challenging as well. You can paint a worst case scenario, but what about direct costs (replacement, clean-up, disclosure, etc.) vs. indirect costs (brand damage, etc.). How much do you model in there?

Again, no one knows the answer here and that’s the point of the post.

Mike.

I don’t think anyone would argue the premise that predictions should meet reality.  And I think there are two basic points here:
1) our predictive models for expected loss stink
2) feedback into those decisions (data gathered from breeches) stink too

Completely agree, and both are required for meaningful change, if I lay out a pretty good process:
Step 1 is we try some predictive process: if a tornado hits, we’ll lose X. 
Step 2: we get a pulse on reality: when tornado hits, count the nickels and dimes, compare to X. 
Step 3: revise original predictive process to deal with second tornado.

The gotcha of course is that a tornado may never come so it would be great if we could learn from neighbors down the road, or update the predictive process for tornados hitting by learning from a tape backup failure (for example).

I don’t disagree at all with what’s been said here, but I do think we’re doing the right thing by a) trying stuff b) getting feedback and c) talking about trying stuff.  Even something as silly as the ALE process provides us a place to start comparing our theory with our reality. 

I don’t think the problem lies in risk models trying to be predictive and place value on loss, but the problem is that there’s no feedback, no process for improvement, and in doing so make more claims to their accuracy than are not supportable by data.

>>>
We don’t have a way to value the vast majority of IP/data in any consistent fashion. It’s like every piece of data is a work of art or family photo.
>>>

While I would agree that it’s difficult to quantify IP/data, the fact remains that much of the IP stolen or involved in breaches is tied to a specific project and there is a distinct valuation on said data based on expected returns.  If there weren’t these “expected returns/value” then the business would never embark on developing or utilizing the IP in the first place.

For example, working with a manufacturer recently, they clearly stated that older projects or products that had been on market for years had a significantly lower value than those that were pre-market.  This may be stating the obvious, but it seems that there a time-valuation and distinct expected revenue from certain forms of IP.  That may complicate the quant side of things, but nevertheless, it seems that there should be some way to get to a basis of loss expectancy from there. 

Then again, maybe I’ve had one too many, but without some basis, we may as well throw darts blindfolded.  Somebody in the business knows what this stuff is worth, otherwise, I should just select all and hit delete and no one will be any worse off.

Well, whether you call it ALE or something else, the basic idea is that we want to predict what something is going to cost us. While I agree that the (current) concept of ALE is misguided, if you replace “guess the value of your data” with “insert actual cost” then it’s basically the same procedure you are going to follow to fulfill Rich’s requirement.
Because somewhere down the line, you are going to have to decide which risks to mitigate and which to ignore. That requires you make up your mind about what is going to happen to you and what is not, thus forcing some more or less accurate estimate of likelihood.

And estimating the worth of your assets is going to be just as tough after the fact as before.
In all likelihood, we must therefore settle for vague estimates for the foreseeable future. Preferably (in my book) without quantitative metrics.

Having said that, the world is not entirely without risk frameworks that move in the right direction. IIRC, Information Security Forum has a risk analysis methodology (FIRM) that bases it’s estimates for likelihood/probability on actual events. Ie. not “how often will this happen?”, but “how often has this happened?”.
If you could extend that to also as the same question for actual loss, we would be getting somewhere.

And whatever way you try to estimate your risk:  the lack of real incident data is going to be Bump-In-the-Road #1.
To change that requires widespread industry information sharing, which I guess is sort of a “holy grail” of infosec.

And if we can achieve that, then I am not so sure that “traditional” risk analysis methods won’t work well after all.

You’ve rather lost me… the post starts out as a criticism of ALE (fine, easy target), but then concludes with “...I have yet to see a risk framework with a value/loss model that meets this basic requirement for information assets.” Rothman further adds on “But I’m sure the risk modelers out there will tell me Bayesian estimation factors all this in, eh?” to suggest that you’re not just focused on the classic ALE approach, but are instead attacking all current risk assessment methodologies. It’s kind of difficult to respond when it’s unclear a) what you’re actually talking about, and b) if you’ve done enough research to make your fire-starter comments. e.g. Have you studied FAIR? Did you know that it does not use a classical ALE approach? For a fire starter post, this one seems to be a weak flame…

All of the concerns that have been raised about estimating impact are legitimate.  Part of the problem with many approaches to-date, however, is that they’ve concentrated on asset value and not clearly differentiated that from asset liability.  Another challenge is that we tend to do a poor job of categorizing how loss materializes. 

What I’ve had success with in FAIR is to carve loss into two components—Primary and Secondary.  Primary loss occurs directly as a result of an event (e.g., productivity loss due to an application being down, investigation costs, replacement costs, etc.), while Secondary loss occurs as a consequence of stakeholder reactions to the event (e.g., fines/judgments, reputation effects, the costs associated with managing both of those, etc.).  I also sub-categorize losses as materializing in one or more of six forms (productivity, response, replacement, competitive advantage, fines/judgments, and reputation). 

With the clarity provided by differentiating between the Primary and Secondary loss components, and the six forms of loss, I find it much easier to get good estimates from the business subject matter experts (e.g., Legal, Marketing, Operations, etc.).  To make effective use of these estimates we use them as input to PERT distribution functions, which then become part of a Monte Carlo analysis.

Despite what some people might think, this is actually a very straightforward process, and simple spreadsheet tools remove the vast majority of the complexity.  Besides results that stand up to scrutiny, another advantage is that a lot of the data you get from the business SME’s is reusable from analysis to analysis, which streamlines the process considerably.

Ben,

I have studied FAIR, OCTAVE, and whatever else I can get my hands on. Every framework has to have a loss/valuation component at some point. ALE is the simple example, but isn’t alone.

Jack,

Good point- I shouldn’t lump FAIR in quite the same way since I like how you’ve split the losses and try to use multiple input points to develop the estimate.

What’s nice is that someone can break out the categories and loss types and then evaluate post-incident losses using the same framework. Have you thought about making this post-incident analysis part of FAIR? (Apologies if I’ve missed that part and it is already in there).

Rich -

It’s too bad you missed MiniMetricon 4.5 as we talked a bit about this very topic. Pete Lindstrom provided a good talk based on Douglass Hubbard’s books (in particular, his “How to Measure Anything”). Ranges and confidence are key, and help shake out much of the concern you’ve expressed.

fwiw.

Rich,

I’m not sure that I follow the point of your response to Ben.  Yes, as you state, every framework has to have a loss component at some point.  So the question becomes whether that component of a framework is reasonably effective.  Do you believe it’s impossible to effectively characterize the loss component of a risk scenario, or do you just think the infosec profession has done a poor job of that to-date?

Rich,

Sorry.  Our posts seem to have “crossed in the mail” so to speak.  You can delete my question about your response to Ben if you like. 

Very glad to hear that my categorization strikes a chord with you.  We’ve had excellent buy-in from business management with the approach, and analysis of losses from actual incidents fits nicely within the framework, which helps validate the categories and allows us to do a decent job of leveraging empirical data where it exists.

Unfortunately, I haven’t had (made?) time to update the documentation I’ve made public about FAIR, so a lot of people aren’t familiar with some of the improvements that have taken place since the original white paper was written. 

Thanks,
Jack

Hi, Rich -

It’s nice to hear you say something somewhat nice about quantitative approaches.

They next thing I hope to hear you say someday is that qualitative approaches are almost completely worthless and misleading.

There are a number of ideas that I might suggest here.

1)  Focusing on the value of assets is not always the right thing to do because it’s not always where the real value/risk is - rather, the value/risk is sometimes the loss exposure, realized or as yet unrealized, of a compromised asset, the value of a lost or compromised business process, data store, protected information, etc.

  As I understand it, and some accounting types might wish to weigh in here, according to GAAP (Generally Accepted Accounting Principles), the book value of “information” is limited to the cost of creation and maintenance of the information.  In the event of the sale of a company, additional value of information may be recognized as “good will”.  This value is in many cases far less than the “value/cost” of the information if it falls into the wrong hands.

  As we know from TJX ($170-250M so far), Heartland ($140M so far), and others, the costs of dealing with a large data breach are huge (even if not close to $200/rec that some assert.)

  I wonder which was greater for TJX or Heartland - the cost of creating and maintaining the information, or the loss exposure that came about because of the breaches?  Just a question - I don’t know the answer.

  With regard to a business process, maybe a company has a $10M investment in IT that generates $250M in revenues.  The value of the asset may not even come close to the exposure created by losing the process.

2)  Concerning models out there, you and I have talked about FAIR, which is one model that produces consistent, reproducible estimates.  There are other ways to do this, too.

  I guess now that I am 60 years old, I might as well say what I think - the lack of broader experience that becomes evident when talking to many infosec practitioners is a big problem.

  An even bigger problem - really appalling in my view, is the willingness of many infosec practitioners to issue “pronouncements” based upon this state of ignorance.  (I am not particularly shooting at you, here)

  And, the lack of intellectual honesty and curiosity that is apparent with many infosec “rock stars” is probably the biggest obstacle of all.

  Actuaries, insurance companies, oil and gas companies (even BP), and many others have for decades been doing the sorts of quantitative risk analyses that infosec says are impossible.

  We need to look outside, as Adam Shostack has advocated, and learn from others before deciding what is or is not possible.

3)  Too many people are looking for “the answer”, rather than a range of reasonable estimates that help to reduce uncertainty.  In my view, the whole purpose of risk analysis is to reduce uncertainty in a way that leads to better decision making.

  If you wish to try to convince me that qualitative methods do a better job, I am willing to listen.

4)  You are absolutely correct that every method needs to be tested against measurable outcomes.  I know how to do this with a quantitative approach.  It is not at all clear to me how this might be accomplished in a meaningful way with non-quantitative methods.

Best regards,

Patrick Florer

Patrick…

I argue that the vast majority of quantitative risk assessments I’ve seen in infosec are little more than qualitative risk assessments with dollar signs added to wild ass guesses. Thus they are even more worthless and deceiving than a model that admits a guess is a guess.

I didn’t reiterate it in this post, but my philosophy on risk assessment is quantify as much as you can, qualify where you can’t accurately quantify, and combine them in a consistent fashion to communicate overall risk. I don’t believe either is “right” on it’s own.

It’s easy to say we should learn from other industries, but it isn’t so easy. As I’ll detail in the next response, information assets are fundamentally different than physical goods, which is why we have the problems we do.

This is turning into a great discussion. Bravo to all those participating. At the risk of being the wet blanket man, I don’t think we’ve addressed Rich’s point about going back and comparing **actual** loss to the loss predicted by the (various) models.

As evidenced by the discussion, there are lots of ways to estimate potential losses. Many will be defendable and pass muster of the business folks. The real question is the accuracy of the estimates. We can provide ranges and confidence levels all day and night, but unless we close the loop and actually figure out the real accuracy of the model, we are still practicing black magic. Not science.

I’m not familiar with any attempts to compare estimated loss to actual loss. Can anyone share an example?

Mike.

I want to clarify something that isn’t as clear as it could be from my original post, which responds to a couple of comments…

1. Calling for a model to validate predicted losses with experienced losses applies to any model or type of loss. This is the non-controversial part of the post, other than almost no one does it.
2. I do believe you can measure a number of loss vectors- costs to replace physical items, response costs, legal costs, etc.
3. I do not believe there is any way (currently) to consistently measure the dollar value of the information asset itself. We can equate its value with the loss categories we *can* measure, but that’s not the real value of the asset. That’s the part we can’t measure, but it’s also where I see a lot of infosec risk assessments get completely derailed as people make up numbers which are essentially qualitative expressed as quantitative.

I only slightly called out point 3 directly in the post since if you agree with the prediction/experienced tenet, 3 emerges naturally.

Mike nailed it, and brings us back to what I intended from the start.

Show me a data valuation model where the predicted value matched the measured value after a loss event. I’m not saying that’s impossible, but I haven’t seen it done.

Actually, Mike, I have (unfortunately?) had an opportunity to validate loss estimates for a couple of events where I’ve worked.  The estimates fit quite well.  If I were still working there, and if you were under NDA, I’d be happy to share the details.  Those who’ve worked with me though, can at least corroborate my assertion.

Of course, one of the challenges with any relatively new method is that it takes time to establish enough history in their use to strongly substantiate (or not) their effectiveness.  In the meantime, we’re left with evaluating them based on the logic/reasonableness of their approach and whatever data are available and become available over time.

With regard to actual loss -
In the cases of the big breaches I have followed from 10-K filings as well as press reports, it’s clear to me that these costs unfold over time - years, in fact - and that the estimates and set-asides change and go up and down.  Just take a look at the TJX 10-K’s for 2007, 2008, and 2009.

In addition, there are capex components that may be involved - accelerated spending, delayed spending, etc., that make it hard to tell what the costs are.

I don’t really agree about learning from other disciplines - although I have worked in IT for 30 years, what really turned on the lights for me with regard to risk was the 17 years of that 30 years that I spent part-time in clinical outcomes research - that’s where I learned my statistics, Bayesian techniques, and decision analysis stuff.

Since then I have accelerated my studies in statistics and quantitative risk analysis.  Monte Carlo techniques and probability distributions are not hard to use correctly, even if you cannot do the math by hand.

Mike, very few things in our modern world aren’t black-box-like, wouldn’t you agree?  Modern cars are a completely mystery to me, as are iPods and even refrigerators.  But they work.

I am probably an exception to the rule - an old-fashioned generalist with fairly deep skill.

I never saw this coming 40 years ago when I graduated from UT Austin with a degree in Classical Greek.

This is a fascination discussion, but one thing jumps out at me. There has been quite a bit of discussion here about determining loss, the probability of bad things happening, and other associated factors when trying to determine risk, but one comment/question was made about determining the value of our information. Not the value of loss, but the value of the information to the organization.

Call me naive, but shouldn’t our business partners be able to tell us what the value of their information is? Notice I said their information.

It would seem to me that since we see constant forecasts for revenue and expectations for profit, we (they) should be able to tie that back to the value of the information they maintain in some meaningful manner.

I’m not saying it should be or is easy, but it seems imminently reasonable to me, not that I have accomplished this in my own organization.

It is definitely giving me something to think about though.

Kevin

Hi Kevin,

I believe you’re right that our business colleagues should be able to tell us (at least roughly) what the business value of the information is.  That said, that information is only relevant if the scenario we’re analyzing involves either the loss of that data (as in, it goes away) or damaged integrity of the information.  If the data is still in our possession and we’re still able to generate/realize its value in our business processes, then losses tend to be associated with liability (i.e., secondary loss from stakeholder reactions) and the costs associated with responding to the event.  Consequently, it becomes important in our analyses to distinguish between the different types of events (confidentiality vs. integrity vs. availability).

Jack

Kevin,

Very interesting!

I would submit that information has no value at all except in its use or mis-use.

On the positive side, it’s the value of the business process that the information enables that matters.

On the loss side, maybe it’s a bit more complicated - your business process could be compromised due to loss of information or processing capability.  Or, your information could fall into the wrong hands and create any number of liability scenarios.  Or both.

IT hardware assets, given the rapid pace of change, also have little value, except in their use or mis-use.

Forget about what is carried on a company’s fixed asset ledger as book and depreciated value.

Once you install a server or a data center, what is it really worth if it isn’t doing anything to support business processes?  Very little.

Have you ever tried to sell a used server or a data center or software?

This point was driven home to me about 25 years ago when the company I worked for shut down suddenly.  It was a small service bureau that had about $1M worth of medium scale mainframe and DEC minicomputer hardware - that was $1M cost carried on the books.

At auction, the $1M of hardware fetched less that $40k.

Patrick

As per usual, context is everything, eh? Letters have little to no value until formed into words. Words have some value, but not nearly as much (generally) as when they’re chained to form sentences and paragraphs and so on. It’s not the representation, but the contextual interpretation or use that is important.