I recently got into a debate with someone about cyber-insurance. I know some companies are buying insurance to protect against a breach, or to contain risk, or for some other reason. In reality, these folks are flushing money down the toilet. Why? Because the insurance companies are charging too much. We’ve already had some brave soul admit that the insurers have no idea how to price these policies because they have no data and as such they are making up the numbers. And I assure you, they are not going to put themselves at risk, so they are erring on the side of charging too much. Which means buyers of these policies are flushing money down the loo.
Of course, cyber-insurance is just one example of trying to quantify risk. And taking the chance that the ALE heads and my FAIR-weather friends will jump on my ass, let me bait the trolls and see what happens. I still hold that risk metrics are crap. Plenty of folks make up analyses in attempts to quantify something we really can’t. Risk means something different to everyone – even within your organization. I know FAIR attempts to standardize vernacular and get everyone on the same page (which is critical), but I am still missing the value of actually building the models and making up plugging numbers in. I’m pretty sure modeling risk has failed miserably over time. Yet lots of folks continue to do so with catastrophic results. They think generating a number makes them right. It doesn’t. If you don’t believe me, I have a tranche of sub-prime mortgages to sell you.
There may be examples of risk quantification wins in security, but it’s hard to find them. Jack is right: The cost of non-compliance is zero* (*unless something goes wrong). I just snicker at the futility of trying to estimate the chance of something going wrong. And if a bean counter has ever torn apart your fancy spreadsheet estimating such risk, you know exactly what I’m talking about.
That said, I do think it’s very important to assess risk, as opposed to trying to quantify it. No, I’m not talking out of both sides of my mouth. We need to be able to categorize every decision into a number of risk buckets that can be used to compare the relative risk of any decision we make against other choices we could make. For example, we should be able to evaluate the risk of firing our trusted admin (probably pretty risky, unless your de-provisioning processes kick ass) versus not upgrading your perimeter with a fancy application aware box (not as risky because you already block Facebook and do network layer DLP).
But you don’t need to be able to say the risk of firing the admin is 92, and the risk of not upgrading the perimeter is 25. Those numbers are crap and smell as bad as the vendors who try to tie their security products to a specific ROI.
BTW, I’m not taking a dump on all quantification. I have always been a big fan of security (as opposed to risk) metrics. From an operational standpoint, we need to measure our activity and work to improve it. I have been an outspoken proponent of benchmarking, which requires sharing data (h/t to New School), and I expect to be kicking off a research project to dig into security benchmarking within the next few weeks.
And we can always default to Shrdlu’s next-generation security metrics, which are awesome.
But I think spending a lot of time trying to quantify risk continues to be a waste. I know you all make decisions every day because Symantec thinks today’s CyberCrime Index is 64 and that’s down 6%. Huh? WTF? I mean, that’s just making sh*t up.
So fire away, risk quantifiers. Why am I wrong? What am I missing? How have you achieved success quantifying risk? Or am I just picking on the short bus this morning?
Photo credits: “Smoking pile of sh*t – cropped” originally uploaded by David T Jones
Reader interactions
8 Replies to “FireStarter: Risk Metrics Are Crap”
“Understand what’s important (by talking to the folks that make the decisions in your organization) and focus on protecting those systems.” Herein lies the rub, having a qualitative discussion about what is important and focus on that is great and all but often times business lines are not able to qualitatively discuss with enough concise language why one thing is more important than another. And often times within a large organization a specific business line does not see the big picture to ensure the right things are focused on from an enterprise perspective. Some level of quantitative analysis is valuable and important in order to ensure that all parties are seeing the risk from the same perspective.
Now I am not certainly advocating just applying a risk number (eg 92) to an asset without solid reasoning, however, in the simplest terms if you can at least justify the assumptions and logic of why a particular number is assigned to an asset you will at least be able to have a business discussion with all parties seeing the same assumptions, leading to better informed risk decisioning.
I am personally of the opinion that Info risk analysis should be a blend of qualitative and quantitative attributes that are well defined for particular use cases and scenarios – the risk for some use cases and scenarios can be very quantitative while others depending on the degree of uncertainly and variability may have more qualitative qualities that best describe the risk. But rather than use debating absolutes (risk analysis versus no risk analysis) wouldn’t an organization be better off to have something to leverage that at least is clear in how the assumptions and risk assessments are done regardless of how sophisticated?
Lastly, I understand that the following statement is said in jest…“And we can always default to Shrdlu’s next-generation security metrics, which are awesome.”
In my opinion posts like this ares a big part of our professions problem, if people continue to play the victim abdicating any personal responsibility for improving the situation we will continue to struggle as an industry for business credibility, and any efforts to progress the cause of informed data driven risk analysis and decisioning will continue to suffer.
My two cents 🙂
While I agree some/most of the stuff I see out there regarding quantified risk and weird scores are nebulous and hard to justify at best, it seems that all the risk analysis stuff is really about trying to make informed decisions and if done properly – it would seem to help in achieving that goal.
Your statement that folks should simply ‘focus on protecting important assets and make sure they don’t go down or get compromised’ – is absolutely true and certainly digestible by the masses, but when it comes down to ‘how much security and of what type’, ‘what technologies to deploy’ etc. etc.- having some rational or logic applied to the impact of that asset going down (primary loss, secondary or tertiary losses through reputation or regulatory impact or whatever) should then help to prioritize efforts around even those ‘important assets’…and isn’t that what risk quantification is trying to achieve? If I think specifically in the realm of application security controls (developer training, WAFs, penetration testing, static code analysis, binary analysis, etc. etc.) all have very different costs associated with them and it would seem that just cause one of the assets that your trying to protect is deemed ‘important’ doesn’t mean that you throw the kitchen sink at it as few organizations would have that kinda budget or capability… in my thought, that is the kinda value that the risk analysis abstract brings to the table (or at least attempts to).
I don’t claim to be an authority on risk or risk analysis stuff, most of my career has been focused on testing and breaking stuff, and to me that kinda technical testing is integral to any kinda IT security risk analysis just to understand the level of exposure in place, susceptibility to attack and what type of attackers could theoretically act on those exposed vulnerabilities (aka does it take a really good bug finder and exploit writer to identify the vuln and craft an 0-day, or can you just pop it using a canned public script). From what I have seen, when you look at that kinda vuln data and then maybe even do a basic high-level threat model, the risk abstract is just another layer on the top of that analysis (obviously the threat modeling kinda analysis comes from the app dev side, but from what i have seen from some risk methodologies, the threat model is basically the first step of a (good) risk analysis).
The first sentence of the last paragraph in your post though is legit – if you are spending a ton of time on quantifying risk, then that is a waste…but I would argue that if it can be done in reasonably short order and provide value, it actually seems pretty useful. And not to be a FAIR follower or anything, but i have actually seen a quick and dirty risk analysis of a target web app using that methodology that was logical, defend-able, fast and provided actionable information for decision making …which again, kinda seems like the point.
—db (Accuvant LABS)
Hi Mike. With regards to being pragmatic – you are preaching to the pragmatic chorus boy. So in the interest of being pragmatic – let’s cut to the chase: some [a lot of] organizations are scrutinizing every dollar of IT spend regardless if it is spend related to maintaining a capability or building a new capability. The days of just do it because: “we need to be secure”, “it is a best practice”, “regulation/standard X say we have to” or “it just feels right” are no longer the norm. Cost / benefit decisions are being made by both IT & non-IT people that want to make well informed decisions and ensure the dollars they are spending are of value (in our case, either reduce frequency or severity of loss). I have observed and participated in such decisions in both large and small organizations. While I do not always agree with the decisions that have been made based off the analysis I presented – that is less of a concern to me – I helped facilitate their decision. The reality is that these decisions are often based on dollars – the language of business – and unless you are using a methodology whose output can speak that language we are relegated to the sidelines. Take Jack up on his offer and if you are ever in Columbus, OH and want to spend a few hours seeing those unicorns you reference – give me a DM.
@Mike,
As usual, you can be counted on for a pragmatic focus. A few thoughts:
* The level of effort in data collection is variable. If you’re looking for a quick-and-dirty analysis, you spend less time/effort. The results will be less reliable and less precise, but certainly no less so than a shoot-from-the-hip qualitative risk statement. Furthermore, much of the data collected are reusable, which streamlines the process over time. For example, if I analyze the loss values associated with a data breach of a certain size/type, then much of those data should be reusable in similar future analyses. Also, as Chris intimates, we have a lot more data available to us than we recognize.
* Love your unicorn analogy. Clearly, marketing is not my strong suit. Still, the word seems to be getting out, largely from organizations who’ve been trained in and started using FAIR, so the fish are still jumping in the boat. And yes, I do believe that most of the frustration out there regarding risk is due to bad methodology.
* We’ve had the dialog before about sophistication, and you’re right that, today, for many organizations it’s simply a matter of triage—keep the airway open, the heart beating, and the major blood vessels sealed. The utility of deeper understanding and analysis can play a key role in instances where management wants to know what they’re getting for their money/pain in more relevant terms, when intuitive prioritization isn’t cutting it, and/or when we’re trying to help our organizations be cost-effective in their security efforts. It’s been my experience that these considerations are becoming more and more important, at least for many organizations. Another thing that shouldn’t be overlooked is how a solid methodology can change perceptions, dialog, and support. My level of influence where I worked increased dramatically when I took a risk-based approach that stood up to critical review.
I’d love to have a chance to sit down with you for half a day and run through the basics and an analysis or two. If we’re ever in the same place at the same time, let’s set it up.
Cheers,
Jack
@chris, I didn’t necessarily say who was on the short bus, did I?
But it’s good to see the RISK zealotry out in force to defend it. Yes, I jest a bit. though I suspect you guys are missing the point. And maybe that’s because I wasn’t exactly clear in the Firestarter. Not the first time I threw a half-baked thought over the transom for folks to shoot at. And thanks for shooting. I do appreciate the feedback.
@jack, to your point about well informed decisions, I’m with you. Obviously you want to make decisions with the best date you can. But at what cost? Maybe FAIR is different (and you do point out I haven’t been trained in the methodology), but everything else I’ve seen requires such a significant data collection/analysis investment that the value of the benefit is outweighed by the cost of gathering the data.
But more importantly, there is a lack of demonstrable success to point to, so that we all (including yours truly) can understand the value of this kind of risk analysis/management. The folks I’ve spoken to tend to give up the ghost after spinning their wheels for a long time. Maybe they are doing it wrong. Maybe it’s not that hard, as Chris says. Unfortunately they don’t see value in line with effort, so they move on to other things. Like those crappy operational metrics that Jack mentions.
I have a different take on operational metrics and that’s mostly because of the reality that security is an overhead function and we aren’t going to get more resources. Operational excellence is critical and those kinds of metrics (though not necessarily useful to gauge risk) to gauge efficiency. Are the controls right? Do they really increase security? The context isn’t there. But we all need to patch, and the mandates require AV. So we should do those functions as efficiently as possible.
I also need to remind everyone that the world is a very unsophisticated place. At least when it comes to security and even more so regarding IT risk. Chris, you ask what is so hard about estimating frequency and severity of loss? Most companies out there don’t even know how many devices they have. Really.
But I titled the post Risk Metrics are Crap. And for most of the world I believe that. So maybe I should have titled the post: “Risk Metrics are Crap, unless you are the 1% of the world that has the resources, expertise, patience and perseverance to make it work.” Though not sure how that title would work for my Google SEO score.
As the steward of FAIR, Jack you really need to start publicizing those “remarkable successes.” I don’t doubt they exist, but I do put them in the camp of unicorns. We’ve all heard of them (mostly because you and Alex talk about them), but almost no one has seen one. All of you guys can put me in a bucket of “folks that just don’t understand” and that’s fine. Not the first time, and won’t be the last.
But don’t mistake your view of my ignorance versus the real problem. And I have the same issue with GRC, SIEM, application security, and host of other security processes or technologies. These are science projects. And the vast majority of the world won’t deal with a science project. IMHO, we need to be focused on bringing these techniques to the masses. Or at least some relevant portion of the technique.
To answer your final question Chris. I’d rather most folks to just be pragmatic. Understand what’s important (by talking to the folks that make the decisions in your organization) and focus on protecting those systems. In my experience, unless you are a Fortune-class large enterprise, that kind of coarse risk filter is about as detailed as you need to get. Find the stuff that when it goes down (or loses data), you get canned. Make sure it doesn’t go down or lose data. And then there is everything else.
Does this coarse filter gloss over a whole mess of complexity and not really reflect the real world? Maybe. Is it applicable to a F500 organization? Not by a long shot. But it’s something everyone can understand. And that’s not something you can say about the current methods of risk quantification.
Hi Mike. I saw a short bus this morning and laughed out loud when I read that in your post. You really are all over the place in this post. In terms of cyber insurance, I think I made some comments on that post you referenced – but they are not there in their entirety. As skeptical as I am about “cyber-insurance” – the products themselves DO meet the characteristics of an insurable risk. While the completeness or accuracy of the dataset that determines cyber-risk premium pricing used by any given insurance carriers that make cyber risk insurance products can be debated – the true litmus test will be when cyber insurance claims are filed and the level of satisfaction of claimants in terms of indemnification. I keep my Internet eyes open for such examples and have only come across a few cases.
“I’m pretty sure modeling risk has failed miserably over time. Yet lots of folks continue to do so with catastrophic results.”
Really? In the modeling world – the old saying goes: All models are wrong – some are just more useful then others (other wise we would not have investment millionaires let alone pay affordable premiums on commodity insurance products). The whole sub-prime debacle was not JUST a failure of a model. It was a failure of a variety of things – bad assumptions, lack of internal oversight in financial firms and the list can go on. BTW, I do recall reading that there were some sub-prime investment models that accounted for the loss reflected in the sub-prime implosion – 27 standard deviations away from expected loss. <- time for a model adjustment. In terms of risk quantification what is so hard about estimating numbers for frequency and severity loss? We have lots of information about data breaches, outages, hacking, and for a lot of these events we know what some of the costs are – why can’t these be combined to make informed estimates in the context of the organization we are doing it on behalf of? There are way too many IT security methods that completely miss the mark on risk analysis let alone risk quantification – and then crappy metrics are created – go figure. My final thoughts: 1. Crappy metrics are probably reflective of crappy methods. 2. Learn more about the discipline of traditional risk management – that has been around for a few hundred years - and then view IT risk (of which security is a component of) through that traditional lens. 3. Forgive the brutally terse nature of what you are about to read but, don’t criticize methods (individual or aggregate modeling) or methodologies (e.g. FAIR) that you really do not know a lot about to begin with. One last thing Mike: From your perspective, what do you consider “achieving success” with regards to risk quantification?
Actually, most security metrics I’ve seen are either operational (e.g., “Our patch levels are 95% current”, or “We’ve had 83% of our employees sign the acceptable use policy”,) or they try to imply significance through broken scoring methods (e.g., CVSS). The operational metrics lack context. “Okay, so we’re at 95% currency on our patches—what’s wrong with 90% or, better yet, why should we shoot for 99%?” Without expressing a value proposition that’s meaningful, it’s not nearly as useful from a decision-making perspective. The broken methods often lead to poor decisions and waste, and are in large part the reason people think risk metrics are crap.
As for scarce wins, I’d argue it’s because much of what’s being used as “risk” methods in our profession is badly broken. So of course it’s scarce. If you’d like, I can put you in touch with people who have had remarkable success.
I assume, Mike, that you support the notion that well-informed decisions are preferable to poorly-informed decisions. Yes? Well, unless I’m mistaken, you haven’t been trained in FAIR (and I’m just citing that because it’s what I’m most qualified to talk about), which means you aren’t qualified to judge it. Of course, I don’t know how much familiarity you have with other methods out there. As a consequence, at least as it relates to FAIR, it seems like you’re spouting… the same stuff you claim everything else is.
Fine Mike, I won’t disappoint.
“BTW, I’m not taking a dump on all aspects of quantification. I’ve always been a big fan of security (as opposed to risk) metrics.”
In the end, the only difference between a “risk” metric and a “security” metric is the fact that there are loss distributions involved. And those loss distributions are actually the EASIEST ones to develop in terms of meaning.
So ultimately, I guess your head is so far up there that you might actually be seeing daylight.