Oh goodie- another religious security debate! We do love our religious arguments so.
This time it’s Amrit taking on Rothman over security metrics. Amrit likes them, Rothman doesn’t. Both of them are funny looking (sorry, it’s not germane to this post, but I figure people should know).
I’m with Amrit on this one- metrics are absolutely critical. But I also agree with Mike, the wrong metrics are worse than no metrics, and pretending everything can be measured is silly. Didn’t we get over that in college?
Amrit and Mike are both right; and despite my attempt to jump the shark and make this sound controversial they both probably agree more than they disagree.
Security metrics are a vital evolution of our industry. We’re not artists, as much as there is an art to our science. We can’t just sit around and tell management to trust us and “no… don’t worry… we’re doing a good job. No viruses this week, right?” By the same token we can’t pretend everything we do can devolve into some simple ROI model to tell the CFO how many people to hire and how many security widgets to buy.
Metrics are a valuable tool to baseline activities and track results. Metrics should help us measure both our activities and the results. Results beyond the number of incidents. Metrics also bring maturity to a discipline by, among other things, allowing that profession to communicate to the outside world. As a paramedic I might have claimed that my only metric was dropping off live bodies (preferably at a hospital), but in reality we tracked dozens of metrics- from response times, to procedural successes, to long term patient outcomes (just keeping you alive to the front door doesn’t always mean you go home).
We need security metrics to:
- Baseline activities and investments
- Track those over time for deviation
- Correlate activities and investments to results
- Optimize to maximize results and minimize waste
- Communicate all of this to external parties
CISOs that can measure and demonstrate program efficiency can more easily obtain budget for necessary improvements. It’s a combination of building trust, and being able to justify new efforts. Metrics should also include qualitative measurements. No Virginia, we can’t measure everything with real numbers, that’s why amps go to 10 (or 11). But if we use consistent qualitative models, we can gain quantitative benefits by still tracking results over time.
Saying, “give us money and you won’t get hacked” won’t help you get money, ensures you lose it when you get hacked (and you will), and doesn’t help you look like a professional.
On the other hand we can’t make up fake ROI models just to keep the CFO happy (one of my biggest pet peeves). You don’t do yourself any favors in the long term if you send off imaginary numbers every time someone asks for the impossible.
Use real metrics. Mix quantitative and structured qualitative. Track yourself over time, correlate results, and use them to optimize efficiency (ooh- I sound like one of those professional speaker types!). Give honest answers to honest questions, and when someone asks for the ROI of a firewall ask them for the ROI on their desk.
Reader interactions
3 Replies to “No Metrics, No Budget (or Paycheck)”
Mogull over at Securosis tries to play mediator by pointing out that the two sides probably agree more than they disagree about. Basically, the argument is that metrics are important, but only the good ones. Rothman isn’t
To follow up on metrics, Amrit pointed out in the comments that we can’t use totally imaginary numbers.
Funny looking? I’‘m taking you off my blogroll, but I doubt your readership of 600 will be negatively impacted by the 3 people who read mine.
You can argue that fake ROI models or useless metrics can be worse than no metrics at all, especially if fake data is used to make decisions – I remember you making this argument before and I completely agree with it.