The FireStarter is something new we are starting here on the blog. The idea is to toss something controversial out into the echo chamber first thing Monday morning, and let people bang on some of our more abstract or non-intuitive research ideas.

For our inaugural entry, I’m going to take on one of my favorite topics – risk management.

There seem to be few topics that engender as much endless – almost religious – debate as risk management in general, and risk management frameworks in particular. We all have our favorite pets, and clearly mine is better than yours. Rather than debating the merits of one framework over the other, I propose a way to evaluate the value of risk frameworks and risk management programs:

  1. Any risk management framework is only as valuable as the degree to which losses experienced by the organization were accurately predicted by the risk assessments.
  2. A risk management program is only as valuable as the degree to which its loss events can be compared to risk assessments.

Pretty simple – all organizations experience losses, no matter how good their security and risk management. Your risk framework should accurately model those losses you do experience; if it doesn’t, you’re just making sh&% up. Note this doesn’t have to be quantitative (which some of you will argue anyway). Qualitative assessments can still be compared, but you have to test.

As for your program, if you can’t compare the results to the predictions, you have no way of knowing if your program works.

Here’s the ruler – time to whip ‘em out…