Security Controls vs. Outcomes

By Rich

One of the more difficult aspects of medical research is correlating treatments/actions with outcomes. This is a core principle of science based medicine (if you’ve never worked in the medical field, you might be shocked at the lack of science at the practitioner level).

When performing medical studies the results aren’t always clean cut. There are practical and ethical limits to how certain studies can be performed, and organisms like people are so complex, living in an uncontrolled environment, that results are rarely cut and dried. Three categories of studies are:

  • Pre-clinical/biological: lab research on cells, animals, or other subsystems to test the basic science. For example, exposing a single cell to a drug to assess the response.
  • Experimental/clinical: a broad classification for studies where treatments are tested on patients with control groups, specific monitoring criteria, and attempts to control and monitor for environmental effects. The classic double blind study is an example.
  • Observational studies: observing, without testing specific treatments. For example, observational studies show that autism rates have not increased over time by measuring autism rates of different age groups using a single diagnostic criteria. With rates holding steady at 1% for all living age groups, the conclusion is that while there is a perception of increasing autism, at most it’s an increase in diagnosis rates, likely due to greater awareness and testing for autism.

No single class of study is typically definitive, so much of medicine is based on correlating multiple studies to draw conclusions. A drug that works in the lab might not work in a clinical study, or one showing positive results in a clinical study might fail to show desired long-term outcomes.

For example, the press was recently full of stories that the latest research showed little to no improvement in long-term patent outcomes due to routine mammograms for patients without risk factors before the age of 50. When studies focus on the effectiveness of mammograms detecting early tumors, they show positive results. But these results do not correlate with improvements in long-term patient outcomes.

Touchy stuff, but there are many studies all over medicine and other areas of science where positive research results don’t necessarily correlate with positive outcomes.

We face the same situation with security, and the recent debate over password rotation highlights (see a post here at Securosis, Russell Thomas’s more-detailed analysis, and Pete Lindstrom’s take).

Read through the comments and you will see that we have good tools to measure how easy or hard it is to crack a password based on how it was encrypted/hashed, length, use of dictionary words, and so on, but none of those necessarily predict or correlate with outcomes. None of that research answers the question, “How often does 90 day password rotation prevent an incident, or in what percentage of incidents did lack of password rotation lead to exploitation?” Technically, even those questions don’t relate to outcomes, since we aren’t assessing the damage associated with the exploitation (due to the lack of password rotation), which is what we’d all really like to know.

When evaluating security, I think wherever possible we should focus on correlating, to the best of our ability, security controls with outcomes. Studies like the Verizon Data Breach Report are starting to improve our ability to draw these conclusions and make more informed risk assessments.

This isn’t one of those “you’re doing it wrong” posts. I believe that we have generally lacked the right data to take this approach, but that’s quickly changing, and we should take full advantage of the opportunity.

No Related Posts

Spaf had a lot of good things to say on the subject recently <> - especially about password change policies and how they, once actually analyzed, generally do not increase security, but do the opposite.

By Marc Farnum Rendino


I think we’re close. I do mean root cause analysis, but also prevention analysis. In what number of incidents do we see X control preventing an event (when we can gather the data), or not preventing an event.

Observational studies- in what percentage of incidents would password rotation have prevented the incident? It’s not completely a predictor, but still useful.

By Rich


By Chris Hayes


I agree (and you can read some old posts)- no way does it all come down to $s. But I think it definitely is in our control to at least correlate a security control with an incident/event, and a qualitative assessment of the nature of that event.

You might agree with that point… let me know if not.

By Rich

@Chris Can you name specific GRC apps “that are incorporating the ability to link loss events to known controls that were compromised as part of incident management”??  I want to follow up on that.

Also, do you know any GRC apps that do *prospective* risk analysis using this information?

By Russell Thomas

Hi Rich. A few thoughts

By Chris Hayes

(I’m getting to like you more and more, too!)

The InfoSec community isn’t alone in facing this challenge.  The Intelligence community faces very similar problems, but they have the advantage of having many more people chewing on it, working for a long time, with very advanced techniques.

Here’s one example: “Formal Methods of Countering Deception and Misperception in Intelligence Analysis”.

They present and critique the “Analysis of Competing Hypothesis” method (similar to the one described in my post), and then augment it with a formal belief logic called “Subjective Logic” to deal with all sorts of uncertainty in the cause-effect relationships between controls and outcomes, and also sources of data, messiness of data, etc.

I may digest this into a tutorial presentation for the spring Mini-Metricon workshop in San Francisco.  Good idea?

By Russell Thomas

If you like to leave comments, and aren’t a spammer, register for the site and email us at and we’ll turn off moderation for your account.