The lack of credible and relevant network security metrics has been a thorn in my side for years. We don’t know how to define success. We don’t know how to communicate value. And ultimately, we don’t even know what we should be tracking operationally to show improvement (or failure) in our network security activities.

But we in the echo chamber seem to be happier bitching about this, or flaming each other on mailing lists, than focusing on finding a solution. Some folks have tried to drive towards a set of metrics that make sense, but I can say most of the attempts are way too academic and also cost too much to collect to be usable in everyday practice. Not to mention that most of our daily activities aren’t even included in the models.

Not to pick on them too much, but I think these issues are highlighted in the way the Center for Internet Security has scoped out network security metrics. Basically, they didn’t. They have metrics on Incident Management, Vulnerability Management, Patch Management, Configuration Change Management, Application Security, and Financial Metrics. So the guy managing the network security devices doesn’t count? Again, I know CIS is working towards a lot of other stuff, but the reality is the majority of security spending is targeted at the network and endpoint domains, and there are no good metrics for those.

So let’s fix it.

Today, we are kicking off the next in our series of Quant projects. This one is called Network Security Operations Quant, and we aim to build a process map and underlying cost model for how organizations manage their network security devices.

The project’s formal objective and scope are:

The objective of Network Security Operations Quant is to develop a cost model for monitoring and managing network security devices that accurately reflects the associated financial and resource costs.

Secondarily, we also want to:

  • Build the model in a manner that supports use as an operational efficiency model to help organizations optimize their network security monitoring and management processes, and compare costs of different options.
  • Heavily engage the community and produce an open model with wide support and credibility, using the Totally Transparent Research process.
  • Advance the state of IT metrics, particularly operational security metrics.

We are grateful to our friends at SecureWorks, who are funding this primary research effort.

As with all our quant processes, our methodology is:

  1. Establish the high level process map via our own research.
  2. Use a broad survey to validate and identify gaps in the process map.
  3. Define a set of subprocesses for each high-level process.
  4. Build metrics for each subprocess.
  5. Assemble the metrics into a model which can be used to track operational improvement.

From a scoping standpoint, we are going to deal with 5 different network security processes:

  1. Monitoring firewalls
  2. Monitoring IDS/IPS
  3. Monitoring server devices
  4. Managing firewalls
  5. Managing IDS/IPS

Yes, we know network security is bigger than just these 5 functions, but we can’t boil the ocean. There is a lot of other stuff we’ll model out using the Quant process over the next year, but this should be a good start.

Put up or shut up

We can’t do this alone. So we are asking for your help. First off, we are going to put together a “panel” of organizations to serve as the basis for our initial primary research. That means we’ll be either doing site visits or detailed phone interviews to understand how you undertake network security processes. We’ll also need the folks on the panel to shoot holes in our process maps before they are posted for public feedback. We are looking for about a dozen organizations from a number of different verticals and company sizes (large enterprise to mid-market).

As with all our research, there will be no direct attribution to your organization. We are happy to sign NDAs and the like. If you are interested in participating, please send me an email directly at mrothman (at) securosis . com.

Once the initial process maps are posted, we will post a survey to find out whether you actually do the steps we identify. We’ll also want your feedback on the process via posts that describe each step in the process. Everyone has an opportunity to participate and we hope you will take us up on it.

This is possibly the coolest research project I’ve personally been involved with and I’m really excited to get moving on it. We look forward to your participation, so we finally can get on the same page, and figure out how to measure how we “network security plumbers” do our business.

Share: