Announcing NetSec Ops Quant: Network Security Metrics Suck. Let’s Fix Them.

By Mike Rothman

The lack of credible and relevant network security metrics has been a thorn in my side for years. We don’t know how to define success. We don’t know how to communicate value. And ultimately, we don’t even know what we should be tracking operationally to show improvement (or failure) in our network security activities.

But we in the echo chamber seem to be happier bitching about this, or flaming each other on mailing lists, than focusing on finding a solution. Some folks have tried to drive towards a set of metrics that make sense, but I can say most of the attempts are way too academic and also cost too much to collect to be usable in everyday practice. Not to mention that most of our daily activities aren’t even included in the models.

Not to pick on them too much, but I think these issues are highlighted in the way the Center for Internet Security has scoped out network security metrics. Basically, they didn’t. They have metrics on Incident Management, Vulnerability Management, Patch Management, Configuration Change Management, Application Security, and Financial Metrics. So the guy managing the network security devices doesn’t count? Again, I know CIS is working towards a lot of other stuff, but the reality is the majority of security spending is targeted at the network and endpoint domains, and there are no good metrics for those.

So let’s fix it.

Today, we are kicking off the next in our series of Quant projects. This one is called Network Security Operations Quant, and we aim to build a process map and underlying cost model for how organizations manage their network security devices.

The project’s formal objective and scope are:

The objective of Network Security Operations Quant is to develop a cost model for monitoring and managing network security devices that accurately reflects the associated financial and resource costs.

Secondarily, we also want to:

  • Build the model in a manner that supports use as an operational efficiency model to help organizations optimize their network security monitoring and management processes, and compare costs of different options.
  • Heavily engage the community and produce an open model with wide support and credibility, using the Totally Transparent Research process.
  • Advance the state of IT metrics, particularly operational security metrics.

We are grateful to our friends at SecureWorks, who are funding this primary research effort.

As with all our quant processes, our methodology is:

  1. Establish the high level process map via our own research.
  2. Use a broad survey to validate and identify gaps in the process map.
  3. Define a set of subprocesses for each high-level process.
  4. Build metrics for each subprocess.
  5. Assemble the metrics into a model which can be used to track operational improvement.

From a scoping standpoint, we are going to deal with 5 different network security processes:

  1. Monitoring firewalls
  2. Monitoring IDS/IPS
  3. Monitoring server devices
  4. Managing firewalls
  5. Managing IDS/IPS

Yes, we know network security is bigger than just these 5 functions, but we can’t boil the ocean. There is a lot of other stuff we’ll model out using the Quant process over the next year, but this should be a good start.

Put up or shut up

We can’t do this alone. So we are asking for your help. First off, we are going to put together a “panel” of organizations to serve as the basis for our initial primary research. That means we’ll be either doing site visits or detailed phone interviews to understand how you undertake network security processes. We’ll also need the folks on the panel to shoot holes in our process maps before they are posted for public feedback. We are looking for about a dozen organizations from a number of different verticals and company sizes (large enterprise to mid-market).

As with all our research, there will be no direct attribution to your organization. We are happy to sign NDAs and the like. If you are interested in participating, please send me an email directly at mrothman (at) securosis . com.

Once the initial process maps are posted, we will post a survey to find out whether you actually do the steps we identify. We’ll also want your feedback on the process via posts that describe each step in the process. Everyone has an opportunity to participate and we hope you will take us up on it.

This is possibly the coolest research project I’ve personally been involved with and I’m really excited to get moving on it. We look forward to your participation, so we finally can get on the same page, and figure out how to measure how we “network security plumbers” do our business.

No Related Posts

One of the biggest challenges for calculating labor associated with delivering networking security services is accurately measuring the amount of “end to end” time spent on a given task.  Many environments are interrupt driven and resources are frequently juggling more than one issue or project at once.  The multi-tasking, toggling of priorities, and overlap of different tasks makings it exceptionally difficult to automate labor tracking.  Leaving it up to the analysts or engineers to gauge the time and effort put into a single task can lead to wide variations of estimates.  There is no magic bullet for accurately tracking labor costs in an interrupt-driven operational environment. You can get a fair idea with the help of automated tracking tools and using estimates given by the engineers performing the work, but each approach has it’s limitations.  Since labor is one of the biggest costs to delivering services, accurate tracking is essential in pricing a service appropriately and enabling profitability in a managed service.

By Mark Matthews

One comment in general with this project, and all processes, is that providing metrics is key to the success of a business, however, it is critical that the right ones are chosen to be built, they are not too labor intensive to maintain, and are confirmed as valuable by the audience receiving them.  Today, many people feel that just having the data available is helpful, however, if they are not using that data to manage their environment, than it is useless! 

What will help throughout this project is to clearly identify what the proper way is to measure the success of an organization.  Without this in tact, a company will not be able to understand what their baseline is and what it will take to improve their organization to get to the next step.  As advice is given to improve processes, metrics will help identify how effective that advice is and what should remain and not be implemented.

By David Collette

Is the book coming out this year?

By Daniel Nunez

I’ve put together a couple documents that kind of relate to this.  One is a security taxonomy and maturity model of security components that we use to gauge the growth and effectiveness of the security program, and the other is a security capability model for a different view from the process side, but I’m not satified that I’ve captured the information we need.  Ideally such a model will answer questions like, “What processes are targets for out sourcing?”, “Who does what?”, “How mature are our processes?”, “How do we do these security things?”. I’ll be interested in following your progress.

By Steve Watne

If you like to leave comments, and aren’t a spammer, register for the site and email us at and we’ll turn off moderation for your account.