An Open Metrics Model for Database Security: Project Quant for Databases

One of the more vexing problems in information security is our lack of metrics models for measuring and optimizing security efforts. We tend to lack frameworks and metrics to help measure the efficiency and effectiveness of security programs. This makes it more difficult both to improve our processes, and to communicate our value to non-technical decision makers.

I'm not saying we don't have any metrics. In recent years we've come a long way, with developments such as the Center for Internet Security's Consensus Metrics and the work of Andrew Jaquith and the Security Metrics community. For the most part these metrics fall into two broad categories: program metrics, and risk/threat models.

One area that has been generally lacking -- not to take anything away from the other two categories -- is detailed process-oriented models for improving efficiency and effectiveness within specific security areas. In other words, instead of just determining whether a particular process is an overall improvement, such as by measuring time to patch managed systems (efficiency) or percentage of overall systems patched (effectiveness), we lack tools for examining the individual steps within the process for finer-grained changes. Such detailed measurements can help us figure out how much it costs to patch, identify where and why our patching might be slower than desired (and thus how to make it faster), and determine why certain systems fall between the gaps and aren't patched. Our higher-level models help us evaluate risk and overall security programs, while detailed metrics would be useful for performance optimization.

Our first attempt at building a security performance optimization model focused on patch management, and we called it Project Quant. Over about 6 months we built a standard process framework for patch management, with heavy community participation, and then identified a series of detailed metrics for each step in the process. We ended up with about 40 steps in 10 min phases, with well over 100 potential metrics, prioritized so you can focus on few key areas because few people have the resources to measure them all.

About a month ago we were approached by a database security vendor to see if we could do the same thing for database security. This vendor, Application Security Inc., wanted an open, public, objective framework to measure the potential costs associated with database security. As with the initial Project Quant, which was sponsored by Microsoft, we agreed to proceed with the project as long as we could maintain our Totally Transparent Research policy. In other words, all the work has to be done in public, and the sponsor must participate through the same public mechanisms (comments and forum posts) as anyone else.

This project aligns very well with our research coverage, and we've been looking for an excuse to build out more-detailed database security process models for some time now. We also realized the format we used for Project Quant works well for other process-based metrics models. Thus we're proud to introduce Project Quant for Database Security, and we will now refer to the initial project as Project Quant for Patch Management.

Based on what we have learned to date in Project Quant, this is how the project will proceed:

1. We will, with community involvement, build out a high-level process framework for database security (see the patch management cycle for an example).
2. Once the high level process looks good, we will build out detailed steps for each phase of the higher-level process, and solicit public feedback and involvement.
3. We will build out sub-phase processes that help define tasks, and identify metrics for each step. Metrics will be hard costs in dollars (hardware/software), or time to complete the step. In some cases we will also include some effectiveness metrics (e.g., success vs. failure rates), but the primary focus of the model is costs/efficiency.
4. We will classify the metrics by importance and identify key metrics. We learned in the first Project Quant that it's easy to identify a large number of potential metrics, but most people need only focus on a few that they can measure with a reasonable investment -- once again, some metrics are expensive enough to measure that they would be a poor investment for some (or even most) organizations.
5. Where possible, we will support the research with open surveys and interviews.
6. Absolutely all the research will be conducted out in the open to maintain objectivity. All public comments will be retained as part of the project record, and no comments will be filtered except for spam and off-topic content. The sponsor is only allowed to participate through the same public mechanisms, so their financial involvement can't influence the result. (As with all our contracts, the sponsor doesn't have to pay if the result doesn't meet their needs due to our objectivity requirements).
7. Anyone can participate -- other security vendors, database and security professionals, database vendors, or anyone with too much time on their hands. If you work for a database or database security vendor, we ask that you disclose the company you are with.
8. All materials will be released under a Creative Commons license.

Since database security is more diverse than patch management, we expect to identify multiple sub-processes as part of an overall program. For example, assessment and monitoring aren't necessarily part of a contiguous cycle like most of the phases of patch management. Because this scope is also wider, we don't plan on delving into the same level of detail on the metrics as we did with patch management. To be honest, we probably went too deep, and included far more metrics than anyone could reasonably collect using current technologies.

In terms of timeline we are shooting to complete this project around the end of January or early February.

So let us know what you think. We'll start posting initial thoughts on the process model tomorrow, and start cranking through it from there. We'll keep all material in the Project Quant site, and will update the Research Library to reflect that we're now expanding Quant into other security areas. You can find a complete Table of Contents in the Process Framework post.

Thanks,

—Rich