Patch Management

Join the Open Patch Management Survey—Project Quant

By Rich

Are you tired of all those BS vendor surveys designed to sell products, and they don't ever even show you the raw data?

Yeah, us too.

Today we're taking the next big step for Project Quant by launching an open survey on patch management. Our goal here is to gain an understanding of what people are really doing with regards to patch management, to better align the metrics model with real practices.

We're doing something different with this survey. All the results will be made public. We don't mean the summary results, but the raw data (minus any private or identifiable information that could reveal the source person or organization). Once we hit 100 responses we will release the data in spreadsheet formats. Then, either every week or for every 100 additional responses, we will release updated data. We don't plan on closing this for quite some time, but as with most surveys we expect an initial rush of responses and want to get the data out there quickly. As with all our material, the results will be licensed under Creative Commons.

We will, of course, provide our own analysis, but we think it's important for everyone to be able to evaluate the results for themselves.

All questions are optional, but the more you complete the more accurate the results will be. In two spots we ask if you are open for a direct interview, which we will start scheduling right away. Please spread the word far and wide, since the more responses we collect, the more useful the results.

If you fill out the survey as a result of reading this post please use SECUROSISBLOG as the registration code (helps us figure out what channels are working best). If you came to this post via twitter, use TWITTER as the reg code. This won't affect the results, but we think it might be interesting to track how people found the survey, and which social media channels are more effective.

Thanks for participating, and click here to fill it out.

(This is a SurveyMonkey survey, so we can't disable the JavaScript like we do for everything here on the main site. Sorry).

–Rich

The CIS Consensus Metrics and Project Quant

By Rich

Just before release, the Center for Internet Security sent us a preview copy of the CIS Consensus Metrics. I'm a longtime fan of the Center, and, once I heard they were starting on this project, was looking forward to the results.

Overall I think they did a solid job on a difficult problem. Is it perfect? Is it complete? No, but it's a heck of a good start. There are a couple things that stand out:

• They do a great job of interconnecting different metrics, and showing you how you can leverage a single collected data attribute across multiple higher-level metrics. For example, a single "technology" (product/version) is used in multiple places, for multiple metrics. It's clear they've designed this to support a high degree of automation across multiple workflows, supporting technologies, and operational teams.
• I like how they break out data attributes from security metrics. Attributes are the feeder data sets we use to create the security metrics. I've seen other systems that intermix the data with the metrics, creating confusion.
• Their selected metrics are a reasonable starting point for characterizing a security program. They don't cover everything, but that makes it more likely you can collect them in the first place. They make it clear this is a start, with more metrics coming down the road.
• The metrics are broken out by business function -- this version covering incident management, vulnerability management, patch management, application security, configuration management, and financial.
• The metric descriptions are clear and concise, and show the logic behind them. This makes it easy to build your own moving forward.

There are a few things that could also be improved:

• The data attributes are exhaustive. Without automated tool support, they will be very difficult to collect.
• The document suggests prioritization, but doesn't provide any guidance. A companion paper would be nice.

This isn't a mind-bending document, and we've seen many of these metrics before, but not usually organized together, freely available, well documented, or from a respected third party. I highly recommend you go get a copy.

Now on to the CIS Consensus Metrics and Project Quant...

I've had some people asking me if Quant is dead thanks to the CIS metrics. While there's the tiniest bit of overlap, the two projects have different goals, and are totally complementary. The CIS metrics are focused on providing an overview for an entire security program, while Quant is focused on building a detailed operational metrics model for patch management. In terms of value, this should provide:

1. Detailed costs associated with each step of a patch management process, and a model to predict costs associated with operational changes.
2. Measurements of operational efficiency at each step of patch management to identify bottlenecks/inefficiencies and improve the process.
3. Overall efficiency metrics for the entire patch management process.

CIS and Quant overlap for the last goal, but not for the first two. If anything, Quant will be able to feed the CIS metrics. The CIS metrics for patch management include:

• Patch Policy Compliance
• Patch Management Coverage
• Mean Time to Patch

I highly suspect all of these will appear in Quant, but we plan on digging into much greater depth to help the operational folks directly measure and optimize their processes.

–Rich

Project Quant: Draft Survey Questions

By Rich

Hey folks,

While we aren't posting everything related to Project Quant here on the site, I will be putting up some major milestones. One of the biggies is to develop a survey to gain a better understanding of how organizations manage their patching processes.

I just completed my first rough draft of some survey questions over in the forums. The main goal is to understand to what degree people have a formal process, and how their processes are structured.

I consider this very rough and in definite need of some help.

Please pop over to this thread in the forums and let me know what you think.

In particular I'm not sure I've actually captured the right set of questions, based on our priorities for the project (I know survey writing is practically an art form).

Please let us know what you think. Once we lock it down we will use a variety of mechanisms to get the survey out there, and will follow it up with some focused interviews.

–Rich

Project Quant: Patch Management Cycle

By Rich

While we don't plan on posting every Project Quant update here on the main blog, we will be cross-posting some of the more significant project updates, as well as other content we relevant to our broader readership. (For these posts we will turn off comments to consolidate them all in the Project Quant area.)

So here is our first pass at defining a patch management process for the project:

Although we posted some of our initial thoughts, and have been getting some great feedback from everyone, Jeff and I realized that we haven't even defined a standard patch management cycle yet to start from. DS, Dutch, and a few others have started posting some metrics/variables, but we didn't have a process to fit them into.

I've been researching other patch management cycles, and here's my first stab at one for the project. You'll notice it's a little more granular than most of the other ones out there -- I think we need to break out phases in more detail to both match the different processes used by different organizations, and to give us cleaner buckets for our metrics.

Here's a quick outline of the steps:

1. Monitor for Release/Advisory: Anything associated with tracking patch releases, since all vendors follow different processes.
2. Acquire: Get the patch.
3. Evaluate: Initial evaluation of the patch. What's it for? Is it security-sensitive? Do we use that software? Is the issue relevant in our environment? Are there workarounds or dependencies?
4. Prioritize/Schedule: Prioritize based on the nature of the patch itself, and your infrastructure/assets. Then build out a deployment schedule, based on your prioritization.
5. Test and Certify/Accredit: Perform any required testing, and certify the patch for release. This could include any C&A requirements for you government types, compliance requirements, or internal policy requirements.
6. Create Deployment Package: Prepare the patch for deployment.
7. Deploy.
8. Confirm Deployment: Verify that patches were properly deployed. This might include use of configuration management or vulnerability assessment tools.
9. Clean up: Clean up any bad deployments, remnants of the patch application procedure, or other associated cruft/detritus.
10. Document and Update Configuration Standards: Document the patch deployment, which may be required for regulatory compliance, and update any associated configuration standards/guidelines/requirements.

This is a quick and dirty pass and meant to capture the macro-level steps in the process. I know not all organizations follow, or need to follow, a process like this, but it will help us organize our metrics.

Let me know what you think -- I'm sure I'm missing something...

To comment on this post, please see the original over in the Project Quant area.

–Rich