The Securosis Nexus

For discussion of the patch management process/cycle phases.

(Had accidentally left this as a comment before noting that it was requested to place feedback here… sorry for the multi-post)

I would suggest re-ordering the process as follows:

1. Monitor
2. Evaluate
3. Acquire: You won’t get a patch if it isn’t relevant as determined by the “Evaluate” process
4. Create Deployment Package:  Likely done as part of, or before, “Test/Certify”, since the deployment package would need testing just as the actual patch would.
5. Test/Certify: Is it disruptive or not?  This will have a lot of impact on schedule.
6. Schedule
7. Deploy
8. Clean up: all installs should be done before validating, else validation would not be as efficient as possible.
9. Confirm Deployment

Thinking about it, Deploy->Clean up->Confirm is really a sub-cycle in the process, since unless you have a 100% first deployment, you will find missed hosts in the confirm phase which need to be re-run through deploy, some of which may require assistance, and so on.

That’s a good idea- I like the evaluate/acquire swap. Reduces time in the process.

I’m starting to think the test/create package structure might vary based on the organization, and they are clearly tightly coupled.

Can we get feedback from any of you end users on how you order your process?

I like the points DS mentioned. Posting some thoughts based on his -

—
1. Monitor
2. Evaluate
3. Acquire: You won’t get a patch if it isn’t relevant as determined by the “Evaluate” process
4. Create Deployment Package:  Likely done as part of, or before, “Test/Certify”, since the deployment package would need testing just as the actual patch would.
5. Test/Certify: Is it disruptive or not?  This will have a lot of impact on schedule.
6. Schedule
7. Deploy
8. Clean up: all installs should be done before validating, else validation would not be as efficient as possible.
9. Confirm Deployment
—

Step (3) might be more a sub point to (2). Depending on the patch management solution ‘acquire’ might occur automatically depending on the vulnerability rating the VM vendor files the package under. I think the actual evaluation (sub) process is the important part due to the many factors need consideration.

The prioritizing process might occur before the actual testing cycle to ensure resources are spend on the high profile/risk vulnerabilities before anything else. I would tend to group Prioritize/Evaluate and Schedule/Deploy i think.

I agree that ‘Test/Certify’ and ’ Create package’ are closely related. I would assume that the native patch will be tested as well as the wrapped up deployment package to ensure neither is causing trouble.

On ‘Confirm Deployment’ i would add to verify not only the deployment was successful but also that the service is actually still working as expected. Could argue that the testing phase should take care of this, but i think its good practice to contact the asset/service owners to verify their service is still performing/delivering as expected.

Daniel

I like the start. How does everyone like the starting framework? I’m a little more ‘task oriented’, so my list looks a little more like:
1) Receive notification
<Comment>  How are organizations notified that patches needed to be installed? Are system owners relying on the communication coming from the vendor?  Is there a 3rd party notification (outside companies or perhaps a need is identified internally for feature patches)
2) Evaluate impact (necessity of patching)
<Comment> Is the patch a security update? Is the exposure being fixed critical to the confidentiality, integrity and availability of the systems being patched? Is it a feature update (Sub question: Is this feature needed?).
3) Determine which systems need to be patched
<Comment>  Pretty self explanatory, but a required step when building a deployment strategy.
4) Acquire Patch
<Comment>  Of course.
5) Perform Regression testing
<Comment>  Does the patch fix one thing and break another? Again, pretty self explanatory
6) Schedule
<Comment> If step 2 determines the patch is critical, it might be scheduled sooner than later. Does the patch need user intervention (and required to be scheduled during working hours)? Is there a policy or contractual requirement that dictates how patches are pushed?
7) Deploy
<Comment>  Git ‘r done ;)
8) Verify deployment and redeploy as needed
<Comment> What are the different verification processes? Perhaps a vulnerability scan for security patches. Polling users for feature patching. More needed.
9) Identify ‘Real World’ issues and fix as issues come up.
<Comment>  What is the help desk getting called about with respect to the patch installed? What is the process for fixing? Redo Steps 5 - 8?

I got some private feedback that this was both a little too security oriented, and a bit too Windows-centric. Since I’m a security guy, and I used to admin Windows systems back in my operational days, I can only blame myself.

How do you think we need to change this to account for servers/non-windows/non security patches? I still think switching around some of the opening steps might help, but what else?

Good feedback. Perhaps capturing the different patches and systems and building the process around the different types?
Patches:
Security
Features Upgrade
Bug fixing
Systems:
Servers
...Windows
...<Your favorite unix>
...Mainframe
Appliances
...Firewalls
...Printers
...Switches
...Routers
...Multi-function devices (like those Xerox all in ones)
...Non-IP based devices (modems, etc)
Workstations
...Windows
...Mac
...<Your favorite unix>
...Isolated systems like Kiosks

Not feeling so creative at the moment, what did I miss?

I’m kind of worried that would overly complicate the model. I do think we may end up with some different metrics for different types, but I’d *way* prefer to run off a single base cycle people can try and plug into.

For instance, what are the real differences between a Unix and MS update? Not the mechanical bits so much as in what ways do they need a different process?

I just got some email feedback from Amrit Williams over at BigFix and he gave me permission to post it here:

At a quick glance here are a couple of comments:

Not all patches are security related and this is clearly designed for a “security” patch management program. You should note the focus or include references to how an org would handle non-security patches, for example they would evaluate before they acquire since they wouldn’t acquire all patches (this may also apply to security patch situations as I mention below) or how many orgs will test before they can prioritize or schedule as they will need to understand the operational impact of a patch

Patching Windows can be radically different from patching Unix and from patching virtual environments to patching databases. I assume this is fairly generic but is clearly has a “Windows Desktop Security Patch Management” focus In many non-window security patch management programs they look for the patch bundles as opposed to trying to quickly patch against all conditions, they also have to deal with dependency resolution issues which is a huge problem for IT dealing with hetergeneous *nix platforms

You miss a very important and key step between the monitor for release/advisory and deployment of the patch and that is the absolutely critical need to determine shielding mechanisms that may be used to quickly shield the environment from exploit prior to patch deployment. For example it may be less impactful and faster to disable RPC over certain ports than to deploy an RPC patch, it may be faster block specific protocol/port ingress traffic to certain parts of the network or machines of certain types as one rolls out the patch. I think this is really important.

There is so much room for detail and metrics here, how does one evaluate impact, how long should it take, what are considered standard SLA’s, etc…

I’m not sure i read anything in the feedback so far that would not fit into the general framework created by Rich or DS.
From a high level view does it really matter if the patch is a critical security fix or application update for Unix? Sure, the patches would be treated differently but the phases it goes through would still fit into the cycle discussed i think.
I agree with Amrit that every phase potentially has many tasks attached to it which will impact the effectiveness and overall cost of the cycle for a certain fix/patch. At this point are you looking for a detailed list of steps involved per phase or more a general indication of the time/resources?

I have to agree with Daniel, above, and also wonder what the person who said this is security/windows specific was thinking. 

My background is in AIX and Solaris as much as it is Windows, and I see nothing in a generic process that is platform specific.  Fine, Oracle bundles patches with generally infrequent release, so does that mean they don’t have to watched for, evaluated, tested and etc?  Bah. 

As for amrit’s feedback about evaluating mitigating actions, that is not (IMO) part of patch management, but instead part of vulnerability management.  The two are related but not the same.  If this is a patch management process, it has one focus: to deploy patches.

@Daniel,

Thanks- at this point I want to lock in the high-level process, then the plan is to break each step out and start producing individual metrics for each phase. I’m thinking we’ll start with a massive list of anything that could be possibly be measured in each phase, then prioritize, organize, and optimize them.

Actually, you bring up a good point- we’ll probably need a more detailed list of steps, then the metrics associated with each step. That’s probably a better way to organize things than how I was originally thinking.

Thanks DS,

I agree- shielding is definitely out of scope for this project, and thanks for helping cover the non-Windows side. I suspect some of those differences will pop up more once we get into the more detailed steps.

And it’s nice to know my background isn’t overly biasing things. While I’ve managed some servers, by far the bulk of my experience as a working sysadmin was on desktops.

Okay- I incorporated everyone’s feedback and adjusted the cycle. The changes are:

1. Swapped evaluate/acquire.
2. Added testing to Create Deployment Package to account for initial testing, then testing of the actual package.
3. Added an arrow to show the deploy-confirm-clean up sub-cycle. In the confirm deployment phase we can include some measure of clean up, but I tend to think that I send out my deployment packages, test to see they installed correctly, then clean up any problems.

What does everyone think?

Just to expound on the issue of servers v. workstations, I think where the differences will come into play will be in the component steps in each phase.  For example, one may use a more rigorous testing process with servers such as full regression testing in a QA lab for a major application server, while a desktop patch may be tested by deploying to IT users ahead of a more general deployment.  Or, desktops may be set to auto-update, while servers may be patched manually during a maintenance window. 

Anyway, I think this process looks very nice and look forward to the next level of detail.

That circular image fits with our patch process.

I like the inclusion of the “Confirm Deployment” piece.

This might be beyond the scope or contained within one of the existing boxes, but if there is anything missing, it would be some way to track or compartmentalize those systems you can’t patch either because the patch disrupts service or the devices aren’t managed the same (i.e. Cisco phone servers on Windows). You run through this cycle, but every now and then a cast-off gets thrown into that bucket.

Also, I can see merit in defining whether this is a security patch cycle or other. While my point of view is that they should be treated the same, there may be some who think a full system version upgrade is similar to a patch. Or that optional functionality (.NET 3.5 in Windows for instance) is part of that. Then again, would it just simply stop at Evaluate anyway? Is it a “patch process” to upgrade from XP to Vista? I’m playing a little bit of a Devil’s Advocate here. :)

Servers:
I’m not dedicated to just security in my company, so we try to make the patching process as simple as possible. Patch anything and everything we can.

We get notification on new patches through our vuln assessment tools and I’ll do some of my own evaluation as well just to point out any extremely bad patches (wormable, remote…) or possible conflicts that might influence our adherence to timelines and testing focus. But honestly, patches should rarely impact us, so we don’t like to spend much time on this. Roll out patches to a test group, and within a week patch all the rest that we manage.

Special Servers/Applications (i.e. SQL):
Again, this fits into your patch wheel, but typically can get hung up on the testing part. Seems SQL patches are perceived as scary! Our DBA tends to dictate when such patching will occur.

Desktops:
The testing on desktop patching is a litle more drawn out, especially with all the end-user app patches these days. This is not necessarily my realm, so that’s all I know.

Network devices:
Network device patching is far less frequent and slower. We typically only patch if we need new functionality or we see some security fixes that are really important. The reason for less patching is these usually incur network outages, are more manual, and tend to be riskier with full OS/IOS replacements. In short: bigger operational impact and risk, based in experience.