As we described in the Introduction to Implementing and Managing Patch and Configuration Management, endpoint hygiene is key to endpoint security management. WIth the product (or service) in hand, it’s time to get the technology implemented and providing value as quickly as possible. You know the old saying, “if you fail to prepare, you prepare to fail.” It’s actually true, and the preparation in this situation involves ensuring your processes are solid, defining device coverage and roll-out priorities, figuring out what’s already out there, and finally going through a testing phase to make sure you are ready to deploy widely. So, let’s revisit the patch and configuration management processes.

Determine Processes

We are process centric at Securosis. We admit it, but only because we understand the folly of trying to implement and manage technology without proper processes and accountabilities defined before products get installed. So we start most activities with a check to ensure the process supports the problem to be solved. With patch and configuration management, you are looking at two distinct but tightly intertwined processes.

To be clear, you don’t have to do all the functions described below. Figure out which will work for your organization. But you do need to make sure everyone understands what they are supposed to do – especially when it comes to remediation. If the operations team is expected to run through the patch process, open up the maintenance windows, and confirm the successful implementation of each patch, they need to know that. Likewise, if the incident response team needs to investigate strange configuration changes found during assessment, the handoffs must be clearly defined, as well as your ability to remediate a device under investigation.

Patch Management

1. Discover and define targets: Before you jump into the Patch Management process you need to define which devices will be included. Is it just endpoints, or do you also need to patch servers? These days you also need to think about cloud instances. The technology is largely the same, but increased numbers of devices make execution more challenging.
2. Obtain patches: You need to monitor for release of relevant patches, and then figure out whether you need each patch, or you can work around the issue.
3. Prepare to patch: Once the patch is obtained you need to figure out how critical the issue is. Is it something you need to fix right now? Can it wait for the next maintenance window? Once priority is established give the patch a final Q/A check to ensure it won’t break anything important.
4. Deploy the patch: Once preparation is complete and your window has arrived, you can install.
5. Confirm the patch: Patches don’t help if the install fails, so confirm that each patch is fully installed.
6. Reporting: Compliance requirements for timely patching make reporting an integral function.

Obviously this is a very high-level process description. If you want a much more granular process map for patch management, with metrics and cost models, check out Patch Management Quant.

Configuration Management

1. Establish configuration baselines and/or benchmarks: First define acceptable secure configurations for each managed device type. Many organizations start with benchmarks from CIS or NIST (PDF) for granular guidance on how devices should be configured.
2. Discover and define targets: Next find the devices that need to be managed. Ideally you can leverage an endpoint security management platform with an integrated asset management repository. You will also want to categorize and group assets to avoid unnecessary services. Engineering workstations, for example, require different configurations than Finance systems.
3. Assess, alert, and report changes: Once devices are discovered and categorized, define a frequency for assessments. How often will you check them against policy? Some vendors use the term “continuous assessment”, but their assessments aren’t really continuous. Fortunately this isn’t normally a problem – not least because most operational groups wouldn’t be able to validate alerts and correct issues in real time anyway.
4. Remediate: Once a problem is identified, either it needs to be fixed or someone needs to grant an exception. You are likely to have too much work to handle it all immediately so prioritization is key. We offered some perspective on prioritization for vulnerability management, but the concepts are the same for configuration management. You will also probably need to verify that changes actually took place for the audit, as well as plan for a roll-back in case the change breaks something.

Define Initial Priorities/Targets and Deployment Model

After gaining consensus on the applicable processes and ensuring everyone knows their roles and responsibilities, it’s time to determine your initial priorities/targets to figure out whether you will start with the Quick Wins process or jump right into Full Deployment. Most organizations have at least a vague sense of what types of devices they need to patch and manage, but translating that into deployment priorities can be tricky. Let’s highlight some of the categories of things you can manage, which should help you figure out the best direction.

• Servers: (OS) Keeping server devices updated is essential for protecting them. Look to group servers logically based on function, so you can identify typical configurations and applicable patch windows for each class of device. Also factor in whether you are dealing with physical servers, private cloud instances, or public cloud instances, because managing each type differs dramatically.
• PCs: (OS) Though non-server PCs are rarely the ultimate target of an attack, they provide a way for attackers to gain a foothold within your organization, so they can jump laterally to attack servers. Group PCs logically based on job function and need for access to critical data stores. Keep in mind that laptops create unique problems for to patch and configuration management because they may connect to the network infrequently, so consider whether you want to tackle that as part of the initial deployment.
• Mobile devices: (OS) Quicker than you can say BYOD, you will need to more effectively manage the mobile devices (including smartphones) that access your network. The smartphone vendors provide utilities to update and enforce configuration policies on their devices, but in heterogeneous environments it is good to provide consistent patch and configuration management across all devices. It may be constrained by organizational structure, particularly if a different group is responsible for the mobile devices and pushes to use their own purpose-built tool.
• Applications: Finally, as applications have emerged as the path of least resistance for many attacks, the need to keep commercial applications patched and properly configured becomes more important. The good news is that patch and configuration management platforms handle applications and operating systems similarly, so supporting applications is not likely to require massive technology changes. But you will again need to decide whether application patching is something to deal with in your initial deployment.

You should now have a sense of what devices to focus on and where to start. The next step is to pick a deployment process. Here are some suggestions for deciding which to start with. The easy answer is almost always the same: start with the Quick Wins process. The only common exception is when you have already prioritized what to manage, have a good sense of where you need to manage it, and believe you understand the scope you need to tackle – then you might be able to jump directly to Full Deployment. This tends to come up when you have a specific compliance deficiency or have tracked back a breach to poor device hygiene, because going all in on patch and configuration management will solve the problem. Otherwise we suggest starting with Quick Wins to highlight what works and what doesn’t, and to help figure out where to focus your full deployment.

Initial discovery

Both the processes described above start with a discovery phase to figure out what’s actually out there. We suggest you kickstart the effort by mining an existing asset management repository – perhaps a CMDB, enterprise directory, or other such data store which lists devices. Of course your asset management function can only provide you with detail on devices you already know about, so it is also important to perform an active scan of applicable IP address ranges to discover what’s really out there. You are likely to be surprised when you compare reports against reality.

Do yourself a favor and be sure to coordinate with the operations team to avoid disrupting production systems. And note that a segmented network architecture (to segregate servers within PCI scope, for instance) will require access to the protected segments for scanning. If you can scan your entire environment from a single location without consulting the operations team, you have bigger problems than patch management, so maybe you need to address those first.

Test and Proof of Concept

We talked a bit about this in the Endpoint Security Management Buyer’s Guide, because a Proof of Concept (PoC) is typically part of the procurement process one way or another. But let’s distinguish between the kind of testing you do before you buy, and the testing you need to do before you implement. During the selection process you focus on user experience, deployment architecture, and device & application coverage – keeping in mind your long-term endpoint security management needs.

The testing before you deploy is all about figuring out what breaks when you implement. Don’t decide to patch all your Windows devices, and then find out that XP isn’t supported as well as it should be. Or that patching Adobe Reader isn’t as reliable as it should be. Make sure you don’t create more problems than you solve. You need to make sure your initial set of supported platforms and apps really do work reliably and save time.

As part of this limited deployment, look for a representative sample of the devices you decided above to support as the implementation pilot. Pick this sample from folks who won’t react too poorly if things don’t work perfectly. IT staffers are usually good guinea pigs, as are more technically sophisticated employees.

The testing process also provides a good mechanism to train staff on the tools – especially whoever wasn’t involved in the proof of concept. Remember, they will have to both run the tool and remediate the issues it uncovers. So having them run through the process during testing will pay dividends when the live ammo is flying.

Further along in this series we will discuss the ongoing management requirements for both patch and configuration, and then we’ll dig into things like monitoring for patches and defining configuration settings for certain device types. For now you need to focus the test deployment on ensuring that you can perform the functions required on the devices you’ve chosen without adversely impacting anything. Assuming testing goes swimmingly, next you will plan out and execute your deployment – the topic of our next post.