NSO Quant: Take the Survey and Win an iPad

By Mike Rothman

One of the coolest things about how we do research at Securosis is the inclusiveness of our findings. We always post our work to the blog first and let the community have at it. In many cases it gets poked and prodded, ridiculed, and broken down. It's certainly a hard process for the ego, but in the end makes the work better.

So we are now asking for more help as we get into the final phase of our Network Security Operations Quant research. We have already posted the high level process maps and broken those into subprocesses for each of the major areas (Monitoring, Manage Firewall, and Manage IDS/IPS). Now it's time for validation, and that means we need your help.

We are kicking off our NSO Quant survey. As with most of our surveys, we've set this up so you can take it anonymously, and all the raw results (anonymized, in spreadsheet format) will be released after our analysis.

This survey is a bit different in that we are really focused on the processes you take to perform these activities, and a bit on the metrics you use to determine success and improve operationally. We also ask a bit about organizational responsibilities and outsourcing.

Click here to take the survey, and please spread the word. This is a pretty wide-ranging research initiative, so the survey is fairly detailed. We hope you can get through it in 20 minutes or so.

Since we understand filling out surveys is a pain in the behind, we want to provide a bit of incentive, so we'll give a 32gb WiFi iPad to a random participant. You don't need to provide an email address to take the survey, but you do if you want the iPad. If we get a lot of recipients (let's say over 200) we'll cough up for more iPads, so the odds stay better than the lottery.

We are tracking where we get our responses from, so if you take the survey in response to this post, please use "Securosis Blog." If you re-post the link, you can make up your own code and email it to us. We'll let you know how many people responded to your referral. If there is sufficient response, we'll be happy to give you a custom slice of the data.

Thanks again for your help. We'll keep the survey open at least 2 weeks, and then we'll begin analysis.

–Mike Rothman

NSO Quant: Manage IDS/IPS Process Revisited

By Mike Rothman

Now that we've been through all the high-level process steps and associated subprocesses for managing IDS/IPS devices, we thought it would be good to summarize with links to the subprocesses and a more detailed diagram. Note that some names of process steps have changed as the process maps have evolved through the research process.

• NSO Quant: Manage IDS/IPS Process Map (Updated)
• NSO Quant: Manage IDS/IPS -- Policy Review
• NSO Quant: Manage IDS/IPS -- Define/Update Policies & Rules
• NSO Quant: Manage IDS/IPS -- Document Policies & Rules
• NSO Quant: Manage IDS/IPS -- Signature Management
• NSO Quant: Manage IDS/IPS -- Process Change Request
• NSO Quant: Manage IDS/IPS -- Test and Approve
• NSO Quant: Manage IDS/IPS -- Deploy
• NSO Quant: Manage IDS/IPS -- Audit/Validate
• NSO Quant: Manage IDS/IPS -- Monitor Issues/Tune

What's missing? The IDS/IPS health subprocesses. But in reality keeping the devices available, patched, and using adequate hardware is the same regardless of whether you are monitoring or managing firewalls and/or IDS/IPS. So we'll refer back to the health maintenance post in the Monitoring step for those subprocesses. The only minor difference, which doesn't warrant a separate post, is the testing phase -- and as you've seen we are testing the IDS/IPS signatures and rules throughout the change process so this doesn't need to also be included in the device health process.

As with all our research, we appreciate any feedback you have on this process and its subprocesses. It's critical that we get this right because we start developing metrics and building a cost model directly from these steps. So if you see something you don't agree with, or perhaps do a bit differently, let us know.

–Mike Rothman

NSO Quant: Manage IDS/IPS - Monitor Issues/Tune

By Mike Rothman

At long last we come to the end of the subprocesses. We have taken tours of Monitoring and Managing Firewalls, and now we wrap up the Manage IDS/IPS processes by talking about the need for tuning the new rules and/or signatures we set up. This step we don't necessarily have to do with firewalls.

IDS/IPS is a different ballgame, though, mostly because of the nature of the detection method. The firewall looks for specific conditions, such as traffic over a certain port, or protocols characteristics, or applications performing certain functions inside or outside a specified time window. In contrast IDS/IPS looks for patterns, and pattern recognition requires a lot more trial and error. So it really is an art to write IDS/IPS rules that work as intended. That process is rather bumpy so a good deal of tuning is required once the changes are made. That's what this next step is all about.

Monitor Issues/Tune

As described, once we make a rule change/update on an IDS/IPS it's not always instantly obvious whether it's working or not. Basically you have to watch the alert logs for a while to make sure you aren't getting too many or too few alerts for the new rule(s), and the conditions are correct when the alerts fire. That's why we've added a specific step for this probationary period of sorts for a new rule.

Since we are tracking activities that take time and burn resources, we have to factor in this tuning/monitoring step to get a useful model of what it costs to manage your IDS/IPS devices. We have identified four discrete subprocesses in this step:

1. Monitor IDS/IPS Alerts/Actions: The event log is your friend, unless the rule change you just made causes a flood of events. So the first step after making a change is to figure out how often an alert fires. This is especially important because most organizations phase a rule change in via a "log only" action initially. Until the rule is vetted, it doesn't make sense to put in an action to block traffic or blow away connections. How long you monitor the rule(s) varies, but within a day or two most ineffective rules can be identified and problems diagnosed.
2. Identify Issues: Once you have the data to figure out if the rule change isn't working, you can make some suggestions for possible changes to address the issue.
3. Determine Need for Policy Review: If it's a small change (threshold needs tuning, signature a bit off), it may not require a full policy review and pass through the entire change management process again. So it makes sense to be able to iterate quickly over minor changes to reduce the amount of time to tune and get the rules operational. This requires defining criteria for what requires a full policy review and what doesn't.
4. Document: This subprocess involves documenting the findings and packaging up either a policy review request or a set of minor changes for the operations team to tune the device.

And there you have it: the last of the subprocess posts. Next we'll post the survey (to figure out which of these processes your organization actually uses), as well as start breaking down each of these subprocesses into a set of metrics that we can measure and put into a model.

Stay tuned for the next phase of the NSO Quant project, which will start later this week.

–Mike Rothman

NSO Quant: Manage IDS/IPS—Audit/Validate

By Mike Rothman

As a result of our Deploy step, we have the rule change(s) implemented on the IDS/IPS devices but it's not over yet. To keep everything aboveboard (and add steps to the process) we need to include a final audit.

Basically this is about having either an external or internal resource, not part of the operations team, validate the change(s) and make sure everything has been done according to policy. Yes, this type of stuff takes time, but not as much as an auditor spending days on end working through every change you made on all your devices because the documentation isn't there.

Audit/Validate

This process is pretty straightforward and can be broken down into 3 subprocesses:

1. Validate Rule/Signature Change: There is no real difference between this Validate step and the Confirm step in Deploy except the personnel performing them. This audit process provides separation of duties, which means someone other than an operations person must verify the change(s).
2. Match Request to Change: In order to close the loop, the assessor needs to match the request (documented in Process Change Request) with the actual change to ensure everything about the change was clean. This involves checking both the functionality and the approvals/authorizations through the entire process resulting in the change.
3. Document: The final step is to document all the findings. This documentation should be stored separately from the policy management and change management documentation to eliminate any chance of impropriety.

Overkill?

For smaller companies this step is a non-starter. Small shops generally have the same individuals define policies and implement the rules associated with them. We do advocate documentation at all stages even in this case because it's critical for passing any kind of audit/assessment. Obviously for larger companies with a lot more moving pieces this kind of granular process and oversight of the changes can identify potential issues early -- before they cause significant damage. The focus on documenting as much as possible is also instrumental for making the auditor go away quickly.

As we've been saying through all our Quant research initiatives, we define very detailed and granular processes, not all of which apply to every organization. So take this for what it is and tailor the process to your environment.

–Mike Rothman

NSO Quant: Manage IDS/IPS—Deploy

By Mike Rothman

In our operational change management phase, we have processed the change request and tested and gotten approval for it. That means we're finally finished with planning and get to actually do something. So now we can dig into deploying the IDS/IPS rule and/or signatures change(s).

Deploy

We have identified 4 separate subprocesses involved in deploying a change:

1. Prepare IDS/IPS: Prepare the target devices(s) for the change(s). This includes activities such as backing up the last known good configuration and rule/signature set, rerouting traffic, rebooting, logging in with proper credentials, and so on.
2. Commit Rule Change: Within the device management interface, make the rule/signature change(s). Make sure to clean up any temporary files or other remnants from the change, and return the system to operational status.
3. Confirm Change: Consult the rule/signature base once again to confirm the change took effect.
4. Test Security: You may be getting tired of all this testing, but ultimately making any changes on critical network security devices can be dangerous business. We advocate constant testing to avoid unintended consequences which could create significant security exposure, so you'll be testing the changes. You have test scripts from the test and approval step to ensure the rule change delivered the expected functionality. We also recommend a general vulnerability scan on the device to ensure the IDS/IPS is functioning and firing alerts properly.

What happens if the change fails the security tests? The best option is to roll back the change immediately, figure out what went wrong, and then repeat the deployment with a fix. We show that as the alternative path after testing in the diagram. That's why backing up the last known good configuration during preparation is critical: so you can go back to a known-good configuration in seconds if necessary.

In the next post we'll continue the Manage IDS/IPS Change Management phase with auditing and validating these changes.

–Mike Rothman

NSO Quant: Manage IDS/IPS—Test and Approve

By Mike Rothman

Still on the operational side of change management, we need to ensure whatever change to the IDS/IPS has been processed won't break anything. That means testing the change and then moving to final approval before deployment.

We should be clear that testing here is different than an operational test. The content management test (in the Define/Update Rules & Policies step) really focuses on functionality and making sure the suggested change solves the problems identified during Policy Review. This operational test is to make sure nothing breaks. Yes, the functionality needs to be confirmed later in the process (during Audit/Validation), but test is about making sure there are no undesired consequences of the requested change.

Test and Approve

We've identified four discrete steps for the Test and Approve subprocess:

1. Develop Test Criteria: Determine the specific testing criteria for the IDS/IPS changes and assets. These should include installation, operation, and performance. The depth of testing varies depending on the assets protected by the device, the risk driving the change, and the nature of the change.
2. Test: Performing the actual tests.
3. Analyze Results: Review the test data. You will also want to document it, both for audit trail and in case of problems later.
4. Approve: Formally approve the rule change for deployment. This may involve multiple individuals from different teams (who hopefully have been in the loop all along), so factor any time requirements into your schedule.

This phase also includes one or more subcycles if a test fails and triggers additional testing, or reveals other issues. This may involve adjusting the test criteria, environment, or other factors to achieve a successful outcome.

We assume that the ops team has a proper test environment(s) and tools, although we are well aware of the hazards of such assumptions. Remember that proper documentation of assets is necessary for quickly finding assets and troubleshooting issues.

Next up is the process of deploying the change and then performing audit/validation (that pesky separation of duties requirement again).

–Mike Rothman

NSO Quant: Manage IDS/IPS—Process Change Request

By Mike Rothman

Now that we've gone through managing the content (policies/rules and signatures) that drive our IDS/IPS devices and developed change request systems for both rule change and signature updates, it's time to make whatever changes have been requested. That means you must transition from a policy/architecture perspective to operational mode. The key here is to make sure every change has exactly the desired impact, and that you have a rollback option in case of an unintended consequence -- such as blocking traffic for a critical application.

Process Change Request

A significant part of the Policy Management section is to document the change request. Let's assume it comes over the transom and ends up in your lap. We understand that in smaller companies the person managing policies and rules may very well also be making the changes, but processing the change needs still requires its own process -- if only for auditing and separation of duties.

The subprocesses are as follows:

1. Authorize: Wearing your operational hat, you need to first authorize the change. That means adhering to the pre-determined authorization workflow to verify the change is necessary and approved. Usually this involves both a senior level security team member and an ops team member formally signing off. Yes, this should be documented in some system to provide an audit trail.
2. Prioritize: Determine the overall importance of the change. This will often involve multiple teams -- especially if the firewall change impacts any applications, trading partners, or key business functions. Priority is usually a combination of factors, including the potential risk to your environment, availability of mitigating options (workarounds/alternatives), business needs or constraints, and importance of the assets affected by the change.
3. Match to Assets: After determining the overall priority of the rule change, match it to specific assets to determine deployment priorities. The change may be applicable to a certain geography or locations that host specific applications. Basically, you need to know which devices require the change, which directly affects the deployment schedule. Again, poor documentation of assets makes this analysis more expensive.
4. Schedule: Now that the priority is established and matched to specific assets, build out the deployment schedule. As with the other steps, quality of documentation is extremely important here -- which is why we continue to focus on it during every step of the process. The schedule also needs to account for any maintenance windows and may involve multiple stakeholders, as it is coordinated with business units, external business partners, and application/platform owners.

Now that the change request is processed and scheduled, we need to test the change and formally approve it for deployment. That's the next step in our Manage IDS/IPS process.

–Mike Rothman

NSO Quant: Manage IDS/IPS—Signature Management

By Mike Rothman

As described in the Manage IDS/IPS process map, we have introduced Content Management: the requirement to manage not only policies and rules but also signatures. The signatures (and other detection techniques) are constantly evolving, and that means the folks responsible for managing the boxes need to keep those detection mechanisms up to date. The Signature Management subprocess helps you do that.

This subprocess is pretty straightforward. Basically you need to find the updates, get them, evaluate their applicability to your environment, and then prepare a change request to add and activate the appropriate signatures.

Monitor for Release/Advisory

Unfortunately there is no stork that delivers relevant and timely signatures to your doorstep as you sleep; so you need to do the grunt work of figuring out which detection techniques need to be added, updated, and turned off on your IDS/IPS devices. The first step is to figure out what is new or updated, and we have identified a couple steps in this subprocess:

1. Identify Sources: You need to identify potential sources of advisories. In most cases this will be the device vendor, and their signature feeds are available through the maintenance relationship. Many organizations use open source IDS engines, most or all of which use Snort rules. So these users need to monitor the Snort updates, which are available via the SourceFire VRT premium feed or 30 days later in the public feed. It also makes sense to build periodic checks into the workflow, to ensure your advisory sources remain timely and accurate. Reassessing your sources a couple times a year should suffice.
2. Monitor Signatures: This is the ongoing process of monitoring your sources for updates. Most of them can be monitored via email subscriptions or RSS feeds.

Once you know you want to use a new or updated signature you need to get it and prepare the documentation to get the change made by the operations team.

Acquire

The third step in managing IDS/IPS signatures is actually getting them. We understand that's obvious, but when charting out processes (in painful detail, we know!) we cannot skip any steps.

1. Locate: Determine the location of the signature update. This might involve access to your vendor's subscription-only support site or even physical media.
2. Acquire: Download or otherwise obtain the new/updated signature.
3. Validate: Determine that the new/updated signature uses proper syntax and won't break any devices. If the signature fails validation, you'll need to figure out whether to try downloading it again, fix it yourself, or wait until it's fixed. If you are a good samaritan, you may even want to let your source know it's broken.

For Snort users, the oinkmaster script can automate many of these monitoring and acquiring processes. Of course commercial products have their own capabilities built into the various management consoles.

Once you have signature updates you'll need to figure out whether you need them.

Evaluate

Just because you have access to a new or updated signature doesn't mean you should use it. The next step is to evaluate the signature/detection technique and figure out whether and how it fits into your policy/rule framework. The evaluation process is very similar to reviewing device policies/rules, so you'll recognize similarities to Policy Review.

1. Determine Relevance/Priority: Now that you have a set of signatures you'll need to determine priority and relevance for each. This varies based on the type of attack the signature applies to, as well as the value of the assets protected by the device. You'll also want criteria for an emergency update, which bypasses most of the change management processes in case of an emergency.
2. Determine Dependencies: It's always a good idea to analyze the dependencies before making changes. If you add or update certain signatures, what business processes/users will be impacted?
3. Evaluate Workarounds: It turns out IDS/IPS signatures mostly serve as workarounds for vulnerabilities or limitations in other devices and software -- such as firewalls and application/database servers -- to handle new attacks (especially in the short term, as adding a signature may be much quicker to implement than the complete fix at the source), but you still need to verify the signature change is the best option.
4. Prepare Change Request: Finally, take the information in the documentation and package it for the operations team. We recommend some kind of standard template, and don't forget to include context (justification) for the change.

We aren't religious about whether you acquire or evaluate the signatures first. But given the ease (and automation) of monitoring and acquiring updates, it may not be worth the effort of running separate monitoring and acquisition processess -- it might be simpler and faster to grab everything automatically, then evaluate it, and discard the signatures you don't need.

So in a nutshell that's the process of managing signatures for your IDS/IPS. Next we'll jump into change management, which will be very familiar from the Manage Firewall process.

–Mike Rothman

NSO Quant: Manage IDS/IPS—Document Policies & Rules

By Mike Rothman

As we conclude the policy management aspects of the Manage IDS/IPS process (which includes Policy Review and Define/Update Policies & Rules), it's time to document the policies and rules you are putting into place.

Document Policies and Rules

Keep in mind the level of documentation you need for your environment varies based on culture, regulatory oversight, and (to be candid) 'retentiveness' of the security team. We are fans of just enough documentation. You need to be able to substantiate your controls (especially to the auditors) and ensure your successor knows how and why you did certain things. But there isn't much point in spending all your time documenting rather than doing. Obviously you have to find the right balance, but clearly you want to automate as much of this process as you can.

We have identified 4 subprocesses in the policy/rule documentation step:

1. Approve Policy/Rule: The first step is to get approval for the policy and/or rule (refer to Define/Update for definitions of policies and rules), whether it's new or an update. We strongly recommend having this workflow defined before you put the operational process into effect, especially if there are operational handoffs required before actually making the change. You don't want to step on a political land mine by going around a pre-determined hand-off in the heat of trying to make an emergency change. That kind of action makes operational people very grumpy. Some organizations have a very formal process with committees, while others use a form within their help desk system to provide very simple separation of duties and an audit trail -- of the request, substantiation, approver, etc. Again, don't make this harder than it needs to be, but you need some formality.
2. Document Policy/Change: Once the change has been approved it's time to write it down. We suggest using a fairly straightforward template which outlines the business need for the policy and its intended outcome. Remember policies consist of high-level, business-oriented statements. The documentation should already be about ready from the approval process. This is a matter of making sure it gets filed correctly.
3. Document Rule/Change: This is equivalent to the Document Policy Change step, except here you are documenting the actual IDS/IPS rules so the operations team can make the change.
4. Prepare Change Request: Finally we take the information from the documentation and package it up for the operations team. Depending on your relationship with ops, you may need to be very granular with the specific instructions. This isn't always the case but we make a habit of not leaving much to interpretation, because that leaves an opportunity for things to go haywire. Again we recommend some kind of standard template, and don't forget to include some context for why the change is being made. You don't need a full business case (as when preparing the policy or rule for approval), but if you include some justification, you have a decent shot at avoiding a request for more information from ops, which would mean delay while you convince them to make the change.

Emergency Updates

In some cases -- including data breach lockdowns, imminent zero-day attacks, and false positives impacting a key business process -- a change to the IDS/IPS ruleset must be made immediately. A process to circumvent the broader change process should be established and documented in advance, ensuring proper authorization for such rushed changes, and that there is a rollback capability in case of unintended consequences.

–Mike Rothman

NSO Quant: Manage IDS/IPS—Define/Update Policies & Rules

By Mike Rothman

As we continue digging into the policy management aspects of managing IDS/IPS gear (following up on Manage IDS/IPS: Policy Review), we need to define the policies and rules that drive the IPS/IDS. Obviously the world is a dynamic place with all sorts of new attacks continually emerging, so defining policies and rules is an iterative process. You need an ongoing process to update the policies as well.

To be clear, the high level policies should be reasonably consistent for all of your network security gear. The scope of this research includes Managing Firewalls and IDS/IPS, but the same high level policies would apply to other devices (like email and web filtering gateways, SSL VPN devices, Network DLP gateways, etc.). What will be different, of course, are the rules that implement the policies. For a more detailed discussion of policies vs. rules, look back at the Manage Firewall: Define/Update Policies and Rules, where a more detailed explanation exists.

Define/Update Policies and Rules

Given the amount of critical data you have to protect, building an initial set of policies can seem daunting. We recommend use cases for building the initial policy set. This means first identify the critical applications, users, and/or data to protect, and the circumstances for allowing or blocking access (location, time, etc.). This initial discovery process will help when you need to prioritize enforcing rules vs. inconveniencing users, because you always need to strike a balance between these two imperatives. Given those use cases you can define the policies, then model the potential threats to the applications, users, and data. Your rules address the attack vectors identified through the threat model. Finally you need to stage/test the rules before full deployment to make sure everything works.

More specifically, we have identified five subprocesses in defining and updating these policies/rules:

1. Identify Critical Applications/Users/Data: Here we discover what we need to protect. The good news is that you should already have at least some of this information, most likely through the Define Policies Subprocess. While this may seem rudimentary, it's important not to assume you know what is important and what needs to be protected. This means doing technical discovery to see what's out there, as well as asking key business users what applications/users/data are most important to them. Take every opportunity you can to get in front of users to listen to their needs and evangelize the security program. For more detailed information on discovery, check out Database Security Quant on Database Discovery.
2. Define/Update Policies: Once the key things to protect are identified, we define the base policies. As described above, the policies are the high-level business-oriented statements of what needs to be protected. For policies worry about what rather than how. It's important to prioritize them as well, because that helps with essential decisions on which policies go into effect and when specific changes happen. This step is roughly the same whether policies are being identified for the first time or updated.
3. Model Threats: Similar to the way we built correlation rules for monitoring, we need to break down each policy into a set of attacks, suspect behavior, and/or exploits which could be used to violate the policy. Put yourself in the shoes of an hacker, and think like them. Clearly there are an infinite number of attacks that can be used to compromise data, so fortunately the point isn't to be exhaustive -- it's to identify the most likely threat vectors for each policy.
4. Define/Update Rules: Once the threats are modeled it's time go one level down and define how you'd detect the attack using the IDS/IPS. This may involve some variety of signatures, traffic analysis, heuristics, etc. Consider when these rules should be in effect (24/7, during business hours, or on a special schedule) and whether the rules have an expiration date (such as when a joint development project ends or a patch is available). This identifies the base set of rules to implement a policy. Once you've been through each policy you need to get rid of duplicates and see where the leverage is.
5. Test Rule Set: The old adage about measure twice, cut once definitely applies here. Before implementing any rules, we strongly recommend testing both the attack vectors and the potential ripple effect to avoid breaking other rules during implementation. You'll need to identify and perform a set of tests for the rules being defined and/or updated. To avoid testing on a production box it's extremely useful to have a network perimeter testbed to implement new and updated rules; this can be leveraged for all network security devices. If any of the rules fail, you need to go back to the define/update rules step and fix. Cycle define/update/test until the tests pass.

Default Rule Sets

Given the complexity of making IDS/IPS rules work, especially in relatively complicated environments, we see many (far too many) organizations just using the default policies that come with the devices. You need to start somewhere, but the default rules usually reflect the lowest common denominator. So you can start with that set, but we advocate clearing out the rules that don't apply to your environment and going through this procedure to define your threats, model them, and define appropriate rules. Yes, we know it's complicated, but you've spent money on the gear, you may as well get some value from it.

–Mike Rothman

NSO Quant: Manage IDS/IPS—Policy Review

By Mike Rothman

Last week we dug into the Manage Firewall Process and updated the Manage IDS/IPS process map with what we've learned through our research. This week we will blow through Manage IDS/IPS because the concepts should be very familiar to those following the series. There are significant synergies between managing firewalls and IDS/IPS. Obviously the objectives of each device type are different, as are the detection techniques and rules options, but the defining policies and rules are relatively similar across device types, as is change management. You need to explicitly manage signatures and other detection content on an IDS/IPS, so that subprocess is new. But don't be surprised if you get a feeling of deja vu through the next few posts.

We've broken up the processes into Content Management (which is a bit broader than the Policy Management concept from Manage Firewall) and Change Management buckets. The next three posts will deal with policy and rule management, which begins with reviewing your policies.

Policy Review

Although it should happen periodically, far too many folks rarely (or even never) go through their IDS/IPS policies and rules to clean up and account for ongoing business changes. Yes, this creates security issues. Yes, it also creates management issues, and obsolete and irrelevant rules can place unnecessary burden on the IDS/IPS devices. In fact, obsolete rules tend to have a bigger impact on an IDS/IPS than on a firewall, due to the processing overhead of IDS/IPS rules. So at a minimum there should be a periodic review (perhaps twice a year) to evaluate the rules and policies, and make sure everything is up to date.

We see two other main catalysts for policy review:

1. Service Request: This is when someone in the organization needs a change to the IDS/IPS rules, typically driven by a new application or trading partner who needs access to something or other. These requests are a bit more challenging than just opening a firewall port because the kind of application traffic and behavior need to be profiled and tested to ensure the IDS/IPS doesn't start blocking legitimate traffic.
2. External Advisory: At times, when a new attack vector is identified, one ways to defend against it is to set up a detection rule on the IDS/IPS. This involves monitoring the leading advisory services and using that information to determine whether a policy review is necessary.

Once you have decided to review policies, we have identified five subprocesses:

1. Review Policies: The first step is to document the latest version of the polices; then you'll research the requested changes. This gets back to the catalysts mentioned above. If it's a periodic review you don't need a lot of prep work, while reviews prompted by user requests require you to understand the request and its importance. If the review is driven by a clear and present danger, you need to understand the nuances of the attack vector to understand how you can make changes to detect the attack and/or block traffic.
2. Propose Policy Changes: Once you understand why you are making the changes, you'll be able to recommend policy changes. These should be documented as much as possible, both to facilitate evaluation and authorization, and also to maintain an audit trail of why specific changes were made.
3. Determine Relevance/Priority: Now that you have a set of proposed changes it's time to determine its initial priority. This is based on the importance of particular assets behind the IDS/IPS and the catalyst for change. You'll also want criteria for an emergency update, which bypasses most of the change management processes in the event of a high-priority situation.
4. Determine Dependencies: Given the complexity and interconnectedness of our technology environment, even a fairly simple change can create ripples that result in unintended consequences. So analyze the dependencies before making changes. If you turn certain rules on or off, or tighten their detection thresholds, what business processes/users will be impacted? Some organizations manage by complaint by waiting until users scream about something broken after a change. That is one way to do it, but most at least give users a "heads up" when they decide to break something.
5. Evaluate Workarounds/Alternatives: An IDS/IPS change may not be the only option for defending against an attack or providing support for a new application. As part of due diligence, you should include time to evaluate workarounds and alternatives. In this step determine any potential workarounds and/or alternatives, and evaluate their dependencies and effectiveness, in order objectively choose the best option.

In terms of our standard disclaimer for Project Quant, we build these Manage IDS/IPS subprocesses for organizations that need to manage any number of devices. We don't make any assumptions about company size or whether a tool set will be used. Obviously the process varies based on your particular circumstances, as you will perform some steps and skip others. We think it's important to give you a feel for everything that is required to manage these devices, so you can compare apples to apples between managing your own vs. buying a product(s) or using a service.

As always, we appreciate any feedback you have on these subprocesses.

Next we'll Define/Update Policies and Rules: roll up our sleeves to maintain the policy set, and take that to the next level by figuring out the rules to implement the policies.

–Mike Rothman

NSO Quant: Manage IDS/IPS Process Map (Updated)

By Mike Rothman

Note: Based on our ongoing research into the process maps, we felt it necessary to update both the Manage Firewall and IDS/IPS process maps. As we built the subprocesses and gathered feedback, it was clear we didn't make a clear enough distinction between main processes and subprocesses. So we are taking another crack at this process map. As always, feedback appreciated.

After banging out the Manage Firewall processes and subprocesses, we now move on to the manage IDS/IPS process. The first thing you'll notice is that this process is a bit more complicated, mostly because we aren't just dealing with policies and rules, but we also need to maintain the attack signatures and other heuristics used to detect attacks. That adds another layer of information required to build the rule base that enforces policies. So we have expanded the definition of the top area to Content Management, which includes both policies/rules and signatures.

Content Management

In this phase, we manage the content that underlies the activity of the IDS/IPS. This includes both attack signatures and the policies/rules that control reaction to an attack.

Policy Management Subprocess

• Policy Review: Given the number of potential monitoring and blocking policies available on an IDS/IPS, it's important to keep the device up to date. Keep in mind the severe performance hit (and false positive issues) of deploying too many policies on each device. It is best practice to review IDS/IPS policy and prune rules that are obsolete, duplicative, overly exposed, prone to false positives, or otherwise not needed. Possible triggers for a policy review include signature updates, service requests (new application support, etc.), external advisories (to block a certain attack vector or work around a missing patch, etc.), and policy update resulting from the operational management of the device (change management process described below).

• Define/Update Policies & Rules: This involves defining the depth and breadth of the IDS/IPS policies, including the actions (block, alert, log, etc.) taken by the device in the event of attack detection, whether via signature or another method. Note that as the capabilities of IDS/IPS devices continue to expand, a variety of additional detection mechanisms will come into play. Time limited policies may also be deployed to activate or deactivate short-term policies. Logging, alerting, and reporting policies are also defined in this step. At this step, it's also important to consider the hierarchy of policies that will be implemented on the devices. The chart at right shows a sample hierarchy including organizational policies at the highest level, which may be supplemented or supplanted by business unit or geographic policies. Those feed into a set of policies and/or rules implemented at a location, which then filter down to the rules and signatures implemented on a specific device. The hierarchy of policy inheritance can dramatically increase or decrease complexity of rules and behaviors. Initial policy deployment should include a Q/A process to ensure none of the rules impact the ability of critical applications to communicate either internally or externally through the device.

• Document Policies and Rules: As the planning stage is an ongoing process, documentation is important for operational and compliance reasons. This step lists and details the policies and rules in use on the device according to the associated operational standards/guidelines/requirements.

Signature Management Subprocess

• Monitor for Release/Advisory: Identify signatures sources for the devices, and then monitor on an ongoing basis for new signatures. Since attacks emerge constantly, it's important to follow an ongoing process to keep the IDS/IPS devices current.

• Evaluate: Perform the initial evaluation of the signature(s) to determine if it applies within your organization, what type of attack it detects, and if it's relevant in your environment. This is the initial prioritization phase to determine the nature of the new/updated signature(s), its relevance and general priority for your organization, and any possible workarounds.

• Acquire: Locate the signature, acquire it, and validate the integrity of the signature file(s). Since most signatures are downloaded these days, this is to ensure the download completed properly.

Change Management

In this phase the IDS/IPS rule and/or signatures additions, changes, updates, and deletes are implemented.

• Process Change Request: Based on either a signature or a policy change within the Content Management process, a change to the IDS/IPS device(s) is requested. Authorization involves both ensuring the requestor is allowed to request the change, as well as the change's relative priority, to slot it into an appropriate change window. The change's priority is based on the nature of the signature/policy update and potential risk of the relevant attack. Then build out a deployment schedule based on priority, scheduled maintenance windows, and other factors. This usually involves the participation of multiple stakeholders, ranging from application, network, and system owners, to business unit representatives if downtime or changes to application use models is anticipated.

• Test and Approve: This step requires you to develop test criteria, perform any required testing, analyze the results, and approve the signature/rule change for release once it meets your requirements. Testing should include signature installation, operation, and performance impact on the device as a result of the change. Changes may be implemented in "log-only" mode to observe the impact of the changes before committing to block mode in production. With an understanding of the impact of the change(s), the request is either approved or denied. Obviously approval may require "thumbs up" from a number of stakeholders. The approval workflow needs to be understood and agreed upon to avoid significant operational issues.

• Deploy: Prepare the target device(s) for deployment, deliver the change, and return the device to service. Verify that changes were properly deployed, including successful installation and operation. This might include use of vulnerability assessment tools or application test scripts to ensure no disruption of production systems.

• Audit/Validate: Part of the full process of making the change is not only having the operational team confirm the change (which happens during the Deploy step), but also having another entity (internal or external, but not part of the ops team) audit it as well for separation of duties. This involves validating the change to ensure the policies were properly updated, as well as matching the change to a specific request. This closes the loop and makes sure there is a documented trail for every change happening on the box.

• Confirm/Monitor for Issues: The final step of the change management process involves a burn-in period, where each rule change should be scrutinized to detect unintended consequences such as unacceptable performance impact, false positives, security exposures, or undesirable application impact. The goal of the testing process in the Test and Approve step is to minimize these issues, but typically there are variances between the test environment and the production network, so we recommend a probationary period for each new or updated rule -- just in case. This is especially important when making numerous changes at the same time, as it requires diligent effort to isolate which rule created any issues.

Other Considerations

Emergency Update

In some cases, including a data breach lockdown or imminent zero day attack, a change to the IDS/IPS signature/rule set must be made immediately. An 'express' process should be established and documented as an alternative to the full normal change process, ensuring proper authorization approves for emergency changes, as well as a rollback capability in case of unintended consequences.

IDS/IPS Health (Monitoring and Maintenance)

This phase involves ensuring the IDS/IPS devices are operational and secure. This involves monitoring for availability and performance, as well as upgrading the hardware when needed. Additionally, software patches (for either functionality or security) are implemented in this phase. We've broken this step out due to its operational nature. This doesn't relate to security or compliance directly, but can be a significant cost component of managing these devices and so gets modeled separately.

Incident Response/Management

For this Quant project we consider the monitoring and management processes separate, although many organizations (especially the service providers that provide managed services) consider device management a superset of device monitoring.

So the IDS/IPS management process flow does not include any steps for incident investigation, response, validation, or management. Please refer to the Monitoring process flow for those activities.

Next week I'll post all the subprocesses, so get ready. The content will be coming fast and furious. Basically, we are pushing to get to the point where we can post the survey to get a more quantitative feel for how many organizations are doing each of these subprocesses, and identify some of their issues. We expect to be able to get the survey going within two weeks, and then we'll start posting the associated operational metrics as well.

–Mike Rothman

NSO Quant: Manage Firewall Process Revisited

By Mike Rothman

Now that we've been through all the high-level process steps and associated subprocesses for managing firewalls, we thought it would be good to summarize with the links to the subprocesses and a more detailed diagram. Note that some of the names of process steps have changed, as the process maps evolve throughout the research process.

• NSO Quant: Manage Firewall Process Map (Updated)
• NSO Quant: Manage Firewall -- Policy Review
• NSO Quant: Manage Firewall -- Define/Update Policies & Rules
• NSO Quant: Manage Firewall -- Document Policies/Rules
• NSO Quant: Manage Firewall -- Process Change Request
• NSO Quant: Manage Firewall -- Test and Approve
• NSO Quant: Manage Firewall -- Deploy
• NSO Quant: Manage Firewall -- Audit/Validate

What's missing? The firewall health maintenance subprocesses. But in reality, keeping the devices available, patched and using adequate hardware is the same regardless of whether you are monitoring or managing firewalls and/or IDS/IPS. So we'll refer back to the health maintenance post in the Monitoring step for those subprocesses. The only minor difference, which doesn't warrant a separate post, is the testing phase -- and as you've seen we are testing the firewall(s) throughout the change process so this doesn't need to also be included in the device health process.

As with all our research, we appreciate any feedback you have on this process and its subprocesses. It's critical that we get this right because since we start developing metrics and building a cost model directly from these steps. So if you see something you don't agree with, or perhaps do things a bit differently, let us know.

–Mike Rothman

NSO Quant: Manage Firewall—Audit/Validate

By Mike Rothman

As a result of our Deploy step, we have the rule change(s) implemented on the firewalls. But it's not over yet. Actually, from an operations standpoint it is, but to keep everything above board (and add steps to the process) we need to include a final audit step.

Basically this is about having either an external or internal resource, not part of the operations team, validate the change(s) and make sure everything has been done according to policy. Yes, this type of stuff takes time, but not as much as an auditor spending days on end working through every change you made on all your devices because the documentation isn't there.

Audit/Validate

This process is pretty straightforward and can be broken down into 3 subprocesses:

1. Validate Rule Change: There is nothing fundamentally different between this validate step and the confirm step in Deploy, except the personnel performing it. This audit process addresses any separation of duties requirements, which means by definition someone other than an operations person must verify the change(s).
2. Match Request to Change: In order to close the loop, the assessor needs to match the request (documented in Process Change Request) with the actual change to once again ensure everything about the change was clean. This involves checking both the functionality and the approvals/authorizations throughout the entire process resulting in the change.
3. Document: The final step is to document all the findings. This documentation should be stored separately from the regarding policy management and change management documentation, to eliminate any chance of impropriety.

Overkill?

For smaller companies this step is a non-starter. For the most part, the same individuals define policies and implement them. We do advocate documentation at all stages regardless, because it's critical to pass any kind of audit/assessment. Obviously for larger companies with a lot more moving pieces this kind of granular process and oversight of the changes can identify potential issues early -- before they cause significant damage. The focus on documenting as much as possible is also instrumental for making the auditor go away as quickly as possible.

As we've been saying through all our Quant research initiatives, we define very detailed and granular processes, not all of which apply to every organization. So take it for what it is, and tailor the process to work in your environment.

–Mike Rothman

NSO Quant: Manage Firewall—Deploy

By Mike Rothman

In our operational change management phase, we have processed the change request and tested and gotten approval for the change. That means it's time to stop this planning stuff and actually do something. So now we can dig into deploying the firewall rule change(s).

Deploy

We have identified 4 separate subprocesses involved in deploying a change:

1. Prepare Firewall: Prepare the target firewall(s) for the change(s). This includes activities such as backing up the last known good configuration and rule set, rerouting traffic, rebooting, logging in with proper credentials, and so on.
2. Commit Rule Change: Within the management interface of the firewall, make the rule change(s). Make sure to clean up any temporary files or other remnants from the change, and return the system to operational status.
3. Confirm Change: Consult the rule base once again to confirm the change has been made.
4. Test Security: You may be getting tired of all this testing, but ultimately making firewall rule changes can be dangerous business. We advocate constant testing to ensure no unintended consequences to the system which could create significant security exposure. So you'll be testing the changes just made. You have test scripts from the test and approval step to ensure the rule change delivered the expected functionality. We also recommend a general vulnerability scan on the device to ensure the firewall is functioning properly.

What happens if the change fails the security tests? The best option is to roll back the change immediately, figure out what went wrong, and then repeat this step with a fix. We show that as the alternative path after testing in the diagram. That's why backing up the last known good configuration during preparation is critical -- so you can go back to a configuration you know works in seconds, if necessary.

Finally, for large enterprises, making rule changes one device at a time probably doesn't make sense. A number of tools and managed services can automate management of a large number of firewalls. Each firewall vendor has a management console to manage their own boxes, and a number of third parties have introduced tools to make managing a heterogeneous firewall environment easier.

Our goal through this Quant research is to provide an organization with a base understanding of the efficiency and cost of managing all these devices, to help track and improve operational metrics, and to provide a basis for evaluating the attractiveness of using a tool or service for these functions.

In the next post we'll finish up the Manage Firewall Change Management phase by auditing and validating these changes.

–Mike Rothman