As we dive back into Firewall Management Essentials, let’s revisit some of the high points from our Introduction:

The firewalls run on a set of rules that basically define what ports, protocols, networks, users, and increasingly applications, can do on your network. And just like a closet in your house, if you don’t spend time sorting through old stuff it can become a disorganized mess, with a bunch of things you haven’t used in years and don’t need any more.

The problem is that, like your closet, this issue just gets worse if you put off addressing the issue. And it’s not like rule bases are static. You have new requests coming in to open this port or allow that group of users to do something new or different pretty much every day. The situation can get out of control quickly, especially as you increase the number of devices in use.

So first we will dig into building a consistent workflow to manage the change process. This process is important for numerous reasons:

1. Accuracy: If you make an incorrect change or have rules which conflict with other rules you can add significant attack surface to your environment. So it is essential to ensure you make the proper changes, correctly.
2. Authorization: It is difficult for many security admins to say no, especially to persuasive business and technology leaders who ‘need’ their stuff done now. So a consistent and fair authorization process eliminates bullying and other shenanigans folks use to get what they want.
3. Verification: Was the change made correctly? Are you sure? The ability to verify the change was correct and successful is important, especially for auditing.
4. Audit trail: Speaking of audit, making sure every change is documented, with details of the requestor and approver, is helpful both when preparing for an audit and for ensuring the audit’s outcome is positive.

Network Security Operations

A few years ago we tackled building a huge and granular process map for network security operations as part of our Network Security Operations Quant research. One of the functions we explicitly described was managing firewalls. Check out the detailed process map:

This can be a bit ponderous for many organizations, and isn’t necessarily intended to be implemented in its entirety. But it illustrates what is involved in managing these devices. To ensure you understand how we define some of these terms, here is a brief description of each step from that report.

Policy, Rule, and Signature Management

In this phase we manage the content that underlies the network security devices. This includes attack signatures and the policies & rules that control response to an attack.

1. Policy Review: Given the number of monitoring and blocking policies available on network devices, it is important to keep rules (policies) current. Keep in mind the severe performance hit (and false positive issues) of deploying too many policies on a device. It is a best practice to review network security device policies and prune rules that are obsolete, duplicative, overly exposed, prone to false positives, or otherwise unneeded. Policy review triggers include signature updates, service requests (new application support, etc.), external advisories (to block a certain attack vector or work around a missing patch, etc.), and policy updates resulting from the operational management of the device (change management process described below).
2. Define/Update Policies & Rules: This involves defining the depth and breadth of the network security device policies, including the actions (block, alert, log, etc.) taken by the device if an attack is detected – whether via rule violation, signature trigger, or another method. Note that as the capabilities of network security devices continue to expand, a variety of additional detection mechanisms come into play. They include increasing visibility into application traffic and identity stores. Time-limited policies may also be deployed to activate or deactivate short-term policies. Logging, alerting, and reporting policies are defined in this step. Here it is important to consider the hierarchy of policies that will be implemented on devices. You will have organizational policies at the highest level, applying to all devices, which may be supplemented or supplanted by business unit or geographic policies. Those highest-level policies feed into the policies and/or rules implemented at a location, which then filter down to the rules and signatures implemented on a specific device. The hierarchy of policy inheritance can dramatically increase or decrease complexity of rules and behaviors. Initial policy deployment should include a Q/A process to ensure none of the rules impacts the ability of critical applications to communicate either internally or externally.
3. Document Policies and Rules: As the planning stage is an ongoing process, documentation is important for operational and compliance reasons. This step lists and details the policies and rules in use on the device according to the associated operational standards, guidelines, and requirements.

Change Management

In this phase rule & signature additions, changes, updates, and deletions are handled.

1. Process Change Request and Authorize: Based on either a signature or policy change within the Content Management process, a change to the network security device(s) is requested. Authorization requires both ensuring the requestor is allowed to request the change, and the change’s relative priority to select an appropriate change window. The change’s priority is based on the nature of the signature/policy update and risk of the relevant attack. Then build out a deployment schedule based on priority, scheduled maintenance windows, and other factors. This usually involves the participation of multiple stakeholders – ranging from application, network, and system owners to business unit representatives if downtime or changes to application use models is anticipated.
2. Test and Approve: This step requires you to develop test criteria, perform any required testing, analyze the results, and approve the signature/rule change for release once it meets your requirements. Testing should include signature installation, operation, and performance impact on the device as a result of the change. Changes may be implemented in ‘log-only’ mode to observe their impact before committing to blocking mode in production. With an understanding of the impact of the change(s), the request is either approved or denied. Obviously approvals may be required from a number of stakeholders. The approval workflow must be understood and agreed on in advance to avoid significant operational issues.
3. Deploy and Confirm: Prepare the target devices for deployment, deliver the change, and return them to normal operation. Verify that changes were properly deployed, including successful installation and operation. This might include use of vulnerability assessment tools or application test scripts to ensure there is no disruption of production systems.
4. Audit/Validate: Part of the full process of making the change is not only having the operations team confirm it during the Deploy step, but also having another entity (internal or external, but not part of the ops team) audit it as well to provide separation of duties. This involves validating the change to ensure the policies were properly updated and matching it to a specific request. This closes the loop and makes sure there is a documentation trail for every change to the box.

Emergency Update

In some cases, including data breach lockdowns and imminent or active zero-day attacks, a change to the network security device’s signature/rule set must be made immediately. An ‘express’ process should be established and documented as an alternative to the full normal change process, ensuring proper authorized approval for emergency changes, as well as a rollback capability in case of unintended consequences.

Key Firewall Management/Change Automation features

As we get into what we need a firewall management tool to do, let’s list a few of the key features:

1. Configurable workflow: Some organizations don’t have any process at all, so they are happy to have the firewall management tool dictate the tasks and approvals. But most already have a well-established process and it probably doesn’t make sense to blow it up. The FM tool should offer a reasonable out-of-the-box process and be configurable to support existing processes.
2. Notification of change request: Once a change request comes in the first task is to actually notify someone. Given the large number of notification options today, any FM tool should support a variety of these options, and allow multiple parties to be notified as needed.
3. Flexible approval/authorization: Once the request is received, the authorization/approval process needs to start. This involves the aforementioned notification, but also support for complicated workflows and multi-stage approvals depending on the nature of the change request.
4. Security posture assessment and policy checking: Once the change has been approved, it would be great to understand how it impacts security posture, as well as check to ensure the change doesn’t violate existing configuration policies. We will discuss this in the next post, on optimizing rules.
5. Change tracking and verification: One of the key values of automating the firewall change management process is to get a better handle on workload, backlog, and responsiveness. The FM tool should track all change requests with a closed loop, as well as verify that each change has been made correctly and successfully.
6. Task management: Hand in hand with closed-loop tracking of firewall change requests is the ability for security administrators and operations folks to track their work queues. This ensures they are working on the most important changes and getting them done in a timely fashion.
7. Integration with help desk systems: Of course some operations groups already have well-established task management systems integrated with their help desk systems. So bi-directional integration with those tools is important to ensure work isn’t duplicated and that both systems a current views of what needs to get done.
8. Audit trail and compliance reporting: Finally, a key value of automation is the ability to maintain an audit trail, so you need to make sure your FM tool tracks everything, can’t be tampered with, and generates reports for both internal use and compliance. Additionally, ensure reports can be customized as necessary.

Of course this isn’t an exhaustive list. But it hits the high points of what to automate for firewall change management.

Automatic provisioning

One of the religious battles brewing on firewall operations involves whether the FM tool should actually commit changes to the firewall. You know, kind of like having Skynet managing missile defense systems. What could go wrong? But all kidding aside, many security administrators are not comfortable with a tool making changes directly, and that is a legitimate cocnern.

We don’t really like wading into religious battles; there is middle ground to be found here. Some changes – simple network changes, reordering existing rules, etc. – present reasonable amounts of risk and are likely candidates for automated change. Others, including supporting new applications and changing entitlements for key organizations, need additional scrutiny.

Either way it is reasonable to expect your tool to generate scripts, command line code, and/or detailed instructions for what is required to commit a change. So even if your folks have hands on keyboards to make changes themselves, they can and should get as much guidance as possible from the FM tool.

Provision Access for Business Applications

Another new function being introduced by firewall management vendors is the ability to have business users request to open up access for specific applications or shut down unsupported applications. The FM tool can translate that request into a set of instructions to be implemented on the firewall to provide or block access as necessary. Given the challenges of translating business-speak into access rules, and the importance of getting applications operational quickly without adding undue attack surface, this capability makes sense as part of the FM tool suite in time.

But for now our research indicates this cutting-edge requirement appeals primarily to very mature operational environments with well-established processes and toolsets. Using a toddler analogy, many organizations are barely able walk, using highly involved processes with few tools to manage network security devices. This capability, in contrast, is more like running, and would only be applicable to organizations with their act together for both workflow and rule optimization – which we will discuss next.