WAFs themselves are an application, and as such they provide additional attack surface for your adversaries. Their goal isn’t necessarily to compromise the WAF itself (though that’s sometimes a bonus) – the short-term need is evasion. If attackers can figure out how to get around your WAF, many of its protections become moot. Your WAF needs to be secured, just like any other device sitting out there and accessible to attackers. So let’s start by discussing device security, including deployment and provisioning entitlements. Then we can get into some evasion tactics you are likely to see, before wrapping up with a discussion of the importance of testing your WAFs on an ongoing basis.

Deployment Options

Managing your WAF pragmatically starts when you plug in the device, which requires you to first figure out how you are going to deploy it. You can deploy WAFs either inline or out of band. Inline entails installing the WAF in front of a web app to block attacks directly. Alternatively, as with Network Access Control (NAC) devices, some vendors to provide an out-of-band option to assess application traffic via a network tap or spanning port, and then use indirect methods (TCP resets, network device integration, etc.) to shut down attack sessions. Obviously there are both advantages and disadvantages to having a WAF inline, and we certainly don’t judge folks who opt for out-of-band deployment rather than risking impact to applications. But as with NAC evasion, out-of-band enforcement can be evaded and presents an additional risk to the application. But balancing risks, such as reduced application protection against possible application disruption, is why you get the big bucks, right?

You will also need to consider high availability (HA) deployment architectures. If a WAF device fails and takes your applications with it, that’s a bad day all around. So make sure you can deploy multiple boxes with consistent policy and utilize some kind of non-disruptive failover option (active/active, active/passive, load balancer front-end, etc.).

Of course some folks opt for a managed WAF service, so the device doesn’t even sit in their data center. This offloads responsibility for scaling up the implementation, providing high availability, and managing devices (patching, etc.) to the service provider. Additionally, the service provider can offer some obfuscation of your IP addresses, complicating attacker reconnaissance and making WAF evasion harder. Depending on how the service is packaged, the service provider may also provide resources to manage policies. Of course they cannot offload accountability for protecting applications, and a service provider cannot be expected to interface directly with your developers. You should also understand the background of your WAF provider. Are they a security company? Does the WAF provide full application security features, or is it a glorified content distribution network (CDN)? Obviously a service provider isn’t likely to offer the full granular capabilities and policy options of a device in your data center, so you need to balance the security capabilities of a managed WAF service against what you can do yourself.

Other Security Considerations

Obviously you need to keep attackers away from the physical devices, so ensuring physical security of devices is the first step, and hopefully already largely covered by existing security measures. After that you need to ensure all credentials stored on the device are protected, including the SSL private keys used for SSL interception.

You will also need to exercise good security hygiene on the device, which means detailed logging of any changes to device configuration and/or policies. Hopefully the logs will aggregated on an external aggregation system (a log management server) to prevent tampering, and alerts should be sent if logging is turned off. That also means keeping the underlying operating system (for software-based WAFs) and the WAF itself patched and up to date. No different than what you should do for every other security device in your environment.

Again, a managed WAF service gets you out of having to update devices and/or WAF software, but make sure you can get access to the appropriate WAF activity logs. Make sure you have sufficient access for forensic investigation, if and when you need to go there.

Finally, keep in mind that Denial of Service (DoS) attacks continue to be problematic, targeting applications with layer 7 attacks, in addition to simpler volume-based attacks. Make sure you have sufficent bandwidth to deal with any kind of DoS attack, a sufficiently hearty WAF implementation to deal with the flood, and a DDoS-focused service provider on call to handle additional traffic if necessary. Protecting against DoS attacks is a discipline unto itself, and we plan a series on that in the near future.

Provisioning and Managing Entitlements

Once you have secured the device, next make sure the policies and device configurations are protected. Take steps to control provisioning and management of entitlements. Given the sensitivity of the WAF, it makes sense to get back to the 3 A’s. Yeah, man, old school.

• Authorization: Who is allowed to access the WAF? Can they set up policies? Change configurations? Is this a group or set of individuals?
• Authentication: Once you know who can legitimately get into the device, how will you ensure it’s really them? Passwords? 2-factor authentication? Digital certificates? Retinal scans? Okay, that last was a joke, but this question isn’t.
• Audit: You want an audit event every time a WAF policy, configuration, entitlement, or anything else is changed.

Note that a managed WAF service will complicate your ability to manage entitlements. The service provider will have the ability to change policies and may even be responsible for managing them. Ensure you adequately vet folks who will have access to your policies, with an audit trail. We know we are beating the audit horse, but it’s particularly important in this context.

An alternative method for managing access to WAF devices is Privileged User Management (PUM). In this scenario administrators log into some sort of proxy, which manages credentials and provides access only to the WAFs each administrator has authorization for. That’s just one of the benefits of PUM, so check out our paper to learn more about these products.

Preventing Evasion

Attacking an application is a means to an end. The attacker is focused on getting the data, and (for now) the path of least resistance is through an application. The WAF is positioned to prevent that attack, so the attacker has a choice. Go through the WAF or around it. In many cases it’s easier to evade a WAF than compromise it. This is another similarity between WAF ans Intrusion Prevention Systems (IPS). Both use negative security policies to detect and block attacks. Obviously WAF is optimized for different attacks against different targets, but they are conceptually very similar.

Secure device evasion isn’t new – attackers (including penetration testers) have been building tools to evade IPS devices for years, and many of them work just as well to evade WAF. Rather than futzing with lower-level network protocols to evade an IPS, an attacker attempting to circumvent a WAF could instead mess with web requests – HTTP parameter manipulation, adding SQL comments in parameters, changing case, encoding URLs or URIs, gaming SSL interception, inserteing random words, packet fragmentation, etc. Any of these tactics might mask an attack pattern a WAF looks for.

The other main tactic used in WAF evasion is to identify semantic gaps. That basically means understanding that an application and/or database will interpret commands and scripts differently than a WAF, and that provides an opportunity for an attacker to game that system. Understanding the details of these semantic gaps is a bit deeper than we can go in this series, but check out Ivan Ristic’s latest research on evasion (PDF) for a better feel for specific tactics seen in the wild.

Reconnaissance

Attackers and security folks play an ongoing cat and mouse game. The attacker starts by poking around, trying to find weaknesses in an application. The good news is that you know how the reconnaissance process works, which enables you to make it a bit harder.

The attacker’s first step is to profile your application. They will try to figure out specific IP addresses, web services, operating systems, session timeouts, cookie settings, application response codes, etc. Anything that can help identify specific technologies can indicate areas of weakness and provides clues on how to attack a system. They will also try to identify the particular WAF in place – bad guys share research on which tactics work against which WAF devices. So they will send a bunch of traffic to the application, looking for clues to the WAF in use in the responses received.

They also refine their profiling to get a feel for which rules are active. A skilled attacker will get a pretty good feel for both your application and defenses. And they will have a good idea of how to beat most systems at the end of the reconnaissance phase – even before they have started trying to get past your defenses. Fortunately this is a resource-heavy approach, but if a serious attacker’s mission requires getting your data, they will put the time in. You can learn a bit more about reconnaissance for WAF evasion (and an approach to protection) at PaulDotCom.

Making Evasion Harder

Fortunately there are some tactics to make this much harder for attackers, starting with obfuscation. In general, security by obscurity doesn’t work very well, but in this case hiding the IP addresses of your applications using a CDN or caching service – especially one that employs a coarse WAF filter – can be very helpful by making it harder attackers to know what they are up against. This is a feature of managed WAF service as well.

You can also rely on having the WAF just drop malformed requests, rather than sending a response code. At times silence is your best defense, and not giving attackers any clues as to what you are doing can be helpful. A more aggressive disinformation tactic involves “active defense” or “intrusion deception,” which involves sending confusing or just wrong response codes to confuse attackers. You can learn more about these tactics in Rich’s great post. Of course you need to be sure you are successfully identifying reconnaissance or attack traffic before you can employ these techniques, because it is unacceptable to incorrectly identify a legitimate user as an attacker and deceive or counterattack them

The point is not to enumerate and detail all possible evasion tactics – they are a moving target. But make sure you factor WAF evasion into your deployment and policy management processes.

To address the semantic gaps you need to really understand each application and do a lot of testing. You need to understand how application servers and databases interpret commands, and also have enough knowledge of the WAF to know whether and how its responses differ. So the process we defined in application lifecycle integration is instrumental to avoiding sophisticated evasion. This is another reason so many customers have had issues making WAF truly useful – WAF admins need to have both application and WAF skills to block this kind of evasion.

And don’t forget the importance of ongoing discovery of web applications. Strangely enough, not all WAF devices automatically detect traffic coming into a new application and immediately provide generic protections – instead you need to explicitly identify each new application to enjoy any application-layer protection. You need to sit on top of the WAF as it learns new URLs, make sure rules are in place, and confirm that learned behaviors are quickly turned into policies to ensure those pages are protected. Your adversaries are doing discovery on your web properties every day, so you need to do the same. Waiting three months for learning won’t cut it, so look for ways to accelerate the process. That may involve a manual review, using your vulnerability scanner to detect new applications, evaluating traffic through the WAF or access firewall, or using a network discovery tool such as nmap. Regardless of technique you need to identify new applications and/or pages and apply rules quickly.

Hacking Yourself and Testing

Despite your best efforts, you still may miss something which leaves your applications exploitable. We advocate an ongoing testing process to ensure your policies are sound, your configuration hasn’t drifted, and ultimately that you are blocking what needs to be blocked. Of course you can use scanners, evasion libraries (more kudos to Ivan Ristic for publishing an Evasion Techniques Catalogue), and other tools (such as Havij and SQLmap) to automate the testing process, but that’s only half the story. To fully provide this kind of assurance, you need actual humans banging away at your applications.

Additionally, a number of compliance mandates require periodic penetration tests. One of the big sticking points is to what degree a pen tester can evade your defenses to do what their work. Talk to enough pen testers, and their frustration at being glorified eunuchs (not being able to really attack a target) becomes evident. We share their frustration that the bad guys don’t operate under the same restrictions – they do not hold back when trying to evade the WAF, crack the application, and exfiltrate your data. We suggest you check out some of the work done under the Penetration Testing Execution Standard (PTES) to get a feel for how to structure your engagement to maximize value without crushing your systems.

None of this is perfect science. And nowhere have we said that WAF management is easy – quite the contrary. But the good news is that most attackers, after running into a well-configured WAF, will go elsewhere for lower-hanging fruit. The majority of attackers don’t have the time or inclination to hammer away at your site indefinitely. Raising the bar and closing the gaps against basic attacks and evasion techniques will save you many headaches. Staying on top of policies and continually probing your own systems for weaknesses give you a much greater chance at success.

And with that we finished up the Pragmatic WAF series. We will assemble the blog posts into a research paper over the next few weeks, so there is still time to weigh in and let us know what’s good (or not so good).