As described in last post, deploying a WAF requires knowledge of both application security and your specific application(s). Management it requires an ongoing effort to keep a WAF current with emerging attacks and frequent application changes. Your organization likely adds new applications and changes network architectures at least a couple times a year. We see more and more organizations embracing continuous deployment for their applications. This means application functions and usage are constantly changing as well. So you need to adjust your defenses regularly to keep pace.

Test & Tune

The deployment process is about putting rules in place to protect applications. Managing the WAF involves monitoring it to figure out how well your rules are actually working, which requires spending a bunch of time examining logs to learn what is working and what’s not.

Tuning policies for both protection and performance is not easy. As we have mentioned, you need someone who understands the rule ‘grammars’ and how web protocols work. That person must also understand the applications, the types of data customers should have access to within them, what constitutes bad behavior/application misuse, and the risks the web applications pose to the business. An application security professional needs to balance security, applications, and business skills, because the WAF policies they write and manage touch all three disciplines.

The tuning process involves a lot of trial and error, figuring out which changes have unintended consequences like adding attack surface or breaking application functionality, and which are useful for protecting applications from emerging attacks. You need dual tuning efforts, one for positive rules which must be updated when new application functionality is introduced, and another for negative rules which protect applications against emerging attacks.

By the time a WAF is deployed customers should be comfortable creating whitelists for applications, having gained a decent handle on application functionality and leveraging the automated WAF learning capabilities. It’s fairly easy to observe these policies in monitor-only mode, but there is still a bit of nail-biting as new capabilities are rolled out. You’ll be waiting for users to exercise a function before you know if things really work, after which reviewing positive rules gets considerably easier.

Tuning and keeping negative security policies current still relies heavily on WAF vendor and third-party assistance. Most enterprises don’t have research groups studying emerging attack vectors every day. These knowledge gaps, regarding how attackers work and cutting-edge attack techniques, create challenges when writing specific blacklist policies. So you are will depend on your vendor for as long as you use WAF, which is why we stress finding a vendor who acts as a partner and building support into your contract.

As difficult as WAF management is, there is hope on the horizon, as firms embrace continuous deployment and DevOps, and accept daily updates and reconfiguration. These security teams have no choice but to build & test WAF policies as part of their delivery processes. WAF policies must be generated in tandem with new application features, which requires security and development teams to work shoulder-to-shoulder, integrating security as part of release management. New application code goes through several layers of functional testing and WAF rules get tested as code goes into a production environment, but before exposure to the general public.

This integrated release process is called Blue-Green deployment testing. In this model both current (Blue) and new (Green) application code are run, on their own servers, in parallel in a fully functional production environment, ensuring applications run as intended in their ‘real’ environment. The new code is gated at the perimeter firewall or routers, limiting access to in-house testers. This way in-house security and application teams can verify that both the application and WAF rules function effectively and efficiently. If either fails the Green deployment is rolled back and Blue continues on. If Green works it becomes the new public production copy, and Blue is retired. It’s early days for DevOps, but this approach enables daily WAF rule tuning, with immediate feedback on iterative changes. And more importantly there are no surprises when updated code goes into production behind the WAF.

WAF management is an ongoing process – especially in light of the dynamic attack space blacklists addresses, false-positive alerts which require tuning your ruleset, and application changes driving your whitelist. Your WAF management process needs to continually learn and catalog user and application behaviors, collecting metrics as part of the process. Which metrics are meaningful, and which activities you need to monitor closely, differs between customers. The only consistency is that you cannot measure success without logs and performance metrics. Reviewing what has happened over time, and integrating that knowledge into your policies, is key to success.

Machine Learning

At this point we need to bring “machine learning” into the discussion. This topic generates confusion, so let’s first discuss what it means to us. In its simplest form, machine learning is looking at application usage metrics to predict bad behavior. These algorithms examine data including stateful user sessions, user behavior, application attack heuristics, function misuse, and high error rates. Additional data sources include geolocation, IP address, and known attacker device fingerprints (IoC). The goal is to detect subtler forms of application misuse, and catch attacks quickly and accurately. Think of it as a form of 0-day detection. You want to spot behavior you know is bad, even if you haven’t seen that kind of badness before.

Machine learning is a useful technique. Detecting attacks as they occur is the ideal we strive for, and automation is critical because you cannot manually review all application activity to figure out what’s an attack. So you’ll need some level of automation, to both scale scarce resources and fit better into new continuous deployment models.

But it is still early days for this technology – this class of protection has a ways to go for maturity and effectiveness. We see varied success: some types of attacks are spotted, but false positive rates can be high. And we are not fans of the term “machine learning” for this functionality, because it’s too generic. We’ve seen some vendors misrepresent their old manual functions as “machine learning” in an attempt to put lipstick on a … well, you know.

These entirely predictable marketing shenanigans make it very difficult for customers to compare apples to apples when selecting a WAF, and it is doubly difficult for people without deep background in security and application development to know which capabilities will be useful. For a real comparison you need vendors to show you how these features make WAF policies easier and better in practice. If these capabilities help tune your WAF policies without a lot of work on your part, you’re getting huge value. If it’s just another way to look at data, and you still need to figure out how to use the data to tune your rules, it’s not worth it.

Security Analytics and Threat Intelligence

We have done a lot of research into the impact of threat intelligence on security operations. You can just Google “Securosis threat intelligence” for enough reading to keep you busy for weeks. We believe in TI because it enables you to benefit from the misfortune of others and learn from attacks being used against other organizations.

In a WAF context these services monitor “threat actors” across the globe, collecting events from thousands of organizations. These events are fed into large data warehouses and mined for application misuse patterns. This data also enables vendors to determine which ‘users’ are regularly scraping content or sending malicious requests to identify devices, domains, and IP addresses involved in attacks. Additionally, security research teams monitor adversary activity and how threat actors are evolving their tactics, techniques, and procedures (TTPs). Patterns and TTPs are usually consistent for each threat actor, so you can look for likely threat actors’ patterns in your applications.

TI offers two key offerings. The first is the source IP addresses of the people and bots involved in attacking web sites – perhaps even those actively attacking your site – which you can incorporate into a blacklist and block. This list changes continually as new devices are compromised and altered to probe and attack web sites, so the feed needs to be directly integrated into your WAF to block dynamically.

The second offering provides the types of exploits and malicious requests currently being seen on other sites, with the feed either providing attack details, or (directly from your WAF vendor) a rule to detect and block a set of attacks. These services help you evolve your rules without watching the entire Internet, or needing to employ people who understand how the attacks work and how to block them.

It’s not exactly a crystal ball, but TI gives you a heads-up about attacks which may be coming your way.

Putting it all together

Once your WAF is deployed you need a strong operational process to keep it up-to-date and current. This requires a consistent and reliable testing process to keep pace with application changes. You also can leverage more automated techniques like machine learning (internal analysis of your own applications), and threat intelligence (external analysis of attacks on other organizations), to scale your processes and increase accuracy.

We won’t pretend WAF is a “set it and forget it” technology. Work is needed before, during, and after deployment. It requires a strong underlying process to make sure you not only get a quick win to maximize short-term value, but also derive long-term sustainable benefit from your WAF investment.