As we have mentioned throughout this series, the purpose of a WAF is to protect web facing applications from attacks. We can debate build-security-in versus bolt-security-on ad infinitum, but ultimately the answer is both. In the last post we discussed how to build and maintain WAF policies to protect applications, but you also need to adapt your development process to incorporate knowledge of typical attack tactics into code development practices to address application vulnerabilities over time. This involves a two-way discussion between WAF administrators and developers. Developers do their part helping security folks understand applications, what input values should look like, and what changes are expected in upcoming releases. This ensures the WAF rules remain in sync with the application.

Granted, not every WAF user will want to integrate their application development and WAF management processes. But separation limits the effectiveness of both the WAF and puts the application at risk. At a minimum, developers (or the DevOps group) should have ongoing communications with WAF managers to avoid having the WAF complicate application deployment and to keep the WAF from interfering with normal application functions. This collaboration is critical, so let’s dig into how this lifecycle should work.

Web applications change constantly, especially given increasingly ‘agile’ development teams – some pushing web application changes multiple times per week. In fact many web application development teams don’t even attempt to follow formal release cycles – effectively running an “eternal beta cycle”. The team’s focus and incentives are on introducing new features as quickly as possible to increase customer engagement. But this doesn’t help secure applications, and poses a serious challenge efforts to keep WAF policies current and effective.

The greater the rate of application change, the harder it is to maintain WAF policies. This simple relationship seriously complicates one of the major WAF selling points: its ability to implement positive security policies based on acceptable application behavior. As we explained in the last post, whitelist policies enumerate acceptable commands and their associated parameters. The WAF learns about the web applications it protects by monitoring user activity and/or by crawling application pages, determining which need protection, which serve static and/or dynamic content, the data types and value ranges for page variables, and other aspects of user sessions. If the application undergoes constant change, the WAF will always be behind, introducing a risky gap between learning and protecting. New application behavior isn’t reflected in WAF policies, which means some new legitimate requests will blocked, and some illegal requests will be ignored. That makes WAF much less useful.

To mitigate these issues we have identified a set of critical success factors for integrating with the SDLC (software development lifecycle). When we originally outlined this process, we mentioned the friction between developers and security teams and how this adversely effects their working relationship. Our goal here is to help set you on the right path and prevent the various groups from feuding like the Hatfields and McCoys – trust us, it happens all to often. Here’s what we recommend:

1. Executive Sponsorship: If, as an organization, you can’t get developers and operations in the room with security, the WAF administrators are stuck on a deserted island. It’s up to them to figure out what each application is supposed to do and how it should be protected, and cannot keep up without sufficient insight or visibility. Similarly, if the development team can say ‘no’ to the security team’s requests, they usually will – they are paid to ship new features, not to provide security. _So either security **is* important, or it isn’t._ To move past a compliance-only WAF, security folks need someone up the food chain – the CISO, the CIO, or even the CEO – to agree that the velocity of feature evolution must give some ground to address operational security. Once management has made that commitment, developers can justify improving security as part of their job. It is also possible – and in some organizational cultures advisable – to include some security into the application specification. This helps guarantee code does not ship until it meets minimum security requirements – either in the app or in the WAF.
2. Establish Expectations: Here all parties learn what’s required and expected to get their jobs done with minimum fuss and maximum security. We suggest you arrange a sit-down with all stakeholders (operations, development, and security) to establish some guidelines on what really needs to happen, and what would be nice to have. Most developers want to know about broken links and critical bugs in the code, but they get surly when you send them thousands of changes via email, and downright pissed when all requests relate to the same non-critical issue. It’s essential to get agreement on what constitutes a critical issue and how critical issues will be addressed among the pile of competing critical requirements. Set guidelines in advance so there are no arguments when issues arise. Similarly, security people hate it when a new application enters production on a site they didn’t know existed, or significant changes to the network or application infrastructure break the WAF configuration. Technically speaking, each party removes work the other does not want to do, or does not have time to do, so position these discussions as mutually beneficial. A true win-win – or at least a reduction in aggravation and wasted time.
3. Security/Developer Integration Points: The integration points define how the parties share data and solve problems together. Establish rules of engagement for how DevOps works with the WAF team, when they meet, and what automated tools will be used to facilitate communication. You might choose to invite security to development scrums, or a member of the development team could attend security meetings. You need to agree upon a communication medium that’s easy to use, establish a method for getting urgent requests addressed, and define a means of escalation for when they are not. Logical and documented notification processes need to be integrated in the application development lifecycle to ensure the security team understands application changes so they can adjust polices in advance. There must be a logical and documented process for application deployment that signals when to update WAF policies and propagate revisions across WAF systems. In those rare cases where a pre-production test environment uses a WAF to help test deployment, take this opportunity to have the WAF learn about the application changes. Don’t forget the beginning and end of the process either – including design, development, and pre-launch validation. Without formal security involvement, inevitably these steps get skipped – especially at crunch time. Go back to point 1 above and start over again if that happens.
4. Automation: This is where you implement and automate the results from steps 2 and 3. Once the initial work is done, these processes need not be painful, and don’t even need to be face to face. After the initial meetings, most of the cooperative efforts can be via less formal mechanisms – email, bug tracking software, trouble ticket systems, or even source code management. Even with executive sponsorship and clearly documented integration points, without sufficient automation of security functions (scanning, testing, regression, etc.), things will be difficult to operationalize. Look to make things easy where possible. Keep in mind the importance of the WAF team filtering out noise before it hits the development team, and of focus on actionable information. On the other hand, the development teams needs to provide enough information so the WAF manager knows what’s going on and can get their job done.
5. Feedback Loops: Once the process is established look for ways to improve it over time. We have seen, after the first few development cycles have passed, the value of discussing ways to make the process easier and more efficient for all parties. Don’t be bashful about sharing what you learned. For example, if the WAFs blocked attacks that could be easily mitigated with a simple change to the application, it’s in everyone’s best interest to share that data. Our research also highlighted the value of this collaboration as the application development team matured their code, which in turn helped the WAF perform better. Nothing turns off either side faster than feeling unappreciated, and this is the time to show your value and show how the process can make things easier. Sure, that’s soft and straight out of a new-age success manual, but it’s still true.

Our focus for this post is on the relationship between app developers and WAF administrators. But many people who end up managing WAFs have some development background because it’s difficult to write WAF policies without understanding the intricacies of web communication protocols and how web services work. And some of the folks we spoke with had no trouble integrating into the DevOps team, partially because their development background provided credibility.

Next we will to cover issues with WAF deployment and how to avoid exposing applications – and the WAF itself – to attackers.