Web Application Firewalls

Building a Web Application Security Program, Part 8: Putting It All Together

By Adrian Lane

'Whew! This is our final post in this series on Building a Web Application Security Program (Part 1, Part 2, Part 3, Part 4, Part 5, Part 6, Part 7), and it's time to put all the pieces together. Here are our guidelines for designing a program that meets the needs of your particular organization. Web application security is not a "one size fits all" problem. The risks, size, and complexity of the applications differ, the level of security awareness among team members varies, and most importantly the goals of each organization are different.

In order to offer practical advice, we needed to approach program development in terms of typical goals. We picked three use cases to represent common challenges organizations face with web app security, and will address those use cases with appropriate program models. We discuss a mid-sized firm tackling a compliance mandate for the first time, a large enterprise looking to improve security across customer-facing applications, and a mid-to-large organization dealing with security for internal applications. Each perspective has its own drivers and assumptions, and in each scenario different security measures are already in place, so the direction of each program will be different. Since we've been posting this over a series of weeks, before you dig in to this post we recommend you review Part 4: The Web Application Security Lifecycle which talks about all tools in all phases. First we describe the environment for each case, then overall strategy and specific recommendations.

Large Enterprise with Customer Facing Web Applications

For our first scenario, let's consider a large enterprise with multiple customer-facing web applications. These applications evolved to offer core business functions and are a principal contact point with customers, employees, and business partners. Primary business drivers for security are fraud reduction, regulatory compliance, and service reliability as tangible incentives. Secondary factors are breach preparedness, reputation preservation, and asset protection secondary -- all considerations for security spending. The question is not whether these applications need to be secured, but how. Most enterprises have a body of code with questionable security, and let's be totally honest here- these issues are flaws in your code. No single off-the-shelf product is going to magically make your application secure, so you invest not only in third-party security products, but also in improvements to your own development process which improve the product with each new release.

We assume our fictitious enterprise has an existing security program and the development team has some degree of maturity in their understanding of security issues, but how best to address problems is up for debate. The company will already have a 'security guy' in place, and while security is this guy's or gal's job, the development organization is not tasked with security assessments and problem identification. Your typical CISO comes from a network security background, lacks a secure code development background, and is not part of this effort. We find their security program includes vulnerability assessment tools, and they have conducted a review of the code for typical SQL injection and buffer overflow attacks. Overall, security is a combination of a couple third-party products and the security guy pointing out security flaws which are patched in upcoming release cycles.

Recommendations: The strategy is to include security within the basic development process, shifting the investment from external products to internal products and employee training. Tools are selected and purchased to address particular deficiencies in team skill or organizational processes. Some external products are retained to shield applications during patching efforts.

Training, Education, and Process Improvements: The area where we expect to see the most improvement is the skill and awareness of the web application development team. OWASP's top flaws and other sources point out issues that can be addressed by proper coding and testing ... provided the team knows what to look for. Training helps staff find errors and problems during code review, and iteratively reduces flaws through the development cycle. The development staff can focus on software security and not rely on one or two individuals for security analysis.

Secure SDLC: Knowing what to do is one thing, but actually doing it is something else. There must be an incentive or requirement for development to code security into the product, assurance to test for compliance, and product management to set the standards and requirements. Otherwise security issues get pushed to the side while features and functions are implemented. Security needs to be part of the product specification, and each phase of the development process should provide verification that the specification is being met through assurance testing. This means building security testing into the development process and QA test scenarios, as well as re-testing released code. Trained development staff can provide code analysis and develop test scripts for verification, but additional tools to automate and support these efforts are necessary, as we will discuss below.

Heritage Applications: Have a plan to address legacy code. One of the more daunting aspects for the enterprise is how to address existing code, which is likely to have security problems. There are several possible approaches for addressing this, but the basic steps are 1) identification of problems in the code, 2) prioritization on what to fix, and 3) planning how to fix individual issues. Common methods of addressing vulnerabilities include 1) rewriting segments of code, 2) method encapsulation, 3) temporary shielding by WAF ("secure & patch"), 4) moving SQL processing & validation into databases, 5) discontinuing use of insecure features, and 6) introduction of validation code within the execution path. We recommend static source code analysis or dynamic program analysis tools for the initial identification step. These tools are cost-effective and suitable for scanning large bodies of code to locate common risks and programming errors. They detect and prioritize issues, and reduce human error associated with tedious manual scanning by internal or external parties. Analysis tools also help educate staff about issues with certain languages and common programming patterns. The resulting arguments over what to do with 16k insecure occurrences of IFRAME are never fun, but acceptance of the problem is necessary before it can be effectively addressed.

External Validation: Periodic external review, through vulnerability assessment, penetration testing or source code review, is highly recommended . Skilled unbiased professionals with experience in threat analysis often catch items items which slip by internal scans, and can help educate development staff on different threat vectors. Plan on external penetration testing on a quarterly or biannual basis- their specific expertise and training goes far beyond the basics threats, and trained humans monitoring the output of sophisticated tools are very useful for detecting weaknesses that a hacker could exploit. We recommend the use of static testing tools for internal testing of code during the QA sweep, with internal penetration testing just prior to deployment so they can fully stress the application without fear of corrupting the production environment. Major releases should also undergo an external penetration test and review before deployment.

Blocking: This is one area that will really depend upon the specifics of your organization. In the enterprise use case, plan on using a Web Application Firewall. They provide basic protection and give staff a chance to remove security issues from the application. You may find that your code base is small and stable enough that you do not need WAF for protection, but for larger organizations this is not an option. Development and patching cycles are too long and cumbersome to counter threats in a reasonable timeframe. We recommend WAF + VA because in combination, they can relieve your organization from much of the threat research and policy development work for firewall rules. If your staff has the skill and time to develop WAF policies specific to your organization, you get customized policies at slightly greater expense in development costs. WAF isn't cheap, so we don't take this recommendation lightly, but it provides a great deal of flexibility in how and when threats are dealt with, today and as new threats evolve.

We recommend you take steps to improve security in every part of the development process. We are focused on improvements to the initial phases of development, as the impact of effort is greatest here, but we also recommend at the very least external assistance, and if budget allows, blocking. These later recommendations fill in other areas that need coverage, with penetration testing and web application firewalls. The risks to the enterprise are greater, the issues to overcome are more complex, and the corresponding security investment will therefore be larger. This workflow process should be formally documented for each stage of an application's lifecycle- from development through ongoing maintenance- with checkpoints for major milestones. Security shouldn't, and can't, be responsible for each stage, but should carry responsibility for managing the program and making sure the proper process is followed and maintained.

Mid-sized firm and PCI Compliance

If we are discussing web application security and compliance, odds are we are talking about the Payment Card Industry's Data Security Standard (PCI-DSS). No other compliance standard specifies steps to secure web applications like the PCI standard does. We can grouse about ambiguities and ways that it could be improved, but PCI is clearly the most widespread driver for web application security today, which is why our second use case is a mid-sized firm that needs to secure its web applications to satisfy PCI-DSS.

The profile for our company is a firm that generates a large portion of their revenue through Internet sales, and recent growth has made them a Tier 3 merchant. The commerce web site is relatively new (< 3 years) and the development team is small and not trained in security. Understanding the nuances of how criminals approach breaking code is not part of the team's skill set. PCI compliance is the mandate, and the team knows that they are both missing the basic requirements and susceptible to some kinds of attacks. The good news is that the body of code is small, and the web application accounts for a significant portion of the company's revenue, so management is supporting the effort.

In a nutshell, the PCI Data Security Standard is a security program specifically for companies that process credit card transactions for Internet commerce. In terms of compliance regulations, PCI-DSS requirements are clearer than most, making specific requirements for security tools and processes around credit card data. However, a company may also it has satisfy the spirit of the requirements in an alternate way, if it can demonstrate that the concern has been addressed. We will focus on the requirements outlined in sections 6.6 & 11.3, but will refer to sections 10 and compensating controls as well.

Recommendations: Our strategy focuses on education and process modifications to bring security into the development lifecycle. Additionally, we suggest assessment or penetration testing services to quickly identify areas of concern. Deploy WAF to address the PCI requirement immediately. Focus on the requirements to start, but plan for a more general program, and use compensating controls as your organization evolves. Use outside help and education to address immediate gaps, both in PCI compliance and more general application security.

Training, Education, and Process Improvements: Once again, we are hammering on education and training for the development team, including project management and quality assurance. While it takes time to come up to speed, awareness by developers helps keep security issues out of the code, and is cost-effective for securing the applications. Altering the process to accommodate fixing the code is essentially free, and code improvements become part of day to day efforts. With a small code base, education and training are easy ways to reap significant benefits as the company and code base grow.

External Help: Make friends with an auditor, or hire one as a consultant to help prepare and navigate the standard. While this is not a specific recommendation for any single requirement in PCI; auditors provide an expert perspective, help address some of the ambiguity in the standard, and assist in strategy and trade-off evaluations to avoid costly missteps.

Section 11.3.2: Section 11.3 mandates penetration testing of the network and the web application. In this case we recommend external penetration testing as an independent examination of the code. It is easy to recommend penetration testing, and not because it is required in the DSS specification, rather the independent & expert review of your application behavior will closely mimic the approach hackers will take. We also anticipate budget will require you make a choice between WAF and code reviews in section 6.6, so this will provide the necessary coverage. Should you use source code reviews, one could argue that acts as a compensating control for this section, but our recommendation is to stick with the external penetration testing. External testers provide much more than just a list of specific flaws, but also identify risky or questionable application behaviors in a near production environment.

Section 6.6: Our biggest debate internally was whether to recommend Web Application Firewall or expert code review to address section 6.6 of the PCI specification. The PCI Security Standards Council recommends that you do both, but it is widely recognized that this is prohibitively expensive. WAF provides a way to quickly meet the letter of 6.6's requirement, if not in spirit, provides basic monitoring, and is a flexible platform to block future attacks. The counter-arguments are significant and include cost, work required to customize policies for the application, and false positives & negatives. Alternatively, a code review by qualified security experts can identify weaknesses in application design and code usage and assist in education of the development team by pointing out specific flaws. Outside review is a very quick way to assess where you are and what you need. Down sides of review include cost, time to identify and fix errors, and that a constantly changing code base presents a moving target and thus requires repeated examinations.

Our recommendation here is deploy a WAF solution. Engaging a team of security professionals to review the code is an effective way to identify issues, but much of the value overlaps with the requirement of Section 11.3.2, periodic penetration testing of the application. The time to fix identified issues (even with a small-to-average body of code), with a development organization which is just coming to terms with security issues, is too long to meet PCI requirements in a timely fashion. Note that this recommendation is specific to this particular fictitious case- in other PCI audit scenarios, with a more experienced staff or a better handle on code quality, we might have made a different recommendation.

Monitoring: Database Activity Monitoring (DAM) is a good choice for Section 10 compliance- specifically by monitoring all access to credit card data. Web applications use a relational database back end to store credit card numbers, transactions, and card related data. DAM products that capture all network and console activity on the database platform provide a focused and cost-effective audit for all access to cardholder data. Consider this option for providing an audit trail for auditors and security personnel.

Internal Web Application Development

Our last use case is an internal web applications that serves employees and partners within a mid-to-large business. While this may not sound like a serious problem, given that companies have on average 1 internal application (web and traditional client/server) per 100 employees, even mid-sized companies have incredible exposure in this area. Using data from workflow, HR, accounting, business intelligence, sales, and other critical IT systems, these internal applications support employees and partners alike. And with access to pretty much all of the data within the company, security and integrity are major concerns. A common assumption is that these systems, behind the perimeter firewall, are not exposed to the same types of attacks as typical web applications, but this assumption has proven disastrous in many cases.

Investment here is motivated by fraud reduction, breach remediation, and avoidance of notification costs- and possibly by compliance. You may find this is difficult to sell to executive management if there is not a compliance mandate and hasn't been a previous breach, but if basic perimeter security is breached these applications need some degree of resiliency rather than blind confidence in network security and access control. TJ Maxx (http://www.tjx.com/) is an excellent illustration of the danger.

Strategy: Determine basic security of the internal application, fix serious issues, and leverage education, training, and process improvements to steadily improve the quality of code. We will assume that budgeting for security in this context is far smaller than for external-facing systems, so look to cooperate between groups and leverage tools and experience.

Vulnerability Assessment and Penetration Testing: Scanning web applications for significant security, patch and configuration issues is a recommended first step in determining if there are glaring issues. Assessment tools are a cost-effective way to establish baseline security and ensure adherence to minimum best practices. Internal penetration testing will help determine the overall potential risk and prioritization, but be extremely cautious of testing live applications.

Training, Education, and Process Improvements: These may be even more important in this scenario than in our other use cases, where the business justification provides incentive to invest in security, internal web applications may not get the same degree of attention. For these applications that have a captive audience, developers have greater controls over the types of environments that they support what can be required in terms of authentication. Use these freedoms to your advantage. Training should focus on common vulnerabilities within the application stack that is being used, and give critical errors the same attention that top priority bugs would receive. Verify that these issues are tested, either as part of the VA sweep, or a a component of regression testing.

Monitoring: Monitoring for suspicious activity and system misuse is a cost-effective way to detect issues and react to them. We find WAF solutions are often too expensive for deployment across hundreds of internal applications distributed across a company, and a more cost-effective approach to collecting and analyzing activity is highly recommended. Monitoring software that plugs into the web application is often very effective for delivering some intelligence at low cost, but the burden of analyzing the data then falls on development team members. Database Activity Monitoring can effectively focus on critical information at the back end and is more mature than Web Application Monitoring.

This segment of the series took much longer to write than we originally anticipated, as our research gave us conflicting answers to some questions, making our choices were far from easy. Our recommendations really depend upon the specifics of the situation and the organization. We approached this with use cases to demonstrate how the motivating factors; combined with the current state of web security; really guides the selection of tools, services, and process changes.

We found in every case that security as part of the overall development process is the most cost-effective and least disruptive to normal operations, and is our principal recommendation for each scenario. However, as transformation of a web application does not happen overnight, we rarely have the luxury of waiting for the development team to address all security issues is not realistic; in the meantime, external third-party services and products are invaluable for dealing with immediate security challenges.

–Adrian Lane

Building A Web Application Security Program: Part 7, Secure Operations

By Adrian Lane

We've been covering a heck of a lot of territory in our series on Building a Web Application Security Program (see Part 1, Part 2, Part 3, Part 4, Part 5, and Part 6). So far we've covered secure development and secure deployment, now it's time to move on to secure operations. This is the point where the application moves out of development and testing and into production.

Keep in mind that much of what we've talked about until now is still in full effect- just because you have a production system doesn't mean you throw away all your other tools and processes. Updates still need to go through secure development, systems and applications are still subject to vulnerability assessments and penetration testing (although you need to use a different process when testing live applications vs. staging), and configuration management and ongoing secure management are more important than ever before.

In the secure operations phase we add two new technology categories to support two additional processes- Web Application Firewalls (WAF) for shielding from certain types of attacks, and monitoring at the application and database levels to support auditing and security alerting.

Before we dig in, we also want to thank everyone who has been commenting on this series as we post it- the feedback is invaluable, and we're going to make sure everyone is credited once we put it into whitepaper format.

Web Application Firewalls (WAF)

The role of a web application firewall is to sit in front of or next to a web application, monitoring application activity, and alerting or blocking on policy violations. Thus it potentially serves two functions- as a detective control for monitoring web activity, and as a preventative control for blocking activity.

A web application firewall is a firewall specifically built to watch HTTP requests and block those that are malicious or don't comply with specific rules. The intention is to catch SQL injection, Cross Site Scripting (XSS), directory traversal, and various HTTP abuses, as well as authorization, request forgeries, and other attempts to alter web application behavior. The WAF rules and policies are effectively consistency checks, for both the HTTP protocol and application functionality. WAFs can alert or block activity based on general attack signatures (such as a known SQL injection attack for a particular database), or application-specific signatures for the web application being protected.

WAF products examine inbound and outbound HTTP requests, compare these with the firewall rules, and create alerts for conditions of concern. Finally, the WAF selects a disposition for the traffic: 1) let it pass, 2) let it pass but audit, 3) block the transaction, or 4) reset the connection.

WAFs typically network appliances. They are normally placed in-line as a filter for the application (proxy mode); or 'out-of-band', receiving traffic from a mirror or SPAN port. In the former scenario, all inbound and outbound requests are intercepted and inspected prior to the web server receiving the request or user receiving the response, reducing load on the web application. For SSL traffic, inline WAFs also need to proxy the SSL connection from the browser so it can decrypt and inspect traffic before it reaches the web server, or after it leaves the web server for responses. In out-of-band mode, there are additional techniques to monitor the encrypted connections by placing a copy of the server certificate on the WAF, or positioning it behind an SSL concentrator. Some vendors also provide WAF capabilities via plug-ins for specific platforms, rather than through external devices.

The effectiveness of any WAF is limited by the quality of the policies it is configured to enforce. Policies are important not merely to ability to recognize and stop known/specific attacks, but also for flexibly dealing with ambiguous and unknown threat types, while keeping false positives manageable and without preventing normal transaction processing. The complexity of the web application, combined with the need for continuous policy updates, and the wide variety of deployment options to accommodate, pose a complex set of challenges for any WAF vendor. Simply dropping a WAF in front of your application and turning on all the default rules in blocking mode is a recipe for disaster. There is no way for black box to effectively understand all the intricacies of a custom application, and customization and tuning are essential for keeping false positives and negatives under control.

When deployed in monitoring mode, the WAF is used in a manner similar to an intrusion detection system (IDS). It's set to monitor activity and generate alerts based on policy violations. This is how you'll typically want to initially deploy the WAF, even if you plan on blocking activity later. It gives you an opportunity to tune the system and better understand application activity before you start trying to block connections. An advantage of monitoring mode is that you can watch for a wider range of potential attacks without worrying that false positives will result in inappropriate blocking. The disadvantages are 1) your incident handlers will spend more time dealing with these incidents and false positives, and 2) bad activity won't be blocked immediately.

In blocking/enforcement mode, the WAF will break connections by dropping them (proxy mode) or sending TCP reset packets (out of band mode) to reset the connection. The WAF can then ban the originating IP, permanently or temporarily, to stop additional attacks from that origin. Blocking mode is most effective when deployed as part of a "shield then patch" strategy to block known vulnerabilities in your application.

When a vulnerability is discovered in your application, you build a specific signature to block attacks on it and deploy that to the WAF (the "shield"). This protects your application as you go back and fix the vulnerable code, or wait for an update from your software provider (the "patch"). The shield then patch strategy greatly reduces potential false positives that interfere with application use and improves performance, but is only possible when you have adequate processes to detect and evaluate these vulnerabilities.

You can combine both approaches by deploying a larger signature set in monitoring mode, but only enabling a few specific policies in blocking mode.

Given these challenges, satisfaction with WAF products varies widely among security professionals who use them. While WAFs are effective against known threats, they are less capable of discovering new issues or handling questionable use cases. Some WAF products are addressing these issues by linking web application firewalls more tightly to the vulnerability assessment process and tools, as we'll discuss in a moment.

Monitoring

Monitoring is primarily used for discovery, both of how an application is used by real users, and also for how it can be misused. The fundamental value of monitoring is to learn what you do not already know- this is important not only for setting up a WAF, but also for tune an application security strategy. Although WAFs provide some level of application activity monitoring, there are three additional ways to monitor web applications, each with a different perspective on application activity:

• Network Monitoring: Monitoring network activity between the user/Internet and the web server. This category includes web application firewalls, intrusion detection systems, packet sniffers, and other external tools. While generic network security and sniffing tools can capture all network traffic, they have much less capability to place it in context and translate network activity into application transactions. Simply viewing HTTP traffic is often insufficient for understanding what users are attempting in an application- this is where interpretation is required. If a solution includes web application specific analysis and the ability to (potentially) audit all web activity, we call it Web Application Monitoring (WAM). While network monitoring is easy to implement and doesn't require application changes, it can only monitor what's going into and coming out of the application. This may be useful for detecting traditional attacks against the application stack, but much less useful than seeing traffic fully correlated to higher-level transactions.
• Application Auditing/Logging: Collection and analysis of application logs and internal activity. Both custom and off-the-shelf applications often include internal auditing capabilities, but there is tremendous variation in what's captured and how it's formatted and stored. While you gain insight into what's actually occurring in the application, not all applications log equally (or at all)- you are limited to whatever the programmers decided to track. For major enterprise applications, such as SAP, we've seen third party tools that either add additional monitoring or can interpret native audits. Log management and SIEM tools can also be used to collect and interpret (to a more limited degree) application activity when audit logs are generated.
• Database Activity Monitoring: DAM tools use a variety of methods to (potentially) record all database transactions and generate alerts on specific policy violations. By monitoring activity between the application and the database, DAM can provide a more precise examination of data elements, and awareness of multi-step transactions which directly correspond to business functions. Some DAM tools have specific plug ins for major application types (e.g., SAP & PeopleSoft) to translate database transactions into application activity. Since the vast majority of web applications run off databases, this is an effective point to track activity and look for policy violations. A full discussion of DAM is beyond the scope of this post, but more information is available in our DAM whitepaper.

WAF can be used for monitoring, but these alternative tools offer a wider range of activity collection, which helps to detect probing which may not be obvious from HTTP requests alone. The focus of web application monitoring is to examine behavior across data sources and provide analysis, recording activity trails and alerting on suspicious events, whereas WAFs are more focused on detection and blocking of known threats. There are significant differences between WAMs and WAFs in areas such as activity storage, aggregation of data from multiple sources, and examination of the collected data, so choosing the best tool depends specifics of the requirement. We must point out that web application monitoring products are not fully mature, and the handful of available products are early in the product evolution cycle.

WAF + VA

Several vendors have begun providing a hybrid model that combines web application vulnerability assessment with a web application firewall. As mentioned earlier, one of the difficulties with a shield then patch strategy is detecting vulnerabilities and building the WAF signatures to defend them. Coupling assessment tools or services with WAFs by feeding the assessment results to the firewall, and having the firewall adjust its policy set accordingly, can makes the firewall more effective. The intention is to fill the gap between exploits discovery/announcement and deployment of tested patches in the application, by instructing the WAF to block access to the particular weakness. In this model the assessment determines that there is a vulnerability and feeds the information to the WAF. The assessment policy contains WAF instructions on what to look for and how to respond. The WAF then dynamically incorporates the policy and protects the application.

This is the last post in this series where we discuss your options at different stages of the application lifecycle. Our next post will discuss which options you should consider, and how to balance your resource expenditures into an overall program.

–Adrian Lane

How The Cloud Destroys Everything I Love (About Web App Security)

By Rich

On Tuesday, Chris Hoff joined me to guest host the Network Security Podcast and we got into a deep discussion on cloud security. And as you know, for the past couple of weeks we've been building our series on web application security. This, of course, led to all sorts of impure thoughts about where things are headed. I wouldn't say I'm ready to run around in tattered clothes screaming about the end of the Earth, but the company isn't called Securosis just because it has a nice ring to it.

If you think about it a certain way, cloud computing just destroys everything we talk about for web application security. And not just in one of those, "oh crap, here's one of those analysts spewing BS about something being dead" ways. Before jumping into the details, in this case I'm talking very specifically of cloud based computing infrastructure- e.g., Amazon EC2/S3. This is where we program our web applications to run on top of a cloud infrastructure, not dedicated resources in a colo or a "traditional" virtual server. I also sprinkle in cloud services- e.g., APIs we can hook into using any application, even if the app is located on our own server (e.g., Google APIs).

Stealing from our yet incomplete series on web app sec and our discussions of ADMP, here's what I mean:

• Secure development (somewhat) breaks: we're now developing on a platform we can't fully control- in a development environment we may not be able to isolate/lock down. While we should be able to do a good job with our own code, there is a high probability that the infrastructure under us can change unexpectedly. We can mitigate this risk more than some of the other ones I'll mention- first, through SLAs with our cloud infrastructure provider, second by adjusting our development process to account for the cloud. For example, make sure you develop on the cloud (and secure as best you can) rather than completely developing in a local virtual environment that you then shift to the cloud. This clearly comes with a different set of security risks (putting development code on the Internet) that also need to be, and can be, managed. Data de-identification becomes especially important.
• Static and dynamic analysis tools (mostly) break: We can still analyze our own source code, but once we interact with cloud based services beyond just using them as a host for a virtual machine, we lose some ability to analyze the code (anything we don't program ourselves). Thus we lose visibility into the inner workings of any third party/SaaS APIs (authentication, presentation, and so on), and they are likely to randomly change under our feet as the providing vendor continually develops them. We can still perform external dynamic testing, but depending on the nature of the cloud infrastructure we're using we can't necessarily monitor the application during runtime and instrument it the same way we can in our test environments. Sure, we can mitigate all of this to some degree, especially if the cloud infrastructure service providers give us the right hooks, but I don't hold out much hope this is at the top of their priorities. (Note for testing tools vendors- big opportunity here).
• Vulnerability assessment and penetration testing... mostly don't break: So maybe the cloud doesn't destroy everything I love. This is one reason I like VA and pen testing- they never go out of style. We still lose some ability to test/attack service APIs.
• Web application firewalls really break: We can't really put a box we control in front of the entire cloud, can we? Unless the WAF is built into the cloud, good luck getting it to work. Cloud vendors will have to offer this as a service, or we'll need to route traffic through our WAF before it hits the back end of the cloud, negating some of the reasons we switch to the cloud in the first place. We can mitigate some of this through either the traffic routing option, virtual WAFs built into our cloud deployment (we need new products for it), or cloud providers building WAF functionality into their infrastructure for us.
• Application and Database Activity Monitoring break: We can no longer use external monitoring devices or services, and have to integrate any monitoring into our cloud-based application. As with pretty much all of this list it's not an impossible problem, just one people will ignore. For example, I highly doubt most of the database activity monitoring techniques will work in the cloud- network monitoring, memory monitoring, or kernel extensions. Native audit might, but not all database management systems provide effective audit logs, and you still need a way to collect them as your app and db shoot around the cloud for resource optimization.

I could write more about each of these areas, but you get the point. When we run web applications on cloud based infrastructure, using cloud based software services, we break much of the nascent web application security models we're just starting to get our fingers around. The world isn't over*, but it sure just moved out from under our feet.

*This doesn't destroy the world, but it's quite possible that the Keanu Reeves version of The Day the Earth Stood Still will.

–Rich

WAF vs. Secure Code vs. Dead Fish

By Rich

I've been slowly catching up on my reading after months of near-nonstop travel, and this post over at Imperviews caught my eye. Ignoring the product promotion angle, it raises one of my major pet peeves these days. I'm really tired of the Web Application Firewall vs. secure coding debate, never mind using PCI 6.6 to justify one over the other for security effectiveness. It's like two drunk cajuns arguing over the relative value of shrimp or pork in gumbo- you need both, and if either is spoiled the entire thing tastes like sh&t. You also can't dress up the family dog and fish in a pinch, use them as substitutes, and expect your kids to appreciate either the results or use of resources (resulting gumbo or the loss of Rover).

Here's the real deal-

Secure coding is awesome and you need to adopt a formal process if you produce any meaningful volume of code. But it takes a ton of resources to get to the old code (which you should still try to do), and can't account for new vulnerability classes. Also, people screw up... even when there are multiple layers to detect or prevent them from screwing up.

On the other hand, WAFs need to get a hell of a lot better. We're seeing some positive advancements, as I've written about before, but they still can't stop all vulnerabilities, can't stop logic flaws and certain other categories of attack, can't deal with the browser end, and I hear a lot of complaints about tuning (while I think liking WAFs with Vulnerability Assessment is a great start on this problem, we're just at the start of that race).

I absolutely hate to tell you to buy more than you need, but if you have a major web presence you likely need both these days, in the right combination (plus a few other things).

If you don't have the resources for both, I suggest two options. First, if you are really on the low end of resources, use hosted applications and standard platforms as much as possible to limit your custom coding. Then, make sure you have kick ass backups. Finally, absolutely minimize the kinds of information and transaction you expose to the risk of web attacks- drop those ad banners, minimize collecting private information, and validate transactions on the back end as much as possible.

If you do have some more resources available, I suggest starting with a vulnerability assessment (not a cheap ass bare-bones PCI scan, but something deeper), and using that to figure out where to go next.

Yes- we are eating our own dog food on this one. The blog is hosted using a standard platform. We know it's vulnerable, so we've minimized the attack surface as best we can and make sure we have backups of all the content. I've been pleasantly surprised we haven't been nailed yet, but I expect it to happen eventually. None of our sensitive operations are on that server, and we've pulled email and our other important stuff in house.

Early next year we're going to be launching some new things, and we will again go with remote hosting (on a more powerful platform). This time, we are switching to a more secure platform than Wordpress (Expression Engine) and will pay for a full vulnerability assessment and penetration test (at least annually, or when any major new components come online). We may perform some financial transactions, and we'll use an external provider for that. A WAF is out of budget for us, so we'll focus on minimizing our exposure and manually fixing problems discovered by ongoing assessments. We also plan on using as little custom code as possible.

But seriously- I'm tired of this debate. Both options have value, they aren't exclusionary, and which you need depends on what you are doing and how many resources you have.

Eventually we'll get a better lock on this problem, but that's a few years out.

–Rich

The Future Of Application And Database Security: Part 2, Browser To WAF/Gateway

By Rich

Since Friday is usually "trash" day (when you dump articles you don't expect anyone to read) I don't usually post anything major. But thanks to some unexpected work that hit yesterday, I wasn't able to get part 2 of this series out when I wanted to. If you can tear yourself away from those LOLCatz long enough, we're going to talk about web browsers, WAFs, and web application gateways. These are the first two components of Application and Database Monitoring and Protection (ADMP), which I define as:

Products that monitor all activity in a business application and database, identify and audit users and content, and, based on central policies, protect data based on content, context, and/or activity.

Browser Troubles

As we discussed in part 1, one of the biggest problems in web application security is that the very model of the web browsers and the World Wide Web is not conducive to current security needs. Browsers are the ultimate mashup tool- designed to take different bits from different places and seamlessly render them into a coherent whole. The first time I started serious web application programming (around 1995/96) this blew my mind. I was able to embed disparate systems in ways never before possible. And not only can we embed content within a browser, we can embed browsers within other content/applications. The main reason, as a developer, I converted from Netscape to IE was that Microsoft allowed IE to be embedded in other programs, which allowed us to drop it into our thick VR application. Netscape was stand alone only; seriously limiting its deployment potential.

This also makes life a royal pain on the security front where we often need some level of isolation. Sure, we have the same-origin policy, but browsers and web programming have bloated well beyond what little security that provides. Same-origin isn't worthless, and is still an important tool, but there are just too many ways around it. Especially now that we all use tabbed browsers with a dozen windows open all the time. Browsers are also stateless by nature, no matter what AJAX trickery we use. XSS and CSRF, never mind some more sophisticated attacks, take full advantage of the weak browser/server trust models that result from these fundamental design issues.

In short, we can't trust the browser, the browser can't trust the server, and individual windows/tabs/sessions in the browser can't trust each other. Fun stuff!

WAF Troubles

I've talked about WAFs before, and their very model is also fundamentally flawed. At least how we use WAFs today. The goal of a WAF is, like a firewall, to drop known bad traffic or only allow known good traffic. We're trying to shield our web applications from known vulnerabilities, just like we use a regular firewall to block ports, protocols, sources, and destinations. Actually, a WAF is closer to IPS than it is to a stateful packet inspection firewall.

But web apps are complex beasts; every single one a custom application, with custom vulnerabilities. There's no way a WAF can know all the ins and outs of the application behind it, even after it's well tuned. WAFs also only protect against certain categories of attacks- mostly some XSS and SQL injection. They don't handle logic flaws, CSRF, or even all XSS. I was talking with a reference yesterday of one of the major WAFs, and he had no trouble slicing through it during their eval phase using some standard techniques.

To combat this, we're seeing some new approaches. F5 and WhiteHat have partnered to feed the WAF specific vulnerability information from the application vulnerability assessment. Imperva just announced a similar approach, with a bunch of different partners.

These advances are great to see, but I think WAFs will also need to evolve in some different ways. I just don't think the model of managing all this from the outside will work effectively enough.

Enter ADMP

The idea of ADMP is that we build a stack of interconnected security controls from the browser to the database. At all levels we both monitor activity and include enforcement controls. The goal is to start with browser session virtualization connected to a web application gateway/WAF. Then traffic hits the web server and web application server, both with internal instrumentation and anti-exploitation. Finally, transactions drop to the database, where they are again monitored and protected.

All of the components for this model exist today, so it's not science fiction. We have browser session virtualization, WAFs, SSL-VPNs (that will make sense in a minute), application security services and application activity monitoring, and database activity monitoring. In addition to the pure defensive elements, we'll also tie in to the applications at the design and code level through security services for adaptive authentication, transaction authentication, and other shared services (happy Dre? :) ). The key is that this will all be managed through a central console via consistent policies.

In my mind, this is the only thing that makes sense. We need to understand the applications and the databases that back them. We have to do something at the browser level since even proper parameterization and server side validation can't meet all our needs. We have to start looking at transactions, business context, and content, rather than just packets and individual requests.

Point solutions at any particular layer have limited effectiveness. But if we stop looking at our web applications as pieces, and rather design security that addresses them as a whole, we'll be in much better shape. Not that anything is perfect, but we're looking at risk reduction, not risk elimination. A web application isn't just a web server, just some J2EE code, or just a DB- it's a collection of many elements working together to perform business transactions, and that's how we need to look at them for effective security.

The Browser and Web Application Gateway

A little while back I wrote about the concept of browser session virtualization. To plagiarize myself and save a little writing time so I can get my behind to happy hour:

What we ideally need is a way to completely isolate our content in the browser. One way to do this is session virtualization, pioneered by GreenBorder, who was later acquired by Google (the GreenBorder site is just in support mode now). When a user connects to our site, we push down some code to create a virtual environment in the browser that we strictly control. We wall off that session, which could just be an isolated iFrame in a page, so that it only accesses content we send it. Basically, we break the normal browser model and hijack what we need. This would, for example, help stop CSRF since other browser elements won't be able to trigger a connection to our application. Done right, it limits man in the middle attacks, even if the user authorizes a bad digital certificate. To work properly, this needs to be tied to a gateway that controls the session. While we could do it from the web/app server itself, I suspect we'll see this as a web application firewall feature, just as we see similar features from SSL-VPNs. I think isolated WAFs have a very limited lifespan, but this is exactly the kind of feature that will extend their value. Better yet, we can tie this in to our Application and Database Monitoring and Protection to build a browser-to-database protected path. We can completely track a transaction or piece of content from the database server to the browser and back. We could even use this to isolate out potentially "bad" content in an in-browser sandbox. For example, it could be a way to enable all those social networking widgets in a more controlled way but locking in potentially bad content instead of known good. Will this protect us from keystroke sniffers or a completely compromised host? Nope, but it will definitely help with a large number of our current browser security issues. If we combine it with full ADMP and additional methods like transaction authentication, I think we can regain a bit of control of the web application security mess.

Thus we see one migration path for a WAF. A user goes to connect to the application and hits the WAF, which is now more of a Web Application Gateway. The gateway, like an SSL-VPN, sends the session virtualization code down to the browser. We do this outside of the web application for performance reasons. The secure virtual session is established and the gateway then allows communications with the application behind it.

For things like retail and financial sites that include only limited third party content if any, we can monitor activity from the browser through to the application and work within the isolated session. It both improves our ability to control what's being sent to the browser, and gives us a higher degree of confidence that what's coming from the browser is safe. We still validate everything, but since we're tied to the application itself we can validate in the browser and at the gateway before we even hit the app (and further validate there). Since in a controlled environment we know what transactions should be allowed (or not), we have more ability to detect and block "bad" transactions like SQL injection from the user.

In less controlled environments- thing MySpace or Gmail and everything in between- the gateway also becomes a filter for third party content. Like Checkpoint's new ForceField. The gateway filters out, to the best of its ability, harmful third party content coming from third party sites. Basically, it becomes an SSL-VPN for secure browsing.

This is obviously not viable for all sites due to bandwidth considerations, and in those circumstances we'll drop this part and stick to the rest of the ADMP stack, or only virtualize our pieces of content knowing the user is at risk for the third party stuff we're still linking them to.

Future of the WAF, Option 2

I've just described a scenario where the WAF extends into a Secure Web Application Gateway that adds virtualization, encryption, and content filtering. That doesn't mean WAFs won't also still exist in non-virtualized situations, since that will still be a massive volume of sites out there.

For these sites the WAF continues to progress with deeper application integration and application understanding, and works with the elements I'll describe later that will be embedded into the applications and databases. Rather than hanging around outside the application with barely any idea what's going on behind it, the WAF will take its cues from the app, help manage sessions, and monitor activity outside the app to block the few things we know we can pick up at that layer.

Why use the WAF at all? To give us a choke point and offload some of the monitoring and processing that could hurt application performance. Let's be honest- maybe WAFs will eventually go away, but performance problems alone will probably keep next-gen WAFs viable for a while. There are also plenty of things we can now block before they ever hit the application controls, which, by nature of being integrated at the app level, will be more complex and delicate.

But again, by tightly integrating with our other layers, instead of naievely assuming that an external black box will magically solve our problems, we get a much higher level of functionality. Feeding in vulnerability data as we're just starting to do is a good beginning, but once we plug in deeper to the application and database servers we'll get entirely new levels of functionality.

Part 2 Conclusions

What I've described today is how we can build a (more) trusted path from the browser to the face of the application. WAFs will add gateway capabilities, both protecting the applications behind them and the browsers in front of them. Since this won't be the right approach in all circumstances, WAFs will also evolve with tighter integration to the application and other ADMP stack components.

Again, this might sound like little more than the usual analyst fiction, but all the components are here today. Also, I don't expect my predictions to be totally accurate. I'm roughly guessing I'm at 85% or so.

Next week I'll start digging into the application and database. We'll talk about application instrumentation, anti-exploitation, DAM, trusted transaction paths, and shared security services.

–Rich

The Future Of Application And Database Security: Part 1, Setting The Stage

By Rich

I've been spending the past few weeks wandering around the country for various shows, speaking to some of the best and brightest in the world of application and database security. Heck, I even hired one of them. During some of my presentations I laid out my vision for where I believe application (especially web application) and database security are headed. I've hinted at it here on the blog, discussing the concepts of ADMP, the information-centric security lifecycle, and DAM, but it's long past time I detailed the big picture.

I'm not going to mess around and write these posts so they are accessible to the non-geeks out there. If you don't know what secure SDLC, DAM, SSL-VPN, WAF, and connection pooling mean, this isn't the series for you. That's not an insult, it's just that this would drag out to 20+ pages if I didn't assume a technical audience.

Will all of this play out exactly as I describe? No way in hell. If everything I predict is 100% correct I'm just predicting common knowledge. I'm shooting for a base level of 80% accuracy, with hopes I'm closer to 90%. But rather than issuing some proclamation from the mount, I'll detail why I think things are going where they are. You can make your own decisions as to my assumptions and the accuracy of the predictions that stem from them.

Also, apologies to Dre's friends and family. I know this will make his head explode, but that's a cost I'm willing to pay. Special thanks to Chris Hoff and the work we've been doing on disruptive innovation, since that model drives most of what I'm about to describe. Finally, this is just my personal opinion as to where things will go. Adrian is also doing some research on the concept of ADMP, and may not agree with everything I say. Yes, we're both Securosis, but when you're predicting uncertain futures no one can speak with absolute authority. (And, as Hoff says, no one can tell you you're wrong today).

Forces and Assumptions

Based on the work I've been doing with Hoff, I've started to model future predictions by analyzing current trends and disruptive innovations. Those innovations that force change, rather than ones that merely nudge us to steer slightly around some new curves. In the security world, these forces (disruptions) come from three angles- business innovation, threat innovation, and efficiency innovation. The businesses we support are innovating for competitive advantage, as are the bad guys. For both of them, it's all about increasing the top line. The last category is more internal- efficiency innovation to increase the bottom line. Here's how I see the forces we're dealing with today, in no particular order:

1. Web browsers are inherently insecure. The very model of the world wide web is to pull different bits from different places, and render them all in a single view through the browser. Images from over here, text from over here, and, using iframes, entire sites from yet someplace else. It's a powerful tool, and I'm not criticizing this model; it just is what it is. From a security standpoint, this makes our life more than a little difficult. Even with a strictly enforced same origin policy, it's impossible to completely prevent cross-site issues, especially when people keep multiple sessions to multiple sites open all at the same time. That's why we have XSS, CSRF, and related attacks. We are trying to build a trust model where one end can never be fully trusted.
2. We have a massive repository of insecure code that grows daily. I'm not placing the blame on bad programmers; many of the current vulnerabilities weren't well understood when much of this code was written. Even today, some of these issues are complex and not always easy to remediate. We are also discovering new vulnerability classes on a regular basis, requiring review and remediation on any existing code. We're talking millions of applications, never mind many millions of lines of code. Even the coding frameworks and tools themselves have vulnerabilities, as we just saw with the latest Ruby issues.
3. The volume of sensitive data that's accessible online grows daily. The Internet and web applications are powerful business tools. It only makes sense that we connect more of our business operations online, and thus more of our sensitive data and business operations are Internet accessible.
4. The bad guys know technology. Just as it took time for us to learn and adopt new technologies, the bad guys had to get up to speed. That window is closed, and we have knowledgeable attackers.
5. The bad guys have an economic infrastructure. Not only can they steal things, but they have a market to convert the bits to bucks. Pure economics give them viable business models that depend on generating losses for us.
6. Bad guys attack us to steal or assets (information) or hijack them to use against others (e.g., to launch a big XSS attack). They also sometimes attack us just to destroy our assets, but not often (less economic incentive, even for DoS blackmail).
7. Current security tools are not oriented to the right attack vectors. Even WAFs offer limited effectiveness since they are more tied to our network security models than our data/information-centric models.
8. We do not have the resources to clean up all existing code, and we can't guarantee future code, even using a secure SDLC, won't be vulnerable. This is probably my most contentious assumption, but most of the clients I work with just don't have the resources to completely clean what they do have, and even the best programmers will still make mistakes that slip through to production.
9. Code scanning tools and vulnerability analysis tools can't catch everything, and can't eliminate all false positives. They'll never catch logic flaws, and even if we had a perfect tool, the second a new vulnerability appeared we'd have to go back and fix everything we'd built up to that point.
10. We're relying on more and more code and web services developed by others. From machine-generated web applications, to frameworks and off-the-shelf web apps we customize, to mashups where we directly pass content generated by someone else to our users.
11. "Web applications" is a misnomer- we mean the entire stack: web servers, web application servers, the databases behind them, and all the various interconnected n tiers. Many of these are internally accessible, creating an additional vector for attack.

To rephrase these a bit:

1. Bad guys are focused on our web applications, and are intelligent and motivated.
2. We have a lot of insecure code, keep generating more, and can't rely on secure development to fix it all.
3. WAFs and code scanning help, but aren't enough.
4. We need to protect a big, complex stack including content and services outside our control.

Following these forces, I'm drawing some assumptions about what any solution needs to look like:

1. We need to include browser elements, but can't trust the browser.
2. We need to monitor and enforce at the transaction level, both for audibility and for logic flaws and other security issues.
3. Such monitoring and enforcement needs to run from the browser to the database.
4. Any solution needs to understand the application and database, not just layer over it.
5. We need to filter anything we pass on to the user.
6. We need to focus on protecting the information.

I'm not totally thrilled with how I've laid this out, but I think it's reasonably understandable. Tomorrow I'll walk through how I think the technology will develop, and where today's tools fit in. I suspect Adrian will start chiming in, once he gets off the road, with his own interpretations.

–Rich

Web Application Security: We Need Web Application Firewalls To Work. Better.

By Rich

Jeremiah Grossman is just finishing up his keynote at the SANS conference on web application security. Jeremiah and I have talked a few times about the future of web application security, and we both agree that many current approaches just can't solve the problem. It's increasingly clear that no matter how good we are at secure programming (SDLC) , and no matter how effective our code scanning and vulnerability analysis tools are, neither approach can "solve" our web application security problem.

We not only develop code at a staggering pace, we have a massive legacy code base. While many leading organizations follow secure software development lifecycles, and many more will be adopting at least some level of code scanning over the next few years thanks to PCI 6.6, it's naive to think that even the majority of software will go through secure development any time soon. On top of that, we are constantly discovering new vulnerability classes that affect every bit of code written in the past. And, truth be told, no tool will ever catch everything, and even well-educated people still make mistakes.

Since these same issues affect non-web software, we've developed some reasonably effective ways to protect ourselves on that side. The key mantra is shield and patch. When we discover a new vulnerability, we (if possible) shield ourselves through firewalls and other perimeter techniques to buy us time to fix (patch) the underlying problem. No, it doesn't always work and we still have a heck of a lot of progress to make, but it is a fundamentally sound approach.

We're not really doing this much in the web application world. The web application firewall (WAF) market is growing, but has struggled for years. Even when WAFs are deployed, they still struggle to provide effective security. If you think about it, this is one big difference between a WAF and a traditional firewall or IPS. With old school vulnerabilities we know the details of the specific vulnerability and (usually) exploit mechanism. With WAFs, we are trying to block vulnerability classes instead of specific vulnerabilitie s . This is a HUGE difference. The WAF doesn't know the details of the application or any application-specific vulnerabilities, and thus is much more limited in what it can block.

I don't think stand-alone external WAFs will ever be effective enough to provide us the security we need for web applications. Rather, we need to change how we view WAFs. They can no longer be merely external boxes protecting against generic vulnerabilities; they need tighter integration into our applications. In the long term, I've branded this Application and Database Monitoring and Protection (ADMP) as we create a dedicated application and database security stack that links from the browser on the front end, to the DB server on the back.

There are a few companies exploring these new approaches today. Jeremiah's company, WhiteHat Security, has teamed up with F5 to provide specific vulnerability data from a web application to the F5 WAF. Fortify is moving down the anti-exploitation path with real-time blocking (and other actions) directly on the web application server. Imperva is tying together their WAF and database activity monitoring. (I'm sure there are more, but these are the web-application specific companies taking this path I can remember offhand). They are all taking different approaches, but all recognize that "static" WAFs or code scanning alone are pretty limited.

–Rich