In our last episode, we continued our series on building a web application security program by looking at the secure development stage (see also Part 1, Part 2, Part 3, Part 4, and Part 5).

Today we’re going to transition into the secure deployment stage and talk about vulnerability assessments and penetration testing. Keep in mind that we look at web application security as an ongoing, and overlapping, process. Although we’ve divided things up into phases to facilitate our discussion, that doesn’t mean there are hard and fast lines slicing things up. For example, you’ll likely continue using dynamic analysis tools in the deployment stage, and will definitely use vulnerability assessments and penetration testing in the operations phase.

We’ve also been getting some great feedback in the comments, and will be incorporating it into the final paper (which will be posted here for free). We’ve decided this feedback is so good that we’re going to start crediting anyone who leaves comments that result in changes to the content (with permission, of course). It’s not as good as paying you, but it’s the best we can do with the current business model (for now- don’t assume we aren’t thinking about it).

As we dig into this keep in mind that we’re showing you the big picture and everything that’s available. When we close the series we’ll talk prioritization and where to focus your efforts for those of you on a limited budget- it’s not like we’re so naive as to think all of you can afford everything on the market.

Vulnerability Assessment

In a vulnerability assessment we scan a web application to identify anything an attacker could potentially use against us (some assessments also look for compliance/configuration/standards issues, but the main goal in a VA is security). We can do this with a tool, service, or combination of approaches.

A web application vulnerability assessment is very different than a general vulnerability assessment where we focus on network and hosts. In those, we scan ports, connect to services, and use other techniques to gather information revealing the patch levels, configurations, and potential exposures of our infrastructure. Since, as we’ve discussed, even “standard” web applications are essentially all custom, we need to dig a little deeper, examine application function and logic, and use more customized assessments to determine if a web application is vulnerable. With so much custom code and implementation, we have to rely less on known patch levels and configurations, and more on actually banging away on the application and testing attack pathways. As we’ve said before, custom code equals custom vulnerabilities. (For an excellent overview of web application vulnerability layers please see this post by Jeremiah Grossman. We are focusing on the top three layers- third-party web applications, and the technical and business logic flaws of custom applications).

The web application vulnerability assessment market includes both tools and services. Even if you decide to go the tool route, it’s absolutely critical that you place the tools in the hands of an experienced operator who will understand and be able to act on the results. It’s also important to run both credentialed and uncredentialed assessments. In a credentialed assessment, the tool or assessor has usernames and passwords of various levels to access the application. This allows them inside access to assess the application as if they were an authorized user attempting to exceed authority.

Tools

There are a number of commercial, free, and open source tools available for assessing web application vulnerabilities, each with varying capabilities. Some tools only focus on a few kinds of exploits, and experienced assessors use a collection of tools and manual techniques. For example, there are tools that focus exclusively on finding and testing SQL injection attacks. Enterprise-class tools are broader, and should include a wide range of tests for major web application vulnerability classes, such as SQL injection, cross site scripting, and directory traversals. The OWASP Top 10 is a good starting list of major vulnerabilities, but an enterprise class tool shouldn’t limit itself to just one list or category of vulnerabilities. An enterprise tool should also be capable of scanning multiple applications, tracking results over time, providing robust reporting (especially compliance reports), and providing reports customized local needs (e.g., add/drop scans).

Tools are typically software, but can also include dedicated appliances. Tools can run either manual scans with an operator behind them, or automatics scans for on a schedule. Since web applications change so often, it’s important to scan any modifications or new applications before deployment, as well as live applications on an ongoing basis.

Services

Not all organizations have the resources or need to buy and deploy tools to assess their own applications, and in some cases external assessments may be required for compliance.

There are three main categories of web application vulnerability assessment services:

• Fully automatic scans: These are machine-run automatic scans that don’t involve a human operator on the other side. The cost is low, but they are more prone to false positives and false negatives. They work well for ongoing assessments on a continuous basis, but due to their limitations you’ll likely still want more in-depth assessments from time to time.
• Automatic scans with manual evaluation: An automatic tool performs the bulk of the assessment, followed by human evaluation of the results and additional testing. They provide a good balance between ongoing assessments and the costs of a completely manual assessment. You get deeper coverage and more accurate results, but at a higher cost.
• Manual assessments: A trained security assessor manually evaluates your web application to identify vulnerabilities. Typically an assessor uses their own tools, then validates the results and provides custom reports. The cost is higher per assessment than the other options, but a good assessor may find more flaws.

Penetration Testing

The goal of a vulnerability assessment is to find potential avenues an attacker can exploit, while a penetration test goes a step further and validates whether attack pathways result in risk to the organization. In a web application penetration test we attempt to penetrate our own applications and determine what an attacker can do, and what the consequences might be.

Vulnerability assessments and penetration tests are highly complementary, and frequently performed together since the first step in any attack is to find vulnerabilities to exploit. The goal during the vulnerability assessment phase is to find as many of those flaws as possible, and during the penetration test to validate those flaws, determine potential damages, and prioritize remediation efforts.

That’s the key value of a penetration test- it bridges the gap between the discovered vulnerability and the exploitable asset so you can make an appropriate risk decision. We don’t just know we’re vulnerable, we learn the potential consequences of those vulnerabilities. For example, your vulnerability scan may show a SQL injection vulnerability, but when you attempt to exploit it in the penetration test it doesn’t reveal sensitive information, and can’t be used to damage the web application. On the other hand, a seemingly minor vulnerability might turn out to allow exploitation of your entire web application.

Some experts consider penetration tests important because they best replicate the techniques and goals an attacker will use to compromise an application, but we find that a structured penetration test is more valuable as a risk prioritization tool. If we think in terms of risk models, a properly performed penetration test helps fill in the potential severity/impact side of the analysis. Of course, this also means you need to focus your penetration testing program on risk assessment, rather than simply “breaking” a web application.

As with vulnerability assessments, penetration tests can include the entire vulnerability stack (from the network and operating system up through your custom application code), and both tools and services are available. Again, we’ll limit this discussion to web application specific aspects of penetration testing.

Tools

A web application penetration testing tool provides a framework for identifying and safely exploiting web vulnerabilities, and measuring or estimating their potential impact. While there are many tools used by penetration testers, most of them are point tools, rather than broad suites. An enterprise-class tool adds features to better assist the risk management process, support internal assessments, and to reduce the costs of internal penetration tests. This includes broad coverage of application vulnerability classes, “safe” techniques to exploit applications without interfering with their ongoing use, workflow, extensive reporting, and automation for ongoing assessments. One advantage of penetration testing tools is you can integrate them into the development and deployment process, rather than just assessing live web applications.

Penetration testing tools are always delivered as software and should be used by a trained operator. While parts of the process can be automated, once you dig into active exploitation and analysis of results, you need a human being involved.

When using penetration testing (or vulnerability assessment) tools, you have the choice of running them in a safe mode that reduces the likelihood of causing a service or application outage (this is true for all kinds of VA and penetration tests, not just web applications). You’ll want to use safe mode on live applications. But these tools are extremely valuable in assessing development/test environments before applications are deployed, and you shouldn’t restrict your exploit attempts to non-production systems.

Services

Unlike vulnerability assessments, we are unaware of any automated penetration testing services (although some of the automatic/manual services come close). Due to the more intrusive and fluid nature of a web application penetration test you always need a human to drive the process. Penetration testing services are typically offered as either one-off consulting projects or a subscription service for scheduled tests. Testing frequency varies greatly based on your business needs, exposure, and the nature of your web applications. Internal applications might only be assessed annually as part of a regularly scheduled enterprise-wide test.

When evaluating a penetration testing service it’s important to understand their processes, experience, and results. Some companies offer little more than a remote scan using standard tools, and fail to provide prioritized results that are usable in your risk assessment. Others simply “break into” web applications and stop as soon as they get access. You should give preference to experienced organizations with an established process that can provide sample reports (and references) that meet your needs and expectations. Most penetration tests are structured as either fixed-time or fixed-depth. In a fixed-time engagement (the most common) the penetration testers discover as much as they can in a fixed amount of time. Engagements may also divide the time used into phases- e.g. a blind attack, a credentialed attack, and so on. Fixed-depth engagements stop when a particular penetration goal is achieved, no matter how much time it takes (e.g. administrative access to the application or access to credit card numbers).

Integrating Vulnerability Assessment and Penetration Testing into Secure Deployment

Although most organizations tend to focus web application vulnerability assessments and penetration tests on externally accessible production applications, the process really begins during deployment, and shouldn’t be limited to external applications. It’s important to test applications in depth before you expose them to either the outside world or internal users. There’s also no reason, other than resources, you can’t integrate assessments into the development process itself at major milestones.

Before deploying an application, set up your test environment so that it accurately reflects production. Everything from system configurations and patch levels, up through application connections to outside services must match production or results won’t be reliable. Then perform your vulnerability assessment and penetration test. If you use tools, you’ll do this with your own (appropriately trained) personnel. If you use a service, you’ll need to grant them remote access to your test environment. Since few organizations can afford to test every single application to the same degree, you’ll want to prioritize based on the exposure of the application, its criticality for business operations, and the sensitivity of the data it accesses.

For any major externally accessible application you’ll want to engage a third party for both VA and penetration testing before deployment. If your business relies on it for critical operations, and it’s publicly accessible, you really want to get an external assessment.

Once an application is deployed, your goal should be to perform at least a basic assessment on any major modifications before they commit. For critical applications, engage a third-party service for ongoing vulnerability assessments, and periodic penetration tests (at least annual or after major updates) on top of your own testing.

We know not all of you have the resources to support internal and external tools and services for VA and penetration testing on an ongoing basis, so you’ll need to adapt these recommendations for your own organizations. We’ve provided a high level overview of what’s possible, and some suggestions on where and how to prioritize.

In our next post we’ll close out our discussion of web application security technologies by looking at web application firewalls and monitoring tools. We’ll then close out the series by showing you how to put these pieces together into a complete program, how to prioritize what you really need, and how to fit it to the wild world of web application development.