Now that we have laid out the Agile process it’s time to discuss where different types of security testing fits within it. Your challenge is not just to figure out what testing you need to identify code issues, but also to smoothly fit tests into the framework to help speed testing. You will incorporate multiple testing techniques into the the process, with each tool or technique focused on finding slightly different issues. Developers are clever, so development teams find ways to circumvent security testing if it interferes with efficient coding. And you will need to accept that some tests simply cannot be performed in certain parts of the process, while others can be incorporated in multiple places. To help you evaluate both which tools to consider and how to incorporate them, we offer several recommendations for designing a security “tool chain”.

Pre-Sprint Tests and Analysis

• Threat modeling: Threat modeling is the act of looking for design-level security problems from the perspective of an attacker, and then designing countermeasures. The process enables designers and developers to think about the big picture security of an application or function, then build in defenses, rather than merely focusing on bugs. The classic vectors include unwanted escalation of user credentials, information disclosure, repudiation (i.e.: injecting false data into logs), tampering, spoofing, and denial of service. For each new feature, all these subversion techniques are evaluated against every place a user or code module communicates with another. If issue are identified, the design is augmented to address the problem. In Agile these changes are incorporated into user stories before task cards are doled out.
• Security defect list: Security defect tracking covers both collecting security defect data and getting the subset of information developers need to address problems. Most organizations funnel all discovered defects into a bug tracking system. These defects may be found in normal testing or through any of the security tests described below. Security testing tools feed defect tracking systems so issues can be tracked, but that does not mean they provide consistent levels of information. Nor do they set the bar for criticality the same. How you integrate and tailor defect feeds from test tools, and normalize those results, is important for effective Agile integration. You need to reach an agreement with the Product Owner on which defects will be addressed in each sprint, and which will be allowed to slide (and for how long). The security defect backlog should be reviewed each sprint.
• Patching and configuration management: Most software uses a combination of open source and/or commercial code to supplement what the in-house development team builds. For example Apache supports most current web services. Keeping these supplementary components patched is just as necessary as fixing issues in your own code. Agile offers a convenient way to integrate patching and configuration changes at the beginning of each sprint: catalog security patches in supporting commercial and open source platforms, and incorporate the changes into the sprint as tasks. This pre-supposes IT and its production systems are as Agile as development systems, which is regrettably not always the case.

Daily Tests

• Unit testing: Development teams use unit tests to prove that delivered code functions as designed. These tests are typically created during the development process; teams using test-driven or behavior-driven development write them before the code. Unit tests are run after each successful build, and help catch any defects that pop up due to recent changes. Unit tests often include attacks and garbage input to verify that the application is resilient to potential issues outlined during threat modeling. The formal requirement for this type of testing needs to be included in the Agile user stories or tasks – they do not magically appear if you fail to specify them as a requirement.
• Security regression tests: Regression tests verify that a code change actually fixes a defect. Like unit testing they run after each successful build. Unlike unit test, regression tests each target a known defect, either in the code or a supporting code module. It is common for development teams to break previous security fixes – usually when merging code branches – so security regression tests are a failsafe. With each security tasks to fix a defect, include a simple test to ensure it stays fixed.
• Manual code inspection: Code reviews, also called peer review, are where a member of the development team examines another developer’s code. They check to ensure that code complies with general standards, but also look for specific implementation flaws such as unfiltered input variables, insufficient user authentication, and unhandled errors. Despite wide adoption of automated testing, about 50% of development shops still leverage manual code review for code quality assessment. Manual efforts may appear inefficient, but manual testing is as Agile as it needs to be. For example, the team chooses whether to perform these tests as part of development, QA, predeployment, or any combination of the above. The task can be assigned to any developer, on any branch of code, and as focused or random as the team decides. We recommend manual testing against critical code modules, supplemented with automated code scanning because manual scanning is repetitive and prone to errors. Manual testing can serve a very valuable security function when properly integrated, by focusing on critical functions (including authentication and encryption), using domain experts to keep an eye on that code and subsequent changes.

Every-Sprint (Commit) Tests

• Static analysis: Static analysis examines the source code of a web application for common vulnerabilities, errors, and omissions within the constructs of the language itself. This provides an automated counterpart to peer review. Among other things these tools generally scan for unhandled error conditions, unfiltered input variables, object availability and/or scoping, and potential buffer overflows. The technique is called “static analysis” because it examines source code rather than execution flow of a running program.

Like manual reviews, static analysis is effective at discovering ‘wetware’ problems: issues in code that are directly attributable to programmer error. Better tools integrate well with various development environments to provide educational feedback and suggest corrective actions; prioritize discovered vulnerabilities based on included or user-provided criteria; and include robust reporting to keep management informed, track trends, and engage the security team in the development process without requiring them to be programmers. Static analysis tools do not account for all pathways within the code, and are blind to certain types of vulnerabilities and problems that only appear at runtime. To address these gaps, dynamic analysis tools (below) often supplement static analysis. Some static analysis tools require all code to be present and compiled, while others can work with isolated components. This trait dictates whether static analysis can run each build, or must wait until the end of the sprint.

Every team is different but many run static analysis as part of the nightly build process, only responding immediately to critical vulnerabilities. These organizations also run static analysis at the end of each sprint, agreeing on which issues must be addressed before code changes can be bundled with the final build.

• Dynamic analysis: Dynamic analysis identifies problems which cannot be detected in source code and issues more visible during execution. Dynamic analysis tools fall into two types: white and black box. The test application that ‘exercises’ the application may lack all prior knowledge – probing the application as a “black box” and testing for exploits as it navigates the application. More often tests are a “white box” derivative of functional tests, traversing known pathways and supplying malicious or garbage input values. Random black box testing better represents an outside hacker without detailed internal knowledge and unburdened by assumptions, but it takes longer to run and cannot focus on known and suspected weak spots. White box tests can prioritize known critical code paths and focus on high-priority issues.
• Component analysis: Most applications are deployed into ecosystems, rather than tiny isolated boxes – and containers aren’t really isolated boxes. They rely on third party code libraries, operating system elements, and a slew of other open source and proprietary tools and components. Any of these can introduce vulnerabilities just as easily as any development error. Component analysis provides security testing of these pieces, ensuring that expected components are in use (such as the proper compiler version or run-time engine).

It is important to test both credentialed and anonymous access. Some vulnerabilities may not be visible to a random attacker, but show up when logging in as a known user. And with either white or black box testing, human examination of results is necessary to distinguish real problems from false positives. Error conditions are easy to find, and most dynamic test tools discover and report many, but behavioral oddities which indicate an attack vector are not usually detected by scanning alone. Some organizations set up credentialed scans to run during automated build testing, but more often they run after delivery of specific functional milestones. Keep in mind that dynamic and static analyses are complimentary, with different strengths, and for slightly different audiences. Some vendors provide both, combining results and reporting.

Additional Pre-deployment Tests

• Vulnerability assessment: These scans applications to identify anything an attacker could potentially use. Application-layer vulnerability assessments are very different than general vulnerability assessments, which focus on networks and hosts. Application-layer assessments validate configurations, detect inclusion of known defective code modules, and evaluate user authentication services. Some dig a little deeper to examine application function and logic, or use customized assessments to determine whether an application is vulnerable.
• Penetration testing: The goal of a vulnerability assessment is to find avenues an attacker can exploit, while a penetration test goes a step further to validate which attack pathways pose actual risk. A penetration tester attempts to exploit applications to determine what a real attacker could do, and what the consequences might be. Penetration tests are important because they closely replicate the techniques and tools attackers use to compromise applications, but we find structured penetration tests more valuable for risk prioritization. Penetration tests can include the entire vulnerability stack from the network and operating system up through custom application code and application firewall layers. Since these tests examine the entire stack, they must be run against live production services.

The earlier a defect is discovered, the easier and cheaper it is to fix. The current trend is to shift all testing, including security, as early in the process as possible. Building up a suite of nightly automated tests is an increasingly common strategy to improve quality and security. Our last post will discuss recent DevOps evolution to Agile development methodologies that facilitate earlier testing and better synchronization between IT, QA and Development teams.