Update: Verisign already closed the hole for RapidSSL.

This morning (in the US- afternoon in Europe), a team of security researchers revealed that they are in possession of a forged Certificate Authority digital certificate that pretty much breaks the whole idea of a trusted website. It allows them to create a fake SSL certificate that your browser will accept for any website.

The short summary is that this isn’t something you need to worry about as an individual, there isn’t anything you can do about it, and the odds are extremely high that the hole will be closed before any bad guys can take advantage of it.

Now for some details and analysis, based on the information they’ve published. Before digging in, if you know what an MD5 hash collision is you really don’t need to be reading this post and should go look at the original research yourself. Seriously, we’re not half as smart as the guys who figured this out. Hell, we probably aren’t smart enough to scrape poop off their shoes (okay, maybe Adrian is, since he has an engineering degree, but all I have is a history one with a smidgen of molecular bio).

This seriously impressive research was released today at the Chaos Computer Congress conference. The team, consisting of Alexander Sotirov, Marc Stevens, Jacob Appelbaum, Arjen Lenstra, David Molnar, Dag Arne Osvik, and Benne de Weger took advantages of known flaws in the MD5 hash algorithm and combined it with new research (and an array of 200 Sony Playstation 3s) to create a forged certificate all web browsers would trust. Here are the important things you need to know (and seriously, read their paper):

• All digital certificates use a cryptographic technique known as a hash function as part of the signature to validate the certificate. Most certificates just ‘prove’ a website is who they say they are. Some special certificates are used to sign those regular certificates and prove they are valid (known as a Certificate Authority, or CA). There is a small group of CAs which are trusted by web browsers, and any certificate they issue is in turn trusted. That’s why when you go to your bank, the little lock icon appears in your browser and you don’t get any alerts. Other CAs can issue certificates (heck, we do it), but they aren’t “trusted”, and your browser will alert you that something fishy might be going on.
• One of the algorithms used for this hash function is called MD5, and it’s been broken since 2004. The role of a hash function is to take a block of information, then produce a shorter string characters (bits) that identifies the original block. We use this to prove that the original wasn’t modified- if we have the text, and we have the MD5 results, we can recalculate the MD5 from the original and it should produce exactly the same result, which must match the hash we got. If someone changes even a single character in the original, the hash we calculate will be completely different from the one we got to check against. Without going into detail, we rely on these hash functions in digital certificates to prove that the text we read in them (particularly the website address and company name) hasn’t been changed and can be trusted. That way a bad guy can’t take a good certificate and just change a few fields to say whatever they want.
• But MD5 has some problems that we’ve known about for a while, and it’s possible to create “collisions”. A collision is when two sources have the exact same MD5 hash. All hash algorithms can have collisions (if they were really 1:1, they would be as long as the original and have no purpose), but it’s the job of cryptographers to make collisions very rare, and ideally make it effectively impossible to force a collision. If a bad guy could force an MD5 hash collision between a real cert and their a fake, we would have no way to tell the real from the forgery. Research from 2004 and then in 2007 showed this is possible with MD5, and everyone was advised to stop using MD5 as a result.
• Even with that research, forging an MD5-based digital certificate for a CA hadn’t ever been done, and was considered very complex, if not impossible. Until now. The research team developed new techniques and actually forged a certificate for RapidSSL, which is owned by Verisign. They took advantage of a series of mistakes by RapidSSL/Verisign and can now fake a trusted certificate for any website on the planet, by signing it with their rogue CA certificate (which carries an assurance of trustworthiness from RapidSSL, and thus indirectly from Verisign).
• RapidSSL is one of 6 root CAs that the research team identified as still using MD5. RapidSSL also uses an automatic system with predictable serial numbers and timing, two fields the researchers needed to control for their method to work. Without these three elements (MD5, serial number, and timing) they wouldn’t be able to create their certificate.
• They managed to purchase a legitimate certificate from RapidSSL/Verisign with exactly the information they needed to use the contents to create their own, fake, trusted Certificate Authority certificate they can then use to create forged certificates for any website. They used some serious math, new techniques, and a special array of 200 Sony PS3s to create their rogue certificate.
• Since browsers will trust any certificate signed by a trusted CA, this means the researchers can create fake certificates for any site, no matter who originally issued the certificate for that site.
• But don’t worry- the researchers took a series of safety precautions, one being that they set their certificate to expire in 2004- meaning that unless you set the clock back on your computer, you’ll still get a security alert for any certificate they sign (and they are keeping it secret in the first place).
• All the Certificate Authorities and web browser companies are now aware of the problem. All they need to do is stop using MD5 (which only a few still were in the first place). RapidSSL only needs to change to using random serial numbers to stop this specific technique.

Thus at this point, your risk is essentially 0, unless Verisign (and the other CAs using MD5) are really stupid and don’t switch over to a different hash algorithm quickly. We are at greater risk of someone like Comodo issuing a bad certificate without all the pesky math.

Nothing to worry about, and hopefully the CAs will avoid SHA1- another hash algorithm that cryptographers believe is prone to collisions.

And I really have to close this out with one final fact:

Chuck Norris collides hash values with his steely stare and power of will.

Update: Yes, if the researchers turn bad or lose control of their rogue cert, we could all be in serious trouble. Or if bad guys replicate this before the CAs fix the hole. I’m qualitatively rating the risk of either event as low, but either is within the realm of possibility.

-rich

We’ve been covering a heck of a lot of territory in our series on Building a Web Application Security Program (see Part 1, Part 2, Part 3, Part 4, Part 5, and Part 6). So far we’ve covered secure development and secure deployment, now it’s time to move on to secure operations. This is the point where the application moves out of development and testing and into production.

Keep in mind that much of what we’ve talked about until now is still in full effect- just because you have a production system doesn’t mean you throw away all your other tools and processes. Updates still need to go through secure development, systems and applications are still subject to vulnerability assessments and penetration testing (although you need to use a different process when testing live applications vs. staging), and configuration management and ongoing secure management are more important than ever before.

In the secure operations phase we add two new technology categories to support two additional processes- Web Application Firewalls (WAF) for shielding from certain types of attacks, and monitoring at the application and database levels to support auditing and security alerting.

Before we dig in, we also want to thank everyone who has been commenting on this series as we post it- the feedback is invaluable, and we’re going to make sure everyone is credited once we put it into whitepaper format.

Web Application Firewalls (WAF)

The role of a web application firewall is to sit in front of or next to a web application, monitoring application activity, and alerting or blocking on policy violations. Thus it potentially serves two functions- as a detective control for monitoring web activity, and as a preventative control for blocking activity.

A web application firewall is a firewall specifically built to watch HTTP requests and block those that are malicious or don’t comply with specific rules. The intention is to catch SQL injection, Cross Site Scripting (XSS), directory traversal, and various HTTP abuses, as well as authorization, request forgeries, and other attempts to alter web application behavior. The WAF rules and policies are effectively consistency checks, for both the HTTP protocol and application functionality. WAFs can alert or block activity based on general attack signatures (such as a known SQL injection attack for a particular database), or application-specific signatures for the web application being protected.

WAF products examine inbound and outbound HTTP requests, compare these with the firewall rules, and create alerts for conditions of concern. Finally, the WAF selects a disposition for the traffic: 1) let it pass, 2) let it pass but audit, 3) block the transaction, or 4) reset the connection.

WAFs typically network appliances. They are normally placed in-line as a filter for the application (proxy mode); or ‘out-of-band’, receiving traffic from a mirror or SPAN port. In the former scenario, all inbound and outbound requests are intercepted and inspected prior to the web server receiving the request or user receiving the response, reducing load on the web application. For SSL traffic, inline WAFs also need to proxy the SSL connection from the browser so it can decrypt and inspect traffic before it reaches the web server, or after it leaves the web server for responses. In out-of-band mode, there are additional techniques to monitor the encrypted connections by placing a copy of the server certificate on the WAF, or positioning it behind an SSL concentrator. Some vendors also provide WAF capabilities via plug-ins for specific platforms, rather than through external devices.

The effectiveness of any WAF is limited by the quality of the policies it is configured to enforce. Policies are important not merely to ability to recognize and stop known/specific attacks, but also for flexibly dealing with ambiguous and unknown threat types, while keeping false positives manageable and without preventing normal transaction processing. The complexity of the web application, combined with the need for continuous policy updates, and the wide variety of deployment options to accommodate, pose a complex set of challenges for any WAF vendor. Simply dropping a WAF in front of your application and turning on all the default rules in blocking mode is a recipe for disaster. There is no way for black box to effectively understand all the intricacies of a custom application, and customization and tuning are essential for keeping false positives and negatives under control.

When deployed in monitoring mode, the WAF is used in a manner similar to an intrusion detection system (IDS). It’s set to monitor activity and generate alerts based on policy violations. This is how you’ll typically want to initially deploy the WAF, even if you plan on blocking activity later. It gives you an opportunity to tune the system and better understand application activity before you start trying to block connections. An advantage of monitoring mode is that you can watch for a wider range of potential attacks without worrying that false positives will result in inappropriate blocking. The disadvantages are 1) your incident handlers will spend more time dealing with these incidents and false positives, and 2) bad activity won’t be blocked immediately.

In blocking/enforcement mode, the WAF will break connections by dropping them (proxy mode) or sending TCP reset packets (out of band mode) to reset the connection. The WAF can then ban the originating IP, permanently or temporarily, to stop additional attacks from that origin. Blocking mode is most effective when deployed as part of a “shield then patch” strategy to block known vulnerabilities in your application.

When a vulnerability is discovered in your application, you build a specific signature to block attacks on it and deploy that to the WAF (the “shield”). This protects your application as you go back and fix the vulnerable code, or wait for an update from your software provider (the “patch”). The shield then patch strategy greatly reduces potential false positives that interfere with application use and improves performance, but is only possible when you have adequate processes to detect and evaluate these vulnerabilities.

You can combine both approaches by deploying a larger signature set in monitoring mode, but only enabling a few specific policies in blocking mode.

Given these challenges, satisfaction with WAF products varies widely among security professionals who use them. While WAFs are effective against known threats, they are less capable of discovering new issues or handling questionable use cases. Some WAF products are addressing these issues by linking web application firewalls more tightly to the vulnerability assessment process and tools, as we’ll discuss in a moment.

Monitoring

Monitoring is primarily used for discovery, both of how an application is used by real users, and also for how it can be misused. The fundamental value of monitoring is to learn what you do not already know- this is important not only for setting up a WAF, but also for tune an application security strategy. Although WAFs provide some level of application activity monitoring, there are three additional ways to monitor web applications, each with a different perspective on application activity:

• Network Monitoring: Monitoring network activity between the user/Internet and the web server. This category includes web application firewalls, intrusion detection systems, packet sniffers, and other external tools. While generic network security and sniffing tools can capture all network traffic, they have much less capability to place it in context and translate network activity into application transactions. Simply viewing HTTP traffic is often insufficient for understanding what users are attempting in an application- this is where interpretation is required. If a solution includes web application specific analysis and the ability to (potentially) audit all web activity, we call it Web Application Monitoring (WAM). While network monitoring is easy to implement and doesn’t require application changes, it can only monitor what’s going into and coming out of the application. This may be useful for detecting traditional attacks against the application stack, but much less useful than seeing traffic fully correlated to higher-level transactions.
• Application Auditing/Logging: Collection and analysis of application logs and internal activity. Both custom and off-the-shelf applications often include internal auditing capabilities, but there is tremendous variation in what’s captured and how it’s formatted and stored. While you gain insight into what’s actually occurring in the application, not all applications log equally (or at all)- you are limited to whatever the programmers decided to track. For major enterprise applications, such as SAP, we’ve seen third party tools that either add additional monitoring or can interpret native audits. Log management and SIEM tools can also be used to collect and interpret (to a more limited degree) application activity when audit logs are generated.
• Database Activity Monitoring: DAM tools use a variety of methods to (potentially) record all database transactions and generate alerts on specific policy violations. By monitoring activity between the application and the database, DAM can provide a more precise examination of data elements, and awareness of multi-step transactions which directly correspond to business functions. Some DAM tools have specific plug ins for major application types (e.g., SAP & PeopleSoft) to translate database transactions into application activity. Since the vast majority of web applications run off databases, this is an effective point to track activity and look for policy violations. A full discussion of DAM is beyond the scope of this post, but more information is available in our DAM whitepaper.

WAF can be used for monitoring, but these alternative tools offer a wider range of activity collection, which helps to detect probing which may not be obvious from HTTP requests alone. The focus of web application monitoring is to examine behavior across data sources and provide analysis, recording activity trails and alerting on suspicious events, whereas WAFs are more focused on detection and blocking of known threats. There are significant differences between WAMs and WAFs in areas such as activity storage, aggregation of data from multiple sources, and examination of the collected data, so choosing the best tool depends specifics of the requirement. We must point out that web application monitoring products are not fully mature, and the handful of available products are early in the product evolution cycle.

WAF + VA

Several vendors have begun providing a hybrid model that combines web application vulnerability assessment with a web application firewall. As mentioned earlier, one of the difficulties with a shield then patch strategy is detecting vulnerabilities and building the WAF signatures to defend them. Coupling assessment tools or services with WAFs by feeding the assessment results to the firewall, and having the firewall adjust its policy set accordingly, can makes the firewall more effective. The intention is to fill the gap between exploits discovery/announcement and deployment of tested patches in the application, by instructing the WAF to block access to the particular weakness. In this model the assessment determines that there is a vulnerability and feeds the information to the WAF. The assessment policy contains WAF instructions on what to look for and how to respond. The WAF then dynamically incorporates the policy and protects the application.

This is the last post in this series where we discuss your options at different stages of the application lifecycle. Our next post will discuss which options you should consider, and how to balance your resource expenditures into an overall program.

-Rich and Adrian

Remember our guest post from Jesse Krembs on the MIT students put under a gag order during DefCon this year for hacking the rail system? And I quote:

Please grow up; in the connected world there are very few ogres in caves any more, and they don’t let you ride their trains. The difference between black hats and white hats is a line, and it’s a gray one. But occasionally it gets a little contrast. When you treat the person or organization with a security problem like a victim or an enemy, then you’re the bad guy. You’re basically fucking them over, sometimes hard, sometimes gently, but it’s still a screw job. When you treat them like a partner, then everyone wins. Sure, sometimes they don’t want partners, and sometimes you have to go public because they put the rest of the world at risk, but you don’t know that until you try talking to them. Finally I should note that in the end the only people winning in this case are the lawyers; the kids won’t win in the way they want, nor will the MBTA. The lawyers, on the other hand, always get paid

Looks like Superman just spun the Earth backwards and turned back time (sort of):

The announcement brings to a close a high profile case that pitted the rights of security researchers to freely discuss their findings against the concerns of one of the country’s largest transit systems, which worried that this type of information could lead to widespread ticket fraud. “I’m really glad to have it behind me. I think this is really what should have happened from the start,” said Zack Anderson, one of the students sued by the MBTA.

…

The settlement ends the matter in an amicable way. “For professional reasons and for public interest reasons, the students wanted to help the MBTA,” said Jennifer Granick, a lawyer with the Electronic Frontier Foundation who represents the students.

The case against the three was finally settled on Oct. 7, but this was not publicly announced until Monday, because it took two months for all parties to schedule a public announcement of the settlement, Granick said. The researchers met with MBTA technical staff on Oct. 21 to discuss their findings and are working to improve the transit authority’s fare collection system, she added.

And all is good in the world again.

Remember our first post that there are no trusted sites? Followed by our second one? Now I suppose it’s time to start naming names in the post titles, since this seems to be a popular trend.

American Express is our latest winner. From Dark Reading:

Researchers have been reporting vulnerabilities on the Amex site since April, when the first of several cross-site scripting (XSS) flaws was reported. However, researcher Russell McRee caused a stir again just a week ago when he reported newly discovered XSS vulnerabilities on the Amex site

The vulnerability, which is caused by an input validation deficiency in a get request, can be exploited to harvest session cookies and inject iFrames, exposing Amex site users to a variety of attacks, including identity theft, researchers say. McRee was tipped off to the problem when the Amex site prompted him to shorten his password — an unusual request in today’s security environment, where strong passwords are usually encouraged.

…

McRee says American Express did not respond to his warnings about the vulnerability. However, in a report issued by The Register on Friday, at least two researchers said they found evidence that American Express had attempted to fix the flaw — and failed.

“They did not address the problem,” says Joshua Abraham, a Web security consultant for Rapid7, a security research firm. “They addressed an instance of the problem. You want to look at the whole application and say, ‘Where could similar issues exist?’”

No, we don’t intend on posting every one of these we hear about, but some of the bigger ones serve as nice reminders that there really isn’t any such thing as a “safe” website.

-Rich

A Microsoft Security Advisory for SQL Server (961040) was posted on the 22nd of December. Microsoft has done a commendable job and provided a lot of information on this page, with a cross reference of the CVE number (CVE-2008-4270) so you can find more details if you need it. Any stored procedure that provide remote code execution can be dangerous and is a target for hackers.

You want to patch as soon as Microsoft releases a patch.

Microsoft states that “… MSDE 2000 or SQL Server 2005 Express are at risk of remote attack if they have modified the default installation to accept remote connections, if they allow untrusted users access to MSDE 2000 or SQL Server 2005 Express …” But I rate the risk higher than they say because of the following:

MSDE 2000 and SQL Server Express 2005 are often bundled/embedded into applications and so their presence is not immediately apparent. There may be copies around that IT staff are not fully aware of, and/or these applications may be delivered with open permissions because the developer of the application was not concerned with these functions.

Second, replication is an administrative function. sp_replwritetovarbin, along with other stored procedures like sp_resyncexecutesql and sp_resyncexecute, runs as DBO or Database Owner, so if they are compromised they expose permissions as well as functions.

Finally, as MSDE 2000 and SQL Server Express 2005 get used by web developers who run the database on the same machine with the same OS/DBA credentials, you server could be completely compromised with this one.

So follow their advice and run the command:

use master
deny execute on sp_replwritetovarbin to public

A couple more recommendations, assuming you are a DBA (which is a fair assumption if you are running the suggested workaround): check master.dbo.sysprotects and master.dbo.sysobjects for public permissions in general. Even if you are patched for this specific vulnerability, or if you are running an unaffected version of the database, you should have this procedure locked down- otherwise you remain vulnerable.

Over and above patching known servers, if you have a scanning and discovery tool, run a scan across your network for the default SQL Server port to see if there are other database engines. That should uncover most undocumented SQL Server databases.

-Adrian

This will be our last Friday Summary for 2008. This afternoon Adrian and I are off to The Office for our Securosis Annual Staff Festivus Party (sorry Chris, but we can drunk dial you if that makes you feel included).

2008 has been an incredibly wild ride. When it started I was just a solo consultant that wasn’t even calling myself an analyst anymore, and wasn’t certain where I wanted to take things. In January I ran a half marathon on a bad knee that mysteriously felt better after the race, but in February I went in for shoulder surgery that I’m still struggling to recover from. Over the summer, Adrian joined Securosis and we moved firmly back into the analyst column. As the year closes we’ve published a ton of free content, multiple vendor-neutral whitepapers, spoken at everything from RSA, to SOURCE Boston, to DefCon, and a few TechTarget and MISTI events (including a show in Moscow), given over a dozen webcasts, and, to be honest, had a heck of a lot of fun in the process. We’ve written articles for everyone from Macworld to Dark Reading, been interviewed by… well, pretty much everyone else, and enjoyed more than a few frothy beverages with our industry friends. For two skinny guys (and a part-time editor/UNIX guru, also skinny) running a small company we really couldn’t have asked for more. We’ve decided to give back, and we’ll announce more on that next week.

And 2009 is looking even crazier. In February we’ll be adding a new staff member, the exact date, gender, length, and weight are still undetermined (if he or she is over 8 lbs, my wife might kill me). We’re also completely redesigning our website as we continue to expand things a bit. This site started as just my personal blog, and as we keep pumping out content it isn’t nearly as well suited as it was at the beginning. The blog won’t change, but we’re going to make content more accessible and start loading up new kinds of materials- like videos of our conference presentations.

We’re also really going to push forward with the ideas of totally transparent and open research. We’re not idiots, and we don’t intend on competing with Gartner, Forrester, and the other large firms, but we still love what we do and think there’s plenty of room for us little guys (and our combined weight is pretty low, not that that’s relevant). We have more flexibility than they do, and you can expect no bullshit research that’s focused on in-depth, practical advice to help you with specific projects. We already have two programs planned- Pragmatic PCI, and Pragmatic Database Security (we’ll have to charge for those, since we have to keep the dogs, cats, and other little ones fed). Finally, we have some new media, social media, and community stuff in the works.

Okay- I realize that all sounded like marketing junk, but I think we’re allowed to be excited about what we’ve done, and what we have planned, from time to time. We are incredibly thankful for the opportunities and support you’ve all given us. And as a preview, here’s the official premier of our new logo (it will look better on the new site template):

Have a wonderful holiday season. We’ll be reducing our posting volume a bit over the holidays, but stay tuned for the end of our web application series and a few other treats.

Here is the week’s security summary:

Webcasts, Podcasts, Outside Writing, and Conferences:

• The Network Security Podcast is a little shorter this week as we finish off the year. As an aside, Martin and I would like to apologize for our recent audio difficulties. We narrowed it down to a bad sound card on Martin’s side, and are changing our recording process for higher quality (we’ll be moving to double-ended recording).
• Via NetworkWorld, I gave a webcast for Oracle on Database Security for Security Professionals. It targets security pros who may be new to databases, and the replay is available here.
• I talked with Forbes about antivirus scanners.
• I debunked some FUD by Bit9 on automatic software updates in enterprises for LinuxInsider. IT departments can turn them off, so I don’t see what the problem is.
• Adrian talked database security with eWeek.
• Adrian on log management for internetnews.com.
• Someone named Adrian Lane is into otters. It’s the UK, so probably a different guy, but we’ll take all the press we can get.

Favorite Securosis Posts:

• Rich: Part 6 of our Building a Web Application Security Program. We really want to get this series (and the eventual paper) right, so any feedback, comments, and (especially) criticisms are very much appreciated.
• Adrian: While my practical experience has come to the same set of conclusions, finding meaning is groups of anonymous statistical patterns to justify database security is a black art I don’t care to dabble in.

Favorite Outside Posts:
{Adrian editorial}- I have been following the series of posts between Alan Shimmel and Andy the IT Guy (links below). They are touching on the very heart of the sales process and common friction between the IT gatekeeper and the salesman. But I thought they both danced around the key point. The sales guy is doing his job by pushing as hard as he can to get the deal done without pissing everyone off to the point the organization gets fed up and will no longer work with you. A good sales guy knows there is always a deal if they can overcome objections (price, support, consultative assistance, etc) because they would not be talking if the need was not there. However buyers buy from people they know, like, and trust, and trampling the gatekeeper is a good way to make enemies. Alan’s comment “Try putting yourself in the other’s shoes to better understand what is involved. Common courtesy and respect would be a good place to start” cuts both ways. Seems to me the sales guy failed to build proper relations before making the ‘hail-Mary’ sales pitch. -Adrian

• Adrian: Humorous series of posts between Shimmel and Andy the IT Guy.
• Rich: Layer 8 on planning your security for 2009. Really great practical advice.

Top News and Posts:

• Looks like automakers will get a bail-out after all.
• Microsoft’s out-of-cycle IE patch.
• Practical advice on dealing with botnets … even if your legal team disavows any such possibility.
• RIAA is finally killing off their incredibly stupid strategy of suing individual file sharers.
• Jeremiah on web application security and history. He’s going to out-analyst us if he keeps this up. Come on Jer, we need to feed our kids too!
• Microsoft releases their Anti-XSS library for .Net. This is a far bigger deal than I think most people realize, and hasn’t been picked up by the press yet.
• My friends, for $20 you could have owned a former McCain campaign Blackberry… with all the contacts. Joe the Plumber’s phone must be ringing off the hook!

Blog Comment of the Week:
EB on Part 6 of our Building a Web Application Security Program:
One piece I thought I would mention under your “Penetration Testing” subheading (which you may have been implying) is the risk of linking vulnerabilities. For example, while one SQL injection flaw may not reveal sensitive information (PII, cc#), it could reveal usernames, password hints, or a list of database tables that may seem trivial at the time; however, as the penetration test continues, and more ‘tidbits’ of information are discovered, it could lead to retrieval of sensitive information. For example, retrieving a list of user accounts and noting that the application requires weak passwords could lead to a valid, authenticated user session by the penetration tester. Another example is vulnerability retrieves a spreadsheet of employees’ last 4 digits of their SSN which can also be used to reset a user’s password on the application. The examples can go on and on…
Linking of vulnerabilities and the ability to assess the application as a whole is a quality that companies should ensure, if a third-party is engaged, will perform.
-Rich

Looks like the RIAA has finally realized that treating customers like criminals isn’t the best strategy in the world. According to the Wall Street Journal (via Slashdot) they are ending their campaign of suing individual file sharers to focus on working with ISPs to reduce illegal sharing.

As much as I like to rip the heck out of the RIAA and MPAA for their draconian views on copyright and enforcement, it really is stealing if you snag something off a file sharing network. Like most people in college I was into the Napster thing for a bit, but quickly realized it was wrong, and I stopped using it. Heck, a friend’s dad who was an FBI agent had her download music for him; that’s how new the concept was and how much it snapped our usual social mores.

But I will admit, here and now, to downloading digital content I already legally access when DRM restrictions interfere with my use of that content. It’s not something I do very often, but I have no qualms about heading to the Pirate Bay and grabbing an episode of a TV show my TiVo won’t let me transfer to my phone (mostly the hi def stuff), I’ll even rent movies, rip them, watch them once on my iPhone, then delete them. If the media companies interfere with my existing rights, I’m more than happy to circumvent them.

I still pay for all my music, movies, and television, and in exchange I use all my technical skills to maintain my rights.

-Rich

Just ran across this ‘new’ SQL Server vulnerability in my news feed. Hopefully this is not an issue because you should not be using this set of functions. If you are using external stored procedures on a production database, stop. In fact, you want to stop using them altogether; either lock them down or remove them entirely. Not just because of this reported instance. External stored procedures exploits are favorites of database hackers, and have been used to alter database functionality and to run arbitrary code, both externally and internally launched attacks! SQL Server has historically had issues with buffer overflow attacks (See Microsoft Technical Bulletin MS02-020) against the pre-built procedures, and while known issued have been cleared up, XP’s are a complex and powerful extension ripe for exploits.

The database vendors in general recommend as a security best practice the restriction of these to administrative use at a minimum. Even then it violates the best practice of segregation of the OS / database functionality required by compliance and operational security. Use of external stored procedures is flagged by all of the database vulnerability assessment tools, as both a security and a compliance issue. And in case you think that I am picking on SQL Server, many similar problems have been reported on Oracle ExtProc as well.

The DBA in me loves the ability to run native platform utilities to support database admin efforts. It’s a really handy extension, and I know it is tempting to leave these on the database so you can make admin  easier, but you will be relying upon security through obscurity. It is a really big risk in a production environment and one that every database hacker will have scripts to find and exploit.

-Adrian

In our last episode, we continued our series on building a web application security program by looking at the secure development stage (see also Part 1, Part 2, Part 3, Part 4, and Part 5).

Today we’re going to transition into the secure deployment stage and talk about vulnerability assessments and penetration testing. Keep in mind that we look at web application security as an ongoing, and overlapping, process. Although we’ve divided things up into phases to facilitate our discussion, that doesn’t mean there are hard and fast lines slicing things up. For example, you’ll likely continue using dynamic analysis tools in the deployment stage, and will definitely use vulnerability assessments and penetration testing in the operations phase.

We’ve also been getting some great feedback in the comments, and will be incorporating it into the final paper (which will be posted here for free). We’ve decided this feedback is so good that we’re going to start crediting anyone who leaves comments that result in changes to the content (with permission, of course). It’s not as good as paying you, but it’s the best we can do with the current business model (for now- don’t assume we aren’t thinking about it).

As we dig into this keep in mind that we’re showing you the big picture and everything that’s available. When we close the series we’ll talk prioritization and where to focus your efforts for those of you on a limited budget- it’s not like we’re so naive as to think all of you can afford everything on the market.

Vulnerability Assessment

In a vulnerability assessment we scan a web application to identify anything an attacker could potentially use against us (some assessments also look for compliance/configuration/standards issues, but the main goal in a VA is security). We can do this with a tool, service, or combination of approaches.

A web application vulnerability assessment is very different than a general vulnerability assessment where we focus on network and hosts. In those, we scan ports, connect to services, and use other techniques to gather information revealing the patch levels, configurations, and potential exposures of our infrastructure. Since, as we’ve discussed, even “standard” web applications are essentially all custom, we need to dig a little deeper, examine application function and logic, and use more customized assessments to determine if a web application is vulnerable. With so much custom code and implementation, we have to rely less on known patch levels and configurations, and more on actually banging away on the application and testing attack pathways. As we’ve said before, custom code equals custom vulnerabilities. (For an excellent overview of web application vulnerability layers please see this post by Jeremiah Grossman. We are focusing on the top three layers- third-party web applications, and the technical and business logic flaws of custom applications).

The web application vulnerability assessment market includes both tools and services. Even if you decide to go the tool route, it’s absolutely critical that you place the tools in the hands of an experienced operator who will understand and be able to act on the results. It’s also important to run both credentialed and uncredentialed assessments. In a credentialed assessment, the tool or assessor has usernames and passwords of various levels to access the application. This allows them inside access to assess the application as if they were an authorized user attempting to exceed authority.

Tools

There are a number of commercial, free, and open source tools available for assessing web application vulnerabilities, each with varying capabilities. Some tools only focus on a few kinds of exploits, and experienced assessors use a collection of tools and manual techniques. For example, there are tools that focus exclusively on finding and testing SQL injection attacks. Enterprise-class tools are broader, and should include a wide range of tests for major web application vulnerability classes, such as SQL injection, cross site scripting, and directory traversals. The OWASP Top 10 is a good starting list of major vulnerabilities, but an enterprise class tool shouldn’t limit itself to just one list or category of vulnerabilities. An enterprise tool should also be capable of scanning multiple applications, tracking results over time, providing robust reporting (especially compliance reports), and providing reports customized local needs (e.g., add/drop scans).

Tools are typically software, but can also include dedicated appliances. Tools can run either manual scans with an operator behind them, or automatics scans for on a schedule. Since web applications change so often, it’s important to scan any modifications or new applications before deployment, as well as live applications on an ongoing basis.

Services

Not all organizations have the resources or need to buy and deploy tools to assess their own applications, and in some cases external assessments may be required for compliance.

There are three main categories of web application vulnerability assessment services:

• Fully automatic scans: These are machine-run automatic scans that don’t involve a human operator on the other side. The cost is low, but they are more prone to false positives and false negatives. They work well for ongoing assessments on a continuous basis, but due to their limitations you’ll likely still want more in-depth assessments from time to time.
• Automatic scans with manual evaluation: An automatic tool performs the bulk of the assessment, followed by human evaluation of the results and additional testing. They provide a good balance between ongoing assessments and the costs of a completely manual assessment. You get deeper coverage and more accurate results, but at a higher cost.
• Manual assessments: A trained security assessor manually evaluates your web application to identify vulnerabilities. Typically an assessor uses their own tools, then validates the results and provides custom reports. The cost is higher per assessment than the other options, but a good assessor may find more flaws.

Penetration Testing

The goal of a vulnerability assessment is to find potential avenues an attacker can exploit, while a penetration test goes a step further and validates whether attack pathways result in risk to the organization. In a web application penetration test we attempt to penetrate our own applications and determine what an attacker can do, and what the consequences might be.

Vulnerability assessments and penetration tests are highly complementary, and frequently performed together since the first step in any attack is to find vulnerabilities to exploit. The goal during the vulnerability assessment phase is to find as many of those flaws as possible, and during the penetration test to validate those flaws, determine potential damages, and prioritize remediation efforts.

That’s the key value of a penetration test- it bridges the gap between the discovered vulnerability and the exploitable asset so you can make an appropriate risk decision. We don’t just know we’re vulnerable, we learn the potential consequences of those vulnerabilities. For example, your vulnerability scan may show a SQL injection vulnerability, but when you attempt to exploit it in the penetration test it doesn’t reveal sensitive information, and can’t be used to damage the web application. On the other hand, a seemingly minor vulnerability might turn out to allow exploitation of your entire web application.

Some experts consider penetration tests important because they best replicate the techniques and goals an attacker will use to compromise an application, but we find that a structured penetration test is more valuable as a risk prioritization tool. If we think in terms of risk models, a properly performed penetration test helps fill in the potential severity/impact side of the analysis. Of course, this also means you need to focus your penetration testing program on risk assessment, rather than simply “breaking” a web application.

As with vulnerability assessments, penetration tests can include the entire vulnerability stack (from the network and operating system up through your custom application code), and both tools and services are available. Again, we’ll limit this discussion to web application specific aspects of penetration testing.

Tools

A web application penetration testing tool provides a framework for identifying and safely exploiting web vulnerabilities, and measuring or estimating their potential impact. While there are many tools used by penetration testers, most of them are point tools, rather than broad suites. An enterprise-class tool adds features to better assist the risk management process, support internal assessments, and to reduce the costs of internal penetration tests. This includes broad coverage of application vulnerability classes, “safe” techniques to exploit applications without interfering with their ongoing use, workflow, extensive reporting, and automation for ongoing assessments. One advantage of penetration testing tools is you can integrate them into the development and deployment process, rather than just assessing live web applications.

Penetration testing tools are always delivered as software and should be used by a trained operator. While parts of the process can be automated, once you dig into active exploitation and analysis of results, you need a human being involved.

When using penetration testing (or vulnerability assessment) tools, you have the choice of running them in a safe mode that reduces the likelihood of causing a service or application outage (this is true for all kinds of VA and penetration tests, not just web applications). You’ll want to use safe mode on live applications. But these tools are extremely valuable in assessing development/test environments before applications are deployed, and you shouldn’t restrict your exploit attempts to non-production systems.

Services

Unlike vulnerability assessments, we are unaware of any automated penetration testing services (although some of the automatic/manual services come close). Due to the more intrusive and fluid nature of a web application penetration test you always need a human to drive the process. Penetration testing services are typically offered as either one-off consulting projects or a subscription service for scheduled tests. Testing frequency varies greatly based on your business needs, exposure, and the nature of your web applications. Internal applications might only be assessed annually as part of a regularly scheduled enterprise-wide test.

When evaluating a penetration testing service it’s important to understand their processes, experience, and results. Some companies offer little more than a remote scan using standard tools, and fail to provide prioritized results that are usable in your risk assessment. Others simply “break into” web applications and stop as soon as they get access. You should give preference to experienced organizations with an established process that can provide sample reports (and references) that meet your needs and expectations. Most penetration tests are structured as either fixed-time or fixed-depth. In a fixed-time engagement (the most common) the penetration testers discover as much as they can in a fixed amount of time. Engagements may also divide the time used into phases- e.g. a blind attack, a credentialed attack, and so on. Fixed-depth engagements stop when a particular penetration goal is achieved, no matter how much time it takes (e.g. administrative access to the application or access to credit card numbers).

Integrating Vulnerability Assessment and Penetration Testing into Secure Deployment

Although most organizations tend to focus web application vulnerability assessments and penetration tests on externally accessible production applications, the process really begins during deployment, and shouldn’t be limited to external applications. It’s important to test applications in depth before you expose them to either the outside world or internal users. There’s also no reason, other than resources, you can’t integrate assessments into the development process itself at major milestones.

Before deploying an application, set up your test environment so that it accurately reflects production. Everything from system configurations and patch levels, up through application connections to outside services must match production or results won’t be reliable. Then perform your vulnerability assessment and penetration test. If you use tools, you’ll do this with your own (appropriately trained) personnel. If you use a service, you’ll need to grant them remote access to your test environment. Since few organizations can afford to test every single application to the same degree, you’ll want to prioritize based on the exposure of the application, its criticality for business operations, and the sensitivity of the data it accesses.

For any major externally accessible application you’ll want to engage a third party for both VA and penetration testing before deployment. If your business relies on it for critical operations, and it’s publicly accessible, you really want to get an external assessment.

Once an application is deployed, your goal should be to perform at least a basic assessment on any major modifications before they commit. For critical applications, engage a third-party service for ongoing vulnerability assessments, and periodic penetration tests (at least annual or after major updates) on top of your own testing.

We know not all of you have the resources to support internal and external tools and services for VA and penetration testing on an ongoing basis, so you’ll need to adapt these recommendations for your own organizations. We’ve provided a high level overview of what’s possible, and some suggestions on where and how to prioritize.

In our next post we’ll close out our discussion of web application security technologies by looking at web application firewalls and monitoring tools. We’ll then close out the series by showing you how to put these pieces together into a complete program, how to prioritize what you really need, and how to fit it to the wild world of web application development.

-Rich and Adrian

Bryan Sullivan’s thought provoking post on Streamlining Security Practices for Agile Development caught my attention this morning. Reading it gave me the impression of a genuine generational divide. If you have ever witnessed a father and son talk about music, while they are talking about the same subject, there is little doubt the two are incompatible.

The post is in line with what Rich and I have been discussing with the web application series, especially in the area of why web apps are different, albeit on a slightly more granular level. This article is about process simplification and integration, and spells out a few of the things you need to consider if moving from more formalized waterfall process into Agile with Security. The two nuggets of valuable information are the risk based inclusion of requirements, where the higher risk issues are placed into the sprints, and the second is how to account for lower priority issues that require periodic inspection within a non-linear development methodology.

The risk based approach of placing higher security issues as code gets created in each sprint is very effective. It requires that issues and threats be classified in advance, but it makes the sprint requirements very clear while keeping security as a core function of the product. It is a strong motivator for code and test case re-use to reduce overhead during each sprint, especially in critical areas like input validation.

Bryan also discusses the difficulties of fitting other lower priority security requirements extracted from SDL into Agile for Web Development. In fact, he closes the post with the conclusion that retrofitting waterfall based approaches to secure Agile development is not a good fit. Bravo to that! This is the heart of the issue, and while the granular inclusion of high risk issues into the sprint works, the rest of the ‘mesh’ is pretty much broken. Checks and certifications triggered upon completed milestones must be rethought. The bucketing approach can work for you, but what you label the buckets and when you give them consideration will vary from team to team. You may decide to make them simple elements of the product and sprint backlog. But that’s the great thing about process is you get to change it to however it suits your purpose.

Regardless, this post has some great food for though and is worth a read.

-Adrian