Research

I realize this might shock our fair readers, but once upon a time I used to get my hands dirty with a little hands on web application development. Back in the heady early days of the mid-1990’s Internet I accidentally transitioned from a systems and network administrator to a web application developer and DBA at the University of Colorado’s Graduate School of Business. It all started when I made the mistake of making an incredibly ugly home page for the school, complete with a tiled background of my Photoslop-embossed version of the CU logo (but, thankfully, no BLINK tag). The University took note, and I slowly migrated out of keeping the network running into developing database driven web applications for a few thousand users. Eventually I ran my own department before setting off into the big bad world of private consulting. To this day I’m still proud of our online education tools that could totally kick Blackboard’s ass, but I think I developed my last application around 2001. I’ll be the first to admit that my skills are beyond stale, and the tools and techniques now available to web application developers are simply astounding. When I first started out in Boulder, Colorado I’d say the majority of web site developers I met were more focused on graphics skills than database design and proper user authentication. Today’s web application developers need a background in everything from structured programming, to application design, to a working knowledge of multiple frameworks and programming languages. Current web applications exist in an environment that is markedly different from the early days of businesses entering the Internet. They’ve become essential business tools interconnecting organizations in ways never anticipated when the first web browsers were designed. These changes have occurred so rapidly that, in many ways, we’ve failed to adapt our operational processes to meet current needs. This is especially apparent with web application security, where although most organizations have some security controls in place, few organizations have a comprehensive web application security program. This is a concern for two reasons. First, the lack of a complete program materially increases the chance of failure resulting in a loss-bearing security breach. Second, the lack of a coordinated program is likely to increase overall costs- not just losses from a breach, but the long term costs of maintaining an adequate security level (adequate being defined as meeting all compliance obligations and reducing losses to an acceptable level). This series of posts will show you how to build a pragmatic web application security program that constrains costs while still providing effective security. Rather than digging into the specific details of any particular technology, we’ll show you all the basic pieces and how to put them together. We’ll start with some background on how web applications are different than traditional enterprise applications or commercial off-the-shelf products. We’ll provide basic business justifications for investments in web application security you can use to gain management support (although we’ll be covering these in more depth in future research). The bulk of this series will then focus of the particular security needs of web applications, before delving into details on the major security components and how to pull them together into a complete program. Eventually we plan on releasing this as a white paper, and we already have one sponsor lined up (sponsors can redistribute the content and are acknowledged in the paper, but have no influence on content- it’s just an advertising spot within the larger paper). As with all of our research we rely on you, our readers, to keep us honest and accurate as we develop the research. Technically all analysts do that, but we actually admit it and prefer to engage you directly out in the open- so please comment away as we post. Since I’ve already wasted a ton of space setting up the series, today we’ll just cover the web application security problem. Our next post will provide business justifications for investing in a web application security program, and guidance on building a structured program. The Web Application Security Problem Enterprise web applications evolved in such a way that they’ve created a bit of a conundrum for security. Although we’ve always been aware of them, we initially treated them as low-risk endeavors almost fully under the control of the developers creating them. But before we knew it, they transitioned from experimental programs to critical business applications. Ask any web application developer and they can tell you the story of their small internal project that became an essential business application once they made the mistake of showing it off to a business unit or the outside world. We can break this general evolution down into some key trends creating the current security situation: Before web applications, few businesses exposed their internal transactional systems to the outside world. Even those businesses which did expose systems to business partners on a restricted basis rarely exposed them directly to customers. Web applications grew organically- starting from informational websites that were little more than online catalogs, through basic services, to robust online applications connected to back end systems. In many cases, this transition was the classic “frog in a frying pan” problem. Drop a frog into a hot frying pan, and it will hop right out. Slowly increase the heat, and it will fry to death without noticing or trying to escape. Our applications developed slowly over time, with increased functionality, leading to increased reliance, often without the oversight they might have gotten had they been scoped as massive projects from the beginning. Web application protocols were designed to be lightweight and flexible, and lacked privacy, integrity, and security checks. Web application development tools and techniques evolve rapidly, but we still rely on massive amounts of legacy code. Both internal and external systems, once deployed, are nearly impossible to simply shut down and migrate to new systems. Web application threats evolve as quickly as our applications, and apply to everything we’ve done in the past.

Brian Krebs posted a follow up article on the takedown of fraudulent hosting provider McColo (facilitated by his initial reporting last week). If you think all the nasties out there are hosted in Russia or China, you should really read his article. McColo’s servers weren’t sending out the actual spam; they functioned as the command and control infrastructure for some of the world’s biggest botnets. For those of you who don’t know, spam is rarely sent from static servers anymore; it originates from botnets scattered around the world that are directed by their control network to issue once in a lifetime offers for the best possible deals on male enhancement products. (It’s nice to know everyone has small weewees and lasts about 8 seconds, since otherwise this stuff wouldn’t be so profitable). Since the spam originates from tens of thousands of different systems, it makes it nearly impossible to blacklist based just on IP address. McColo hosted major components of the command infrastructure for spewing out your totally legitimate university diplomas (for a small fee). All those little bots are still out there, but no one is telling them what to do. As Krebs reports, it’s only a matter of time before the network owners reassert control and we can get back to purchasing discount medications and finding true love in former Soviet countries. But what if we took control ourselves and locked out the network? Those servers are still sitting in some building in California, and the ISPs still control the IP addresses. Imagine what we could do if we sent in a research team (or law enforcement) to commandeer all those bots and lock the bad guys out. Yes folks, this is just fantasy today. We don’t have the legal framework to execute such a project without creating risk for the good guys involved. Sure, we could use the botnet to patch all the compromised systems, but that’s effectively breaking into someone’s computer and making changes. I dream of a day when we can more effectively take the fight to the bad guys without worrying about going to jail ourselves. There’s absolutely no chance we can continue this fight indefinitely if we’re always on the defense. But we’re a long way off from having the legal framework and institutions to effectively stand up for ourselves. Share:

Edited: I stupidly credited Nate Lawson for Mark Dowd’s work with Sotirov. Dumb mistake, and I apologize. Since my travel is slowing down a bit, I’m finally able catch up a little on my reading. Two articles this week reminded me of something I’ve been meaning to talk about. First, Chris Wysopal talks about how we’ve reached an application security tipping point. How the OS vendors are doing such a (relatively) good job at hardening the operating system that it’s become easier and more lucrative for attackers to go after common applications. Since nearly everyone online has a reasonably common set of Internet-enabled desktop apps running, it’s nearly as effective as targeting the OS. Heck, in some cases these apps are cross platform, and in a few cases we even see cross platform exploits. To top it off, many of these applications do not activate or use anti-exploitation features like ASLR or DEP, even when it’s little more than a checkbox during the development process. Thus, as we saw during Alex Sotirov and Mark Dowd’s demo at Black Hat this year, you can use these applications to totally circumvent host operating system security, even through the web browser. As Chris states: Whoa. Millions of dollars spent on securing the most prevalent piece of software and it could be meaningless? Yes, it’s true. Since attackers typically only need one vulnerability, if it isn”t in the network, and it isn”t in the host configuration, and it isn”t in the OS, they will happily exploit a vulnerability in an application. Mike Andrews also nails it: They don’t just go away, they go to the next level of lowest hanging fruit. It might be other vendors (Apple, Adobe, Google for example) which may not have the focus that Microsoft has been forced to have, or even worse, smaller players like custom websites or things like WordPress, Movable Type, phpbb, vbulletin, etc — software that has a huge install base, but perhaps not the resources to deal with a full-frontal attack. …snip… Think of it security”s own Hydra — cut of one head (vulns in a major vendor), two grow back (vulns in smaller vendors), and that”s a worrying proposition. As I often say on this blog we have a term for this… it’s called job security. Share:

Here’s a valuable lesson for you college students out there, from Dave Meizlik: if your professor is married to one of the leads at a DLP vendor, think twice before plagiarizing a published dissertation. We talked generally for a while about the problem and then it hit me what if I downloaded a bunch of relevant dissertations, fingerprinted them with a DLP solution, and then sent the girls dissertation through the systems analysis engine for comparison? Would the DLP solution be able to detect plagiarism? It almost seemed too simple. … So when we got home from lunch I started up my laptop and RDP”d into my DLP system. I had my wife download a bunch of relevant dissertations from her school”s database, and within minutes I fingerprinted roughly 50 dissertation files, many of which were a couple hundred pages in length, and built a policy to block transmission of any of the data in those files. I then took her students dissertation and emailed it from a client station to my personal email. Now because the system was monitoring SMTP traffic it sent the email (with the student”s paper as an attachment) to the content analysis engine. I waited a second another and then I impatiently hit send receive and there it was, an automated notification telling me that my email had violated my new policy and had been blocked. I suspect that’s one grad student who’s going to be serving fries soon… Share:

Look I understand too little too late I realize there are things you say and do You can never take back But what would you be if you didn’t even try You have to try So after a lot of thought I’d like to reconsider Please If it’s not too late Make it a… cheeseburger -Here I Am by Lyle Lovett Sometimes I get a little too smart for my own good and think I can pull a fast one over you fair readers. Sure enough, Hoff called me out over my Cloud Computing Macro Layers post from yesterday. And I quote: I’d say that’s a reasonable assertion and a valid set of additional “layers.” There also not especially unique and as such, I think Rich is himself a little disoriented by the fog of the cloud because as you’ll read, the same could be said of any networked technology. The reason we start with the network and usually find ourselves back where we started in this discussion is because the other stuff Rich mentions is just too damned hard, costs too much, is difficult to sell, isn’t standardized, is generally platform dependent and is really intrusive. See this post (Security Will Not End Up In the Network) as an example. Need proof of how good ideas like this get mangled? How about Web 2.0 or SOA which is for lack of a better description, exactly what RIch described in his model above; loosely coupled functional components of a modular architecture. My response is… well… duh. I mean we’re just talking distributed applications, which we, of course, have yet to really get right (although multiplayer games and gambling are close). But I take a little umbrage with Chris’s assumptions. I’m not proposing some kind of new, modular structure a la SOA, I’m talking more about application logic and basic security than anything else, not bolt-on tools. Because the truth is, it will be impossible to add these things on after the fact; it hasn’t worked well for network security, and sure as heck won’t work well for application security. These aren’t add-on products, they are design principles. They aren’t all new, but as everyone jumps off the cliff into the cloud they are worth repeating and putting into context for the fog/cloud environment. Thus some new descriptions for the layers. Since it’s Friday and all I can think about is the Stone Brewery Epic Vertical Ale sitting in my fridge, this won’t be in any great depth: Network: Traditional network security, and the stuff Hoff and others have been talking about. We’ll have some new advances to deal with the network aspects of remote and distributed applications, such as Chris’ dream of IF-MAP, but we’re still just talking about securing the tubes. Service: Locking down the Internet exposed APIs- we have a fair bit of experience with this and have learned a lot of lessons over the past few years with work in SOA- SOAP, CORBA, DCOM, RPC and so on. We face three main issues here- first, not everyone has learned those lessons and we see tons of flaws in implementations and even fundamental design. Second, many of the designers/programmers building these cloud services don’t have a clue or a sense of history, and thus don’t know those lessons. And finally, most of these cloud services build their own kinds of APIs from scratch anyway, and thus everything is custom, and full of custom vulnerabilities from simple parsing errors, to bad parameterization, to logic flaws. Oh, and lest we not forget, plenty of these services are just web applications with AJAX and such that don’t even realize they are exposing APIs. Fun stuff I refer to as, “job security”. User: This is an area I intend to talk about in much greater depth later on. Basically, right now we rely on static authentication (a single set of credentials to provide access) and I think we need to move more towards adaptive authentication (where we provide an authentication rating based on how strongly we trust that user at that time in that situation, and can thus then adjust the kinds of allowed transactions). This actually exists today- for example, my bank uses a username/password to let me in, but then requires an additional credential for transactions vs. basic access. Transaction: As with user, this is an area we’ve underexplored in traditional applications, but I think will be incredibly valuable in cloud services. We build something called adaptive authorization into our applications and enforce more controls around approving transactions. For example, if a user with a low authentication rating tries to transfer a large sum out of their bank account, a text message with a code will be send to their cell phone with a code. If they have a higher authentication rating, the value amount before that back channel is required goes up. We build policies on a transaction basis, linking in environmental, user, and situational measurements to approve or deny transactions. This is program logic, not something you can add on. Data: All the information-centric stuff we expend endless words on in this blog. Thus this is nearly all about design, with a smidge of framework and shared security services support we can build into common environments (e.g. an adaptive authentication engine or encryption services in J2EE). No loosely coupled functional components, just a simple focus on proper application design with awareness of the distributed environment. But as Chris also says, It should be noted, however that there is a ton of work, solutions and initiatives that exist and continue to be worked on these topics, it’s just not a priority as is evidenced by how people exercise their wallets. Exactly. Most of what we write about cloud computing security will be ignored… … but what would we be if we didn’t even try. You have to try. Share:

I have to say, Moscow was definitely one of the more interesting, and difficult, places I’ve traveled to. The city wasn’t what I expected at all- everywhere you look there’s a park or big green swatch down major streets. The metro was the cleanest, most fascinating of any city (sorry NY). I never waited more than 45 seconds for a car, and many of the stations are full of beautiful Soviet-era artwork. In other ways it was more like traveling to Japan- a different alphabet, the obvious recognition of being an outsider, and English (or any Western European language) is tough to find outside the major tourist areas. Eating was sometimes difficult as we’d hunt for someplace with an English menu or pictures. But the churches, historical sites, and museums were simply amazing. We did have one amusing (after the fact) incident. I was out there for the DLP Russia conference, at a Holiday Inn outside of Moscow proper. We requested a non-smoking room, which wasn’t a problem. Of course we’re in a country where the average 3 year old smokes, so we expected a little bleed-over. What we didn’t expect was the Philip-Morris conference being held at the same hotel. So much for our non-smoking room, and don’t get me started on the smoking-only restaurant. Then there was my feeble attempt to order room service that led to the room service guy coming to our room, me pointing at things on the menu, and him bringing something entirely different. Oh well, it was a good trip anyway. Now on to the week’s security summary: Webcasts, Podcasts, Outside Writing, and Conferences: I spoke at a DLP Executive Forum in Dallas (a Symantec event). Over at TidBITS I explain how my iPhone rescued me from a travel disaster. Although I wasn’t on the Network Security Podcast this week, Martin interviewed Homeland Security Secretary Chertoff. Favorite Securosis Posts: Rich: I finally get back to discussing database encryption in Data Encryption, Option 1- Media Protection. (Adrian’s follow up is also great stuff). Adrian: Examining how hackers work and think as a model for approaching Data Discovery & Classification. Favorite Outside Posts: Adrian: Nothing has been more fascinating this week than to watch the Spam stories on Brain Krebs blog. Rich: Kees Leune offers up advice for budding security professionals. Top News: One small step at the ISP, one giant leap for the sanity of mankind. 8e6 and Mail Marshal merge. AVG Flags Windows DLL as a virus, scrambles to fix false positive Jeremiah Grossman on how browser security evolves. Apple updates Safari. Oh yeah, and Google Chrome and Firefox also issued updates this week. Google also fixes a critical XSS vulnerability in its login page. Microsoft patches a 7 year old SMB flaw, which leads Chris Wysopal to talk about researcher credit. Researchers hijack storm worm. I think it’s this kind of offensive computing we’ll need to manage cybercrime- you can’t win on defense alone. Blog Comment of the Week: Ted on my Two Kinds of Security Threats post: You get more of what you measure. It’s pretty easy to measure noisy threats, but hard to measure quiet ones. Fundamentally this keeps quiet threats as a “Fear” sell, and nobody likes those. Share:

First he exposes the Russian Business Network and forces them to go underground, now he nearly single-handedly stops 2/3rds of spam. Most tech journalists, myself included merely comment on the latest product drivel or market trends. Brian is one of the only investigative journalists actually looking at the roots of cybercrime and fraud. On Tuesday, he contacted the major ISPs hosting McColo– a notorious hosting service whose clients include a long roster of cybercriminals. At least one of those ISPs pulled the plug, and until McColo’s clients are able to relocate we should enjoy some relative quiet. Congrats Brian, awesome work. This also shows the only way we can solve some of these problems- through proactive investigation and offense. We can’t possibly win if all we do is run around and try and block things on our side. Share:

I was reading this article over at NetworkWorld today on a study by a commercial DNS vendor that concluded 1 in 4 DNS servers is still vulnerable to the big Kaminsky vulnerability. The problem is, the number is more like 4 in 4. The new attack method that Dan discovered is only slowed by the updates everyone installed, it isn’t stopped. Now instead of taking seconds to minutes to compromise a DNS server, it can take hours. Thus if you don’t put compensating security in place, and you’re a target worth hitting, the attacker will still succeed. This is a case where IDS is your friend- you need to be watching for DNS traffic floods that will indicate you are under attack. There are also commercial DNS solutions you can use with active protections, but for some weird reason I hate the idea of paying for something that’s free, reliable, and widely available. On that note, I’m going to go listen to my XM Radio. The irony is not lost on me. Share:

There’s been a lot of discussion on cloud computing in the blogosphere and general press lately, and although I’ll probably hate myself for it, it’s time to jump in beyond some sophomoric (albeit really funny) humor. Chris Hoff inspired this with his post on TCG IF-MAP; a framework/standard for exchanging network security objects and events. It’s roots are in NAC, although as Alan Shimel informs us there’s been very little adoption. Since cloud computing is a crappy marketing term that can mean pretty much whatever you want, I won’t dig into the various permutations in this post. For the purposes of this post I’ll be focusing on distributed services (e.g. grid computing), online services, and SaaS. I won’t be referring to in the cloud filtering and other network-only services. Chris’s posting, and most of the ones I’ve seen out there, are heavily focused on network security concepts as they relate to the cloud. but if we look at cloud computing from a macro level, there are additional layers that are just as critical (in no particular order): Network: The usual network security controls. Service: Security around the exposed APIs and services. User: Authentication- which in the cloud word will need to move to more adaptive authentication, rather than our current username/password static model. Transaction: Security controls around individual transactions- via transaction authentication, adaptive authorization, or other approaches. Data: Information-centric security controls for cloud based data. How’s that for buzzword bingo? Okay, this actually includes security controls over the back end data, distributed data, and any content exchanged with the user. Down the road we’ll dig into these in more detail, but anytime we start distributing services and functionality over an open public network with no inherent security controls, we need to focus on the design issues and reduce design flaws as early as possible. We can’t just look at this as a network problem- our authentication, authorization, information, and service (layer 7) controls are likely even more important. This gets me thinking it’s time to write a new framework… not that anyone will adopt it. Share:

I do believe I am officially setting a personal best for the most extended blog series. Way back in February, before my shoulder surgery, I started a series on database encryption. I not only don’t expect you to remember this, but I’d be seriously concerned about your mental well being if you did. In that first post I described the two categories of database encryption- media protection, and separation of duties. Today we’re going to talk more about media encryption, and the advantages of combining it with database activity monitoring. When encrypting a database for media protection our goal is to protect the data from physical loss or theft (including some forms of virtual theft). This won’t protect sensitive content in the database if someone has access to the DB, but it will protect the information in storage and archive, and may offer realtime protection from theft of the database files. The advantage of encryption for media protection is that it is far easier to implement than encryption for separation of duties, which involves mucking with the internal database structures. The disadvantage is that it provides no internal database controls, and thus isn’t ideal for things like credit card numbers where you need to restrict even an administrator’s ability to see them. Database encryption for media protection is performed using the following techniques/technologies: Media encryption: This includes full drive encryption or SAN encryption; the entire storage media is encrypted, and thus the database files are protected. Depending on the method used and the specifics of your environment, this may or may not provide protection for the data as it moves to other data stores, including archival (tape) storage. For example, depending on your backup agent, you may be backing up the unencrypted files, or the encrypted storage blocks. This is best suited for high performance databases where the primary concern is physical loss of the media (e.g. a database on a managed SAN where the service provider handles failed drives potentially containing sensitive data). Any media encryption product supports this option. External File/Folder Encryption: The database files are encrypted using an external (third party) file/folder encryption tool. Assuming the encryption is configured properly, this protects the database files from unauthorized access on the server and those files are typically still protected as they are backed up, copied, or moved. Keys should be stored off the server and no access provided to local accounts, which will offer protection should the server become compromised and rooted by an external attacker. Some file encryption tools, such as Vormetric or BitArmor, can also restrict access to the protected files based on application. Thus only the database processes can access the file, and even if an attacker compromises the database’s user account, they will only be able to access the decrypted data through the database itself. File/folder encryption of the database files is a good option as long as performance is acceptable and keys can be managed externally. Any file/folder encryption tool supports this option (including Microsoft EFS), but performance needs to be tested since there is wide variation among the different tools. Remember that any replication or distribution of data handled from within the database won’t be protected unless you also encrypt those destinations. Native Database Object Encryption: Most current database management system versions, such as Oracle, Microsoft SQL Server, and IBM DB2 include capabilities to encrypt either internal database objects (tables and other structures) or the data stores (files). This is managed from within the database, and keys are typically stored internally. This is overall good option in many scenarios as long as performance meets requirements. Depending on the platform, you may be able to offload key management to an external key management solution. The disadvantage is that it is specific to each database platform, and isn’t even always available. The decision on which option to choose depends on your performance requirements, threat model, exiting architecture, and security requirements. Unless you have a high-performance system that exceeds the capabilities of file/folder encryption, I recommend you look there first. If you are managing heterogeneous database, you will likely look at a third party product over native encryption. In both cases, it’s very important to use external key management and not allow access by any local accounts. The security of database encryption for media protection is greatly enhanced when combined with database activity monitoring. In this scenario, the database content is protected from loss via encryption, and internal data protected against abuse by database activity monitoring. I’ve heard of this combination being used as a compensating control for PCI- the database files are encrypted to prevent loss, while database activity monitoring is used to track all access to credit card numbers and generate alerts for unapproved access, such as a DBA running a SELECT query. Share: