Remember the dead or alive game Howard Stern used to do? I think it was Stern. Not sure if he’s still doing it because I’m too cheap to subscribe to Sirius for the total of 5 minutes I spend in the car driving between coffee shops. Pen testing has been under fire lately. Ranum has been talking for years about how pen testing sucks. Brian Chess also called pen testing dead at the end of 2008.
It’s almost two years later and the death of pen testing has been greatly exaggerated. Pen testing is not dead. Not by a long shot. But it is changing. And we have plenty of folks weighing in on how this evolution is taking place.
First off is the mouth from the South, Dave Maynor. OK, one of the mouths from the South, because I suspect I am another. Dave made some waves regarding whether to use 0-day exploits in a pen test, and then had to respond when everyone started calling him names. Here’s the thing. Dave is right. The bad guys don’t take an oath when they graduate from bad guy school that they won’t use 0-days. They can and do, and you need to know how you’ll respond. Whether it’s part of a pen test or incident response exercise doesn’t matter to me. But if you think you don’t need to understand how you’ll respond under fire, you are wrong.
Second, I got to attend a great session by Dave Kennedy and Eric Smith at BSides Atlanta about strategic pen testing. It was presented from the viewpoint of the pen tester, but you can apply a lot of those lessons to how a practitioner runs a pen test in their organization. First off, a pen test is about learning where you can be exploited. If you think it’s about checking a box (for an audit) or making yourself and your team look good, you’ve missed the point. These guys will break your stuff. The question is what can you learn and how will that change your defensive strategies?
The pen testers need to operate in a reasonable semblance of a real wold scenario. Obviously you don’t want them taking down your production network. But you can’t put them in a box either. The point is to learn and unless their charter is broad enough to make a difference, again you are wasting your time.
Finally, I’ll point to a presentation by Josh Abraham, talking about his “Goal Oriented Pentesting” (PDF) approach. It’s good stuff. Stuff you should know, but probably don’t do.
What do all these things have in common? They talk about the need for pen testing to evolve. But by no means are they talking about its death. Listen – at the end of the day, whether you are surprised by what an attacker does to your network is your business.
I still believe pen testing can provide insights you can’t get any other way. I think those insights are critical to understanding your security posture. Those enlightened organizations whihc don’t pen test do so at their own risk. And the rest of us should thank them – they are the slow gazelles and the lions are hungry.
Reader interactions
2 Replies to “Dead or Alive: Pen Testing”
Talk to Jack Mannino about how application security testing has evolved. What Jack has done in the past is provide functional component-level testing scripts to clients after an app assessment. These scripts leverage the testing activities that would be performed by penetration-testers in order to find a proverbial rabbit hole.
However, they technically shouldn’t be called scripts. They should be called a test harness. These tools are exactly like the ones used by software quality engineers.
In the book, “The Art of Software Security Assessment”, there are several code-audit strategies lined out in chapter 4. The authors present 3 primary categories in order to complete an app assessment: code comprehension, candidate points, and design generalizations. The authors appear to recommend leveraging existing test harnesses (i.e. built by developers or software quality engineers) and using black-box “hits” to determine where in the code to focus attention and risk management (i.e. find that proverbial rabbit hole).
Jack Mannino did this, but also took it a step further by providing a regression test suite based on his findings. I suggest using an open format for the test cases when doing web application assessments, such as the OWASP Webscarab format, or perhaps the Content App Tool byte-level HTTP Request/Response format.
Veracode and WhiteHat Security do not provide test harness or test cases to customers. However, they do provide free regressions for a year (or thereafter as long as you are a paying customer of theirs). This is how pen-tests have already changed.
Brian Chess had a plan to replace pen-testing. It did not work out, and I remain skeptical that it will. His plan was to provide “security coverage”, mapping the dynamic testing results of a web application security manual pen-test to the static analysis results of the Fortify SCA engine’s ability to see sources and sinks. The “coverage” would tell a black-box tester how much coverage (e.g. 9/100 or 9%) his or her tests were achieving tied to the way that the SCA saw both the sources and sinks in the app.
Unfortunately, the tool that Fortify used to do this was ripe with issues, including the fact that it only worked on Java Enterprise and .NET — and even then only on some platforms. It could not be used in production because the technology slowed down the application server. There was no way to run a web application security scanner or other fast, automated test harness against the app when in this mode. There were other performance and optimization issues — perhaps some of which have been solved (or are being worked on) now that Hybrid 2.0 with HP is becoming more of a reality.
However, even if HP and Fortify can pull off this magical feat — it will not replace penetration-testing, nor may it become very popular. The problem with appsec products is that they are driven by customer demand. The problem with our appsec industry is that it is extremely immature and filled with too many charlatans (one could say that penetration-testing has the same exact problem — except that it is a little more mature). Thus, the customers are not very witty or capable. They haven’t read or internalized “The Art of Software Security Assessment” so they are at least 4 years behind public information, which was probably available at ISS 14 years ago.
Worse, the security boutiques that can fluently speak both penetration-testing and appsec guard their secrets very carefully. They don’t work together or co-operate. Clients don’t get the Gotham Digital Science people working with their own internal people — let alone working with the iSec Partners people on some big project. You see Cigital with their snobby “we invented everything” attitude bumping heads against Trustwave with their “our clients are the most important” attitude. You see BT with their “we’re larger than everyone” evil grin. You see IOActive with the “We used to have Dan Kaminsky working for us” idiocracy. Then you’ve got appsec or pen-test teams from FishNet or Neohapsis trying to undercut everyone — or places like Accuvant trying to get into business that they shouldn’t because they don’t have the talent to back it up. Every single one of these shops are elitist and annoying. They all have at least one huge character flaw — a hubris. Thus, the immaturity of these industries permeates.
We don’t even use the right language to describe this stuff.
Goal-oriented pen-testing is pen-testing. Pen-testing is never time-boxed. You hire a red team, always at a flat fee, and they break into the target that you propose using an agreed-upon set of guidelines (i.e. no killing or cutting off people’s hands or forcing them to say their passphrase at gunpoint, thank you to Sneakers). IT DOESN’T MATTER IF THEY TAKE 2 DAYS, 2 WEEKS, or 2 YEARS. That’s the whole point. You are trying to see how long it takes them!
What most people refer to penetration-testing is actually Ethical Hacking. Ethical hacking is a time-boxed activity where the tester(s) simulate what an actual hacker would do in order to break into the target scoped network(s) or app(s). Unfortunately, you have places like FishNet claiming that they can perform ethical hacking activity that is as complete as a more thorough security boutique in 2 days for $2k — no matter the size of the target scoped network(s) or app(s).
Worse, you have shops like WhiteHat Security who attempt to sell “levels” of ethical hacking — where they have tiered structures that aim to protect against specific threats: script kiddies, hacking groups, or APT. Unfortunately, that’s not how the underground economy works. In the underground hacking forums — everything is for sale by script kiddies, by hacking groups, by APT — for use by anyone willing to rent a botnet or trade a secret.
My summary point here is that ethical hacking and security boutique penetration-testing is NOT changing. Online criminals work together and are growing and sharing more than ever. Security boutiques and big shops DO NOT share information, do NOT work together, and we are not maturing.
How about a change to how we pay pentesters? If you don’t root our DB, you don’t get paid.