A Question
If you can tell, with absolute certainty, that systems are vulnerable to an exploit without needing to test the mechanism, what good is served by releasing weaponized attack code immediately after patches are released, but before most enterprises can patch?
Unless you’re the bad guy, that is.
-rich
Andre Gironda Jul 24
1) No exploit; No IPS “virtual-patch” for said exploit. No IPS “virtual patch” for said exploit; no way to keep IPS vendors in business. Remember that IPS is both about offensive and defensive computing in the security market!
2) No way to create a snort filter or other IDS or network monitoring way of dealing with the problem
3) For those who actually care about things like how code works, how exploits work, etc - they absolutely must see it to believe it and understand how it works properly. “Loose standards and running code” is what makes the Internet work if you weren’t aware of the IETF/Linux/FOSS mantra
4) Reverse engineering goes both ways. For those who have DNS servers that need to be fixed, they can reverse the exploit into a patch/fix.
Look for number 4, it gets really complicated. Maybe there is an easy way to block this attack. If we don’t let everyone understand how it works, then those minds cannot understand how the exploit works, thus they also can’t imagine a fix.
This concept goes much more deep that one would believe at first glance. How many implementations of a DNS server are out there? 5? More like 500. How many one-off patches have been applied to existing source code bases?
Ever hear of http://www.nlnetlabs.nl/projects/nsd/
?
I can’t tell from their mailing-list or release notes if they have fix or not. Or even if they know about the security problem yet.
Do these patches break djbdns? http://www.fefe.de/dns/
?
I don’t know; maybe they do. Maybe others break something.
Apparently MaraDNS and Deadwood are ok. http://www.maradns.org
But how about your home-rolled DNS server? Is is ok?
I colo’d with a guy once who wrote his own nameserver implementation. I wonder how his code is fairing up right now. Maybe with this exploit, he can figure out how to fix his code so that he isn’t vulnerable. He tends to keep up on this sort of thing. It must be really annoying for him to have to buy a last-minute ticket to BlackHat in order to learn about the details of the vulnerability. Now, he doesn’t have to do that!
Question: What if Dan’s test software has false positives, false negatives, or Type III errors (aka false-false positives)? How do we know that Dan’s code works universally with everything? Has his test-software been tested?
I hate to be the devils advocate here, but what Dan Kaminsky did (and you with him) really fucking pisses me off. I expected as much from Paul Vixie, but he’s not even in the security industry. The real losers here are you. Egg on your face!
TJ Jul 24
An excellent question indeed…
dave Jul 24
clearly full disclosure has its (un)intended consequences…
Jack Daniel Jul 24
Even though it was a rhetorical question, I’ll answer-
Ego.
Either that, or the embarrassment with the size of…
rmogull Jul 24
Dre-
–This concept goes much more deep that one would believe at first glance. How many implementations of a DNS server are out there? 5? More like 500. How many one-off patches have been applied to existing source code bases?
Ever hear of http://www.nlnetlabs.nl/projects/nsd/
?–
Doesn’t matter. If there isn’t source port randomization, it’s vulnerable. If there is, it’s not (okay, less vulnerable).
–Do these patches break djbdns? http://www.fefe.de/dns/
?–
djbdns isn’t vulnerable. It uses source port randomization by default.
–But how about your home-rolled DNS server? Is is ok?–
Only if it uses source port randomization.
–Question: What if Dan’s test software has false positives, false negatives, or Type III errors (aka false-false positives)? How do we know that Dan’s code works universally with everything? Has his test-software been tested?–
Doesn’t matter- look for source port randomization, that’s al he does.
–I hate to be the devils advocate here, but what Dan Kaminsky did (and you with him) really fucking pisses me off. I expected as much from Paul Vixie, but he’s not even in the security industry. The real losers here are you. Egg on your face!–
Us with egg? That’s amusing. Why are you pissed off, because no one let you know? Guess what- you didn’t need to know. IDS vendors were given signatures, DNS vendors patched, and the rest of the info was coming out in 30 days. You, like many others, just seem pissed no one catered to your ego. A patch was chosen that didn’t directly correlate to the vulnerability, purposely to give the good guys time to patch. WTF, you wanted metasploit exploits out the day after the patches were released? Who would that help?
This was a weird case- no bindiff to immediately reverse the patch and fgure out an exploit. Why the hell do we need to produce weaponized code before the bad guys? Most of the people pissed at Dan seem to be jealous or religious about FD and unwilling to change their philosophy.
They can go to church- this is the real world.
CG Jul 24
easy. To keep pentesters, researchers, bloggers, journalists, security vendors, software vendors, bad guys, etc in business. To give full disclosure people stuff to bitch about, to give no disclosure people stuff to bitch about and everyone in between.
And to keep everyone is this business employed and the public scared of the bad bad hackers out there in the world just like everyone else does with terrorists so they can continue to buy product X,Y, or Z or training on X,Y, or Z, or the book about X, Y, or Z or to support the podcast talking about X, Y, Z. Without our little “emergencies” no one would regularly want to spend the cash.
But probably mostly just to see if they could do it. What fun is it to figure something out and not share it?
windexh8er Jul 24
To respond to the base question…
If the vulnerability is publicized in such a manner that you give the community a teaser you will provoke interest. Not only from a malicious side but equally from an understanding side. You can’t keep things like this under wrap for long and since you have both sides knowing the end result because the vulnerability is ultimately trivial it makes more sense to create PoC code so that others can properly defend themselves or see the inherent risk. This is what Andre already stated and is how the world actually works — we don’t live in a state of utopia where you can politely ask everyone to just ignore it, but patch it until someone wants to release it. If anyone thinks it will work any differently, given the way it was presented to the world, they’re high as a kite.
To me, the way it was presented to the public and the people doing the presenting are on some levels hypocrites. Let’s make a huge deal about this, press release, succumb to industry pokings but then get mad about the fact that it eventually came out before the 30 day (enter eternity in Internet-land when you consider the landscape of knowledge traversal here). I’d have to say WTF were you thinking when you thought a BlackHat press release was good for this? That target audience is the people you know are the ones who will present their case if they find it. I commend Dan up until the point of the saying too much through the PR channels and asking for 30 days grace. He did a fabulous job — but that was his fatal move, and whoever talked him into making that move, if it wasn’t his own conscience, is the one to blame here.
So, basically what happened is someone wanted to be in the spotlight… They wanted everyone else to know that they knew and that they knew how insanely evil this was. I’m not sure this was at all in the best interest of the community as a whole, but more so to get in on the limelight.
In the end it should have been communicated through vendor channels as a high risk with little to no information around what it actually was. Obfuscation would have lasted longer in this case and the deadline for patching shouldn’t have been revolving around a BlackHat conference (whos bright idea was that?), it should have been meeting a quantifiable goal as a percentage of major outlets being fixed prior to really going public.
This was a PR gold mine for certain people. I think the only one that should have been in that light in the end was Dan. The rest of ya’ll f’d it up. I could have cared less about the vulnerability and happily patched. But, you guys put the cookie next to the milk and told the 5 year old to wait until it was stale before they could eat it and walked away expecting something you knew wouldn’t happen.
Oh and, NLnet is not vulnerable to clarify:
http://www.kb.cert.org/vuls/id/MIMG-7EMJVA
Andre Gironda Jul 24
@ Rich
You said, “Why are you pissed off, because no one let you know? Guess what- you didn’t need to know.”
I am going to try and go real slow here for you with this. I hate to be the harbinger of evil, but it’s really the opposite. You assume too much. Your side assumes too much. The reason that the big shots in this industry (e.g. Tom Ptacek, Halvar Flake, |)ruid, hdm, et al) are on one side of this full-disclosure issue — while they are normally on the responsible disclosure issue — is because of the severity of this vulnerability and the partial disclosure of the details.
Rich, you don’t realize how much software is out there. When you say, “IDS vendors were given signatures, DNS vendors patched, and the rest of the info was coming out in 30 days”: it is clear to me that you don’t understand software. What you mean to say is that “… some IDS vendors” … “some DNS vendors” (why does one have to be a vendor to have software?!).
Then you say, “the rest of the info was coming out in 30 days”? What is that about? 30 days is way too long to know about such a severe problem. For every DNS implementer or hobbyist to have to buy a ticket to BlackHat just to be the first to hear about it is certainly more tied to ego and helping the “wrong people” (special note to people not going to BlackHat for free: it’s EXPENSIVE) than Halvar Flake’s discovery or |)ruid/hdm/Desfossez’s weaponized exploits.
Also note the difference here, for people who don’t know -
Weaponized exploit: works for attackers
PoC exploit: a proof-of-concept that is broken to only show understanding and not be an attacker tool
I would have rather seen PoC code. Kaminsky never released it. Maybe because he didn’t have it? Maybe because it didn’t work? Maybe because he couldn’t make any money or fame out of it?
What HalvarF/hdm/|)ruid/Desfossez have done helped much more than any of the work Dan Kaminsky, Paul Vixie, or thousands of other elitist “vendors” could have ever done. They identified the problem and told the world about it. I think perhaps the issue with Dan Kaminsky is that he had no PoC. He had no proof. If he did have proof, then where was it when Halvar discovered the vulnerability? Releasing proof now is hearsay in my eyes. It’s too little; too late.
You are correct that I didn’t need to know. I’m not an operator right now, nor do I maintain any DNS server software. I simply don’t care about the vuln, how it works, or whatever else. However, I am merely trying to get you (and possibly Dan, the Hoff, and others on your side) to understand what you have done wrong. I know that Paul Vixie will never come around, but he isn’t a “security analyst” or “vulnerability researcher”.
You also said to me, “You, like many others, just seem pissed no one catered to your ego.”
In reality, my care around this issue is about how it was handled. The immaturity of the people at the top (i.e. the “decision-makers”) is astounding. It’s like you’ve never handled or confronted this problem before, and didn’t bother to ask experts or listen to their advice. Let me say from experience that I’ve handled 10, maybe 15, of these global-reaching critical incidents as both an operator and vulnerability researcher (sometimes both on the same exploit). It really hits home to me. Maybe next time you’ll know how to do the right thing.
Experience lets me have an opinion on this. Your lack of experience means that you must be the one with the ego problem. Try to fit in this category for once and then you’ll definitely understand:
http://spiresecurity.typepad.com/spire_security_viewpoint/2007/01/lowering_the_ba.html
Rich, you continue to misunderstand by stating, “A patch was chosen that didn’t directly correlate to the vulnerability, purposely to give the good guys time to patch. WTF, you wanted metasploit exploits out the day after the patches were released? Who would that help?”
I never wanted anyone to speak about “vulnerability this” or “exploit that”. Dan Kaminsky started it, Halvar Flake “speculated”, and hdm and |)ruid ended it.
I personally would have rather that Dan Kaminsky kept his mouth shut and realized that the vulnerability really isn’t at all that serious. Compared to Slipping the Window, the Cisco IPv4 vulnerability, the SNMP PROTOS vulnerability, many various IPv4/IPv6 vulnerabilities (e.g. land, teardrop, et al), SSH vulnerabilities, the TCP SYN attack, TCP/UDP/ICMP amplification attacks, and “real” critical infrastructure CRASHES in software such as Apache, SCADA, or BGP… Dan’s bug is Kindergarten-class and over-hyped to the max. Compared to something called THE MORRIS WORM, none of these compare in the slightest. You should know that “it always could be worse”.
If he contacted everyone he possibly could under the radar and stayed quiet for two years or so, how different could this have been? If Dan had just kept his mouth shut, kept this to himself without “testing” the problem, only making sure that the code itself was “obviously secure”, then we wouldn’t need a weaponized exploit to demonstrate how it’s “obviously insecure”.
Once he started with his mouth (and you as well) is when the world had to know — or else suffer the consequences. You are the ones that broke the disclosure. What the exploits writers did is just empower everyone that you missed, all the people that you forgot about, and anyone lower on the totem pole than your “elite crew” to understand the problem as an attacker would.
You/Dan == disservice
Halvar/hdm/others == aid to those who most need it
This is not about full-disclosure to me and many others. This is about proper disclosure when disclosure is partially available.
If you want to answer my retort properly, you’ll address these parts:
“3) For those who actually care about things like how code works, how exploits work, etc - they absolutely must see it to believe it and understand how it works properly. “Loose standards and running code” is what makes the Internet work if you weren’t aware of the IETF/Linux/FOSS mantra
4) Reverse engineering goes both ways. For those who have DNS servers that need to be fixed, they can reverse the exploit into a patch/fix.
Look, for number 4, it gets really complicated. Maybe there is an easy way to block this attack. If we don’t let everyone understand how it works, then those minds cannot understand how the exploit works, thus they also can’t imagine a fix.”
Michael R. Farnum Jul 24
You know, maybe I am making this way to simple, but I have to say that most people who care about this exploit in a real world fashion are operators. And truthfully, how many of those guys and gals need to know the distinct and exact details of this attack? Does every server admin need to know every nitty gritty about flaws that are patched in Windows or Linux? No. They need a patch. They need something they can throw on a test bed (if they are lucky) to make sure it won’t crash their shit, then it goes into production.
So this whole debate about what Dan did AFTER he announced this is fruitless. Maybe this went a little far, but holy crap, I am tired of hearing about it! Let Dan have his day, and then move on…
Pepper Jul 24
“If you can tell, with absolute certainty”
Unfortunately, that isn’t good enough. Dan was sure. You believed him. I believe him. At the time, lots of others disbelieved. HF & Matasano made a lot of believers, as would an exploit.
But as a rule, there are always doubters. An exploit is the proof they demand and require (and some will still be skeptics). Not that it’s always worth delivering, but that’s the main (non-ego) reason one would release an exploit (assuming not a “bad guy”).
Andrew Jaquith Jul 25
As usual, this debate ignores the only question that matters: does weaponized attack code, distributed well before most systems could be reasonably expected to be patched, help or harm customers? It clearly harms them. Arguing that PoC/exploit code is necessary to convince Thomas, Dre or other researchers is a red herring.
It is telling that the word “customer” is not found in anywhere on this page before now.
One thing I do agree with: teasing everyone with “I’ve got a vuln but I can’t tell you what it is” was what set much of this in motion. That smacks of ego, just as much as HD’s Metasploit release does.
Christofer Hoff Jul 25
As is usually the case, I agree with AJ.
Dan’s presentation of the vuln. is as much an issue as HD’s release of the exploit code. Let’s put that to bed already.
As Andy stated and I mentioned in my poem:
When researchers consider
how to disclose and thus when
will you think of the users?
How it might affect them?
This ego-fueled rush
to put your name on a vuln
has a much bigger impact
than you might have known
If the point here is really
to secure and protect
then consider what image
you really project
…I’m debating via email with a very well known researcher at the moment to try and gain better perspective, but I honestly believe that *my* customers and my customer’s customers are done a disservice when exploit code is released before they have a chance to properly assess and enforce testing, change control, SLA and legal issues.
Dre, from your operations background you *have* to know that you’re ignoring this side of the issue for sake of propping up the disclosure argument…”do no harm” means exactly that.
It just ain’t as easy as pushing go on a patch 10 seconds after it’s issued. If researchers would understand that, we’d be closer in being able to discuss this rationally.
This disconnect MUST be resolved.
/Hoff
ivan Jul 25
Hi Rich.
First of all, let me point out that your question is not a valid one:
“If you can tell, with absolute certainty, that systems are vulnerable to an exploit without needing to test the mechanism what good is served by releasing weaponized attack code…”
You are stating and hypothesis, assuming that it holds true and then following up with a question based on your assumption.
I hate to break the news to you but… guess what? you CANT tell with absolute certainty that systems are vulnerable to an exploit if you don’t have such exploit. An “exploit” is not an abstract concept or a theoretical attack. It is an actual software artifact: real code that runs on real systems and targets other real systems or persons.
If you are so oriented I invite you to consider what are the implications of Rice’s theorem to the field of vulnerability research and claiming proof of exploitability or non-exploitability.
Nonetheless here are some of my answers to your question:
1. How can you tell beyond any doubt (not in theory) that a specific environment is vulnerable (even if it using vulnerable software)
2. How can you tell that the patch actually fixes the problem?
3. How can you tell that you are no longer vulnerable after installing a patch?
4. How can you differentiate systems that were fixed from those that were not?
5. How can you tell if any mitigation strategy other than deploying a patch is actually working?
6. How can you tell if you can detect a attack that exploits the bug?
Ok, let’s talk about customers now or rather “affected users” because the term “customer” implies some form of commercial relationship that may be too restrictive for an issue that affects almost all TCP/IP networks on a global scale (a superset of the 1nternecz ).
You said:
“Doesn’t matter. If there isn’t source port randomization,
it’s vulnerable. If there is, it’s not (okay, less vulnerable).”
oh really? prove both statements or indicate that you are just asking me to trust your claims but not stalking about factual data.
Now also prove that source port randomization is the only solution to this problem.
Finally, also prove the source port randomization is the best possible solution to the problem in every possible scenario (and therefore just the patch is all I need)
BTW, while you are at that also please let me know what is the potential harm that source port randomization could cause to my environment and how to address it.
I, a very busy non-criminal security practitioner with lots of security fires to put off (aka “customer” aka “affected user”) may legitimately say:
*prove* you claims about my network infrastructure or I wont be able to redirect resources to deploy patches and then have my team cleanup all the mess that the patch will bring to all my other network security controls.
Good luck doing that without an exploit…
Now, lets look beyond the specific case of Dan Kamisnky’s latest DNS attack technique (hello? hello? it is not a new vuln, it is a new attack technique can anybody tell the difference?) and beyond the source port randomizing patch that some people may think to be “sub-optimal” (shhh, I will tell you who, how or why at BAcon 2008).
Not all attackers are criminals therefore you cant conclude that attack tools are built exclusively for criminals, it then follows that there must be some legitimate, non-criminal use for attack code must it not?
If that is the case we would then enter the realm of discussion about who benefits the most from attack code, criminals or non-criminals? (disregarding if they are in an attacker or defender role).
Andrew Jaquith says:
“As usual, this debate ignores the only question that matters: does weaponized attack code, distributed well before most systems could be reasonably expected to be patched, help or harm customers?”
@Andrew: what do you consider “reasonably expected”? Is your consideration universally valid or are others allowed to consider differently?
Do you consider that systems could be “reasonable expected to be patched” if a patch that is disruptive to network security devices is thrown at admins and security staff without any tool or technical details that let them assess the impact of deploying it on their infrastructure? Would the lack of such assessment tool accelerate or delay patch deployment?
You answered your own question with the following conclusion:
“It clearly harms them.”
“clearly” where? Either *substantiate* your claim or indicate that it is simply a wild guess with no relevant data or facts to support it.
BTW, I would be looking more more than a few examples of correlation of events to substantiate your conclusion. I would be looking for proof of causality nor correlation.
As a “customer” I believe I’m being told “install patch or fear the wrath of the Big Bad Guy That Will Boil the 1nternekz, trust us, we are 16 and we know how what is best for you and everybody else”
Forcing a single binary option upon customers is a very particular way of “helping” them. In fact giving them a potentially broken patch as their only option may actually hurt rather than help them. How do I tell that the patch is not broken? (oh yeah, right, because it was developed by the same people that developed the broken software that it patches… no, no wait, it is not that, it is because it relies on security assumptions that are dependent on the underlying OS and network protocol layers.. hmm right I should have known)
Exploit code helps “customers” to test their systems and to evaluate other options. It also helps the entire security community (other than the issuers of the official patch) to provide alternative solutions to their customers.
Arguing that exploit code does not help non-criminal users can be debunked easily: You cant prove that all users of exploit code are criminals and therefore you should consider them innocent until you can prove otherwise.
Andrew, you talk about “customers” but when referring to Metasploit who are exactly these customers? Should we assume that users that are not paying “customers” are more likely to be criminals and therefore not entitled to the receive attack code that “paying customers” consider valuable for their legitimate purposes?
ivan Jul 25
@Hoff: *my* customers and *my* customer’s customers believe that they are given a very valuable service when they receive exploit code that lets them manage their IT security risk better. I am sure that some of HD Moore’s “customers” feel likewise. So where does that leave us?
Hey, it may even be the case that the intersection of our respective customer sets is not an empty set. So what do we do now?
Incidentally, the software vendors that we report vulnerabilities to invariably ask us for exploit code to prove our claims. Some of them may even secretly feel we’ve done them a good service by reporting bugs to them and providing exploit code. Some others say so explicitly.
If vendors that have total access to the vulnerable software’s source code, design documentation and development teams ask for exploit code how could that same exploit code not be potentially useful and valuable to the vendor’s customers? Their customers are vulnerable organizations that do not have access to any of the vendor’s proprietary resources, they are less equipped to assess the risk and potential impact of the vulnerability.
Is the vendor the one and only organization entitled to provide assistance to the vendor’s customers?
It is unfair to claim ownership of the global customer population’s position on this topic and deny the same type of ownership to “researchers”.
It is also unfair to present this debate as a simple matter of your or mine commercial relationship with “customers” rather than as a multi-faceted issue that touches on business, public welfare, scientific research, national and international law and social sciences. Yes, among other things that includes the vulnerability researcher’s egotistical cravings as well.
Allen Baranov Jul 29
Simple.
If we allow the blackhats to make their own exploit then we’ll only know what it looks like once it has done the damage.
And, there may be a few different versions of the exploit.
If we create the exploit ourselves then we know exactly what it looks like and we can block it easily.
Of course, there is nothing stopping the blackhats from creating their own exploit but they are lazy and don’t need to reinvent the wheel.
Ariel Jul 29
Hi all. Can somebody please provide logs where I)ruid’s and hdm’s Metasploit exploit are being used in the wild? This would surely end the discussion. Also, it would be a great motive for all DNS servers to patch or circumvent the issue. Else, we might find out that the exploit hurt no one, and it only helped the not-so-bad guys to understand the issue.
I don’t think that ignorance is a bliss. Conversely, I applaud Halvar, I)ruid and hdm for researching and publishing their findings. They did help me to understand the then-potential DNS vulnerability and understand its severity.
Andrew Jaquith Jul 29
@Ivan –
In my post, I made one factual statement and two assertions. The factual statement was that the word “customer” appeared nowhere in the discussion of motives, decision-making, and consequences until I raised it. I thought that was relevant to bring up, because I thought it highlighted a blind spot. The lack of the word “customer” in the comments prior to mine is a fact, and beyond dispute.
Now, on to the things we can actually argue about. My first assertion was that “does this particular exploit code release hurt customers?” is the only question that matters. I assert that it trumps security researcher curiosity and self-serving “you-need-to-convince-me arguments.” We appear to disagree about this, in part because our definition of “customer” differs. We also seem to disagree that this is even a valid question to raise. Fine. We shall agree to disagree.
My second assertion was that “customers” (by which I meant the Silent Majority of non-enlightened lümpenadministators) are harmed by the early release of exploit code in a tool like Metasploit. You challenged me to substantiate that assertion. I can’t do that definitively, as you might imagine. But let’s look, first, at the role that automation, through publicly available tools like Metasploit, plays in attacks (and this one in particular):
1. My previous research into the Veritas Netbackup vulnerability (CAN-2005-0773) concluded that Metasploit was responsible for causing a 1000x increase of hostile scans on port 10000. (http://www.securitymetrics.org/content/Wiki.jsp?page=Welcome_blogentry_061205_1). If a similar pattern emerges for the latest exploit, one would expect to see a surge in DNS attacks after 7/24.
2. Dan announced the vulnerability (without details) on 7/9. Details of the exploit were leaked on 7/21. The Metasploit exploit was released on 7/24.
3. Arbor’s data show a substantial increase in DNS attacks on 7/9 — the day Dan announced the flaw. Because I do not have the raw data. It is hard to tell the magnitude of the increase, but it’s something like 25X compared to the previous day.
4. Arbor also shows a slight uptick in DNS poisoning attacks on the date of the release of the exploit. (Source: http://asert.arbornetworks.com/2008/07/30-day-of-dns-attack-activity/). However, Arbor hasn’t provided current data to allow me to assess the impact of the exploit code. The data SHOULD show, when it becomes available, that the attack rate increases at least one or two more orders of magnitude.
That might not meet your standard of “harm,” in the sense that the data about attacks are incomplete. But let’s look at this another way, by reviewing what is known about WHO is currently vulnerable, and the patch rate:
1. As of today, the ISP I am using right now, at the **Usenix security conference,** is vulnerable to DNS poisoning. My company’s internal DNS is also vulnerable, but I have received pushback from my CTO about patching it because it “isn’t accessible from the outside.” My home ISP’s DNS (Comcast) is NOT vulnerable.
2. According to the Register, as of 7/25 major ISPs like Time Warner, Bell Canada, AT&T, T-Mobile and others were still vulnerable. (Source: http://www.theregister.co.uk/2008/07/25/isps_slow_to_patch/)
3. As of 7/26, Dan Kaminsky tells us that 52% of DNSes were vulnerable (Source: http://www.doxpara.com/?p=1191)
4. On July 7th, according to DNS-OARC, 2/3 of DNSes were vulnerable. As of today, only 1/3 are. (Source: https://www.dns-oarc.net/node/131). If you check the graph, the patch rate for DNS does not appear to increase on or after the release of the Metasploit module on 7/24.
I contend that the potential for harm clearly exists. Despite having several months to patch, between one-third and one-half of ISPs have still not patched. Moreover, there is no relationship between the appearance of the automated exploit on 7/24 and the patch rate. I conclude that how and when admins fix things is NOT related to whether or not they are “convinced” by a tool that proves the flaw actually exists. I believe was your core contention.
Obviously, we can continue this debate ad infinitum. We are clearly on different sides of the debate.
ivan Jul 30
@Andrew
We do agree that the most relevant question here is about customers. It is not the only question but it is the most relevant one. I did not agree with the way you formulated the question because I believe you riddled it with an opinionated sub-statement such as “well before most systems could be reasonably expected to be patched”. That turned it into a purely rhetorical formulation and you answered it with a prepackaged opinion: “clearly, it harms”. Well it was not “clear” or evident to me and it is still not so. To transform your rhetorical question into one that I would consider answerable I asked you to properly define what you meant with “well before”, “most systems” and “reasonably expected”. You may decide not to do so but then you are not going to be very effective at changing some people’s perception (including mine) of what is best for customers. I assumed that was the purpose of your comment as opposed to just venting about those evil so-called researchers that publish exploit code to the criminal masses.
Incidentally, customers are much more likely to change my perception of what is best for them anyway…
I do not understand why you label “you-need-to-convince-me arguments” as self-serving. If you actually want HD Moore or Druid or anybody else that you may label as security researcher to stop releasing exploit code to the public then you need to convince them that doing so is more harmful then helpful. They will not yield to authoritarian or dogmatic arguments, they *may* yield to rational reasoning and factual data.
On the other hand, it seems to me that you are claiming representation of a “Silent Majority of non-enlightened lumpen-administrators”. Unfortunately I did not receive the news of your appointment as speaker for the Silent Majority so I ask why should I assume that you speak for them any more than HD Moore does? btw, did you really you want to qualify your constituency as “lumpen”?
I am very familiar with your previous blogpost from December 2005 about this topic because back then you used it to publicly criticize policies of my employer’s that you conveniently miss-represented. For the purpose you used foregone conclusions that were supported by little if any factual data. Namely:
You showed correlation between two specific events:
1. public availability of a Metasploit exploit for CVE-2005-0773
2. a spike in port scanning activity targeting port 10000
However you did not point out the correlation that also existed with two other previous events:
3. publication of a security advisory disclosing the vulnerability, and;
4. publication of non-Metasploit functional proof-of-concept code.
Events 3 and 4 happened just two and three days before event 1 respectively yet you only pointed out the correlation between 2 and 1 and not between 2 and 3, 2 and 4 or 4 and 1.
From that chosen correlation you derived causality: Since 2 occurred after 1 you concluded that 1 caused 2. This is clearly invalid reasoning.
Furthermore you presented the increase in port scanning activity as equivalent to an increase of the number of attacks or actual incidents and you attributed those scans to hostile behavior a conclusion that was (and still is) not evident from the facts.
You are using a similar line of reasoning with your DNS examples now. For example the Arbor Networks report that you quoted says:
“Given that this vulnerability was partially disclosed on July 8, I suspect a great deal of this traffic is name server vulnerability scanning, as opposed to malicious cache poisoning attempts, although there may well be a mix of the latter.”
Yet you chose to classify it above as a “substantial increase in DNS attacks”. Also in you comment you attributed the slight uptick in traffic on the 21st to be an uptick in DNS poisoning attacks even tho the report says that 87% of all monitored traffic on port 53/udp are version queries not DNS poisoning attempts.
An interesting additional source is the report from SANS’s ISC published on the 25th (http://isc.sans.org/diary.html?storyid=4780) which states that DNS traffic that seemed to be actually hostile did not seem to be based on the publicly available exploits.
Regarding your alternative way of looking at the issue by looking at estimations of patch deployment rates and allegedly vulnerable DNS servers I also have several relevant points to make but I will refrain from doing so now. I’ve already abused Rich’s blog more than enough with my endless comments (sorry Rich, this is my final one…I promise)
As a final note: I agree with you that the potential for harm exists but that is substantially different from saying that it is “clearly more harmful than helpful” to publish exploit code for an OSS tool used by thousands of users worldwide that are not proven to be criminals.
We are not on different sides, we are both concerned about helping customers (with slightly different definitions of “customer”) we simply do not agree on how to do that. Our disagreement does not entitle either of us to make uncontestable the accusation that HD Moore, Druid or anybody else is actually helping criminals more than non-criminals by making their code freely available.
rmogull Jul 30
Ivan,
Keep posting- this is a very important debate. I’ve been out of town, so here’s my response to much of what’s been said.
First of all, I want to reiterate that I consider security (including vulnerability) research to be extremely valuable (I’m no Lindstrom). I support and use Metasploit and Core Impact and would want both at my disposal if I were an operator again. Heck, I’m using Metasploit in my Defcon presentation. That said, I do believe the stakes have changed in recent years, but our debate hasn’t. Also remember that in this debate Impact is very different than Metasploit since it’s a commercial tool with a vetted customer base, and not something any kid can run. A few points, many of them opinion:
1. A bunch of researchers, analysts, and pen testers arguing about this is asinine. None of us are operators anymore and we’re all speaking for that silent majority. We all need to admit that none of us has the data to support our arguments on their behalf.
2. I can say that most of my end user customers (going back to Gartner days) do not want PoC code. But I will admit that I do not have metrics to back that up, and thus don’t consider it completely valid yet.
3. We also need more metrics along the lines that Andrew presented- on both sides we’ve only ever presented samples, not performed a real epidemiological study.
4. Opinion: we need to continue to release full vulnerability details when patches are released to support detection and prevention of attacks. I’ve publically called many vendors to task for trying to hide these details.
5. But releasing exploit code in a free, simple to use tool that everyone has access to makes it easier to attack than patch. Loading a Metasploit module usually takes far less time than patching a production system.
6. There is a difference between regular PoC code and a Metasploit module. A module has much greater potential to cause serious damage out of the box, PoC code is often more limited because it lacks many of the additional tools and features to support full exploitation.
I just got word that I’ll be able to run a survey on this as part of my monthly Dark Reading column, so we can get a little more information. Also, it’s clear none of us have done rigorous research, but I agree with Andrew that what I’ve seen does suggest that exploit code results in more exploits.
Thus we all need to do two things- ask the end users, preferably in a statistically meaningful way, and perform deeper analysis of any potential causality between the time of exploit release and the increase in successful attacks.
ivan Jul 30
@Rich:
This is a good discussion and I hope we can continue it over time. I will not address your comments now but simply add an additional one:
We need to get past the idea that patching is the only possible solution or way to address vulnerabilities. The vulnerability research and disclosure debate cannot be entered exclusively around patching.
BTW, I also agree that the scenario has changed steadily over the past years and to that I’d add that the growing adoption of attack tools for legitimate business purposes since 2002 to date is immersed in that wave of change and we need to reconcile our views and opinions with that reality.
rmogull Jul 30
Agree completely that patching isn’t the only answer- I’d like to see continued evolution in anti-exploitation.
Andrew Jaquith Jul 30
@Ivan
We both have our “prepackaged” opinions on this issue. Those opinions are rooted in our perceptions of customers. My phrase “Silent Majority of lümpenadministrators” was deliberately chosen, as hyperbole, to draw maximum contrast with the sort of customer you seem to be talking about — the sort that wants and needs exploit code. “My type” of customer does not want exploit code made freely available to the general public. A senior security manager at a large investment bank once put it this way: “I don’t want exploits made public. But we do need ways of assessing our vulnerability.” Those are not incompatible goals.
As for your critique of my December 2005 post — your comments are fair ones. I did indeed assert causality between the release of the Metasploit module and scans on port 10000. I felt confident in doing so, in part, because this exact linkage had been examined before by Arbaugh, Fithen and McHugh in the paper “Windows of Vulnerability” (IEEE, 2000). Their conclusions were similar to mine. PS if you have not read the paper, you should.
That said, I agree that the appearance of the advisory and non-working exploit code, just before the port 10000 spike, could also have a causal relationship.
The fact that you are contesting my analysis of the data in this case is a good thing. Arguing about data is where this debate needs to move next. And I appreciate that you are willing to consider that there might, in fact, BE a relationship between exploit code and tools and attacks.
Want to do a little research project together? I would like to see if we (you, Rich and I? other interested parties?) could find more examples that might show correlation (positive, negative, none) between public POC/exploit code availability and attacks. For inclusion in the study, we would look for 1) vulnerabilities for which 2) freely-available POC or attack exploits have been made. Evidence of harm could be inferred by examining 3) port activity or IDS data gathered by 4) parties we agree on. Those parties might include DShield/SANS, Arbor, Narus etc.
That, combined with Rich’s Dark Reading survey of admins (lümpen- and otherwise), might advance the debate. In the absence of further empirical work, you, me, and everyone else in this thread will continue to be, as Tovalds put it, “people wanking around with their opinions.”
Sharon Aug 5
Here’s the short version of my opinion (FWIW …)
IMHO Full disclosure when ALL the vulnerability details are revealed is a bad idea from a security stand point. I would like to see the community develop other methods to credit the important work of security researchers.