Breach

Analysis of Trustwave’s 2010 Breach Report

By Rich

Trustwave just released their latest breach (and penetration testing) report, and it's chock full of metrics goodness. Like the Verizon Data Breach Investigations Report, it's a summary of information based on their responses to real breaches, with a second section on results from their penetration tests.

The breach section is the best part, and I already wrote about one lesson in a quick post on DLP. Here are a few more nuggets that stood out:

1. It took an average of 156 days to detect a breach, and only 9% of victims detected the breach on their own -- the rest were discovered by outside groups, including law enforcement and a credit card companies.
2. Chip and PIN was called out for being more resilient against certain kinds of attacks, based on global analysis of breaches. Too bad we won't get it here in the US anytime soon.
3. One of the biggest sources of breaches was remote access applications for point of sale systems. Usually these are in place for third party support/management, and configured poorly.
4. Memory parsing (scanning memory to pull sensitive information as it's written to RAM) was a very common attack technique. I find this interesting, since certain versions of memory parsing attacks have virtualization implications... and thus cloud implications. This is something to keep an eye on, and an area I'm researching more.
5. As I mentioned in the post 5 minutes ago, encryption was only used once for exfiltration.

Now some suggestions to the SpiderLabs guys:

1. I realize it isn't politically popular, but it would totally rock if you, Verizon, and other response teams started using a standard base of metrics. You can always add your own stuff on top, but that would really help us perform external analysis across a wider base of data points. If you're interested, we'd totally be up for playing neutral third party and coordinating and publishing a recommended metrics base.
2. The pen testing section would be a lot more interesting if you released the metrics, as opposed to a "top 10" list of issues found. We don't know if number 1 was 10x more common than issue 10, or 100x more common.

It's great that we now have another data source, and I consider all these reports mandatory reading, and far more interesting than surveys.

–Rich

Recent Breaches: We May Have All the Answers

By Rich

You know how sometimes you read something and then forget about it until it smacks you in the face again?

That's how I feel right now after @BreachSecurity reminded me of this advisory from February.

To pull an excerpt, it looks like we now know exactly how all these recent major breaches occurred:

Attacker Methodology: In general, the attackers perform the following activities on the networks they compromise:

1. They identify Web sites that are vulnerable to SQL injection. They appear to target MSSQL only.

2. They use "xp_cmdshell", an extended procedure installed by default on MSSQL, to download their hacker tools to the compromised MSSQL server.

3. They obtain valid Windows credentials by using fgdump or a similar tool.

4. They install network "sniffers" to identify card data and systems involved in processing credit card transactions.

5. They install backdoors that "beacon" periodically to their command and control servers, allowing surreptitious access to the compromised networks.

6. They target databases, Hardware Security Modules (HSMs), and processing applications in an effort to obtain credit card data or brute-force ATM PINs.

7. They use WinRAR to compress the information they pilfer from the compromised networks.

No surprises. All preventable, although clearly these guys know their way around transaction networks if they target HSMs and proprietary financial systems.

Seems like almost exactly what happend with CardSystems back in 2004. No snarky comment needed.

–Rich

Heartland Hackers Caught; Answers and Questions

By Rich

UPDATE: follow up article with what may be the details of the attacks, based on the FBI/Secret Service advisory that went out earlier this year.

The indictment today of Albert Gonzales and two co-conspirators for hacking Hannaford, 7-Eleven, and Heartland Payment Systems is absolutely fascinating on multiple levels. Most importantly from a security perspective, it finally reveals details of the attacks. While we don't learn the specific platforms and commands, the indictment provides far greater insights than speculation by people like me. In the "drama" category, we learn that the main perpetrator is the same person who hacked TJX (and multiple other retailers), and was the Secret Service informant who helped bring down the Shadowcrew.

Rather than rehashing the many articles popping up, let's focus on the security implications and lessons hidden in the news reports and the indictment itself. Let's start with a short list of the security issues and lessons learned, then dig into more detail on the case and perpetuators themselves:

To summarize the security issues:

• The attacks on Hannaford, Heartland, 7-Eleven, and the other 2 retailers used SQL injection as the primary vector.
• In at least some cases, it was not SQL injection of the transaction network, but another system used to get to the transaction network.
• In at least some cases custom malware was installed, which indicates either command execution via the SQL injection, or XSS via SQL injection to attack internal workstations. We do not yet know the details.
• The custom malware did not trigger antivirus, deleted log files, sniffed the internal network for card numbers, scanned the internal network for stored data, and exfiltrated the data. The indictment doesn't reveal the degree of automation, or if it was more manually controlled (shell).

The security lessons include:

• Defend against SQL injection -- it's clearly one of the top vectors for attacks. Parameterized queries, WAFs, and so on.
• Lock databases to prevent command execution via SQL. Don't use a privileged account for the RDBMS, and do not enable the command execution features. Then, lock down the server to prevent unneeded network services and software installation (don't allow outbound curl, for example).
• Since the bad guys are scanning for unprotected data, you might as well do it yourself. Use DLP to find card data internally. While I don't normally recommend DLP for internal network traffic, if you deal with card numbers you should considering using it to scan traffic in and out of your transaction network.
• AV won't help much with the custom malware. Focus on egress filtering and lockdown of systems in the transaction network (mostly the database and application servers).
• Don't assume attackers will only target transaction applications/databases with SQL injection. They will exploit any weak point they can find, then use it to weasel over to the transaction side.
• These attacks appear to be preventable using common security controls. It's possible some advanced techniques were used, but I doubt it.

Now let's talk about more details:

• This indictment covers breaches of Heartland, Hannaford, 7-Eleven, and two "major retailers" breached in 2007 and early 2008. Those retailers have not been revealed, and we do not know if they are in violation of any breach notification laws.
• This is the same Albert Gonzales who was indicted last year for breaches of TJ Maxx, Barnes & Noble, BJ's Wholesale Club, Boston Market, DSW, Forever 21, Office Max, and Sports Authority.
• A co-coconspirator referred to in the indictment as "P.T." was not indicted. While it's pure conjecture, I won't be surprised if this is an informant who help break the case.
• Gonzales and friends would identify potential targets, then use a combination of online and physical surveillance to identify weaknesses. Physical visits would reveal the payment system being used (via the point of sale terminals), and other relevant information. When performing online reconnaissance, they would also attempt to determine the payment processor/processing system.
• In the TJX attacks it appears that wireless attacks were the primary vector (which correlates with the physical visits). In this series, it was SQL injection.
• Multiple systems and servers scattered globally were used in the attack. It is quite possible that these were the part of the web-based exploitation service described in this article by Brian Krebs back in April.
• The primary vector was SQL injection. We do not know the sophistication of the attack, since SQL injection can be simple or complex, depending on the database and security controls involved.
• It's hard to tell from the indictment, but it appears that in some cases SQL injection alone may have been used, while in others it was a way of inserting malware.
• It is very possible that SQL injection on a less-secured area of the network was used to install malware, which was then used to attack other internal services and transition to the transaction network. Based on information in various other interviews and stories, I suspect this was the case for Heartland, if not other targets. This is conjecture, so please don't hold me to it.
• More pure conjecture here, but I wonder if any of the attacks used SQL injection to XSS internal users and download malware into the target organization?
• Custom malware was left on target networks, and tested ensure it would evade common AV engines.
• SQL injection to allow command execution shouldn't be possible on a properly configured financial transaction system. Most RDBMS systems support some level of command execution, but usually not by default (for current versions of SQL Server and Oracle after 8 -- not sure about other platforms). Thus either a legacy RDBMS was used, or a current database platform that was improperly configured. This would either be due to gross error, or special requirements that should have only been allowed with additional security controls, such as strict limits on the RDBMS user account, server lockdown (everything from application whitelisting, to HIPS, to external monitoring/filtering).
• In one case the indictment refers to a SQL injection string used to redirect content to an external server, which seems to indicate that malware wasn't necessarily always used.
• The malware attempted to hide itself. While details aren't available, the indictment indicates it probably erased log files, at a minimum.
• The attacks both sniffed traffic and attempted to identify stored card numbers. They targeted data at rest and in motion.
• I've heard rumors from trusted sources that the exfiltrated data was not encrypted or otherwise obfuscated in at least one case.

Just as the TJX hack was based on well known issues with wireless security, these attacks seem to use well known SQL injection techniques. In a way I really hope I'm wrong, and that some new kind of advanced SQL injection attack was involved, but I think we'd see more bigger victims if that were the case. I also find it fascinating that a single individual is at the crux of multiple cases, and used to be a Secret Service informant. I hope more information is revealed about the "hacking platform" that may refer to systems referenced in the Washington Post article back in April.

Finally, does this mean we have two major retailers in violation of breach disclosure laws?

As is almost always the case, preventing these attacks wouldn't necessarily have required rocket science or millions of dollars in specialized security tools.

Wired also has a good article.

–Rich

Our Financial System is Under a Coordinated, Sophisticated Attack

By Rich

This is a great day for security researchers, and a bad day for anyone with a bank account.

First up is the release of the 2009 Verizon Data Breach Investigations Report. This is now officially my favorite breach metrics source, and it's chock full of incredibly valuable information. I love the report because it's not based on bullshit surveys, but on real incident investigations. The results are slowly spreading throughout the blogosphere, and we won't copy them all here, but a few highlights:

1. Verizon's team alone investigated cases that resulted in the loss of 285 million records. That's just them, never mind all the other incident response teams.
2. Most organizations do a crap job with security- this is backed up with a series of metrics on which security controls are in place and how incidents are discovered.
3. Essentially no organizations really complied with all the PCI requirements- but most get certified anyway.

Liquidmatrix has a solid summary of highlights, and I don't want to repeat their work. As they say,

Read pages 46-49 of the report and do what it says. Seriously. It’s the advice that I would give if you were paying me to be your CISO.

And we'll add some of our own advice soon.

Next is an article on organized cybercrime by Brian Krebs THAT YOU MUST GO READ NOW. (I realize it might seem like we have a love affair with Brian or something, but he's not nearly my type). Brian digs beyond the report, and his investigative journalism shows what many of us believe to be true- there is a concerted attack on our financial system that is sophisticated and organized, and based out of Eastern Europe.

I talked with Brain and he told me,

You know all those breaches last year? Most of them are a handful of groups.

Here are a couple great tidbits from the article:

For example, a single organized criminal group based in Eastern Europe is believed to have hacked Web sites and databases belonging to hundreds of banks, payment processors, prepaid card vendors and retailers over the last year. Most of the activity from this group occurred in the first five months of 2008. But some of that activity persisted throughout the year at specific targets, according to experts who helped law enforcement officials respond to the attacks, but asked not to be identified because they are not authorized to speak on the record.

...

One hacking group, which security experts say is based in Russia, attacked and infiltrated more than 300 companies -- mainly financial institutions -- in the United States and elsewhere, using a sophisticated Web-based exploitation service that the hackers accessed remotely. In an 18-page alert published to retail and banking partners in November, VISA described this hacker service in intricate detail, listing the names of the Web sites and malicious software used in the attack, as well as the Internet addresses of dozens of sites that were used to offload stolen data.

...

Steve Santorelli, director of investigations at Team Cymru, a small group of researchers who work to discover who is behind Internet crime, said the hackers behind the Heartland breach and the other break-ins mentioned in this story appear to have been aware of one another and unofficially divided up targets. "There seem, on the face of anecdotal observations, to be at least two main groups behind many of the major database compromises of recent years," Santorelli said. "Both groups appear to be giving each other a wide berth to not step on each others' toes."

Keep in mind that this isn't the same old news. We're not talking about the usual increase in attacks, but a sophistication and organizational level that developed materially in 2007-2008.

To top it all off, we have this article over at Wired on PIN cracking. This one also ties in to the Verizon report. Another quote:

"We're seeing entirely new attacks that a year ago were thought to be only academically possible," says Sartin. Verizon Business released a report Wednesday that examines trends in security breaches. "What we see now is people going right to the source ... and stealing the encrypted PIN blocks and using complex ways to un-encrypt the PIN blocks."

If you read more deeply, you learn that the bad guys haven't developed some quantum crypto, but are taking advantage of weak points in the system where the data is unencrypted, even if only in memory.

Really fascinating stuff, and I love that we're getting real information on real breaches.

–Rich

Did They Violate Breach Disclosure Laws?

By Rich

There's been an extremely interesting, and somewhat surprising, development in the TJX case the past couple weeks. No, I'm not talking about one of the defendants pleading guilty (and winning the prisoners dilemma), but the scope of the breach.

Based on the news reports and court records, it seems TJX wasn't the only victim here. From ComputerWorld:

Toey was one of 11 alleged hackers arrested last month in connection with a series of data thefts and attempted data thefts at TJX and numerous other companies. Besides TJX and BJ's, the list of publicly identified victims of the hackers includes DSW, OfficeMax, Boston Market, Barnes and Noble, Sports Authority and Forever 21.

Huh. Wacky. I don't seem to recall seeing breach notifications from anyone other than TJX. Since I've been out for a few weeks, I decided to hunt a bit and learned the Wall Street Journal beat me to the punch on this story:

That's because only four of the chains clearly alerted their customers to breaches. Two others -- Boston Market Corp. and Forever 21 Inc. -- say they never told customers because they never confirmed data were stolen from them. The other retailers -- OfficeMax Inc., Barnes and Noble Inc., and Sports Authority Inc. -- wouldn't say whether they made consumer disclosures. Computer searches of their Securities and Exchange Commission filings, Web sites, press releases and news archives turned up no evidence of such disclosures. The other companies allegedly targeted by the ring charged last week were: TJX Cos., BJ's Wholesale Club Inc., shoe retailer DSW Inc., and restaurant chain Dave and Buster's Inc. They each disclosed to customers they were breached shortly after the intrusions were discovered.

The blanket excuse from these companies for not disclosing? "We couldn't find any definite information that we'd been breached".

Seems to me someone has a bit of legal exposure right now. I wonder if is greater or less than the cost of notification? And don't forget, thanks to TJX seeing absolutely no effect on their business after the breach, we can pretty effectively kill off the reputation damage argument.

–Rich

Comments on Security Breach Statistics

By Adrian Lane

I still have not quite reached complete apathy regarding breach statistics, but I am really close. The Identity Theft Resource Center statistics made their way into the Washington Post last week, and were reposted on the front page of The Arizona Republic business section this morning. In a nutshell they are saying the number of breaches was up 69% for the first half of 2008 over the first half of 2007.

I am certain no one is surprised. As a security blogging community we have been talking about how the custodians of the information fail to address security, how security products are not all that effective, how the 'bad guys' are creative, opportunistic, and committed to finding new exploits, and my personal favorite, how the people who set up the (financial, banking, heath care, government, insert your favorite here) systems have a serious financial stake in things being quick and easy rather than secure. Ultimately, I would have been surprised if the number had gone down.

I used to do a presentation called "Dr. Strangelog or; How I stopped worrying and loved the breach". No, I was not advocating building subterranean caverns to wait this out; rather a mental adjustment in how to approach security. For the corporate IT audience, the premise is that you are never going to be 100% secure, so plan to do the best you can, and be prepared to react when a breach happens. And I try to point out some of the idiocy in certain policies that invite unnecessary risk ... like storing credit card numbers when it is unnecessary, not encrypting backup tapes, and allowing all your customer records to ever be on a laptop outside the company. While we have gone well beyond these basics, I still think that contrarian thinking is in order to find new solutions, or to redefine the problem itself as it seems impossible to stop the breaches at this point.

As an individual, as opposed to as a security practitioner, Is there anything meaningful in these numbers? Is there any value what so ever? Is it going to be easier to quantify the records that have not been breached? Are we getting close to having every personal record compromised at least once? The numbers are so large that they start to lose their meaning. Breaches are so common that they have spawned several secondary markets in areas such as tools and techniques for fraudulently gaining additional personal information, partial personal information useful for the same purpose, and of course various anti-fraud tools and services. I start to wonder if the corporations and public entities of the world have already effectively wiped out personal privacy.

–Adrian Lane

ATM PIN Thefts

By Adrian Lane

The theft of Citibank ATM PINs is in the news again as it appears that indictments have been handed down on the three suspects. This case will be interesting to watch, to see what the fallout will be. It is not still really clear if the PINs were leaked in transit, or if the clearing house servers were breached.

There are a couple of things about this story that I still find amusing. The first is that Fiserv, the company that operates the majority of the network, is pointing fingers at Cardtronics Inc. The quote by the Fiserv representative "Fiserv is confident in the integrity and security of our system" is great. They both manage elements of the 'system'. When it comes down to it, this is like two parties who are standing in a puddle of gasoline, accusing each other of lighting a match. It won't matter who is at fault when they both go up in flames. In the public mind, no one is going to care, and they will be blamed equally and quite possibly both go out of business if their security was shown to be grossly lacking.

My second though on this subject was, once you breach the 'system', you have to get the money out. In this case, it has been reported that over $2M was 'illegally gained'. If the average account is hacked for $200.00, we are talking about at least 10,000 separate ATM withdrawals. That is a lot of time spent at the 7-11! But seriously, that is a lot of time to spend making ATM withdrawals. I figure that they way they got caught is that the thief's picture keept turning up on security cameras ... otherwise this is a difficult crime to detect and catch.

I also got to thinking about ATMs and the entire authentication process is not much more than basic two factor authentication combined with some simple behavioral checks at the back end. The security of these networks is really not all that advanced. Typically PIN codes are four digits in length, and it really does not make a lot of sense to use hash algorithms given the size of the PIN and the nature of the communications protocol. And while it requires some degree of technical skill, the card itself can be duplicated, making a fairly weak two factor system. Up until a couple years ago, DES was still the typical encryption algorithm in use, and only parts of the overall transaction processing systems keep the data encrypted. Many of the ATMs are not on private networks, but utilize the public Internet and airwaves. Given the amount of money and the number of transactions that are processed around the world, it is really quite astonishing how well the system as a whole holds up.

Finally, while I have been known to bash Microsoft for various security miscues over the years, it seems somewhat specious to state "Hackers are targeting the ATM system's infrastructure, which is increasingly built on Microsoft Corp.'s Windows operating system." Of course they are targetting the infrastructure; that is the whole point of electronic fraud. They probably meant the back end processing infrastructure. And why mention Windows? Windows may make familiarity with the software easier; this case does not show that any MS product was at fault for the breach. Throwing that into the story seems like they are trying to cast blame on MS software without any real evidence.

–Adrian Lane