Heartland

New Details, and Lessons, on Heartland Breach

By Rich

Thanks to an anonymous reader, we may have some additional information on how the Heartland breach occurred. Keep in mind that this isn't fully validated information, but it does correlate with other information we've received, including public statements by Heartland officials.

On Monday we correlated the Heatland breach with a joint FBI/USSS bulletin that contained some in-depth details on the probable attack methodology. In public statements (and private rumors) it's come out that Heartland was likely breached via a regular corporate system, and that hole was then leveraged to cross over to the better-protected transaction network.

According to our source, this is exactly what happened. SQL injection was used to compromise a system outside the transaction processing network segment. They used that toehold to start compromising vulnerable systems, including workstations. One of these internal workstations was connected by VPN to the transaction processing datacenter, which allowed them access to the sensitive information. These details were provided in a private meeting held by Heartland in Florida to discuss the breach with other members of the payment industry.

As with the SQL injection itself, we've seen these kinds of VPN problems before. The first NAC products I ever saw were for remote access -- to help reduce the number of worms/viruses coming in from remote systems.

I'm not going to claim there's an easy fix (okay, there is, patch your friggin' systems), but here are the lessons we can learn from this breach:

1. The PCI assessment likely focused on the transaction systems, network, and datacenter. With so many potential remote access paths, we can't rely on external hardening alone to prevent breaches. For the record, I also consider this one of the top SCADA problems.
2. Patch and vulnerability management is key -- for the bad guys to exploit the VPN connected system, something had to be vulnerable (note -- the exception being social engineering a system 'owner' into installing the malware manually).
3. We can't slack on vulnerability management -- time after time this turns out to be the way the bad guys take control once they've busted through the front door with SQL injection. You need an ongoing, continuous patch and vulnerability management program. This is in every freaking security checklist out there, and is more important than firewalls, application security, or pretty much anything else.
4. The bad guys will take the time to map out your network. Once they start owning systems, unless your transaction processing is absolutely isolated, odds are they'll find a way to cross network lines.
5. Don't assume non-sensitive systems aren't targets. Especially if they are externally accessible.

Okay -- when you get down to it, all five of those points are practically the same thing.

Here's what I'd recommend:

1. Vulnerability scan everything. I mean everything, your entire public and private IP space.
2. Focus on security patch management -- seriously, do we need any more evidence that this is the single most important IT security function?
3. Minimize sensitive data use and use heavy egress filtering on the transaction network, including some form of DLP. Egress filter any remote access, since that basically blows holes through any perimeter you might think you have.
4. Someone will SQL inject any public facing system, and some of the internal ones. You'd better be testing and securing any low-value, public facing system since the bad guys will use that to get inside and go after the high value ones. Vulnerability assessments are more than merely checking patch levels.

–Rich

Recent Breaches: We May Have All the Answers

By Rich

You know how sometimes you read something and then forget about it until it smacks you in the face again?

That's how I feel right now after @BreachSecurity reminded me of this advisory from February.

To pull an excerpt, it looks like we now know exactly how all these recent major breaches occurred:

Attacker Methodology: In general, the attackers perform the following activities on the networks they compromise:

1. They identify Web sites that are vulnerable to SQL injection. They appear to target MSSQL only.

2. They use "xp_cmdshell", an extended procedure installed by default on MSSQL, to download their hacker tools to the compromised MSSQL server.

3. They obtain valid Windows credentials by using fgdump or a similar tool.

4. They install network "sniffers" to identify card data and systems involved in processing credit card transactions.

5. They install backdoors that "beacon" periodically to their command and control servers, allowing surreptitious access to the compromised networks.

6. They target databases, Hardware Security Modules (HSMs), and processing applications in an effort to obtain credit card data or brute-force ATM PINs.

7. They use WinRAR to compress the information they pilfer from the compromised networks.

No surprises. All preventable, although clearly these guys know their way around transaction networks if they target HSMs and proprietary financial systems.

Seems like almost exactly what happend with CardSystems back in 2004. No snarky comment needed.

–Rich

Heartland Hackers Caught; Answers and Questions

By Rich

UPDATE: follow up article with what may be the details of the attacks, based on the FBI/Secret Service advisory that went out earlier this year.

The indictment today of Albert Gonzales and two co-conspirators for hacking Hannaford, 7-Eleven, and Heartland Payment Systems is absolutely fascinating on multiple levels. Most importantly from a security perspective, it finally reveals details of the attacks. While we don't learn the specific platforms and commands, the indictment provides far greater insights than speculation by people like me. In the "drama" category, we learn that the main perpetrator is the same person who hacked TJX (and multiple other retailers), and was the Secret Service informant who helped bring down the Shadowcrew.

Rather than rehashing the many articles popping up, let's focus on the security implications and lessons hidden in the news reports and the indictment itself. Let's start with a short list of the security issues and lessons learned, then dig into more detail on the case and perpetuators themselves:

To summarize the security issues:

• The attacks on Hannaford, Heartland, 7-Eleven, and the other 2 retailers used SQL injection as the primary vector.
• In at least some cases, it was not SQL injection of the transaction network, but another system used to get to the transaction network.
• In at least some cases custom malware was installed, which indicates either command execution via the SQL injection, or XSS via SQL injection to attack internal workstations. We do not yet know the details.
• The custom malware did not trigger antivirus, deleted log files, sniffed the internal network for card numbers, scanned the internal network for stored data, and exfiltrated the data. The indictment doesn't reveal the degree of automation, or if it was more manually controlled (shell).

The security lessons include:

• Defend against SQL injection -- it's clearly one of the top vectors for attacks. Parameterized queries, WAFs, and so on.
• Lock databases to prevent command execution via SQL. Don't use a privileged account for the RDBMS, and do not enable the command execution features. Then, lock down the server to prevent unneeded network services and software installation (don't allow outbound curl, for example).
• Since the bad guys are scanning for unprotected data, you might as well do it yourself. Use DLP to find card data internally. While I don't normally recommend DLP for internal network traffic, if you deal with card numbers you should considering using it to scan traffic in and out of your transaction network.
• AV won't help much with the custom malware. Focus on egress filtering and lockdown of systems in the transaction network (mostly the database and application servers).
• Don't assume attackers will only target transaction applications/databases with SQL injection. They will exploit any weak point they can find, then use it to weasel over to the transaction side.
• These attacks appear to be preventable using common security controls. It's possible some advanced techniques were used, but I doubt it.

Now let's talk about more details:

• This indictment covers breaches of Heartland, Hannaford, 7-Eleven, and two "major retailers" breached in 2007 and early 2008. Those retailers have not been revealed, and we do not know if they are in violation of any breach notification laws.
• This is the same Albert Gonzales who was indicted last year for breaches of TJ Maxx, Barnes & Noble, BJ's Wholesale Club, Boston Market, DSW, Forever 21, Office Max, and Sports Authority.
• A co-coconspirator referred to in the indictment as "P.T." was not indicted. While it's pure conjecture, I won't be surprised if this is an informant who help break the case.
• Gonzales and friends would identify potential targets, then use a combination of online and physical surveillance to identify weaknesses. Physical visits would reveal the payment system being used (via the point of sale terminals), and other relevant information. When performing online reconnaissance, they would also attempt to determine the payment processor/processing system.
• In the TJX attacks it appears that wireless attacks were the primary vector (which correlates with the physical visits). In this series, it was SQL injection.
• Multiple systems and servers scattered globally were used in the attack. It is quite possible that these were the part of the web-based exploitation service described in this article by Brian Krebs back in April.
• The primary vector was SQL injection. We do not know the sophistication of the attack, since SQL injection can be simple or complex, depending on the database and security controls involved.
• It's hard to tell from the indictment, but it appears that in some cases SQL injection alone may have been used, while in others it was a way of inserting malware.
• It is very possible that SQL injection on a less-secured area of the network was used to install malware, which was then used to attack other internal services and transition to the transaction network. Based on information in various other interviews and stories, I suspect this was the case for Heartland, if not other targets. This is conjecture, so please don't hold me to it.
• More pure conjecture here, but I wonder if any of the attacks used SQL injection to XSS internal users and download malware into the target organization?
• Custom malware was left on target networks, and tested ensure it would evade common AV engines.
• SQL injection to allow command execution shouldn't be possible on a properly configured financial transaction system. Most RDBMS systems support some level of command execution, but usually not by default (for current versions of SQL Server and Oracle after 8 -- not sure about other platforms). Thus either a legacy RDBMS was used, or a current database platform that was improperly configured. This would either be due to gross error, or special requirements that should have only been allowed with additional security controls, such as strict limits on the RDBMS user account, server lockdown (everything from application whitelisting, to HIPS, to external monitoring/filtering).
• In one case the indictment refers to a SQL injection string used to redirect content to an external server, which seems to indicate that malware wasn't necessarily always used.
• The malware attempted to hide itself. While details aren't available, the indictment indicates it probably erased log files, at a minimum.
• The attacks both sniffed traffic and attempted to identify stored card numbers. They targeted data at rest and in motion.
• I've heard rumors from trusted sources that the exfiltrated data was not encrypted or otherwise obfuscated in at least one case.

Just as the TJX hack was based on well known issues with wireless security, these attacks seem to use well known SQL injection techniques. In a way I really hope I'm wrong, and that some new kind of advanced SQL injection attack was involved, but I think we'd see more bigger victims if that were the case. I also find it fascinating that a single individual is at the crux of multiple cases, and used to be a Secret Service informant. I hope more information is revealed about the "hacking platform" that may refer to systems referenced in the Washington Post article back in April.

Finally, does this mean we have two major retailers in violation of breach disclosure laws?

As is almost always the case, preventing these attacks wouldn't necessarily have required rocket science or millions of dollars in specialized security tools.

Wired also has a good article.

–Rich