Picking Apart The Hannaford Breach- What Might Have Happened
There goes another one.
According to multiple sources, the Hannaford Brothers grocery chain suffered a major breach with 4.2 million credit cards exposed. Hannaford had published an FAQ for their customers. Odds are it will be months until we find out what really happened, but I’m going to speculate anyway, pick apart the press coverage and FAQ, and see if we can learn something from this now.
As usual, the information released is incomplete and contradictory.
PORTLAND, Maine (AP) - A security breach at an East Coast supermarket chain exposed 4.2 million credit and debit card numbers and led to 1,800 cases of fraud, the Hannaford Bros. grocery chain announced Monday.
Hannaford said credit and debit card numbers were stolen during the card authorization process and about 4.2 million unique account numbers were exposed.
The breach affected all of its 165 stores in the Northeast, 106 Sweetbay stores in Florida and a smaller number of independent groceries that sell Hannaford products.
This is interesting since there is a direct tie to fraud, as opposed to many other breaches. This often means the fraud was detected in the credit system and then traced back to the retailer, which seems to be what happened based on the FAQ. As a researcher it’s always helpful to be able to tie the breach to illegal activity. This does, of course, suck for the victims, but as long as it’s credit card fraud they are protected.
Since the information was stolen during the authorization process, and was distributed over many locations, it means a compromise of the central authorizations system or the credit card processor. It could be as simple as sniffing unencrypted communications, or a more complex compromise of a database or application. My money is 70% on sniffing, 30% on something in the database.
No personal data such as names, addresses or telephone numbers were divulged - just account numbers.
This can’t be true. Without names, the card numbers are unusable.
Hannaford became aware of the breach Feb. 27. Investigators later discovered that the data breach began on Dec. 7; it wasn’t contained until March 10, said Carol Eleazer, Hannaford’s vice president of marketing in Scarborough.
“We have taken aggressive steps to augment our network security capabilities,” Hannaford president and CEO Ronald C. Hodge said in a statement released Monday. “Hannaford doesn’t collect, know or keep any personally identifiable customer information from transactions.”
This reinforces the likelihood of a network breach and sniffing, assuming the statement is true. How was the network breached? Could be any one of hundreds of ways. Targeted phishing and compromise of the central network from a remote location are common. I can’t add anything more than pure speculation on this one.
The company urged its customers to monitor their credit and debit cards for unusual transactions and report any problems to authorities.
Actually, card issuers should reissue the cards and just eliminate the chance of greater fraud. This is irresponsible. Since this is just loss of credit cards, there is no need for identity theft protection.
Mark Walker, an attorney for the Maine Bankers Association, said his organization sent an advisory to member banks Friday after learning of the breach. Only a few had reported suspicious activity involving the credit and debit cards they had issued customers, Walker said.
“I had expected there would be more than we’ve heard of,” Walker said. “But it’s still too early for us to tell.”
Strange- I consider 1,800 to be a large number. It could be that the fraud was performed directly in the Hannaford system or something. Or this is an erroneous statement.
The FAQ gives us a little more information and narrows things down.
What happened?
Hannaford announced containment of a data intrusion into its computer network that resulted in the theft of customer credit and debit card numbers. This data was illegally accessed from Hannaford’s computer systems during the card verification transmission process in transactions. Further, Hannaford is cooperating with credit and debit card issuers to ensure those customers who may be affected by the theft are protected
Somewhat contradictory, with a mention of data security and network, but I don’t expect everyone to be as picky about those details as we are. I suspect the last sentence means fraud alerts are in place, and cards are probably being reissued to some extent.
When did you discover the intrusion?
Hannaford was first made aware of suspicious credit card activity on Feb. 27, and immediately initiated a comprehensive investigation with the assistance of leading computer security experts
Bingo. It was detected by the banks or credit card companies, then brought to Hannaford.
Is it safe to continue shopping in your stores?
We have continually devoted significant round-the-clock resources to ensure Hannaford has comprehensive data security systems in place. For example, our security measures meet industry compliance standards and many go above and beyond what is required by industry standards.
In other words, PCI is worthless.
In conclusion, it looks like some sort of a network breach (which could be anything from phishing/malware to compromise from a retail location to a full network hack). A sniffer was possibly installed, since it seems they don’t keep credit card information (again, assuming statements are true). The fraud was detected by the banks or credit card companies, then it took a little under two weeks to contain. Not great, and indicative of either a little sophistication on the attacker’s part, or a lack of sophistication on Hannaford’s part.
How to prevent this?
We won’t know until more information is out, but since they shouldn’t be PCI compliant if they transmitted credit card numbers in the clear, perhaps my guess of sniffing is off. I’m still laying odds on that, and if so, encryption is the answer.
Technorati Tags: Data Breach, Hannaford
Mike Mar 18
Agreed, PCI is basically worthless. I’ve seen retailers (not this one) store credit card transactions in a temporary daily flat file in unencrypted plaintext. This file is stored on a local server for a period of time of up to 48 hours before the entire file is processed and sent to the appropriate credit card processor (albeit over an encrypted connection). I believe that because of the semi-temporary nature of the file, by the letter of the law/standard, they were considered ‘PCI compliant.’
To be fair they did go above and beyond monitoring that particular server in a variety of ways, but that doesn’t change the fact that anyone with access to that single point of failure basically had the keys to the kingdom.
Tom Mahoney Mar 18
A great analysis - but I disagree on one point.
You said:
No personal data such as names, addresses or telephone numbers were divulged - just account numbers.
—-
This can’t be true. Without names, the card numbers are unusable.
You’d be partially correct if you said that they could not be used successfully. In a CNP transaction, the bad guys could certainly attempt to use just the number and expire date. You’re ‘partially correct’ in that there are still a large number of on-line merchants that don’t verify the CVV2 and we all know that AVS is essentially broken. This being the case, I see no reason why a certain percentage of the accounts could not be used - especially against CNP merchants.
Tom Mahoney Mar 18
A great analysis - but I disagree on one point.
You said:
No personal data such as names, addresses or telephone numbers were divulged - just account numbers.
—-
This can’t be true. Without names, the card numbers are unusable.
You’d be partially correct if you said that they could not be used successfully. In a CNP transaction, the bad guys could certainly attempt to use just the number and expire date. You’re ‘partially correct’ in that there are still a large number of on-line merchants that don’t verify the CVV2 and we all know that AVS is essentially broken. This being the case, I see no reason why a certain percentage of the accounts could not be used - especially against CNP merchants.
Tom Mahoney, Director
Merchant911.org
Leprechaun Mar 18
There is one point where the data might not be encrypted and it isn’t the retailers fault. I know of several retailers whose acquiring bank cannot process the transactions transmitted to them unless they are decrypted. These companies cannot understand why they are required to keep the data encrypted at all possible points, but they have to decrypt it to send it over the frame network to the acquirer. (this is not the transaction processing stage, but the later reconciliation stage between HQ and the bank.
Jim Troutman Mar 18
I don’t think PCI is worthless. It is better than no standard at all. It does have flaws, but it is a good starting point. Achieving compliance is hard work, and you have to do it on a everyday basis, and this is where companies fail. It is easy to be PCI complaint on a certain day, or even for a week, but hard in the long term (log monitoring, adds/moves/changes, staffing to continue to audit and test changes, etc.).
Mike: I can’t imagine anyone passing a PCI audit if that stored PANs in any sort of unencrypted fashion, no matter how ephemeral. Data MUST be encrypted “at rest” (on disk) and encrypted “in motion” (over the network). It is not optional for PCI DSS.
Tom Mahoney Mar 18
Leprechaun;
I would agree that it isn’t the retailers fault if decyption is done in transit during the rerconcilliation, but I’d be willing to bet that the card company would still hold the retailer responsible. As I understand it, retailers are held jointly liable if PCI compliance is broken anywhere in the chain. That culpability might not result in sanctions for a given incident, but would probably be looked at as part of the retailer’s overall compliance record.
Tom Mahoney, Director
Merchant911.org
Scott K Mar 18
Often with PCI its a question and answer checkbox race…
Consultant: Do you encrypt the data at rest?
Client: Yes
Consultant: Check!
Consultant: Do you encrypt the data in motion?
Client: Yes
Consultant: Check!
Even if this is not the case with some auditors/consultants and they actually ask to see proof. Proof is not that hard to provide in a one time case. As Jim Troutman said, “It is easy to be PCI compliant on a certain day, or even for a week, but hard in the long term.”
Mike Mar 18
http://pcianswers.com/2008/03/18/hannaford-data-breach-and-pci/
Allen Baranov Mar 19
@Mike
I just took a look at the PCI standard and it says that at the very least the “card number” must be made unreadable wherever it is “at rest” . It doesn’t state a time limit or excuse data that is at rest for a short amount of time and I would imagine that trying to prove that a temporary file that is around for 48 hours is not “at rest” would be very difficult to do whereas data in a network card’s cache (or such) for a few seconds would be easier to prove.
I think though that the PCI standard does fall apart in the restrictions on data in transit. Data in transit needs to be encrypted only on networks that are “easy and common for a hacker to intercept, modify, and divert data while in transit” which pretty. It goes on to detail what these networks are and an internal ethernet network is excluded.
Considering the amount of bots in the world today it may be time for the PCI guys to reconsider what type of network is at risk from a hacker, especially if ,as Hannaford claims, they were PCI compliant as per the current PCI standard.
Rani Mar 19
Good analysis Rich. We’ve been looking at this and drew many of the same conclusions. The piece of info that stands out is the bit about CC numbers being breached but not personal details. I read elsewhere that PIN numbers were also breached, though I’m not sure. In that case the fraud could have been committed using duplicate CCs for withdrawals from ATMs.
Because if this partial info, it seems unlikely that this is a straight database breach, and also unlikely that it was some kind of “man in the middle” attack on network communications between Hannaford and their CC processor (and these communications are always encrypted).
Regarding how it was done, I think Mike’s comment on flat files is spot on. We’ve seen that too. That information is then either transmitted for log aggregation over an unsecured part of the network, where it could be picked up by a sniffer, or compromised on the host itself.
The PCI discussion is a different topic altogether. I don’t think PCI is worthless, but it’s not perfect either. The issues I see are mostly not with PCI DSS but with how it is being audited and enforced, a process that is still undergoing many changes.
Mike Mar 19
@Allen the PCI standard evolves and changes with the ever changing risk environment. Data in transit was not a big issues several years back but it is now. I can expect the standard to change in response to the emerging attack patterns.
That said, to discount the program as “worthless” makes me question how informed the person saying it really is about this topic.
Matt Mar 19
There are two very serious problems with PCI. First of all, it doesn’t really kick-in until post authorization. And there is the problem, as pointed out with the flat files, that the banks are no accommodating the clients to send encrypted files - which also ties to the first problems since the banks can’t/won’t handle encrypted authorizations. In the first case, Hannaford may be 100% compliant and get off the hook if this was all pre-auth sniffing. In the second case, the bank is on the hook.
PCI is still very good for businesses who would have NO security without PCI. But it’s not bulletproof. As you see, it covers that 98% in the merchants hands and not that first or last 1% that is still susceptible. Hypothetically, this breach is related to WEP encryption and hypothetically it was passed by their auditor. Hence, the transmission issue. Based on the volume, I would guess WEP at corporate, not each store.
rmogull Mar 19
Im writing up a separate PCI post instead of responding here- I know it’s a contentious statement that deserves a more thoughtful response.
Should be up in about 15…
CPineda Mar 19
Will wait for your response Rich. Good analysis but disagree a bit on PCI. Like the others have said, it’s good rather than nothing at all. PCI has been a good step. It takes a while but we’re all getting there. We just need to fix the process of verifying the controls.
LonerVamp Mar 19
Rich, nice post. I think all of your statements above are what I read as well when this news broke. I was especially disturbed inferring that Hannaford didn’t even know a lick about this until the fraud cases built up and someone else notified them. So much for detection. And of course prevention was incompete.
Like others have mentioned above, so I won’t wax long on it: PCI is not worthless. It won’t stop a breach, but it does have value to shops that otherwise have less security than what PCI requires. I’ll have to say it again: PCI does not stop a breach. Whether that means it is worthless to security or not depends on how you define security, as an all or nothing situation or a point on a scale. So, to some, yes PCI is worthless, but I’d bet those people are also already beyond the low denominator PCI sets.
Adrian Lane Mar 19
Nice post Rich!
-
Agree with you this had to be some central HQ breach. Too many numbers gathered too quickly to be anything else. Disagree with the statement that PCI is worthless, unless you are also making the statement merchants should not be storing the numbers in the first place. A PCI audit on the other hand, as a bare minimum lowest common denominator set of practices, is worthless. If you need someone to certify your doing the absolute minimum, you’re really missing the entire point of the exercise.
TimC Mar 19
Great analysis, I have something to add that may not have been considered.
If the data was somehow sniffed during transit between the Hannaford network and the CC processor, considering the business model of a grocery store——-one has to assume that these are almost all ‘Card Present’ transactions.
This being true; the mag-stripe may have also be captured which contains, Cardholder Name, Card Verification Value (CVV), and PIN verification data. In fact, with mag-stripe data, bad-guys can create their own cards out of hotel room keys and commit fraud.
This is much more that just card numbers and expiration dates as described in the press releases.
Wackawacka Mar 19
We had to contact our bank about this. Wachovia doesn’t seem to know a thing about this situation. Our cards are being reissued and I’m never shopping with them again.
missing link Mar 19
Seems like an inside job to me. The data transmission was probably clear text over a private line and redirected to an unauthorized url after the authotization came back. My bet is weak log review.
Concerned shopper Mar 20
I am deeply hurt by what happened at Hannaford. I am truly a frequent shopper, especially during these dates mentioned.
I only like Hannaford because it has all the natural products I prefer and they sell Goat Milk all the time. I also like the organic breads with no HFCS. Throughout the store now are many alternative choices of Nature’s Place which I prefer in many cases. I do not really like PC that much, yet have sort of switched for a day. If I go back to Hannaford, should I get $200 at the ATM and have a wad of twenties in my purse so I can get conked on the head? Or should I write a check, in which my signature can be frauded? Is that the right word? This credit/debit card solution must be made secure as it is suppose to be a private transaction. I do want to continue my shopping at Hannaford, yet when? This problem was bound to happen with hackers all over the world having a great time using their brains for all the wrong reasons and considering it success when they can get into a private system. I truly am saddened at the state of technology and the world.
Bitten by the hand that feeds me Mar 20
Bank responses have been all over the map. When I heard the news I stopped in at my local branch - they ran my card and confirmed it was ‘on the list’. I asked them when they were going to tell me and they gave the the bank president’s phone number (I kid you not!). His representative said that they weren’t going to tell customers, and weren’t going to re-issue cards unless a customer ‘asked’ - that they were confident they could monitor card activity and react in the case of a compromise. I asked what that reaction might be - and they said ‘cancel the card’ - great! it normally takes 7-10 days to get one, and if you are on the road and it happens to be your only card you are out of luck. They seemed remarkably unmoved by my argument that we should have the right to know and then decide how we wanted to handle it.
Other local banks have stepped up to the plate and are letting customers know immediately - including one which is issuing new cards, then monitoring the old ones, but not canceling them until the new ones get activated by the customer. How’s that for great customer service!
David Navetta, Esq. Mar 28
I think we don’t have all the information so we everybody is engaging in various levels of speculation (which is probably more worthless than PCI is alleged). More info on all of this from a legal standpoint at my blog: http://www.infoseccompliance.blogspot.com
However, we do know two facts: (1) compliance with PCI was represented in Hannaford’s privacy policy (last visited 3-21-2008); and (2) there was a breach exposing cardholder data.
In my view, here are some of the possibilities (in no particular order of likelihood, and by no means an exclusive ilst):
(1) the qualified security assessor (QSA) (or internal assessor) may have misinterpreted or loosely interpreted a section of the PCI standard (and the reality was there were security weaknesses); potential culprit is 4.1. if unencrypted data was swiped in transit;
(2) the PCI compliance may have been old or outdated (e.g. they may have been PCI compliant 9 months ago, but perhaps added new systems that were not secured consistently with PCI);
(3) Hannaford may not have provided all of the information to the QSA (assuming one was used) that it needed to validate its decision (e.g. this could include mistakes in defining which parts of Hannaford’s networks were in-scope/out-of-scope);
(4) Hannaford may have been 100% PCI compliant and reasonably secure in general and just got unlucky (e.g. there is no such thing as 100% perfect security). Under this scenario, Hannaford would argue that it was not negligent because it did all the right things and that unfortunately these things just happen.
(5) Hannaford and/or its QSA may have had a security weakness or questions about an ambiguity and may have had either the PCI Council, its upstream payment processor or its merchant bank give a bad interpretation.
(6) Hannaford may have been perfectly PCI compliant, but nonehtless engaged in “negligent security” practices (e.g. under the law, the industry standard is a necessary, but not necessarily adequate — see T.J. Hooper)
The interesting issue will be, assuming that some sort of negligence is shown, who was/is ultimately responsible? Hannaford? The QSA? A merchant bank that accepted Hannaford’s certification? The standards setting body?
Bloggerjogger Mar 28
Why not just face the facts?
Hackers will get card numbers in a variety of ways in a variety of exposure levels (Hannaford= 4M card numbers; TJX= 90+M? cards; your irresponsible son = 1 card), no matter the security steps.
As a consumer, if you use cards, you have to review your account activity and statements carefully and regularly. You cannot expect someone to do it for you. If you are unable or unwilling to do this, get rid of your cards and stop whining.
Bankers and card issuers, get smart about your risk management policies and practices. Is blanket reissues of millions of cards on suspect lists cheaper than your insurance premiums, consumer fraud chargebacks, risk mitigation and fraud detection costs? The bottom line: there is is baseline cost of playing this game; make some decisions on objective facts and wise up: how much are you willing to pay and how will you be recovering these expenses?
Merchants: see the previous paragraphs and consider how consumer fraud reports will impact the issuers and insurers; consider how the issuers and insurers will recover their losses. How much are you willing to pay and how will you recover these expenses?
In the spirit of the fuel cost adjustment fee we are now seeing regularly, I propose a data security risk fee to be applied to all at-risk transactions for greater transparency.
Jim Apr 4
I read an article that Hannaford admitted that malware was installed on their store servers and that was the vehicle for their compromise. Now PCI-DSS compliance requires that anti-virus software must be installed on all systems that store or processes cardholder data. Futher, they must stay current on virus definition updates. So, how could they have been deemed PCI compliant? Did they lapse on staying current on the definition updates or is it that anti-virus software can only protect you from “known” virus & malware programs? So does this mean if I create a virus today and load it on a system that has anti-virus software and up to date definition files that anti-virus software will not detect it since the AV software and definitions do not have it in their list of “known” malicious programs?
rmogull Apr 4
Actually, most AV can’t stop a targeted attack. They could easily be in compliance, yet still suffer from malware.
You have it pegged- AV software kind of sucks.
Matt Apr 6
Credit card number are NOT useless without the name.
You can use a credit card without having to know the name of the card holder. I worked at a store where I took payments over the phone. We simply asked for the CCN and exp date. No name was required. The criminals need to own or have access to one of those credit card machines, they can simply type in the numbers and charge the customer for a fake service or product and pocket the money. Or, they could find some other sort of way to convert a credit charge to cash, services, or products through a third party. For instance, they could call a 900 number that the criminals ultimately own. That way they filter the money through the phone company.
e9an Apr 7
Why is no one considering the idea that First Data Corp. could be the culprit here and not Hannaford at all. They use PCI to validate the security of their systems as well.
e9an
rmogull Apr 15
It’s possible, but all the reports specifically state that retail locations were compromised.
Interesting on the CNP transactions- I thought you always needed a name, not just the card number. Apoligies for getting that wrong.
Gregg Shupe Apr 25
As more stories like this unfold, the questions on the minds of data security professionals are “How do I mitigate the risk of data loss in general?” and “How do I prevent this type of loss (insider attacks, data transmissions or storing sensitive information) in particular?”
The answer to both questions is to keep your data out of a position where it’s vulnerable to theft or other loss by protecting it, everywhere. In this particular incident, the real sin committed was not the loss of the data itself, but rather the fact that the company passed the PCI standards, thus thinking they were protected. The comfort in the compliance was the great downfall; they lost sight of what is really important…protecting the data itself. Why take unnecessary risks when solutions exist to completely eliminate this type of exposure?
There are technologies specifically developed to secure the data itself for transport, storage or backup. A strong encryption solution coupled with network security solutions will protect the data itself and keep hackers out of the network. Also, with the new advancements in Policy and Key management solutions, organizations can encrypt all of their data as it travels the network, end-to-end. With these solutions in place, the data is securely transported electronically, drastically reducing the chances of being stolen; and because the data is encrypted, it is useless to anyone except the intended recipient.
AnalystPCI May 6
I’m an analyst following the PCI landscape and I have heard from reliable sources that the Point of Sale systems were compromised by worms. We are also seeing a lot of enquiries for products that lockdown Point of Sale systems. Vendors like Tripwire and Solidcore are experiencing double-digit growth fueled by this requirement. News on the grapevine is that Solidcore has closed at least 5 mega deals and is preparing for an IPO.
rmogull May 12
It still seems a little too coordinated/advanced to be a random worm, but that’s some very interesting information. Could have been something custom then.
Jason Ramsey Jul 23
It is against PCI-DSS to store ANY secure credit card data (Credit Card Numbers, Expiration Data) unencrypted. It is also against policy to store CVV2 Numbers ANYWHERE.
If they were PCI-DSS compliant the only place to steal that information is out of program memory by compromising the application, prior to encryption (at the terminal), or breaking the encryption.
PCI specifies the concepts of securing a system not the details, and they shouldn’t. The details change with increased technology.