Login  |  Register  |  Contact

An Open Letter to Robert Carr, CEO of Heartland Payment Systems

Mr. Carr,

I read your interview with Bill Brenner in CSO magazine today, and I sympathize with your situation. I completely agree that the current system of standards and audits contained in the Payment Card Industry Data Security Standard is flawed and unreliable as a breach-prevention mechanism. The truth is that our current transaction systems were never designed for our current threat environment, and I applaud your push to advance the processing system and transaction security. PCI is merely an attempt to extend the life of the current system, and while it is improving the state of security within the industry, no best practices standard can ever fully repair such a profoundly defective transaction mechanism as credit card numbers and magnetic stripe data.

That said, your attempts to place the blame of your security breach on your QSAs, your external auditors, are disingenuous at best.

As the CEO of a large public company you clearly understand the role of audits, assessments, and auditors. You are also fundamentally familiar with the concepts of enterprise risk management and your fiduciary responsibility as an officer of your company. Your attempts to shift responsibility to your QSA are the accounting equivalent of blaming your external auditor for failing to prevent the hijacking of an armored car.

As a public company, I have to assume your organization uses two third-party financial auditors, and internal audit and security teams. The role of your external auditor is to ensure your compliance with financial regulations and the accuracy of your public reports. This is the equivalent of a QSA, whose job isn’t to evaluate all your security defenses and controls, but to confirm that you comply with the requirements of PCI. Like your external financial auditor, this is managed through self reporting, spot checks, and a review of key areas. Just as your financial auditor doesn’t examine every financial transaction or the accuracy of each and every financial system, your PCI assessor is not responsible for evaluating every single specific security control.

You likely also use a public accounting firm to assist you in the preparation of your books and evaluation of your internal accounting practices. Where your external auditor of record’s responsibility is to confirm you comply with reporting and accounting requirements and regulations, this additional audit team is to help you prepare, as well as provide other accounting advice that your auditor of record is restricted from. You then use your internal teams to manage day to day risks and financial accountability.

PCI is no different, although QSAs lack the same conflict of interest restrictions on the services they can provide, which is a major flaw of PCI. The role of your QSA is to assure your compliance with the standard, not secure your organization from attack. Their role isn’t even to assess your security defenses overall, but to make sure you meet the minimum standards of PCI. As an experienced corporate executive, I know you are familiar with these differences and the role of assessors and auditors.

In your interview, you state:

The audits done by our QSAs (Qualified Security Assessors) were of no value whatsoever. To the extent that they were telling us we were secure beforehand, that we were PCI compliant, was a major problem. The QSAs in our shop didn’t even know this was a common attack vector being used against other companies. We learned that 300 other companies had been attacked by the same malware. I thought, ‘You’ve got to be kidding me.’ That people would know the exact attack vector and not tell major players in the industry is unthinkable to me. I still can’t reconcile that.”

There are a few problems with this statement. PCI compliance means you are compliant at a point in time, not secure for an indefinite future. Any experienced security professional understands this difference, and it was the job of your security team to communicate this to you, and for you to understand the difference. I can audit a bank one day, and someone can accidently leave the vault unlocked the next. Also, standards like PCI merely represent a baseline of controls, and as the senior risk manager for Heartland it is your responsibility to understand when these baselines are not sufficient for your specific situation.

It is unfortunate that your assessors were not up to date on the latest electronic attacks, which have been fairly well covered in the press. It is even more unfortunate that your internal security team was also unaware of these potential issues, or failed to communicate them to you (or you chose to ignore their advice). But that does not abrogate your responsibility, since it is not the job of a compliance assessor to keep you informed on the latest attack techniques and defenses, but merely to ensure your point in time compliance with the standard.

In fairness to QSAs, their job is very difficult, but up until this point, we certainly didn’t understand the limitations of PCI and the entire assessment process. PCI compliance doesn’t mean secure. We and others were declared PCI compliant shortly before the intrusions.

I agree completely that this is a problem with PCI. But what concerns me more is that the CEO of a public company would rely completely on an annual external assessment to define the whole security posture of his organization. Especially since there has long been ample public evidence that compliance is not the equivalent of security. Again, if your security team failed to make you aware of this distinction, I’m sorry.

I don’t mean this to be completely critical. I applaud your efforts to increase awareness of the problems of PCI, to fight the PCI Council and the card companies when they make false public claims regarding PCI, and to advance the state of transaction security. It’s extremely important that we, as an industry, communicate more and share information to improve our security, especially breach details. Your efforts to build an end to end encryption mechanism, and your use of Data Loss Prevention and other technologies, are an important contribution to the industry.

Unless your QSAs were also responsible for your operational security, the only ones responsible for your breach are the criminals, and Heartland itself. I cannot possibly believe that you trusted your PCI audit to determine if you were secure from attack; considering all we know, and all the information available on PCI, that would be borderline negligence. Even if your QSAs were completely negligent and falsified your compliance, that would not make them responsible for your breach.

Rather than blaming your QSAs, I hope you take this opportunity to encourage other executives to treat their PCI assessment as merely another compliance initiative – one that does not, in any way, ensure their security. As an industry professional I see all too many organizations do the minimum for PCI compliance, and ignore the other security risks their organizations face, even when properly informed by their internal security professionals. This is the single greatest problem with PCI, and one you have an opportunity to help change.

If I misread your statements or the article was inaccurate, I apologize for my criticism. If any of my prior criticisms of your organization were unfounded, I take full responsibility and also apologize for those.

But, based on your prior public statements and this interview, you appear to be shifting the blame to the card companies, your QSA, and the PCI Council. From what’s been released, your organization was breached using known attack techniques that were preventable using well-understood security controls.

As the senior corporate officer for Heartland, that responsibility was yours.

Rich Mogull,



No Related Posts
Previous entry: Understanding and Choosing a Database Assessment Solution, Part 1: Introduction | | Next entry: It’s Thursday the 13th—Update Adobe Flash Day


If you like to leave comments, and aren't a spammer, register for the site and email us at info@securosis.com and we'll turn off moderation for your account.

By John Baxter  on  08/12  at  11:07 PM

Strong stuff, Rich. And I hope it is read and taken to heart by the people who should do so. —John (who found the post via Twitter)

By Interested Reader  on  08/13  at  12:00 AM

Well said, Rich.

Keep in mind that he’s not saying any of this under oath and likely will never admit they did something wrong because their General Counsel would advise against it. (Lucky for them, their QSAs are probably gagged from responding under confidentiality clauses in their contract.)  Admitting any wrongdoing exposes the company (and himself) to a variety of legal scenarios, which is why companies will often “settle” with regulators and others without admitting guilt.  Corporations and individuals have both been socialized into never admitting wrongdoings as it can only make things worse from the legal standpoint they fear so much. 

I’d wager that the Security team asked for budget to shore up their defenses and, as often happens, it was rejected by someone in the chain of command as “too expensive” without even running the numbers.  Besides, as long as they had the shiny “PCI Compliant” sticker, they believed they had mitigated the risk of increased transaction costs.  So what justification is there to spend the money on security?

I believe that after the writedown they just took in their Q2 earnings, a shareholder class action lawsuit may get to the bottom of what really happened, but as long as it gets settled out of court, it would only benefit the lawyers.

// Chris

By Tom Olzak  on  08/13  at  12:34 PM

Well said, Rich.  This letter is more than a response to one clueless CEO.  It should serve as the basis for any discussion on balancing compliance, operational effectiveness, and actual security.

By windexh8er  on  08/13  at  02:34 PM

Very well put Rich.  I found it laughable at first glance and severely disheartening the more I thought about what Mr. Carr had said to the public.  The thing is I think Mr. Carr knew exactly what he was doing.  If the mainstream press caught wind of the story it would help offset blame of the internal problem for public consumption.  Mr. Carr, to me, is in CYA mode right now as maybe his indefinite future is on the line.  Maybe not, but at the end of the day Mr. Carr feels as if they bought offset of risk through the actions of the QSA.  An auditors job is, at best, a spot check (as noted).  If you want a full blown penetration test / ethical hack performed against your infrastructure then Mr. Carr should have paid for it.

In all reality the director of internal security should end up with the firestorm—sure, not everything can always be prevented even in the best of designed systems (and I’m not just talking about exploit attack vectors), but that particular person is to blame—not the QSA.

If we can learn anything from this it’s that PCI, HIPAA, GLBA, SOX, etc are all minimum levels of security engagement.  IF, you don’t pass PCI with *flying* colors then you have much bigger issues than the QSA will have pointed out to you.  I find that all of my clients are only trying to meet the compliance minimums and I’m trying to tell them that every year they scramble (in some way shape or form) to get “ready” for the next audit.  If you build it in, you do it right, you proactively think about security as an operational requirement you don’t end up in that lather, rinse and repeat mode of security remediation every year.

I liked the open letter, glad someone took the time to address Mr. Carr directly.


By Robi Papp  on  08/13  at  02:50 PM

I disagree with parts of your criticism.  Heartland hired a third party firm to audit their IT controls against a standard list.  That organization did indeed fail them by either being too lenient with their assessment of those controls or failing to advise their client of known risks.  This situation is analogous to taking your auto to a mechanic for a tune up and only checking the fluids, while missing cracks in your timing belt or a leaky coolant system.  There needs to be a way for corporations to understand the expected quality of their audits and assessments.  I just had a similar conversation with a client regarding how to evaluate assessment proposals.  Maybe there needs to be a Yelp! for QSAs?

Also, there is no indication that this is the case:

“But what concerns me more is that the CEO of a public company would rely completely on an annual external assessment to define the whole security posture of his organization.”

By Rich  on  08/13  at  02:57 PM


Considering the complexity of IT systems I don’t think your mechanic analogy holds true. It’s more like a single mechanic evaluating an entire fleet of vehicles. There’s just too much to cover.

This isn’t an audit quality issue, it’s an audit scope issue, and I have no doubts Mr. Carr was aware of the limitations of the process.

By Interested Reader  on  08/13  at  03:05 PM

I’d actually take that a step further, Rich, and say that it’s like expecting a single mechanic to evaluate the every aspect of an entire fleet of vehicles, but only hiring him to check the brakes and air bags in a couple days.  PCI’s breadth is so narrow and QSAs are hired for so little a period of time that you can’t legitimately expect it to be a full security assessment.  You didn’t pay him to do it, so blame him when one of the cars’ fuel line leaks on the muffler. 

// Chris

By Jeff Allen  on  08/13  at  07:36 PM

Very interesting take, Rich. I heard Mr. Carr present their story at the Gartner IT Security Summit last month, and I have to say, despite everything I know about PCI, I was compelled by his argument that PCI and Heartland’s QSA let him down. I think it’s easy to get caught up in his argument when the reality is, as you point out, that this breach was outside of the scope of what the QSA was looking for in the first place.

I see the disconnect caused by the differences between two perspectives: I think it’s easy to look down from the top and say, “I don’t like spending money to comply with this reg, but at least we will know we’re secure”. Unfortunately, the folks on the ground supporting the audit are thinking something very different a lot of the time. They are thinking, “how do we get this auditor out of here as quickly as possible with as few new ‘to-do items’ at the end as possible.” With the guys in the trenches looking at pass/fail grading, it’s unlikely that they will communicate that they got a D+ (pass) on their audit. Meanwhile, the guys upstairs see “pass” and they think “we got an A”. Lots of room for holes between those two views.

Still, I really admire Carr for getting out and telling his story and for the way he’s leading his company out of this morass. Besides, how many other CEOs would agree to take the stage at that show?

By Rich  on  08/13  at  08:27 PM


I’d be more impressed if he released the actual breach details, which will be invaluable to the rest of the industry.

Any CEO knows better than to think an audit for a compliance initiative makes you secure. I think he’s just misdirecting.

By Jeff Allen  on  08/13  at  09:56 PM

I agree, I hope they release more details. I haven’t done extensive research on it, but I haven’t seen much explanation beyond it being a SQL injection attack in their corporate network that gave some rogue entity access to hunt around their systems for a few months until it found access to their payment network.

I certainly hope they’re more open about this than they appear because in his talk, Carr made a big deal about the idea that perhaps some of his competitors had information about the attack vector that they were not sharing because they considered it a competitive advantage. He also said that Heartland has since worked with FS-ISAC to organize the Payment Processors Information Sharing Council (PPISC) to distribute malware and attack vectors.

By Pete  on  08/14  at  05:34 PM

I agree with Rich.

Using PCI compliance to internally justify conforming to good security practices is counter-intuitive. If a company is trying to implement good security they want to uncover any possible vulnerability in their systems. When trying to pass a compliance audit you provide the auditor the minimal amount of information possible so they can complete the task. No smart business would “air their dirty laundry” to their auditor, so it would be foolish to use regulatory compliance as justification of good security practices.

By LonerVamp  on  08/17  at  03:22 PM

Nice piece, Rich, and I also agree with the angles you examined in it.

The interview does have nice pieces, and it’s nice to see Carr getting some things right after the fact (too bad it’s after the fact), but his continuing to blame others was already annoying when they first announced the breach let alone these many months later. That’s certainly one thing he has remained consistent on.

I’m not sure we can nitpick too many details about who or what failed. Maybe a QSA did let them down, maybe not. But that level of analysis probably requires more details about what the attackers did, what weaknesses were present in their systems, and what the QSA results/checks had been.

Curiously, that level of disclosure is exactly another thing Carr has been very vocal about: communication and openness with information and attacks so others can learn from it and be properly secure. Sadly, he himself has not followed through on that by sharing information on his breach (at least not public).

Oh well, it’s sad that all of this had to have a breach involved for someone to find their security religion, but it would do well for him to admit the organization (and he personally) made a mistake and have now moved on.

By James Anderson  on  08/17  at  05:38 PM

Disagree strongly with the tone and some of the facts in the letter.  Carr never blames his QSAs or shirks responsibility.  Maybe his QSAs *should have* done more to alert Heartland to the risk.  http://blog.emagined.com/
Jim Anderson

By Rich  on  08/17  at  06:08 PM


We must have read different interviews, because at no point did I see Carr ever take responsibility for the breach. Actually, if you look across multiple interviews he consistently focuses on PCI, the payment industry and his QSAs. He also has yet to tell anyone how the breach happened, which could clear all of this up.

You also must have read a bizzaro version of my post, since you call it “shrill” and a “rant”, while most readers see it as quite balanced (based on feedback).

If you have a reference where Carr takes responsibility, please send it over and I may change my position.

By Hmmmm  on  08/17  at  10:15 PM

An organization that processes hundreds of millions of transactions each month gets hit with an initial SQL Injection breach?

PCI? QSA? What ever happened to reviewing all code before releasing into production?


Can someone tell me what I am missing here?

By Mike Katz  on  08/19  at  08:12 AM

I agree that the QSA is not at fault.  But what about the company that performed the penetration test.  Heartland should have had an annual penetration test (PCI requirement 11.3).  It looks like whoever did the penetration test, missed the SQL injection vulnerability(ies).  While a penetration test may miss some issues, SQL injection is common enough that it should have been caught during a penetration test.

By Ramachandra Putti  on  08/26  at  05:55 PM

Well said Rich.

I agree with you that PCI compliance is at a point of time. They cannot safeguard breaches and its like the work of anexternal auditor.

This makes me say should corporates be driven or should they drive? Unless corporates take ownership of issues be it security issues of today, no standard or certification makes any sense. Certification is as you said an external auditor based on inputs given to him.

Nevertheless, I also would like to say that QSA’s should not just ask for compliance of minimum standards or settle for compensating controls (based on scenarios).

Corporates should ask “What does it take me to safegaurd the entrusted card holder data? ” and not to settle for certification as the first plug !!

By Ramachandra Putti  on  08/26  at  06:04 PM

For question by “Hmmmm” and comment by Mike Katz

If the application a website, there are good many testing tools for SQL Injection vulnerabilities. What is important is in configuring the tool to check for such vulnerabilities and this is often missed. (IBM Appscan is one such tool)

I liked what my QSA did. Track the change control documents from initial application scan to the last one and verify the configuration of the testing tool.

I would say that apart from SQL Injection, flaws by XSS/CRLF are also to be remediated which requires more effort. Probably you would need to scan the applications for these vulnerabilities.



Remember my personal information

Notify me of follow-up comments?