By Adrian Lane
My mentors in engineering management used to define their job as managing people, process, and technology. Those three realms, and how they interact, are a handy way to conceptualize organizational management responsibilities. We use process to frame how we want people to behave – trying to promote productivity, foster inter-group cooperation, and minimize mistakes. The people are the important part of the equation, and the process is there to help make them better as a group. How you set up process directly impacts productivity, arranges priority, and creates or reduces friction. Subtle adjustments to process are needed to account for individuals, group dynamics, and project specifics.
I got to thinking about this when reading Microsoft’s Simple Implementation of SDL. I commented on some of the things I liked about the process, specifically the beginning steps of (paraphrased):
- Educate your team on the ground rules.
- Figure out what you are trying to secure.
- Commit to gate insecure code.
- Figure out what’s busted.
Sounds simple, and conceptually it is, but in practice this is really hard. The technical analysis of the code is difficult, but implementing the process is a serious challenge. Getting people to change their behavior is hard enough, but with diverging company goals in the mix, it’s nearly impossible. Adding the SDL elements to your development cycle is going to cause some growing pains and probably take years. Even if you agree with all the elements, there are several practical considerations that must be addressed before you adopt the SDL – so you need more than the development team to embrace it.
The Definition of Insanity
I heard Marcus Ranum give a keynote last year at Source Boston on the Anatomy of The Security Disaster, and one of his basic points was that merely identifying a bad idea rarely adjusts behavior, and when it does it’s generally only because failure is imminent. When initial failure conditions are noticed, as much effort is spent on finger-pointing and “Slaughter of the Innocents” as on learning and adjusting from mistakes. With fundamental process re-engineering, even with a simplified SDL, progress is impossible without wider buy-in and a coordinated effort to adapt the SDL to local circumstances.To hammer this point home, let’s steal a page from Mike Rothman’s pragmatic series, and imagine a quick conversation:
CEO to shareholders: “Subsequent to the web site breach we are reacting with any and all efforts to ensure the safety of our customers and continue trouble-free 24x7 operation. We are committed to security … we have hired one of the great young minds in computer security: a talented individual who knows all about web applications and exploits. He’s really good and makes a fine addition to the team! We hired him shortly after he hacked our site.”
Project Manager to programmers: “OK guys, let’s all pull together. The clean-up required after the web site hack has set us back a bit, but I know that if we all focus on the job at hand we can get back on track. The site’s back up and most of you have access to source code control again, and our new security expert is on board! We freeze code two weeks from now, so let’s focus on the goal and …
Did you see that? The team was screwed before they started. Management’s covered as someone is now responsible for security. And project management and engineering leadership must get back on track, so they begin doing exactly what they did before, but will push for project completion harder than ever. Process adjustments? Education? Testing? Nope. The existing software process is an unending cycle. That unsecured merry-go-round is not going to stop so you can fix it before moving on. As we like to say in software development: we are swapping engines on a running car. Success is optional (and miraculous, when it happens).
Break the Process to Fix It
The Simplified SDL is great, provided you can actually follow the steps. While I have not employed this particular secure development process yet, I have created similar ones in the past. As a practical matter, to make changes of this scope, I have always had to do one of three things:
- Recreate the code from scratch under the new process. Old process habits die hard, and the code evaluation sometimes makes it clear a that a retrofit would require more work than a complete rewrite. This makes other executives very nervous, but has been the most efficient path from practical experience. You may not have this option.
- Branch off the code, with the sub-branch in maintenance while the primary branch lives on under the new process. I halted new feature development until the team had a chance to complete the first review and education steps. Much more work and more programming time (meaning more programmers committed), but better continuity of product availability, and less executive angst.
- Moved responsibility of the code to an entirely new team trained on security and adhering to the new process. There is a learning curve for engineers to become familiar with the old code, but weaknesses found during review tend to be glaring, and no one’s ego get hurts when you rip the expletive out of it. Also, the new engineers have no investment in the old code, so can be more objective about it.
If you don’t break out of the old process and behaviors, you will generally end up with a mixed bag of … stuff.
As in the first post, I assume the goal of the simplified version of the process is to make this effort more accessible and understandable for programmers. Unfortunately, it’s much tougher than that. As an example, when you interview engineering candidates and discuss their roles, their skill level is immediately obvious. The more seasoned and advanced engineers and managers talk about big picture design and architecture, they talk about tools and process, and they discuss the tradeoffs of their choices. Most newbies are not even aware of process. Here is Adrian’s handy programmer evolution chart to show what I mean:
Many programmers take years to evolve to the point where they can comprehend a secure software development lifecycle, because it is several years into their career before they even have a grasp of what a software development lifecycle really means. Sure, they will learn tactics, but it just takes a while to put all the pieces together. A security process that is either embedded or wrapped around and existing development process is additional complexity that takes more time to grasp and be comfortable with. For the first couple years, programmers are trying to learn the languages and tools they need to do their jobs. Having a grasp of the language and programming style comes with experience. Understanding other aspects of code development such as design, architecture, assurance and process takes more time.
One of a well-thought-out aspects of the SDL is appointing knowledgeable people to champion the efforts, which gets around some of the skills barrier and helps compensate for turnover. Still, getting the team educated and up to speed will take time and money.
A Tree without Water
Every CEO I have ever spoken with is all for security! Unquestioningly, they will claim they are constantly improving security. The real question is whether they will fund it. As with all engineering projects where it is politically incorrect to say ‘no’, security improvement is most often killed through inaction. If your firm will not send programming team members for education, any security process fails. If your firm will not commit the time and expense to change the development process, by including additional security testing tasks, security processes fail. If your firm will not purchase fuzzing tools or take the time for proper code review, the entire security effort will wither and die. The tendrils of the development process, and any security augmentation efforts, must extend far beyond the development organization itself.
Posted at Monday 8th February 2010 5:00 pm
(5) Comments •
By Mike Rothman
It seems I’ve been preoccupied lately with telling all of you about the things you shouldn’t do anymore. Between blowing away firewall rules and killing security technologies, I guess I’ve become that guy. Now get off my lawn!
But why stop now – I’m on a roll. This week, let’s take on another common practice that ends up being an extraordinarily bad idea – running user devices with administrator access. Let’s slay that sacred cow.
Once again, most of you security folks with any kind of kung fu are already here. You’d certainly not let an unsophisticated user run with admin rights, right? They might do something like install software or even click on a bad link and get pwned. Yeah, it’s been known to happen.
Thus the time has come for the rest of the great unwashed to get with the program. Your standard build (you have a standard build, right? If not, we’ll talk about that next week in the Endpoint Fundamentals Series) should dictate user devices run as standard users. Take that and put it in your peace pipe.
Who cares? Why are we worried about whether a user runs as a standard or admin user? To state the obvious: admin rights on endpoint devices represent the keys to the kingdom. These rights allow users to install software, mess with the registry under Windows, change all sorts of config files, and generally do whatever they want on the device. Which wouldn’t be that bad if we only had to worry about legitimate users.
Today’s malware doesn’t require user interaction to do its damage, especially on devices running with local admin access. All the user needs to do is click the wrong link and it’s game over – malware installed and defenses rendered useless, all without the user’s knowledge. Except when they are offered a great deal on “security software,” get a zillion pop-ups, or notice their machine grinding to a halt.
To be clear, users can still get infected when running in standard mode, but the attack typically ends at logout. Without access to the registry (on Windows) and other key configuration files, malware generally expires when the user’s session ends.
As with most of the tips I’ve provided over the past few weeks, user grumpiness may result once they start running in standard mode. They won’t be able to install new software and some of their existing applications may break. So use this technique with care. That means you should actually test whether every application runs in standard user mode before you pull the trigger. Business critical applications probably need to be left alone, as offensive as that is.
Most applications should run fine, making this decision a non-issue, but use your judgement on allowing certain users to keep admin rights (especially folks who can vote you off the island) to run a specific application. Yet I recommend you stamp your feet hard and fast if an application doesn’t play nicely in standard mode. Yell at your vendors or internal developers to jump back into the time machine and catch up with the present day. It’s ridiculous that in 2010 an end-user facing application would require admin rights.
You also have the “no soup for you” card, which is basically the tough crap response when someone gripes about needing admin rights. Yes, you need to have a lot of internal mojo to pull that off, but that’s what we are all trying to build, right? Being able to do the right security thing and make it stick is the hallmark of an effective CISO.
So there you have it. This is a FireStarter, so fire away, sports fans. Why won’t this work in your environment? What creative, I mean ‘legitimate’, excuses are you going to come up with now to avoid doing proper security hygiene on the endpoints? This isn’t that hard, and it’s time. So get it done, or tell me why you can’t…
Posted at Monday 8th February 2010 2:30 pm
(10) Comments •
By Mike Rothman
I tend to be master of the obvious. Part of that is overcoming my own lack of cranial horsepower (especially when I hang out with serious security rock stars), but another part is the reality that we need someone to remind us of the things we should be doing. Work gets busy, shiny objects beckon, and the simple blocking and tackling falls by the wayside.
And it’s the simple stuff that kills us, as evidenced once again by the latest data breach study from TrustWave.
Over the past couple months, we’ve written a bunch of times about the need to move to the latest versions of the key software we run. Like browsers and Adobe stuff. Just to refresh your memory:
Of course, the news item that got me thinking about reiterating such an obvious concept was that Google is pulling support for IE6 for all their products this year. Google Docs and Google sites loses IE6 support on March 1. Not that you should be making decisions based on what Google is or is not supporting, but if ever there was a good backwards looking indicator, it’s this. So now is the time.
To be clear, this isn’t news. The echo chamber will beat me like a dog because who the hell is still using IE6? Especially after our Aurora Borealis escapade. Right, plenty of folks, which brings me back to being master of the obvious. We all know we need to kill IE6, but it’s probably still somewhere in the dark bowels of your organization. You must find it, and you must kill it.
Not in the Excuses Business
Given that there has been some consistent feedback every time we say to do anything, that it’s too hard, or someone will be upset, or an important application will break, it’s time to man (or woman) up. No more excuses on this one. Running IE6 is just unacceptable at this point.
If your application requires IE6, then fix it. If your machines are too frackin’ old to run IE7 or IE8 with decent performance, then get new machines. If you only get to see those remote users once a year to update their machines, bring them in to refresh now.
Seriously, it’s clear that IE6 is a lost cause. Google has given up on it, and they won’t be the last. You can be proactive in how you manage your environment, or you can wait until you get pwned. But if you need the catalyst of a breach to get something simple like this done in your environment, then there’s something fundamentally wrong with your security program.
But that’s another story for another day.
- Comment from Rich: As someone who still develops web sites from time to time, it kills me every time I need to add IE6 compatibility code. It’s actually pretty hard to make anything modern work with IE6, and it significantly adds to development costs on more-complex sites. So my vote is also to kill it!
Posted at Friday 5th February 2010 4:07 pm
(3) Comments •
I think I need to stop feeling guilty for trying to run a business.
Yesterday we announced that we’re trying to put together a list of end users we can run the occasional short survey past. I actually felt guilty that we will derive some business benefit from it, even though we give away a ton of research and advice for free, and the goal of the surveys isn’t to support marketing, but primary research.
I’ve been doing this job too long when I don’t even trust myself anymore, and rip apart my own posts to figure out what the angle is. Jeez – it isn’t like I felt guilty about getting paid to work on an ambulance.
It is weird to try to build a business where you maintain objectivity while trying to give as much away for free as possible. I think we’re doing a good job of managing vendor sponsorship, thanks to our Totally Transparent Research process, which allows us to release most white papers for free, without any signup or paywall. We’ve had to turn down a fair few projects to stick with our process, but there are plenty of vendors happy to support good research they can use to help sell their products, without having to bias or control the content. We’re walking a strange line between the traditional analyst model, media sponsorship, research department, and… well, whatever else it is we’re doing. Especially once we finish up and release our paid products.
Anyway, I should probably get over it and stop over-thinking things. That’s what Rothman keeps telling me, not that I trust him either.
Back to that user panel – we’d like to run the occasional (1-2 times per quarter) short (less than 10 minutes) survey to help feed our research, and as part of supporting the OWASP survey program. We will release all the results for free, and we won’t be selling this list or anything. If you are interested, please email us at email@example.com. End users only (for now) please – we do plan on expanding to vendors later. If you are at a vendor and are responsible for internal security, that’s also good. All results will be kept absolutely anonymous.
We’re trying to give back and give away as much as we can, and I have decided I don’t need to feel guilty for asking for a little of your time in exchange.
On to the Summary:
Webcasts, Podcasts, Outside Writing, and Conferences
Favorite Securosis Posts
Other Securosis Posts
Favorite Outside Posts
Project Quant Posts
Top News and Posts
- House passes cybersecurity bill. This hit right as we were going to press, so we’ll provide analysis later.
- PGP Acquires TC TrustCenter & Chosen Security. If a PKI falls in the woods, does anyone hear it?
- David Litchfield hangs up the gloves. David is an exceptional researcher who was a powerful counterbalance to Oracle marketing. Sad to see him go, but I think he had a great positive effect.
- A perfect APT rant. Okay, it’s only near perfect, and some of the advice is a bit vanilla, but you gotta love a good rant.
- Social Security Awards finalists revealed. Congrats to the nominees.
- Rsnake releases massive list of Remote File Inclusion vulns. As in, over 1,000.
- Mandiant releases info on APT attacks. These are the guys that actually investigate these attacks.
- Bank sues cybercrime victim.
- Mass data protection law going into effect. Don’t get too excited, at least until we see enforcement.
- Google to pay for bugs in Chromium. Methinks they might want to pay a bit more, considering current market value.
- Security B-Sides lining up for San Francisco. It’s a great event, and free!
- iPhones vulnerable due to certificate handling flaw.
- Need a security vendor? There’s a site for that. Boaz’s new site categorizes 600+ security vendors – should be a huge time saver.
Blog Comment of the Week
Remember, for every comment selected, Securosis makes a $25 donation to Hackers for Charity. This week’s best comment goes to Bryan Sullivan, in response to FireStarter: Agile Development and Security.
Great timing, my Black Hat talk this week (http://blackhat.com/html/bh-dc-10/bh-dc-10-briefings.html#Sullivan) covers exactly this topic. If you’re coming to the briefings, stop by for the talk and we’ll get some conversation going.
It’s definitely not impossible to do secure Agile dev. I would say that certain Agile tenets present challenges to secure dev, but you can say the same thing about waterfall. The trick is overcoming those challenges, and I think we’ve done a pretty good job with SDL-Agile. Of course, if you’re using SDL-A and finding shortcomings with it I’d love to know about those too so I can address them.
Posted at Friday 5th February 2010 5:05 am
(0) Comments •
By Adrian Lane
I spent the last couple hours pouring over the Simplified Implementation of the Microsoft SDL. I started taking notes and making comments, and realized that I have so much to say on the topic it won’t fit in a single post. I have been yanking stuff out of this one and trying to just cover the highlights, but I will have a couple follow-ups as well. But before I jump into the details and point out what I consider are a few weaknesses, let me just say that this is a good outline. In fact, I will go so far as to say that if you perform each of these steps (even half-assed), your applications will more secure. Much more secure, because the average software development shop is not performing these functions. There is a lot to like here but full adoption will be difficult, due to the normal resistance to change of any software development organization. Before I come across as too negative, let’s take a quick look at the outline and what’s good about it.
For the sake of argument, let’s agree that the complexity of MS’s full program is the motivation for this simplified implementation of SDL. Lightweight, agile, simple, and modular are common themes for every development tool, platform, and process that
enjoys mainstream success. Security should be no different, and let’s say this process variant meets our criteria.
Microsoft’s Simplified Implementation of Secure Development Lifecycle (SDL) is a set of companion steps to augment software development processes. From Microsoft’s published notes on the subject, it appears they picked the two most effective security tasks from each phase of software development. The steps are clear, and their recommendations align closely with the web application security research we performed last year.
What I like about it:
- Phased Approach: Their phases map well to most development processes. Using Microsoft’s recommendations, I can improve every step of the development process, and focus each person’s training on the issues they need to account for.
- Appropriate Guidelines: Microsoft’s recommendations on tools and testing are sound and appropriate for each phase.
- Education: The single biggest obstacle for most development teams is ignorance of how threats are exploited, and what they are doing wrong. Starting with education covers the biggest problem first.
- Leaders and Auditors: Appointing someone as ‘Champion’ or leader tasks someone with the responsibility to improve security, and having an auditor should ensure that the steps are being followed effectively.
- Process Updates and Root Cause Analysis: This is a learning effort. No offense intended, but the first cycle through the process is going to be as awkward as a first date. Odds are you will modify, improve, and streamline everything the second time through.
Here’s what needs some work:
- Institutional Knowledge: In the implementation phase, how do you define what an unsafe function is? Microsoft knows because they have been tracking and classifying attacks on their applications for a long time. They have deep intelligence on the subject. They understand the specifics and the threat classes. You? You have the OWASP top ten threats, but probably less understanding of your code. Maybe you have someone on your team with great training or practical experience. But this process will work better for Microsoft because they understand what sucks about their code and how attackers operate. Your first code analysis will be a mixed bag. Some companies have great engineers who find enough to keep your entire development organization busy for ten years. Others find very little and are dependent on tools to tell them what to fix. Your effectiveness will come with experience and a few lumps on your forehead.
- Style: When do you document safe vs. unsafe style? Following the point above, your code development team, as part of their education, should have a security style guide. It’s not just what you get taught in the classroom, but proper use of the specific tools and platforms you rely on. New developers come on board all the time, and you need to document so they can learn the specifics of your environment and style.
- Metrics: What are the metrics to determine accountability and effectiveness? The Simplified Implementation mentions metrics as a way to guide process compliance and retrospective metrics to help gauge what is effective. Metrics are also the way to decide what a risky interface or unsafe function actually is. But the outline only pays lip service to this requirement.
- Agile: The people who most need this are web developers using Agile methods, and this process does not necessarily map to those methods. I mentioned this earlier.
And the parts I could take or leave:
- Tools: Is the latest tool always the best tool? To simplify their process, Microsoft introduced a couple security oversimplifications that might help or hurt. New versions of linkers & compilers have bugs as well.
- Threat Surface: Statistically speaking, if you have twice as many objects and twice as many interfaces, you will generally make more mistakes and have errors. I get the theory. In practice, or at least from my experience, threat surface is an overrated concept. I find that the issue is test coverage of the new APIs and functions, which is why you prioritize tests and manual inspection for new code as opposed to older code.
- Prioritization: Microsoft has a bigger budget than you do. You will need to prioritize what you do, which tools to buy first, and how to roll out tools in production. Some analysis is required on what training and products will be most effective.
All in all, a good general guide to improving development process security, and they have reduced the content to a manageable length. This provides a useful structure, but there are some issues regarding how to apply this type of framework to real world programming environments, which I’ll touch on tomorrow.
Posted at Friday 5th February 2010 1:00 am
(0) Comments •
The NSA is going to work with Google to help analyze the recent Chinese (probably) hack. Richard Bejtlich predicted this, and I consider it a very positive development.
It’s a recognition that our IT infrastructure is a critical national asset, and that the government can play a role in helping respond to incidents and improve security. That’s how it should be – we don’t expect private businesses to defend themselves from amphibious landings (at least in our territory), and the government has political, technical, and legal resources simply not available to the private sector.
Despite some of the more creative TV and film portrayals, the NSA isn’t out to implant microchips in your neck and follow you with black helicopters. They are a signals intelligence collection agency, and we pay them to spy on as much international communication as possible to further our national interests. Think that’s evil? Go join Starfleet – it’s the world we live in. Even though there was some abuse during the Bush years, most of that was either ordered by the President, or non-malicious (yes, I’m sure there was some real abuse, but I bet that was pretty uncommon). I’ve met NSA staff and sometimes worked with plenty of three-letter agency types over the years, and they’re just ordinary folk like the rest of us.
I hope we’ll see more of this kind of cooperation.
Now the one concern is for you foreigners – the role of the NSA is to spy on you, and Google will have to be careful to avoid potentially uncomfortable questions from foreign businesses and governments. But I suspect they’ll be able to manage the scope and keep things under control. The NSA probably pwned them years ago anyway.
Good stuff, and I hope we see more direct government involvement… although we really need a separate agency to handle these due to the conflicting missions of the NSA.
- Note: for those of you that follow these things, there is clear political maneuvering by the NSA here. They want to own cybersecurity, even though it conflicts with their intel mission. I’d prefer to see another agency hold the defensive reins, but until then I’m happy for any
Posted at Thursday 4th February 2010 5:45 pm
(7) Comments •
Trustwave just released their latest breach (and penetration testing) report, and it’s chock full of metrics goodness. Like the Verizon Data Breach Investigations Report, it’s a summary of information based on their responses to real breaches, with a second section on results from their penetration tests.
The breach section is the best part, and I already wrote about one lesson in a quick post on DLP. Here are a few more nuggets that stood out:
- It took an average of 156 days to detect a breach, and only 9% of victims detected the breach on their own – the rest were discovered by outside groups, including law enforcement and a credit card companies.
- Chip and PIN was called out for being more resilient against certain kinds of attacks, based on global analysis of breaches. Too bad we won’t get it here in the US anytime soon.
- One of the biggest sources of breaches was remote access applications for point of sale systems. Usually these are in place for third party support/management, and configured poorly.
- Memory parsing (scanning memory to pull sensitive information as it’s written to RAM) was a very common attack technique. I find this interesting, since certain versions of memory parsing attacks have virtualization implications… and thus cloud implications. This is something to keep an eye on, and an area I’m researching more.
- As I mentioned in the post 5 minutes ago, encryption was only used once for exfiltration.
Now some suggestions to the SpiderLabs guys:
- I realize it isn’t politically popular, but it would totally rock if you, Verizon, and other response teams started using a standard base of metrics. You can always add your own stuff on top, but that would really help us perform external analysis across a wider base of data points. If you’re interested, we’d totally be up for playing neutral third party and coordinating and publishing a recommended metrics base.
- The pen testing section would be a lot more interesting if you released the metrics, as opposed to a “top 10” list of issues found. We don’t know if number 1 was 10x more common than issue 10, or 100x more common.
It’s great that we now have another data source, and I consider all these reports mandatory reading, and far more interesting than surveys.
Posted at Thursday 4th February 2010 12:23 am
(1) Comments •
They both work a heck of a lot better if you use them ahead of time.
I just finished reading the Trustwave Global Security Report, which summarizes their findings from incident response and penetration tests during 2009.
In over 200 breach investigations, they only encountered one case where the bad guy encrypted the data during exfiltration. That’s right, only once. 1. The big uno.
This makes it highly likely that a network DLP solution would have detected, if not prevented, the other 199+ breaches.
Since I started covering DLP, one of the biggest criticisms has been that it can’t detect sensitive data if the bad guys encrypt it. That’s like telling a cop to skip the body armor since the bad guy can just shoot them someplace else.
Yes, we’ve seen cases where data was encrypted. I’ve been told that in the recent China hacks the outbound channel was encrypted. But based on the public numbers available, more often than not (in a big way) encryption isn’t used. This will probably change over time, but we also have other techniques to try to detect such other exfiltration methods.
Those of you currently using DLP also need to remember that if you are only using it to scan employee emails, it won’t really help much either. You need to use promiscuous mode, and scan all outbound TCP/IP to get full value. Also make sure you have it configured in true promiscuous mode, and aren’t locking it to specific ports and protocols. This might mean adding boxes, depending on which product you are using. Yes, I know I just used the words ‘promiscuous’ and ‘condom’ in a blog post, which will probably get us banned (hopefully our friends at the URL filtering companies will at least give me a warning).
I realize some of you will be thinking, “Oh, great, but now the bad guys know and they’ll start encrypting.” Probably, but that’s not a change they’ll make until their exfiltration attempts fail – no reason to change until then.
Posted at Wednesday 3rd February 2010 11:57 pm
(11) Comments •
By Adrian Lane
Deploying a WAF does not fall on the shoulders of database administrators. And it’s not really something one normally thinks of when itemizing database security tasks, but that is beginning to change. With database support for web and mobile applications, and PCI requirements overshadowing most commerce sites, WAF is an important consideration for guarding the symbiotic combination of web and database applications.
For this step in the series we are not fully fleshing out a process for WAF deployment, but picking those tasks where database administrative experience is relevant. For some of you this step will be entirely optional. Others will be working with security and application developers to refine rule sets based on query and parameter profiles, and verifying transactional consistency in cases where blocking is employed.
- Time to identify which databases support web applications.
- Time to gather application query & parameter profiles. Collect queries and activity. Analyze query and parameter distribution. Provide profiles to WAF team.
- Time to analyze pre-deployment tests for query/transaction failures.
- Optional: time to re-test new rules or code.
- Variable: Log file review. Find individual requests that are part of more complex transactions, so blocking activity produces side effects. Every time new web application code is released, r a WAF rule changed, it has the potential to cause issues with the database queries as well as the application function. Periodically review log files for anomalies.
- Time to document findings.
As you can see in the diagram, there is a sub-cycle to adjust the database (or provide information back to the WAF team) if the WAF breaks anything at the database level.
Posted at Wednesday 3rd February 2010 10:55 pm
(0) Comments •
By Adrian Lane
This is part 2 of the Database Security Fundamentals series. In part 1, I provided an overview. Here I will cover basic access and authorization issues.
First, the basics:
- Reset Passwords: Absolutely the first basic step is to change all default passwords. If I need to break into a database, the very first thing I am going to try is to log into a known account with a default password. Simple, fast, and it rarely gets noticed. You would be surprised (okay, maybe not surprised, but definitely disturbed) at how often the default SA password is left in place.
- Public & Demonstration Accounts: If you are surprised by default passwords, you would be downright shocked by how effectively a skilled attacker can leverage ‘Scott/Tiger’ and similar demonstration accounts to take control of a database. Relatively low levels of permissions can be parlayed into administrative functions, so lock out unused accounts or delete them entirely. Periodically verify that they have not reverted because of a re-install or account reset.
- Inventory Accounts: Inventory the accounts stored within the database. You should have reset critical DBA accounts, and locked out unneeded ones previously, but re-inventory to ensure you do not miss any. There are always service accounts and, with some database platforms, specific login credentials for add-on modules. Standard accounts created during database installation are commonly subject to exploit, providing access to data and database functions. Keep a list so you can compare over time.
- Password Strength: There is lively debate about how well strong passwords and password rotation improve security. Keystroke loggers and phishing attacks ignore these security measures. On the other hand, the fact that there are ways around these security precautions doesn’t mean they should be skipped, so my recommendation is to activate some password strength checks for all accounts. Having run penetration tests on databases, I can tell you from first-hand experience that weak passwords are pretty easy to guess; with a little time and an automated login program you can break most in a matter of hours. If I have a few live databases I can divide the password dictionary and run password checks in parallel, with a linear time savings. This is really easy to set up, and a basic implementation takes only a couple minutes. A couple more characters of (required) password length, and a requirement for numbers or special characters, both make guessing substantially more difficult.
- Authentication Methods: Choose domain authentication or database authentication – whichever works for you. I recommend domain authentication, but the point is to pick one and stick with it. Do not mix the two or later on, confusion and shifting responsibilities will create security gaps – cleaning up those messes is never fun. Do not rely on the underlying operating system for authentication, as that would sacrifice separation of duties, and OS compromise would automatically provide control over the data & database as well.
- Educate: Educate users on the basics of password selection and data security. Teach them how to pick a word or phase that is easy to remember, such as something they see visually each day, or perhaps something from childhood. Now show them simple substitutions of the letters with special characters and numbers. It makes the whole experience more interesting and less of a bureaucratic annoyance, and will reduce your support calls.
All these steps are easy to do. Everything I mentioned you should be able to tackle in an afternoon for one or two critical databases. Once you have accomplished them, the following are slightly more complicated, but offer greater security. Unfortunately this is where most DBAs stop, because they make administration more difficult.
- Group and Role Review: List out user permissions, roles, and groups to see who has access to what. Ideally review each account to verify users have just enough authorization to do their jobs. This is a great idea, which I always hated. First, it required a few recursive queries to build the list, and second it requires a huge list for non-trivial numbers of users. And actually using the list to remove ‘extraneous’ permissions gets you complaining users, such as receptionists who run reports on behalf of department administrators. Unfortunately, this whole process is time consuming and often unpleasant, but do it anyway. How rigorously you pursue excess rights is up to you, but you should at least identify glaring issues when normal users have access to admin functions. For those of you with the opportunity to work with application developers, this is your opportunity to advise them to keep permission schemes simple.
- Division of Administrative Duties: If you did not hate me for the previous suggestion, you probably will after this one: Divide up administrative tasks between different admins. Specifically, perform all platform maintenance under an account that cannot access the database and visa-versa. You need to separate the two and this is really not optional. For small shops it seems ridiculous to log out as one user and log back in as another, but negates the domino effect: when one account gets breached it does not mean every system must be considered compromised. If you are feeling really ambitious, or your firm employs multiple DBAs, relational database platforms provide advanced access control provisions to segregate database admin tasks such as archival and schema maintenance, improving security and fraud detection.
Posted at Wednesday 3rd February 2010 9:30 pm
(1) Comments •
As part of our support for the Open Web Application Security Project (OWASP), we participate in their survey program which runs quarterly polls on various application security issues. The idea is to survey a group of users to gain a better understanding of how they are managing or perceiving web application security.
We also occasionally run our own surveys to support research projects, such as Project Quant. All these results are released free to the public, and if we’re running the survey ourself we also release the raw anonymized data.
One of our ongoing problems is getting together a good group of qualified respondents. It’s the toughest part of running any survey. Although we post most of our surveys directly in the blog, we would also like to run some closed surveys so we can maintain consistency over time.
We are going to try putting together a survey board of people in end user organizations (we may also add a vendor list later) who are willing to participate in the occasional survey. There would be no marketing to this list, and no more than 1-2 short (10 minutes or less is our target) surveys per quarter. All responses will be kept completely anonymous (we’re trying to set it up to scrub the data as we collect it), and we will return the favor to the community by releasing the results and raw data wherever possible. We’re also working on other ideas to give back to participants – such as access to pre-release research, or maybe even free Q&A emails/calls if you need some advice on something.
No marketing. No spin. Free data.*
If you are interested please send an email to firstname.lastname@example.org and we’ll start building the list. We will never use any email addresses sent to this project for anything other than these occasional short surveys. Private data will never be shared with any outside organization.
We obviously need to hit a certain number of participants to make this meaningful, so please spread the word.
*Obviously we get some marketing for ourselves out of publishing data, but hopefully you don’t consider that evil or slimy.
Posted at Wednesday 3rd February 2010 7:06 pm
(0) Comments •
By Mike Rothman
I was at dinner over the weekend with a few buddies of mine, and one of my friends asked (again) which AV package is best for him. It seems a few of my friends know I do security stuff and inevitably that means when they do something stupid, I get the call.
This guy’s wife contracted one of the various Facebook viruses about a month ago and his machine still wasn’t working correctly. Right, it was slow and sluggish and just didn’t seem like it used to be. I delivered the bad news that he needed to rebuild the machine. But then we got talking about the attack vectors and how the bad guys do their evil bidding, and the entire group was spellbound.
Then it occurred to me: security folks could be the most interesting guy (gal) in the room at any given time. Like that dude in the Tecate beer ads. Being a geek – especially with security cred – could be used to pick up chicks and become the life of the party. Of course, the picking up chicks part isn’t too interesting to me, but I know lots of younger folks with skillz that are definitely interested in those mating rituals young people or divorcees partake in.
Basically, like every other kind of successful pick-up artist, it’s about telling compelling (and funny) stories. At least that’s what I’ve heard. You have to establish common ground and build upon that. Social media attacks are a good place to start. Everyone (that you’d be interested in anyway) uses some kind of social media, so start by telling stories of how spooky and dangerous it is. How they can be owned and their private data distributed all over China and Eastern Europe.
And of course, you ride in on your white horse to save the day with your l33t hacker skillz. Then you go for the kill by telling compelling stories about your day job. Like finding pr0n on the computer of the CEO (and having to discreetly clean it off). Or any of the million other stupid things people in your company do, which is actually funny if you weren’t the one with the pooper scooper expected to clean it up.
You don’t break confidences, but we are security people. We can tell anecdotes with the best of them. Like AndyITGuy, who shares a few humorous ones in this post. And those are probably the worst stories I’ve heard from Andy. But he’s a family guy and not going to tell his “good” stories on the blog.
Now to be clear (and to make sure I don’t sleep in the dog house for the rest of the week), I’m making this stuff up. I haven’t tried this approach, nor did I have any kind of rap when I was on the prowl like 17 years ago.
But maybe this will work for you, Mr. (or Ms.) L33t Young Hacker trying to find a partner or at least a warm place to hang for a night or three. Or you could go back to your spot on the wall and do like pretty much every generation of hacker that’s come before you. P!nk pretty much summed that up (video)…
Photo credit: “Tecate Ad” covered by psfk.com
Incite 4 U
I’m not one to really toot my own horn, but I’m constantly amazed at the amount and depth of the research we are publishing (and giving away) on the Securosis blog. Right now we have 3-4 different content series emerging (DB Quant, Pragmatic Data Security, Network Security Fundamentals, Low Hanging Fruit, etc.), resulting in 3-4 high quality posts hitting the blog every single day. You don’t get that from your big research shops with dozens of analysts.
Part of this is the model and a bigger part is focusing on doing the right thing. We believe the paywall should be on the endangered species list, and we are putting our money where our mouths are. If you do not read the full blog feed you are missing out.
Prognosis for PCI in 2010: Stagnation… – Just so I can get my periodic shout-out to Shimmy’s new shop, let me point to a post on his corporate blog (BTW, who the hell decided to name it security.exe?) about some comments relative to what we’ll see in PCI-land for 2010. Basically nothing, which is great news because we all know most of the world isn’t close to being PCI compliant. So let’s give them another year to get it done, no? Oh yeah, I heard the bad guys are taking a year off. No, not really. I’m sure all the 12 requirements are well positioned to deal with threats like APT and even new fangled web application attacks. Uh-huh. I understand the need to let the world catch up, but remember that PCI (and every other regulation) is a backward looking indicator. It does not reflect what kinds of attacks are currently being launched your way, much less what’s coming next. – MR
This Ain’t Your Daddy’s Cloud – One of the things that annoys me slightly when people start bringing up cloud security is the tendency to reinforce the same old security ideas and models we’ve been using for decades, as opposed to, you know, innovating. Scott Lupfer offers some clear, basic cloud security advice, but he sticks with classic security terminology, when it’s really the business audience that needs to be educated. Back when I wrote my short post on evaluating cloud risks I very purposely avoided using the CIA mantra so I could better provide context for the non-security audience, as opposed to educating them on our fundamentals. With the cloud transition we have an opportunity to bake in security in ways we missed with the web transition, but we can’t be educating people about CIA or Bell La Padula – RM
Cisco milking the MARS cash cow? – Larry Walsh posted a bit on Cisco MARS Partners getting the ‘roving eye’, with nervous VARs and customers shopping around (or at the very least kicking some tires) of other SIEM and Log Management solutions. I have had some conversations with customers in regard to switching platforms, which support Larry’s premise, but there is far more speculation about Cisco investing in another vendor to prop up their aging SIEM product. And more still about Cisco’s lack of commitment to security altogether. Are either of those rumors true? No idea, but either premise makes sense given the way this is being handled. Reducing investment and letting a product slowly die while milking an entrenched customer base is a proven cash flow strategy. CA did it successfully for better than a decade. This is one of the themes I am interested in following up on at RSA. – AL
How long before Oracle gets into Network Security? – So Oracle completes the Sun deal, and now they are in the hardware business for real. The point of this cnet article is that Oracle’s competitors are not SAP and Sybase anymore, but rather HP and IBM. It also means Oracle needs to keep buying hardware products to build up a broad and competitive offering. They’ve got the servers and the storage, but they have nothing for network or security, for that matter. HP has some (security devices in their ProCurve line), IBM has killed the network security stuff a few times (not really, but where is ISS again?), and Oracle needs some to really be considered a broad Big IT provider. So investment bankers, listen up – now you can go to Oracle City and try to sell that crappy UTM company everyone else passed on. But seriously, Oracle has no issue writing billion-dollar checks, and over the long run that’s a good thing for folks in the security industry. – MR
Monitor Servers too – Hopefully you’ve been following the Network Security Fundamentals series I’ve been posting over the past week. One of the points I made is to Monitor Everything and that really focused on the network layer. Yet server (and device) change detection is also a key data source for figuring out when you are attacked. This article from Mike Chapple on SearchMidMarketSecurity does a good job of highlighting the key points of host integrity monitoring (registration required). It covers topics such as selecting a product, developing a baseline, and tuning performance. The piece is pretty high level, but it’s a good overview. – MR
Brain Science – I’ve found the relatively recent “security is psychology” trend to be pretty amusing. Anyone with any background in physical security learned this lesson a long time ago. When I was running security for concerts and football games you learn really fast that the only advantage your crew of 250 has over 53,000 drunk fans is your ability to fool them into thinking you have the advantage. And the term “magical thinking” has roots in the scientific skeptical community whom uses it to describe our gullibility for things like worthless supplements (Airborne, ginko, and anything “homeopathic”). As I like to say, people are people, and don’t ever expect human behavior to change. Thus I enjoy reading posts like Lang’s compilation over at Financial Cryptography, but instead of fighting the masses I think we will be better served by understanding why people act the way they do, and adjusting our strategies to take advantage of it. – RM
That analysis doesn’t happen by itself… – Cutaway makes a good point in this post about the analysis of log files relative to incident response. He points out that without a well worn and practiced IR plan, you will be hosed. That key, and highlighted in the P-CSO relative to containing the damage when you are attacked. Cutaway points out that just to wade through firewall logs after an attack can take weeks without significant automation and an familiar process. And to be clear, you don’t have weeks when you are being attacked. – MR
It’s not the answer, it’s the discussion… – The cnet article Which is more secure, Mac or PC make me ask again: Why is this a question we are asking? The answer is totally irrelevant. First, because no one is going to choose a Mac over a PC because of their relative security. Second, even if it is more secure today, who knows what exploits will show up tomorrow? But I do love this conversation. I love it because we are actually discussing the merits of computer platform security, in the open, on a web site read by average computer users. That’s a good thing! And I love it because there are some great quotes from security practitioners and researchers. I especially enjoyed Chris Wysopal’s comment. Check it out! – AL
Posted at Wednesday 3rd February 2010 4:45 am
(1) Comments •
By Mike Rothman
As we continue on our journey through the fundamentals of network security, the idea of network monitoring must be integral to any discussion. Why? Because we don’t know where the next attack is coming, so we need to get better at compressing the window between successful attack and detection, which then drives remediation activities. It’s a concept I coined back at Security Incite in 2006 called React Faster, which Rich subsequently improved upon by advocating Reacting Faster and Better.
React Faster (and better)
I’ve written extensively on the concept of React Faster, so here’s a quick description I penned back in 2008 as part of an analysis of Security Management Platforms, which hits the nail on the head.
New attacks are happening at a fast and furious pace. It is a fool’s errand to spend time trying to anticipate where the issues are. REACT FASTER first acknowledges that all attacks cannot be stopped. Thus, focus remains on understanding typical traffic and application usage trends and monitoring for anomalous behavior, which could indicate an attack. By focusing on detecting attacks earlier and minimizing damage, security professionals both streamline their activities and improve their effectiveness.
Rich’s corollary made the point that it’s not enough to just react faster, but you need to have a plan for how to react:
Don’t just react – have a response plan with specific steps you don’t jump over until they’re complete. Take the most critical thing first, fix it, move to the next, and so on until you’re done. Evaluate, prioritize, contain, fix, and clean.
So monitoring done well compresses the time between compromise and detection, and also accelerates root cause analysis to determine what the response should involve.
Network Security Data Sources
It’s hard to argue with the concept of reacting faster and collecting data to facilitate that activity. But with an infinite amount of data to collect, where do we start? What do we collect? How much of it? For how long? All of these are reasonable questions that need answers as you construct your network monitoring strategy. The major data sources from your network security infrastructure include:
- Firewall: Every monitoring strategy needs to correspond to the most prevalent attack vectors, and that means from the outside in. Yes, the insider threat is real, but script kiddies are alive and well and that means we need to start by looking at our Internet-facing devices. First we pull log and activity information from our firewalls and UTM devices on the perimeter. We look for strange patterns, which usually indicate something is wrong. We want to keep this data long enough to ensure we have sufficient data in the event of a well-executed low and slow attack, which means months rather than days.
- IPS: The next layer in tends to be IPS, looking for patterns of traffic that indicate a known attack. We want the alerts first and foremost. But we also want to collect the raw IPS logs as well. Just because the IPS doesn’t think specific traffic is an attack doesn’t mean it isn’t. It could be a dreaded 0-day, so we want to pull all the data we can off this box as well, since the forensic analysis can pinpoint when attacks first surfaced and also provide guidance as to the extent of the compromise.
- Vulnerability scans: Are those devices vulnerable to a specific attack? Vulnerability scan data is one of the key inputs to SIEM/correlation products. The best way to reduce false positives is not to fire an alert if the target is not vulnerable. Thus we keep scan data on hand, and use it both for real-time analysis and also forensics. If an attack happens during a window of vulnerability (like while you debate the merits of a certain patch with the ops guys), you need to know that.
- Network Flow Data: I’ve always been a big fan of network flow analysis and continue to be mystified that market never took off, given the usefulness of understanding how traffic flows within and out of a network. All is not lost, since a number of security management products use flow data in their analyses and a few lower end management products use flow data as well. Each flow record is small, so there is no reason not to keep a lot of it. Again, we use this data to both pinpoint potential badness, and also replay attacks to understand how they spread within the organization.
- Device Change Logs: If your network devices get compromised, it’s pretty much game over. Traffic can be redirected, logging suppressed, and lots of other badness can result. So keep track of device configuration and more importantly when those changes happen – which helps isolate the root causes of breaches. Yes, if the logs are turned off, you lose visibility, which can itself indicate an issue. Through the wonders of SNMP, you should collect data from all your routers, switches, and other pipes.
- Content security: Now we can climb the stack a bit to pull information off the content security gateways, since a lot of attacks still show up via phishing emails and malware-laden web links. Again, we aren’t trying to pull this data in necessarily to stop an attack (hopefully the anti-spam box will figure out you aren’t interested in the little blue pill), but rather to gather more information about the attack vectors and how an attack proliferates through your environment. Reacting faster is about learning all we can about what is compromised and responding in the most efficient and effective manner.
Keeping things focused and pragmatic, you’d like to gather all this data all the time across all the networks. Of course, Uncle Reality comes to visit and understandably, collection of everything everywhere isn’t an option. So how do you prioritize? The objective is to use the data you already have. Most organizations have all of the devices listed above. So all the data sources exist, and should be prioritized based on importance to the business.
Yes, you need to understand which parts of the infrastructure are most important. I’m not a fan of trying to “value” assets, but a couple of categories can be used. Like “not critical,” “critical,” and “if this goes down I’m fired.” It doesn’t have to be a lot more complicated than that. Thus, start aggregating the data for the devices and segments where availability, performance, or security failures will get you tossed. Then you go to the critical devices, and so on. Right, you never get to the stuff that isn’t some form of critical.
Collection, Aggregation and Architecture
So where do we put all this data? You need some kind of aggregation platform. Most likely it’s a log management looking thing, at least initially. Those data sources listed above are basically log records and the maturity of log management platforms mean you can store a lot of data pretty efficiently.
Obviously collecting data doesn’t make it useful. Thus you can’t really discuss log aggregation without a discussion of correlation and analysis of that data. But that is a sticky topic and really warrants its own discussion in the Network Security Fundamentals series. Additionally, it’s now feasible to actually buy a log aggregation service (as opposed to building it yourself), so in future research I’ll also delve into the logic of outsourcing log aggregation. For the purposes of this post, let’s assume you are building your own log collection environment. It’s also worth mentioning that depending on the size of your organization (and collection requirements) there are lower cost and open source options for logging that work well.
In terms of architecture, you want to avoid a situation where management data overwhelms the application traffic on a network. To state the obvious, the closer to the devices you can collect, the less traffic you’ll have running all over the place. So you need to look at a traditional tiered approach. That means collector(s) in each location (you don’t want to be sending raw logs across lower speed WAN links) and then a series of aggregation points depending on the nature of your analysis.
Since most monitoring data gets used for forensic purposes as well, you can leave the bulk of the data at the aggregation points and only send normalized summary data upstream for reporting and analysis. To be clear, a sensor in every nook and cranny of your network will drive up costs, so exercise care to gather only as much data as you need, within the cost constraints of your environment.
As you look to devices for collection, one of the criteria to consider is compression and normalization of the data. For most compliance purposes, you’ll need to keep the raw logs and flows, but can achieve 15-20:1 compression on log data, as well as normalizing where appropriate to facilitate analysis. And speaking of analysis…
Full Packet Capture
We’ve beaten full packet capture into submission over the past few weeks. Rich just posted on the topic in more detail, but any monitoring strategy needs to factor in full network packet capture. To be clear, you don’t want to capture every packet that traverses your network. Rather the network traffic coming into or leaving the really important parts of your environment.
We believe the time is right for full packet capture for most larger organizations that need to be able to piece together an attack quickly after an compromise. At this point, doing any kind of real time analysis on a full packet stream isn’t realistic (at least not in a sizable organization), but this data is critical for forensics purposes. Stay tuned – this is a topic we’ll be digging much deeper into over the rest of the year.
Point of diminishing returns
As with everything in your environment, there can be too much of a good thing. So you need to avoid the point of diminishing returns, where more data becomes progressively less useful. Each organization will have its own threshold for pain in terms of collection, but keep an eye on a couple of metrics to indicate when enough is enough:
- Speed: When your collection systems start to get bogged down, it’s time to back off. During an incident, you need searching speed and large datasets can hinder that. “Backing off” can mean a lot of different things, but mostly it means reducing the amount of time you keep the data live in the system. So you can play around with the archiving windows to find the optimal amount of data to keep.
- Accuracy: Yes, when a collection system gets overwhelmed it starts to drop data. Vendors will dance on a thimble to insist this isn’t the case, but it is. So periodically making sure all the cross-tabs in your management reports actually add up is a good idea. Better you identify these gaps than your ops teams do. If they have to tell you your data and reports are crap, you can throw your credibility out the window.
- Storage overwhelm: When the EMC rep sends you a case of fancy champagne over the holidays, you may be keeping too much data. Keep in mind that collecting lots of data requires lots of storage and despite the storage commodity curve, it still can add up. So you may want to look at archiving and then eventually discarding data outside a reasonable analysis window. If you’ve been compromised for years, no amount of stored data will save your job.
Remember, this post dealing with the data you want to collect from the network, but that’s not the only stuff we should be monitoring. Not by a long shot, so we’ll be discussing collection at other layers of the computing stack: servers, databases, applications, etc. over time.
Next in the Network Security Fundamentals series I’ll tackle the “C” word of security management – correlation – which drives much of the analysis we do with all of this fancy data we collect.
Posted at Tuesday 2nd February 2010 12:22 am
(1) Comments •
When Mike was reviewing the latest Pragmatic Data Security post he nailed me on being too apologetic for telling people they need to spend money on data-security specific tools. (The line isn’t in the published post).
Just so you don’t think Mike treats me any nicer in private than he does in public, here’s what he said:
Don’t apologize for the fact that data discovery needs tools. It is what it is. They can be like almost everyone else and do nothing, or they can get some tools to do the job. Now helping to determine which tools they need (which you do later in the post) is a good thing. I just don’t like the apologetic tone.
As someone who is often a proponent for tools that aren’t in the typical security arsenal, I’ve found myself apologizing for telling people to spend money. Partially, it’s because it isn’t my money… and I think analysts all too often forget that real people have budget constraints. Partially it’s because certain users complain or look at me like I’m an idiot for recommending something like DLP.
I have a new answer next time someone asks me if there’s a free tool to replace whatever data security tool I recommend:
Did you build your own Linux box running
ipfw to protect your network, or did you buy a firewall?
The important part is that I only recommend these purchases when they will provide you with clear value in terms of improving your security over alternatives. Yep, this is going to stay a tough sell until some regulation or PCI-like standard requires them.
Thus I’m saying, here and now, that if you need to protect data you likely need DLP (the real thing, not merely a feature of some other product) and Database Activity Monitoring. I haven’t found any reasonable alternatives that provide the same value.
There. I said it. No more apologies – if you have the need, spend the money. Just make sure you really have the need, and the tool you are looking at really delivers the value, since not all solutions are created equal.
Posted at Monday 1st February 2010 5:54 pm
(3) Comments •
In the Discovery phase we figure where the heck our sensitive information is, how it’s being used, and how well it’s protected. If performed manually, or with too broad an approach, Discovery can be quite difficult and time consuming. In the pragmatic approach we stick with a very narrow scope and leverage automation for greater efficiency. A mid-sized organization can see immediate benefits in a matter of weeks to months, and usually finish a comprehensive review (including all endpoints) within a year or less.
Discover: The Process
Before we get into the process, be aware that your job will be infinitely harder if you don’t have a reasonably up to date directory infrastructure. If you can’t figure out your users, groups, and roles, it will be much harder to identify misuse of data or build enforcement policies. Take the time to clean up your directory before you start scanning and filtering for content. Also, the odds are very high that you will find something that requires disciplinary action. Make sure you have a process in place to handle policy violations, and work with HR and Legal before you start finding things that will get someone fired (trust me, those odds are pretty darn high).
You have a couple choices for where to start – depending on your goals, you can begin with applications/databases, storage repositories (including endpoints), or the network. If you are dealing with something like PCI, stored data is usually the best place to start, since avoiding unencrypted card numbers on storage is an explicit requirement. For HIPAA, you might want to start on the network since most of the violations in organizations I talk to relate to policy violations over email/web/FTP due to bad business processes. For each area, here’s how you do it:
- Storage and Endpoints: Unless you have a heck of a lot of bodies, you will need a Data Loss Prevention tool with content discovery capabilities (I mention a few alternatives in the Tools section, but DLP is your best choice). Build a policy based on the content definition you built in the first phase. Remember, stick to a single data/content type to start. Unless you are in a smaller organization and plan on scanning everything, you need to identify your initial target range – typically major repositories or endpoints grouped by business unit. Don’t pick something too broad or you might end up with too many results to do anything with. Also, you’ll need some sort of access to the server – either by installing an agent or through access to a file share. Once you get your first results, tune your policy as needed and start expanding your scope to scan more systems.
- Network: Again, a DLP tool is your friend here, although unlike with content discovery you have more options to leverage other tools for some sort of basic analysis. They won’t be nearly as effective, and I really suggest using the right tool for the job. Put your network tool in monitoring mode and build a policy to generate alerts using the same data definition we talked about when scanning storage. You might focus on just a few key channels to start – such as email, web, and FTP; with a narrow IP range/subnet if you are in a larger organization. This will give you a good idea of how your data is being used, identify some bad business process (like unencrypted FTP to a partner), and which users or departments are the worst abusers. Based on your initial results you’ll tune your policy as needed. Right now our goal is to figure out where we have problems – we will get to fixing them in a different phase.
- Applications & Databases: Your goal is to determine which applications and databases have sensitive data, and you have a few different approaches to choose from. This is the part of the process where a manual effort can be somewhat effective, although it’s not as comprehensive as using automated tools. Simply reach out to different business units, especially the application support and database management teams, to create an inventory. Don’t ask them which systems have sensitive data, ask them for an inventory of all systems. The odds are very high your data is stored in places you don’t expect, so to check these systems perform a flat file dump and scan the output with a pattern matching tool. If you have the budget, I suggest using a database discovery tool – preferably one with built in content discovery (there aren’t many on the market, as we’ll mention in the Tools section). Depending on the tool you use, it will either sniff the network for database connections and then identify those systems, or scan based on IP ranges. If the tool includes content discovery, you’ll usually give it some level of administrative access to scan the internal database structures.
I just presented a lot of options, but remember we are taking the pragmatic approach. I don’t expect you to try all this at once – pick one area, with a narrow scope, knowing you will expand later. Focus on wherever you think you might have the greatest initial impact, or where you have known problems. I’m not an idealist – some of this is hard work and takes time, but it isn’t an endless process and you will have a positive impact.
We aren’t necessarily done once we figure out where the data is – for approved repositories, I really recommend you also re-check their security. Run at least a basic vulnerability scan, and for bigger repositories I recommend a focused penetration test. (Of course, if you already know it’s insecure you probably don’t need to beat the dead horse with another check). Later, in the Secure phase, we’ll need to lock down the approved repositories so it’s important to know which security holes to plug.
Unlike the Define phase, here we have a plethora of options. I’ll break this into two parts: recommended tools that are best for the job, and ancillary tools in case you don’t have a budget for anything new. Since we’re focused on the process in this series, I’ll skip definitions and descriptions of the technologies, most of which you can find in our Research Library
- Data Loss Prevention (DLP): This is the best tool for storage, network, and endpoint discovery. Nothing else is nearly as effective.
- Database Discovery: While there are only a few tools on the market, they are extremely helpful for finding all the unexpected databases that tend to be floating around most organizations. Some offer content discovery, but it’s usually limited to regular expressions/keywords (which is often totally fine for looking within a database).
- Database Activity Monitoring (DAM): A couple of the tools include content discovery (some also include database discovery). I only recommend DAM in the discover phase if you also intend on using it later for database monitoring – otherwise it’s not the right investment.
- IDS/IPS/Deep Packet Inspection: There are a bunch of different deep packet inspection network tools – including UTM, Web Application Firewalls, and web gateways – that now include basic regular expression pattern matching for “poor man’s” DLP functionality. They only help with data that fits a pattern, they don’t include any workflow, and they usually have a ton of false positives. If the tool can’t crack open file attachments/transfers it probably won’t be very helpful.
- Electronic Discovery, Search, and Data Classification: Most of these tools perform some level of pattern matching or indexing that can help with discovery. They tend to have much higher false positive rates than DLP (and usually cost more if you’re buying new), but if you already have one and budgets are tight they can help.
- Email Security Gateways: Most of the email security gateways on the market can scan for content, but they are obviously limited to only email, and aren’t necessarily well suited to the discovery process.
- FOSS Discovery Tools: There are a couple of free/open source content discovery tools, mostly projects from higher education institutions that built their own tools to weed out improper use of Social Security numbers due to a regulatory change a few years back.
Discover: Case Study
Frank from Billy Bob’s Bait Shop and Sushi Outlet decides to use a DLP tool to help figure out where any unencrypted credit card numbers might be stored. He decides to go with a full suite DLP tool since he knows he needs to scan his network, storage, servers in the retail outlets, and employee systems.
Before turning on the tool, he contacts Legal and HR to set up a process in case they find any employees illegally using these numbers, as opposed to the accidental or business-process leaks he also expects to manage. Although his directory servers are a little messy due to all the short-term employees endemic to retail operations, he’s confident his core Active Directory server is relatively up to date, especially where systems/servers are concerned.
Since he’s using a DLP tool, he develops a three-tier policy to base his discovery scans on:
- Using the one database with stored unencrypted numbers, he creates a database fingerprinting policy to alert on exact matches from that database (his DLP tool uses hashes, not the original values, so it isn’t creating a new security exposure). These are critical alerts.
- His next policy uses database fingerprints of all customer names from the customer database, combined with a regular expression for generic credit card numbers. If a customer name appears with something that matches a credit card number (based on the regex pattern) it generates a medium alert.
- His lowest priority policy uses the default “PCI” category built into his DLP tool, which is predominantly basic pattern matching.
He breaks his project down into three phases, to run during overlapping periods:
- Using those three policies, he turns on network monitoring for email, web, and FTP.
- He begins scanning his storage repositories, starting in the data center. Once he finishes those, he will expand the scans into systems in the retail outlets. He expects his data center scan to go relatively quickly, but is planning on 6-12 months to cover the retail outlets.
- He is testing endpoint discovery in the lab, but since their workstation management is a bit messy he isn’t planning on trying to install agents and beginning scans until the second year of the project.
It took Frank about two months to coordinate with other business/IT units before starting the project. Installing DLP on the network only took a few hours because everything ran through one main gateway, and he wasn’t worried about installing any proxy/blocking technology.
Frank immediately saw network results, and found one serious business process problem where unencrypted numbers were included in files being FTPed to a business partner. The rest of his incidents involved individual accidents, and for the most part they weren’t losing credit card numbers over the monitored channels.
The content discovery portion took a bit longer since there wasn’t a consistent administrative account he could use to access and scan all the servers. Even though they are a relatively small operation, it took about 2 months of full time scanning to get through the data center due to all the manual coordination involved. They found a large number of old spreadsheets with credit card numbers in various directories, and a few in flat files – especially database dumps from development.
The retail outlets actually took less time than he expected. Most of the servers, except at the largest regional locations, were remotely managed and well inventoried. He found that 20% of them were running on an older credit card transaction system that stored unencrypted credit card numbers.
Remember, this is a 1,000 person organization… if you work someplace with five or ten times the employees and infrastructure, your process will take longer. Don’t assume it will take five or ten times longer, though – it all depends on scope, infrastructure, and a variety of other factors.
Posted at Monday 1st February 2010 5:39 pm
(0) Comments •