Login  |  Register  |  Contact
Friday, April 16, 2010

ESF: Endpoint Incident Response

By Mike Rothman

Nowadays, the endpoint is the path of least resistance for the bad guys to get a foothold in your organization. Which means we have to have a structured plan and process for dealing with endpoint compromises. The high level process we’ll lay out here focuses on: confirming the attack, containing the damage, and then performing a post-mortem.

To be clear, incident response and forensics is a very specialized discipline, and hairy issues are best left to the experts. But that being said, there are things you as a security professional need to understand, to ensure the forensics guys can do their jobs.

Confirming the attack

There are lots of ways your spidey-sense should start tingling that something is amiss. Maybe it’s the user calling up and saying their machine is slow. Maybe it’s your SIEM detecting some weird log records. It could be your configuration management system reverting inexplicable changes or noting the presence of strange executables. Or perhaps your network flow analysis shows some reconnaissance activities from the device. A big part of the security management process is about being able to fire alerts when something suspicious is happening.

Then we make like bloodhounds and investigate the issue. We’ve got to find the machine and isolate it. Yes, that usually means interrupting the user and ‘inviting’ them to grab a cup of coffee, while you figure out what a mess they’ve made. The first step is likely to do a scan and compare with your standard builds (you remember the standard build, right?). Basically we look for obvious changes that cause issues.

If it’s not an obvious issue (think tons of pop-ups), then you’ve got to go deeper. This usually requires forensics tools, including stuff to analyze disks and memory to look for corruption or other compromise. There are lots of good tools – both open source and commercial – available for your forensics toolkit.

We do recommend you take a course in simple forensics as you get started, for a simple reason. You can really screw up an investigation by doing something wrong, in the wrong order, or using the wrong tools. If it’s truly an attack, your organization may want to prosecute at some point, and that means you have to maintain chain of custody on any evidence you gather. You should consult a forensics expert and probably your general counsel to identify the stuff you need to gather from a prosecution standpoint.

Containing the damage

“Houston, we have a problem…” Yup, your fears were justified and an endpoint or 200 have been compromised – so what to do? First off, you should inherently know what to do because you have a documented incident response plan, and you’ve practiced the process countless times, and your team springs into action without prompting, right? OK, this is the real world, so hopefully you have a plan and your team doesn’t look at you like an alien when you take it to DEFCON 4.

In all seriousness, you need to have an incident response plan. And you need to practice it. The time to figure out your plan stinks is not while a worm is proliferating through your innards at an alarming rate. We aren’t going to go into depth on that process (we’ll be doing a series later this year on incident response), but the general process is as follows:

  • Quarantine – Bad stuff doesn’t spread through osmosis – you need a network in place to allow malware to find new targets and spread like wildfire, so first isolate the compromised device. Yes, user grumpiness may follow, but whatever. They got pwned, so they can grab a coffee while you figure out how to contain the damage.
  • Assess – How bad is it? How far has it spread? What are your options to fix it? The next step in the process is to understand what you are dealing with. When you confirm the attack, you probably have a pretty good idea what’s going on. But now you have to figure out what the best option(s) is to fix it.
  • Workaround – Are there settings that can be deployed on the perimeter or at the network layer that can provide a short term fix? Maybe it’s blocking communication to the botnet’s command and control. Or possibly blocking inbound traffic on a certain port or some specific non-standard protocol that is the issue. Obviously be wary of the ripple effect of any workaround (what else does it break?), but allowing folks to get back to work quickly is paramount, so long as you can avoid the risk of further damage.
  • Remediate – Is it a matter of changing a setting or uninstalling some bad stuff? That would be optimistic, eh? Now is when you figure out how to fix the issue, and increasingly these days re-imaging is the best answer. Today’s malware hides so well it’s almost impossible to entirely inoculate a compromised device, and impossible to know you got it all. Which means part of your incident response plan should be a leveraged way to re-image machines.

At some point you have to figure out if this is an incident you can handle yourself, or if you need to bring in the artillery, in the form of forensics experts or law enforcement. Your IR plan needs to be identify scenarios which call for experts, and which call for the law. You don’t want that to be a judgement call in the heat of battle. So define the scenarios, establish the contacts (at both forensics firms and law enforcement), and be ready. That’s what IR is all about.

Post mortem

Once most folks get done cleaning up an incident, they think the job is done. Well, not so much. The reality is that the job has just begun, since you need to figure out what happened and make sure it doesn’t happen again. It’s OK to get nailed by something you haven’t seen before (fool me once, shame on you). It’s not OK to get nailed by the same thing again (fool me twice, shame on me). So you’ve got to schedule a post-mortem.

The post-mortem is not about laying blame – it’s about making sure it doesn’t happen again. So you need someone to candidly and in great detail understand what happened and where the existing defenses failed. Again, it is what it is and it’s critical that the organization can accept failures and move on. But not before you figure out whether process, controls, product, or people need to change.

By the way, it’s very hard to fight human nature and build a culture where failure is OK and post mortems are learning experiences, as opposed to a venue for everyone to cover their respective asses. But we don’t believe you can be successful at security without a strong incident response plan and that requires unemotional post-mortem analysis.

And with that, we come to the conclusion of the Endpoint Security Fundamentals series. We’ll be packaging it up in white paper form over the next week, and it will then be posted to the research library. As always, if there are things we missed or ideas you disagree with, please continue to contribute. Securosis research is an ongoing process, so things will change and we’ll update the documents as required.


Other posts in the Endpoint Security Fundamentals Series

  1. Introduction
  2. Prioritize: Finding the Leaky Buckets
  3. Triage: Fixing the Leaky Buckets
  4. Controls: Update and Patch
  5. Controls: Secure Configurations
  6. Controls: Anti-Malware
  7. Controls: Firewall, HIPS, and Device Control
  8. Controls: Full Disk Encryption
  9. Building the Endpoint Security Program
  10. Endpoint Compliance Reporting

—Mike Rothman

Public Goods

By Adrian Lane

Chris Pepper tweeted a very cool post on Why Content is a Public Good. The author, Milena Popova, provides an economist’s perspective on market forces and digital goods. Her premise is that in economic terms, many types of electronic content are “public goods” – that being a technical term for objects with infinite supply and no good way to control consumption. She makes the economic concepts of ‘rival’ and ‘excludable’ very easy to understand, and by breaking it down into rudimentary components, makes a compelling argument that content is a public good:

It means that old business models based on content being a club good simply don’t work. It means we have to rethink our relationship with content – as creators, as distributors and as consumers. It means that there are a lot of giants in the content distribution industry whose livelihoods (profit margins) are being pulled out from under them faster than they can say “illegal downloads”, and they are fighting it. Of course they’re fighting it. They’ve had an incredibly profitable business model for about a century and suddenly they don’t. Let’s face it, human beings don’t like change at the best of times, and we sure as hell don’t like it when it means less cash in our pockets.

I have written many posts on how economics affect DRM, RIAA, and ‘piracy’; and on the difference between actual security and security marketing, so I won’t rehash those subjects here; but note the common theme is that a busted business model is the root of the problem. Right now I want to stay away from some of the negativity of those posts, and instead focus on the economic drivers. Ms. Popova does a much better job than I of isolating the underlying forces, and discusses the factors in a way that helps us begin to visualize possible solutions.

A lot of people have a hard time with the concept of free and how you can actually make money in a world with so much free stuff. In a capitalist society we all have trouble with this. I talk to people in IT who still don’t think Linux and Java are viable technologies, and no one could make money with those products. But the availability of free stuff requires you to think a little differently about value – fewer people will pay money for the everyday and ordinary stuff because they don’t have to, but they will pay for things they perceive as special. In fact, I don’t think I fully grasped the concept and implications until I started working at Securosis. We are a research company that gives away most of our products for free, but charges for services and engagements.

One area I where was at odds with Popova was on the concept of “price discrimination”. From my perspective this looks more like the market being able to set the price, but do so far more efficiently: person to person, item by item, and adjusted over time. This is a very cool concept if you think about something like television: If you pay channel by channel, how many channels would you pay for? You have 400 or so, but I bet when it came to spending money, very few would get your hard-earned dollars. The NFL knows this, as football not only drives huge ad revenue, but single-handedly the bulk of hi-def television sales and additional add-on packages. If it was not for bundling into programming packages, many (most?) other channels would not be able to survive.

All in all, one of the better posts I have seen on the problems of dealing with consumer media.

—Adrian Lane

Friday Summary: April 16, 2010

By Adrian Lane

I am sitting here staring at power supplies and empty cases. Cleaning out the garage and closets, looking at the remnants from my PC building days. I used to love going out to select new motherboard and chipset combinations, hand-selecting each component to build just the right database server or video game machine. Over the years one sad acknowledgement needed to be made: after a year or so, the only pieces worth a nickel were the power supply and the case. Sad, but you spend $1,500.00 and after a few months the freaking box that housed the parts was the only remaining item of value.

I was thinking about this during some of our recent meetings with clients and would-be clients. Rich, Mike, and I are periodically approached by investors to review portfolio companies. We look both at their technology and market opportunities, and determine whether we feel the product is hitting the mark, and reaching buyers with the right product and message. We are engaged in response to either mis-aligned vision between investors and company operators (shocker, I know), or more commonly to give the investors some understanding of whether the company is worth salvaging through additional investment and a change of focus. Sometimes the company has followed market trends to preserve value, but quite a few turn out to be just a box of old parts. If this is not a direct consequence of Moore’s law, it should be.

We see cases where technologies were obsolete before the hit they market. In a handful of instances, we do find one or two worth salvaging, but not for the reasons they thought. In those cases the engineering staff was smart enough, or lucky enough, to build a deployment model or architecture that is currently relevant. The core technology? Forget it. Pitch it in the dust bin and take the write-off because that $20M investment is spent. But the ‘box’ was valuable enough to salvage, invest in, or sell. It’s ironic, and it goes to show how tough technology start-ups are to get right, and that luck is often better than planning.

On to the Summary:

Webcasts, Podcasts, Outside Writing, and Conferences

Favorite Securosis Posts

Other Securosis Posts

Favorite Outside Posts

Project Quant Posts

Top News and Posts

Blog Comment of the Week

Remember, for every comment selected, Securosis makes a $25 donation to Hackers for Charity. This week’s best comment goes to LonerVamp, in response to Security Incite: Just Think.

Nice opening post, especially with the kicker at the end that you’re actually writing it on a plane! :) I definitely find myself purposely unplugging at times (even if I’m still playing with something electronic) and protecting my private time when reasonable. I wonder if this ultimately has to do with the typically American concept of super-efficiency…milking every waking moment with something productive…at the expense of the great, relaxing, leisure things in life. I could listen to a podcast during this normally quiet hour in my day! And so on…
@Porky Risk…Pig: It doesn’t help that every security professional shotguns everyone else’s measurements…and with good reason! At some point, I think we’ll have to accept that every company CEO is different, and it takes a person to distill the necessary information down for her consumption. No tool will ever be enough, just like no tool ever determines if a product will be successful.
@My dad can beat up your dad: Ugh…I have some pretty strong opinions about cyberbullying and home internet monitoring…but I tend to shut up a bit because I don’t have kids and I’ll likely rub parents wrong (not that my humor isn’t dark enough to do that anyway!). While social networking didn’t exist when I was in school, I certainly have to give a lot of caution to taking away a child’s privacy. Protection is one thing, but being overly controlling and in their business is often a recipe for a lifetime of resentment…especially in their most carefree years of life where they have hours upon hours of free time without the burden of responsibilities or death anywhere on the horizon. I think it is simply most important to keep honest communication open (both ways) and make sure actions are taken when something awry happens. If someone isn’t your friend, why are you “friends” on Facebook with them? Masochistic? Sinister plans?
With rare, very saddening exceptions, a vast majority of kids will survive despite the rigors of childhood and school and self-discovery outside protective umbrellas. (I do make one exception. If there is monitoring and intrusion of privacy, they should never find out and you should never make them suspect it. It should be a silent angel watching over. But once you flaunt it or they find out, that is when the resentment and deeper hiding will happen…imo. Again, this is from a one-sided perspective and does not apply to all personalities. :) )

—Adrian Lane

Thursday, April 15, 2010

ESF: Endpoint Compliance Reporting

By Mike Rothman

You didn’t think we could get through an 11-part series about anything without discussing compliance, did you? No matter what we do from a security context – whatever the catalyst, budget center, or end goal – we need to substantiate implemented controls. We can grind out teeth and curse the gods all we want, but security investments are contingent on some kind of compliance driver.

So we need to focus on documentation and reporting for everything we do. Further, we discussed operational efficiencies in the security programs post, and the only way to get any kind of leverage from an endpoint security program is to automate the reporting.

Document what?

First we need to understand what needs to be documented from an endpoint perspective for the regulations/standards/guidance you deal with. You must be able to document the process/procedures of your endpoint program, as well as the data substantiating the controls. Process either exists or it doesn’t, so that documentation should be straightforward to produce.

On the other hand, figuring out which data types corroborate which controls and apply to which standards requires a big matrix to handle all the permutations and combinations. There are two ways to do this:

  1. Buy it – Many of the IT GRC tools out there (and don’t get us started on the value of IT GRC tools) help to manage the workflow of a compliance program. The key capability here is the built-in map, which connects technical controls to regulations, ostensibly so you don’t have to. But these tools cost money and provide limited value.
  2. Build it – The other option involves going through your regulations and figuring out relevant controls. This is about as fun as a root canal, but it has to be done. More likely, you can start with something your buddies, auditor, or vendors have. Vendors have excellent motivation to figure out how their products – representing a variety of security controls – map to the various regulations their customers need to address. The data is out there – you just have to assemble it.

Actually, there is a third option: to just license the content from an organization like the Unified Compliance Framework folks. They license a big-ass spreadsheet with the map, and their pricing is rather economical.

Packaging

Now that you know what you need to report on, how do you do it? This question is bigger than your endpoint security program, and applies to every security program you run. We recommend you think architecturally. You’ve got certain domains of controls – think network, endpoint, data center, applications, etc. You want to put together a few things for each domain to make the auditor happy:

  • Control list – Go back to your control maps and make a big list of all the controls required for the auditor’s checklist (they all have checklists). Make sure the auditor buys into your list, and that you aren’t missing anything.
  • Logical architecture – Show graphically (a picture is worth a thousand words) how your controls are implemented. Right – every control on the list should appear on the logical architecture.
  • Data – You didn’t really think the auditor would just believe your architecture diagram, did you? Now you need the data from each of your systems (endpoint suite, configuration management, full disk encryption, etc.) to show that you’ve actually implemented the controls. Your vendor likely has a pre-built report for your regulation, so you shouldn’t have to do a lot of manual report generation.

To be clear, one of the value propositions of IT GRC and other compliance automation products like log management/SIEM is to aggregate all this information (not just from the endpoint program, but from all your programs) and spit out an integrated report. For the most part, with a bit of angst in deployment, these tools can help reduce the burden of preparing for frequent audits across multiple regulations for global enterprises. The question to answer is whether the tool can pay for itself in terms of saved time and effort – is the ROI sufficient?

Dealing with deficiencies

One other thing to consider is the reality of an audit pointing out a specific deficiency in your endpoint security program. The auditor/assessor is there to find problems, and likely they will. But that doesn’t mean the auditor is right.

Yes, we said it. Sometimes auditors take liberties and/or subjectively decide how to interpret a specific regulation. If there is a specific reason you decided to either bypass a control – or more likely, implement a compensating control – make your case.

In the event (however unlikely) there is a legitimate deficiency, you need to fix it. Welcome, Captain Obvious! During the next audit, first go through the list of previous deficiencies and how you’ve remediated them. Make a big deal of addressing the deficiencies, which will get the audit off on the right foot.

What’s next? We’ll cap off the Endpoint Security Fundamentals series by talking about incident response. Stay tuned for the exciting conclusion. ;-)


Other posts in the Endpoint Security Fundamentals Series

—Mike Rothman

Wednesday, April 14, 2010

ESF: Building the Endpoint Security Program

By Mike Rothman

Over the previous 8 posts in this Endpoint Security Fundamentals series, we’ve looked at the problem from the standpoint of what to do right awat (Prioritize and Triage) and the Controls (update software and patch, secure configuration, anti-malware, firewall, HIPS and device control, and full disk encryption). But every experienced security professional knows a set of widgets doesn’t make a repeatable process, and it’s really the process and the people that makes the endpoints secure.

So let’s examine how we take these disparate controls and make them into a program.

Managing Expectations

The central piece of any security program is making sure you understand what you are committing to and over-communicating your progress. This requires a ton of meetings before, during, and after the process to keep everyone on board. More importantly, the security team needs a standard process for communicating status, surfacing issues, and ensuring there are no surprises in task completion or results.

The old adage about telling them what you are going to do, doing it, and then telling them what you did, is absolutely the right way to handle communications. Forget the ending at your own peril.

Defining success

The next key aspect of the program is specifying a definition for success. Start with the key drivers for protecting the endpoints anyway. Is it to stop the proliferation of malware? To train users? To protect sensitive data on mobile devices? To improve operational efficiency? If you are going to spend time and money, or to allocate resources you need at least one clear driver / justification.

Then use those drivers to define metrics, and operationalize your process based on them. Yes, things like AV update efficiency and percentage of mobile devices encrypted are uninteresting, but you can trend off those metrics. You also can set expectations at the front end of the process about acceptable tolerances for each one.

Think about the relevant incident metrics. How many incidents resulted from malware? User error? Endpoint devices? These numbers have impact – whether good or bad. And ultimately it’s what the senior folks worry about. Operational efficiency is one thing – incidents are another.

These metrics become your dashboard when you are communicating to the muckety-mucks. And remember to use pie charts. We hear CFO-types like pie charts. Yes, I’m kidding.

User training

Training is the third rail of security, and needs discussion. We are fans of training. But not crappy, check-the-box-to-make-the-auditor-happy training. Think more like phishing internal users and, using other social engineering tactics to show employees how exposed they are. Good training is about user engagement. Unfortunately most security awareness training sucks.

Keep the goals in mind. The end user is the first line of defense (and for mobile professionals, unfortunately also the last) so we want to make sure they understand what an attack looks like and what to do if they think they might have a problem. They don’t have to develop kung fu, they just need to understand when they’ve gotten kicked in the head. For more information and ideas, check out Rich’s FireStarter from Monday.

Operational efficiencies

Certainly one of the key ways to justify the investment in any kind of program is via operational efficiencies, and in the security business that means automation whenever and wherever possible. So just think about the controls we discussed through this series, and how to automate them. Here’s a brief list:

  • Update and Patch, Secure Configurations – Tools to automate configuration management kill these two birds with one stone. You set a policy, and can thus both enforce standard configurations and keep key software updated.
  • Anti-malware, FW/HIPS – As part of the endpoint suites, enforcing policies on updates, software distribution and policy settings are fairly trivial. Here is the leverage (and the main justification) for consolidating vendors on the endpoint – just beware folks who promise integration, but fail to deliver useful synergy.
  • Device control, full disk encryption, application white listing – These technologies are not as integrated into the endpoint suites at this point, but as the technologies mature, markets consolidate, and vendors actually get out of their own way and integrate the stuff they buy, this will get better.

Ultimately, operational efficiencies are all about integrating management of the various technologies used to protect the endpoint.

Feedback loops

The other key aspect of the program is making sure it adapts to the dynamic nature of the attack space. Here are a few things you should be doing to keep the endpoint program current:

  • Test controls – We are big fans of hacking yourself, which means using hacking tools to test your own defenses. Check out tools like Metasploit and the commercial offerings, and send phishing emails to your employees trying to get them to visit fake sites – which presumably would pwn them. This is a critical aspect of the security program.
  • Endpoint assessment – Figure out to what degree your endpoints are vulnerable, usually by scanning devices on connect with a NAC-type product, or with a scanner. Look for patterns to identify faulty configuration, ineffective patching, or other gaps in the endpoint defenses.
  • Configuration updates – A couple times a year new guidance emerges (from CIS and NIST, etc.) recommending changes to standard builds. Evaluating those changes, and figuring out whether and how the builds should change, helps to ensure the endpoint protection is always adapted to current attacks.
  • User feedback – Finally, you need to pay attention to the squeaky wheels in your organization. Well, not entirely, but you do have to factor in whether they are complaining about draconian usage policies – and more importantly whether some the controls are impeding their ability to do their jobs. That’s what we are trying to avoid.

The feedback loops will indicate when it’s time to revisit the controls, perhaps changing standard builds or considering new controls or tools. Keep in mind that without process and failsafes to keep the process relevant, all the technology in the world can’t help.

We’ll wrap up the series in the next two posts by discussing the compliance reporting requirements of the endpoint security program, and then endpoint-centric incident response.


Other posts in the Endpoint Security Fundamentals Series

—Mike Rothman

Incite 4/14/2010: Just Think

By Mike Rothman

As numb as we are to most advertising (since we are hit with thousands of advertising exposures every day), sometimes an ad campaign is memorable and really resonates. No, seeing Danica Patrick on a massage table doesn’t qualify. But Apple’s Think Different campaign really did. At that point, Apple was positioning to the counter-culture, looking for folks who didn’t want to conform. Those who had their own opinions, but needed a way to set them loose on the world.

Now that is definitely thinking different... Of course, we all want to think we are more than just cogs in the big machine and that we do matter. So the campaign resonated.

But nowadays I’m not so worried about thinking differently, but just thinking at all. You see, we live in a world of interruption and multitasking. There is nowhere to hide any more, not even at 35,000 feet.

Flying used to be my respite. 2-5 hours of solitude. You know, put in the ear buds, crank up the iPod, and tune out. Maybe I’d catch up on some writing or reading. Or even at the risk of major guilt, I’d get some mental floss (I’m plowing through the Daniel Silva series now) and crank through some fiction on the flight. Or Lord help me, sometimes I’d just sit and think. An indulgence I don’t partake in nearly enough during my standard routine.

Yet through the wonders of onboard WiFi, you can check email, surf the Web, tweet to your friends (yo, dog, I’m at 30K feet - and you are not!) or just waste time. All for $9.95 per flight. What a bargain.

And if you absolutely, positively need to send that email somewhere over Topeka, then the $9.95 is money well spent. Yet in reality, I suspect absolutely, positively means when you get to your destination.

What you don’t see is the opportunity cost of that $9.95. Not sure you can put a value on spending 3 hours battling spies or catching up on some journaling or even revisiting your plans for world domination. On my way back from RSA, I did use the on-board WiFi and to be honest it felt like a piece of me died. I was pretty productive, but I didn’t think, and that upset me. The last bastion of solitude was gone, but certainly not forgotten.

So yes, I’m writing this from 34,701 feet somewhere above New Mexico. But my battery is about dead, and that means it’s time to indulge. There are worlds to dominate, windmills to chase, strategies to develop, and I can’t do that online. Now quiet down, I need to think…

– Mike.

PS: Good luck to AndyITGuy, who is leaving his ATL digs to head to Cincy. Hopefully he’ll keep writing and plug into the security community in Southern Ohio.

Photo credits: “think___different” originally uploaded by nilson


Incite 4 U

  1. Porky, Risk Management, Pig – Let’s all welcome Jack Jones back to bloggy land, and he restarts with a doozy – basically saying risk management tools are like putting lipstick on a pig. Being a vegetarian and thinking about actually putting lipstick on a pig, I can only think of the truism from Jules in Pulp Fiction, “we’d have to be talkin’ about one charming motherf***ing pig.” But I’ll summarize Jack’s point more succinctly. Garbage In = Garbage Out. And even worse, if the analysis and the outcomes and the quantification are lacking, then it’s worse than garbage out: it’s sewage out. But senior management wants a number when they ask about risk, and the weak security folks insist on giving it to them, even though it’s pretty much arbitrary. Off soapbox now. – MR

  2. Craponomics – Repeat after me – it’s all about the economics. (I’m starting to wish I took one of those econ classes in school). According to the New York Times, lenders sort of ignore many of the signs of ID theft because they’d rather have the business. The tighter the fraud controls, the fewer people (legitimate and criminal) they can lend to, and the lower the potential profits. It’s in their interest to tolerate a certain level of fraud, even if that hurts ID theft victims. Remember, the lenders are out to protect themselves, not you. Can you say moral hazard? – RM

  3. Partly Paranoid, with a chance of PR – From what I hear, Google is now paranoid about security. Call me a cynic, but when someone trashes your defenses, is your response to use more web-based computing products and services, like Google Chrome? I am sure they are thinking they’ll modernize their defenses with Chrome, and all those old threat vectors will be magically corrected. You know, like XSS and injection attacks. I am thinking, “Someone broke into my company, now Chrome’s source code is suspect until I can prove attackers did not gain access to the source code control system.” Give Google credit for disclosure, and odds are they will be more secure with Chrome, but that was just a stepping stone in the process. I am far more interested in the steps taken to provide redundant security measures and perhaps some employee education on anti-phishing and security. Something that helps after an employee’s browser is compromised. There will always be another browser hack, so don’t tell me the answer is Chrome. Blah. – AL

  4. Yes, you are an addict… – It was funny to read Chris Nickerson’s post on FUDSEC about being a security addict, especially since that’s the entire premise of the Pragmatic CSO. But we look at the problem from different perspectives. Chris is right in pointing out that although we security folks tend to be powerless, that doesn’t mean we are helpless. It’s an important nuance. Personally I found his 12 steps lacking, especially compared to mine. But he also was working within the context of one blog post, and I wrote a book. All kidding aside, there are things we can control and things we can’t. Security is a hard job and you have to have the right mindset to survive. Whether you subscribe to Chris’ 12-step philosophy or mine or none at all, realizing that you do what you can will make all the difference to your happiness in security. – MR

  5. Accidental Developer – Marisa Fagan of Errata Security conducted a useful survey a few months back on security and software development. When I was taking the survey, the question “What Software Development Methodology do you follow” had me thinking “I don’t”. Sure enough, “Ad-hoc” was the most popular answer. Does this surprise you? I think the popularity of that choice is a good illustration of how early we are in the evolution of software security. I have never followed anyone’s secure development cycle, as I haven’t yet seen a framework I was interested in until recently. This will change with the work coming out of Microsoft and other firms as templates for good development practices. I think that with the industry settling into the – dare I say it – “trough of disillusionment” with Agile, it’s going to take a while before Agile is sufficiently developed to embrace widely accepted principles for secure code development. What cracked me up when taking the survey was, looking for the “Ad-Hoc” check box, I saw “Securosis Secure SDL” as an option. I asked Rich if we had ever written a secure software development lifecycle, to which he responded something like “Yeah, stupid, you did.” I went back and searched the blog and, sure enough, I am stupid. Out of frustration at the lack of practical guidance I had cobbled together a rudimentary process, but not fully fleshed it out. I think it’s time to take a fresh look at the process, and come up with some applied practices for resilient code within an Agile methodology. – AL

  6. Why sell it when you can give it away? – In the endpoint fundamentals series, I talked about anti-malware and touched on free AV. In the corporate context, you get what you pay for given the non-existent advanced detection techniques and management. But from a consumer perspective, is free AV such a bad thing? Not if the advanced stuff you want is there, which makes Check Point’s one-day give-away of ZoneAlarm and an identity service from Intersections, timed to coincide with Patch Tuesday, interesting. On the other hand, this reeks of desperation from CHKP, which can’t seem to make inroads into the endpoint space. Anyhow, if you’re looking for a full endpoint suite for your mother-in-law’s computer, have at it – even if you like your mother-in-law. – MR

  7. Hardening the Cloud – Yesterday I was talking with an investment friend about how I expect to see big hardware refreshes marching hand in hand with cloud computing adoption. It turns out there are some problems related to cloud computing security that are difficult or impossible to solve in software alone. Chris Hoff discusses Trusted Execution Technology, which basically requires new hardware and software, some of which isn’t available yet. (TXT ideally allows you to only execute a VM on hardware that passes assurance checks). There are also memory protection enhancements important to cloud computing that will require hardware upgrades. As Chris says, it doesn’t look like any public cloud providers are using these boxes yet, but make sure you take them into account when planning internal deployments. – RM

  8. My Dad can beat up your Dad – I remember those childish elementary school taunts like it was yesterday. I always thought my Dad was pretty tough, but I also knew he carried heat when he was working in pharmacies in “undesirable areas”. Hard to outrun a bullet. But the rules have changed now, and the bullies (and their parents) who my kids will have to deal with are different in nature. Physical bullies I can deal with, but the email (and Facebook) gladiators are much tougher. Some tactics are discussed on the McAfee blog, as well as in NetworkWorld (featuring our buddy Martin McKeay). They have good ideas there, but I’m still a fan of monitoring – at least to provide an audit trail in case things do go south. And also education – I spend a lot of time teaching my kids wrong from right, so why wouldn’t I do the same for their Internet use? – MR

—Mike Rothman

Tuesday, April 13, 2010

ESF: Controls: Full Disk Encryption

By Mike Rothman

It happens quickly. An end user just needed to pick up something at the corner store or a big box retailer. He was in the store for perhaps 15 minutes, but that was plenty of time for a smash and grab. And then your phone rings, a laptop is gone, and it had information on about 15,000 customers. You sigh, hang up the phone and call the general counsel – it’s disclosure time.

Sound familiar? Maybe this has been you. It likely will be, unless you proactively take action to make sure that the customer data on those mobile devices cannot be accessed by whoever buys the laptop on the gray market. That’s right, you need to deploy full disk encryption (FDE) on the devices. Unless you enjoy disclosure and meeting with lawyers, that is.

Features

Ultimately, encryption isn’t very novel. But managing encryption across an enterprise is, so key management and ease of use end up being the key features that generally drive FDE. As we’ve harped throughout this series, integration of that management with the rest of the endpoint functions is critical to gaining leverage and managing all the controls implemented on the endpoints.

Of course, that’s looking at the issue selfishly from the security professional’s perspective. Ultimately the success of the deployment depends on how transparent it is to users. That means it needs to fit in with the authentication techniques they already use to access their laptops. And it needs to just work. Locking a user out of their data, especially an important user at an inopportune, time will make you a pretty unpopular person.

Finally, don’t forget about those backups or software updates. If your encryption breaks your backups (and you are backing up all those laptops, right?) it’s a quick way to find yourself in the unemployment line. Same goes for having to tell the CIO everyone needs to bring their laptops back to the office every Patch Tuesday to get those updates installed.

Integration with Endpoint Suites

Given the natural order of innovation and consolidation, the industry has seen much consolidation of FDE solutions by endpoint vendors. Check Point started the ball rolling by acquiring Pointsec; shortly afterwards Sophos acquired Utimaco and McAfee acquired SafeBoot, which of course gives these vendors the ability to bundle FDE with their endpoint suites.

Now bundling on the purchase order is one thing, but what we are really looking for is bundling from a management standpoint. Can the encryption keys be managed by the endpoint security management console? Is your directory supported natively? Can the FDE policies be set up from the same interface you use for host firewalls and HIPS policies? Unless this level of integration is available, there is little leverage in using FDE from your endpoint vendor.

Free (as in beer?)

Like all good innovations, the stand-alone companies get acquired and then the capability tends to get integrated into the operating system – which is clearly the case with FDE. Both Microsoft BitLocker and Apple FileVault provide the capability to encrypt at the operating system level (Bitlocker is full drive, FileVault is OS). Yes, it’s free, but not really. As mentioned above, encryption isn’t really novel anymore, it’s the management of encryption that makes the difference. Neither Microsoft nor Apple currently provides adequate tools to really manage FDE across an enterprise.

Which means there will remain a need for third party managed FDE for the foreseeable future, and that also means the endpoint security suite is the best place to manage it. We expect further integration of FDE into endpoint security suites, further consolidation of the independent vendors, and ultimately commoditization of the capability. So we’ll joke over beers in a few years about how you use to pay separately for full disk encryption.

Now that we’ve examined the controls we use to protect the endpoints, we need to build a systematic program to ensure these controls are deployed, enforced, and reported on. That’s our topic for the next two posts, as we build the endpoint security program also consider what kind of reporting we need to keep the auditors happy.

Other posts in the Endpoint Security Fundamentals Series

—Mike Rothman

Monday, April 12, 2010

ESF: Controls: Firewalls, HIPS, and Device Control

By Mike Rothman

Popular perception of endpoint security revolves around anti-malware. But they are called suites for a reason – other security components ship in these packages, which provide additional layers of protection for the endpoint. Here we’ll talk about firewalls, host intrusion prevention, and USB device control.

Firewalls

We know what firewalls do on the perimeter of the network: selectively block traffic that goes through gateways by port and protocol. The functionality of a host firewall on an endpoint is similar. They allow an organization to enforce a policy governing what traffic the device can accept (ingress filtering) and transmit (egress filtering).

Managing the traffic to and from each endpoint serves a number of purposes, including hiding the device from reconnaissance efforts, notifying the user or administrators when applications attempt to access the Internet, and monitoring exactly what the endpoints are doing. Many of these capabilities are available separately on the corporate network, but when traveling or at home and not behind the corporate perimeter, the host firewall is the first defense against attacks.

Of course, a host firewall (like everything else that runs on an endpoint) takes up resources, which can be a problem on older or undersized machines. It also bears consideration that alerts multiply, especially when you have a couple thousand endpoints forwarding them to a central console, so some kind of automated alert monitoring becomes critical.

Although pretty much every vendor bundles a host firewall with their endpoint suite nowadays, the major operating systems also provide firewall options. Windows has included a firewall since XP, but keep in mind that the XP firewall does not provide egress (outbound) filtering – remedied in Windows Vista. Mac OS X 10.5 Leopard added a ‘socket’ firewall to manage application listeners (ingress), and deprecated the classic ipfw network firewall, which is still available.

As with all endpoint capabilities, just having the feature isn’t enough, since the number of endpoints to be managed puts a real focus on managing the policies. This makes policy management more important than firewall engine details.

Host Intrusion Prevention Systems (HIPS)

We know what network intrusion detection/prevention products do, in terms of inspecting network traffic and looking for attacks. Similarly, host intrusion detection/prevention capabilities look for attacks by monitoring what’s happening on the endpoint. This can include application behavior, activity logs, endpoint network traffic, system file changes, Windows registry changes, processes and/or threads, memory allocation, and pretty much anything else.

The art of making host intrusion prevention work is to set up the policies to prevent malware infection, without badly impacting the user experience or destroying the signal-to-noise ratio of alerts coming into the management console. Yes, this involves tuning, so you start with the product’s default settings (hopefully on a test group) and see what works and what doesn’t. You should be able to quickly optimize the policy.

Given the number of applications and activities at each endpoint, you can go nuts trying to manage these policies, which highlights the importance of the standard builds (as described in Controls: Secure Configurations). Starting with 3-4 different policies, and then you can manage others by exception. Keep in mind that tuning the product for servers is totally different, as the policies will need to be tailored for very different applications running on servers.

Currently, all the major endpoint suites include simple HIPS capabilities. Some vendors also offer a more capable HIPS product – typically targeting server devices, which are higher profile targets and subject to different attacks.

USB Device Control

Another key attack vector for both data compromise and malware proliferation is the USB ports on endpoint devices. In the old days, you’d typically know when someone brought in a huge external drive to pilfer data. Nowadays many of us carry a 16GB+ drive at all times (yes, your smartphone is a big drive), so we’ve got to control USB ports to address this exposure.

Moreover, we’ve all heard stories of social engineers dropping USB sticks in the parking lot and waiting for unsuspecting employees to pick them up and plug them in. Instant pwnage 4U! So another important aspect of protecting endpoints includes defining which devices can connect to a USB port and what those devices can do.

This has been a niche space, but as more disclosure is required for data loss, organizations are getting more serious about managing their USB ports. As with all other endpoint technologies, device control adds significant management overhead for keeping track of all the mobile devices and USB sticks, etc. The products in the space include management consoles to ease the burden, but managing thousands of anything is non-trivial.

Right now device control is a discrete function, but we believe these niche products will also be subsumed into the endpoint suites over the next two years. In the meantime, you may be able to gain some leverage by picking a device control vendor partnered with your endpoint suite provider. Then you should at least be able centralize the alerts, even if you don’t get deeper management integration.

Management Leverage

Though we probably sound like a broken record at this point, keep in mind that each additional security application/capability (control) implemented on the endpoint devices increases the management burden. So when evaluating technology for implementation, be sure to assess the additional management required and the level of integration with your existing endpoint management workflow.

We’ll wrap up our discussion of Endpoint Controls in the next post, as we discuss full disk encryption, which disclosure laws have shifted from nice-to-have to something you need deployed – immediately.

Other posts in the Endpoint Security Fundamentals Series

—Mike Rothman

FireStarter: No User Left Behind

By Rich

Not to bring politics into a security blog, but I think it’s time we sit down and discuss the state of education in this country… I mean industry.

Lance over at HoneyTech went off on the economics/metrics paper from Microsoft we recently discussed. Basically, the debate is over the value of security awareness training. The paper suggests that some training isn’t worth the cost. Lance argues that although we can’t always directly derive the desired benefits, there are legitimate halo effects. Lance also points out that the metrics chosen for the paper might not be the best.

I’d like to flip this a little bit. The problem isn’t the potential value of awareness training – it’s that most training is total crap.

When we improperly design the economics, incentives and/or metrics of the system, we fail to achieve desired objectives. Right, it’s not brain surgery.

Let’s use the No Child Left Behind act here in the US as an example. This law requires standardized testing in schools, and funding is directly tied to test results. Which means teachers now teach to do well on an exam, rather than to educate. Students show up at universities woefully unprepared due to lack of general knowledge and possession of a few rote skills. It is the natural outcome of the system design.

Most security awareness training falls into the same trap. The metrics tend to be test scores and completion rates. So organizations dutifully make sure employees sit through the training (if that’s what you want to call it) and get their check mark every year. But that forgets why we spend time and money to perform the training in the first place. What we really care about is improving security outcomes, which I’ll define as a reduction in frequency and severity of security incidents. Not about making sure every employee can check a box.

Thus we need outcome-based security awareness programs, which means we have to design our metrics and economics to support measurable improvements in security.

Rather than measuring how many people took a class or passed a stupid test, we should track outcomes such as:

  • For a virus infection, was user interaction involved?
  • User response rates to phishing spam.
  • Results from authorized social engineering penetration tests.

Tracking incidents where the user was involved can determine whether the incident was reasonably preventable, and whether the user was (successfully) trained to avoid that specific type of incident. Follow the trends over time, and feed this back into your awareness program.

If all you do is force people to sit through boring classes or follow mind-numbing web-based training, and then say your training was successful if they can answer a few multiple choice questions, you are doing it wrong.

So we have not given up hope on the impact of security awareness training. If we focus on tracking real world outcomes, not auditor checklist garbage like how many people signed a policy or sat in a chair for a certain number of hours, it can make a difference. Are taking too many hits off the peace pipe? Have any of you had a measurable impact of training in your environment? Speak up in the comments.

—Rich

Friday, April 09, 2010

ESF: Controls: Anti-Malware

By Mike Rothman

As we’ve discussed throughout the Endpoint Security Fundamentals series, adequately protecting endpoint devices entails more than just an endpoint security suite. That said, we still have to defend against malware, which means we’ve got to figure out what is important in an endpoint suite and how to get the most value from the investment.

The Rise of Socially-Engineered Malware

To state the obvious, over the past few years malware has dramatically changed. Not just the techniques used, but also the volume. It’s typical for an anti-virus companies to identify 1-2 million new malware samples per month. Yes, that’s a huge amount. But it gets worse: a large portion of malware today gets obfuscated within legitimate looking software packages.

A good example of this is fake anti-virus software. If one of your users happens to click on a link and end up on a compromised site (by any means), a nice little window pops up telling them they are infected and need to download an anti-virus program to clean up the attack. Part of that is true – upon visiting the site a drive-by attack did compromise the machine. But in this case, the antidote is a lot worse than the system because this new “anti-virus” package leaves behind a nasty trojan (typically ZeuS or Conficker).

The folks at NSS Labs have dubbed this attack “socially-engineered malware,” because it hides the malware and preys upon the user’s penchant to install the compromised payload with disastrous results. That definition is as good as any, so we’ll go with it.

Cloud and reputation

The good news is that the anti-malware companies are not sitting still. They continue to make investments in new detection techniques to try to keep pace. Some do better than others (check out NSS Labs’ comparative tests for the most objective and relevant testing – in our opinion anyway), but what is really clear is how broken the old blacklist, signature-based model has gotten. With 2 million malware samples per month, there is no way keeping a list of bad stuff on each device remains feasible.

The three main techniques added over the past few years are:

  • Cloud-based Signatures – Since it’s not possible to keep a billion signatures in an endpoint agent, the vendors try to divide and conquer the issue. So they split the signature database between the agent and an online (cloud) repository. If an endpoint encounters a file not in its local store, it sends a signature to the cloud for checking against the full list. This has given the blacklist model some temporary legs, but it’s not a panacea, and the AV vendors know it.
  • Reputation – A technique pioneered by the anti-spam companies a few years ago involves inferring the intent of a site by tracking what that site does and assigning it a reputation score. If the site has a bad reputation, the endpoint agent doesn’t let the site’s files or executables run. Obviously this is highly dependent on the scale and accuracy of the reputation database. Reputation has become important for most security offerings, including perimeter and web filtering, in addition to anti-spam and endpoint security.
  • Integrated HIPS – Another technique in use today is host intrusion prevention. But not necessarily signature-based HIPS, which was the first generation. Today most HIPS looks more like file integrity monitoring, so the agent has a list of sensitive system files which should not be changed. When a malware agent runs and tries to change one of these files, the agent blocks the request – detecting the attack.

So today’s anti-malware agents attempt to detect malware both before execution (via reputation) and during execution (signatures and HIPS), so they can block attacks. But to be clear, the industry is always trying to catch up with the malware authors.

Making things even more difficult, users have an unfortunate tendency to disregard security warnings, allow the called-out risky behavior, and then get pwned. This can be alleviated slightly with high-confidence detection (if we know it’s a virus, we don’t have to offer the user a chance to run it) or stronger administrative policies which authorize not even letting users override the anti-malware software. But it’s still a fundamentally intractable problem.

Management is key

Selecting an anti-malware agent typically comes down to two factors: price and management. Price is obvious – plenty of upstarts want to take market share from Symantec and McAfee. They use price and an aggressive distribution channel to try and displace the incumbents. All the vendors also have migration tools, which dramatically lower switching costs.

In terms of management, it usually comes down to personal preference, because all the tools have reasonably mature consoles. Some use open data stores, so customers can build their own reporting and visualization tools. For others, the built-in stuff is good enough. Architecturally, some consoles are more distributed than others, and so scale better to large enterprise operations. But anti-malware remains a commodity market.

One aspect to consider is the size and frequency of signature and agent updates, especially for larger environments. If the anti-malware vendor sends 30mb updates 5 times a day, that will create problems in low-bandwidth environments such as South America or Africa.

Free AV: You get what you pay for…

Another aspect of anti-malware to consider is free AV, pioneered by folks like AVG and Avast, who claim up to 100 million users of their free products. To be clear, in a consumer context free AV can work fine. But it’s not a suite, so you won’t get a personal firewall or HIPS. There won’t be a cloud-based offering behind the tool, and it won’t use new techniques like reputation to defend against malware. Finally, there are no management tools, so you’ll have to manage every device individually, which loses feasibility past a handful.

For a number of use cases (like your mom’s machine), free AV should be fine. And to be clear, the entire intent of these vendors in giving away the anti-malware engine is to entice you to upgrade to their paid products. That said, I use free AV on my remaining PC, and also in the virtual Windows images running on my Macs, and it works fine. But free AV is generally a poor fit for organizations.

White listing: disruptive or niche?

You can’t really talk about anti-malware without mentioning Application White Listing (AWL). This approach basically only allows authorized executables to run on endpoint devices, thus blocking any unauthorized applications – which includes malware. AWL has the rap of being very disruptive to the end-user experience, by breaking lots of authorized applications and the like. Part of that rep is deserved, but we believe the technology has the potential to fundamentally change how we fight malware.

To be clear, AWL is not there yet. For some use cases (like embedded devices, kiosks, and control systems), the technology is a no-brainer. For general purpose PCs, it comes back to how much mojo security has in dictating what can run and what can’t. Though there are management capabilities (like trusting certain application updates) emerging to address the user disruption issue, we believe AWL will remain a niche technology for the next 2-3 years.

But as AWL matures and the traditional ways of detecting and blocking malware increasingly fail, we expect AWL to become a key technique and appear in more and more of the existing anti-malware suites.

Layers of defense

Yet, we still default to the tried and true method of layering security defenses. Anti-malware agents cannot be the only defense against the bad stuff out there – not if you actually want to protect your devices, anyway. We’ve harped on this throughout the series, relative to the importance of using other tactics on the endpoints (including running updated software and secure configurations) and within the network to compensate for the fact that anti-malware is an inexact science. And don’t forget about the importance of monitoring everything, given that as much as we try to prevent, in many cases reacting faster is the only option we have.

Next we’ll wrap up the controls portion of this series by talking about personal firewalls, a deeper dive on HIPS, and USB device control.


Other posts in the Endpoint Security Fundamentals Series

—Mike Rothman

Friday Summary: April 9, 2010

By Rich

So I’m turning 39 in a couple of weeks. Not that 39 is one of those milestone birthdays, but it leaves me with only 365 days until I can not only no longer trust myself (as happened when I turned 30), but I supposedly can’t even trust my bladder anymore.

I’m not really into birthdays with ‘0’ at the end having some great significance, but I do think they can be a good excuse to reflect on where you are in life. Personally I have an insanely good life – I run my own company, have a great family, enjoy my (very flexible) job, and have gotten to do some pretty cool things over the years. Things like “fly a jet,” “drive over 100 MPH with lights and sirens on,” “visit 6 of 7 continents,” “compete in a national martial arts tournament” (and lose to a 16 year old who hadn’t discovered beer yet), “rescue people from mountains,” “get choppered into a disaster,” “ski patrol at a major resort,” “meet Jimmy Buffett,” and even “write a screenplay” (not a good screenplay, but still).

But there are a few things I haven’t finished yet, and that last year before 40 seems like a good chance to knock one or two off. Here are my current top 5, and I’m hoping to finish at least one:

  • Get my pilot’s license.
  • Visit Antarctica (the only continent I haven’t been on).
  • Sail the Caribbean Captain Ron style.
  • Run a marathon.
  • Finish an Olympic-distance triathlon (I’ve done sprint distance already).

I’m open to suggestions, and while the marathon/triathlon are the cheapest, I’d kind of like to get that pilot’s license.


On to the Summary:

Webcasts, Podcasts, Outside Writing, and Conferences

Favorite Securosis Posts

Other Securosis Posts

Favorite Outside Posts

Project Quant Posts

Top News and Posts

Blog Comment of the Week

Remember, for every comment selected, Securosis makes a $25 donation to Hackers for Charity. This week’s best comment goes to Paul Simmonds, in response to FireStarter: Nasty or Not, Jericho is Irrelevant.

Having just read the RFI response from a major software vendor, who’s marketing BS manages to side-step all the questions designed to get to the bottom of “is this secure”, then the answer is YES, we do need the nasty questions. More importantly they may be obvious but we as purchasers are not asking them, and the vendors are not volunteering the information (mainly because what they supply is inherently insecure). And then we wonder why we are in the state we are in??

—Rich

Thursday, April 08, 2010

Database Security Fundamentals: Auditing Transactions

By Adrian Lane

I am now switching gears to talk about some of the ‘detective’ measures that help with forensic analysis of transactions and activity. The preventative measures discussed previously are great for protecting your system from known attacks, but they don’t help detect fraudulent misuse or failure of business processes. For that we need to capture the events that make up the business processes and analyze them. Our basic tool is database auditing, and they provide plenty of useful information.

Before I get too far into this discussion, it’s worth noting that the term ‘transactions’ is an arbitrary choice on my part. I use it to highlight both that audit data can group statements into logical sequences corresponding to particular business functions, and that audit trail analysis is most useful when looking for insider misuse – rather than external attacks. Audit trails are much more useful for detecting what was changed, rather than what was accessed, and for forensic examination of database ‘state’. There are easier and more efficient ways of cataloging SELECT statements and simple events like failed logins.

Usually at this point I provide a business justification for auditing of transactions or specific events, and some use cases as examples of how it helps. I will skip that this time, as you already know that auditing is built into every database; and captures database queries, transactions, and important system changes. You probably already use audit logs to see what actions are most common so you can set up indices and tune your most common queries. You may even use auditing to detect suspect activity, to perform forensic audits, or even to address a specific compliance mandate. At the very least you need to have some form of database auditing enabled on production databases to answer the question “What the &!$^% happened” after a database crash. Regardless of your reasons, auditing is essential for security and compliance.

In this post I will focus on capturing transactions and alterations to the database. What type of analysis you do, how long you keep the data, and what reports you create are secondary. I am focusing on gathering the audit trail rather than what do with it next. What’s critical is here understanding what data you need, and how best to capture it.

All databases have some type of audit function. The ‘gotcha’ is that use of database auditing needs careful consideration to avoid storage, performance, and management nightmares. Depending on the vendor and how the feature is deployed, you can gather detailed transactional information, filter unwanted transactions to get a succinct view of database activity, and do so with only modest performance impact. Yes, I said modest performance impact. This remains a hot-button issue for most DBAs and is easy to mess up, so planning and basic tests form a bulk of this phase.

  1. Benchmark: Find a test system, gather a bunch of queries that reflect user activity on your system, and run some benchmarks. Turn on the auditing or tracing facility and rerun the benchmark. Wince and swear repeatedly at the performance degradation you just witnessed. Aren’t you glad you did this on a test system? You have a baseline for best and worst case performance.
  2. Tune:
    1. Select Audit options. Oracle, SQL Server, DB2, and Sybase have multiple options for generating audit trails. Don’t use Oracle’s fine-grained auditing when normal auditing will suffice. Don’t use event monitors on DB2 if you have many different types of events to collect. Which auditing option you choose will dramatically affect performance and data volumes.
    2. Examine the audit capture options, and select only the event types you need. If you only care about user events, don’t bother collecting all the object events. If you only care about failed logins and changes to system privileges, filter out meta-data and data changes.
    3. Examine buffer space, tablespace, block utilization, and other resource tuning options. Audit data are static in size, so their data blocks can be set to ‘write-only’, thus saving space. For audit trails that store data within database tables, you can pre-allocate table space and blocks to reduce latency from space allocation.
    4. Rerun the benchmarks and see what helps performance. Generally these steps provide significant performance gains. No more cursing the database vendor should be needed.
  3. Filter: Get rid of specific actions you don’t need. For example, batch updates may fall outside your area of interest, yet comprise a significant fraction of the audit log, and can therefore be parsed out. Or you may want to audit all transactions while the databse is running, but not need events from database startup. In some scenarios the database can filter these events, which improves performance. If the database does not provide this type of row-level filtering, you can add a WHERE clause to the extraction query or use a script to whittle down the extracted data.
  4. Implement: Take what you learned and apply it in the production environment. Verify that the audit trail is being collected.
  5. Ad-hoc Analysis: Review the logs periodically, focusing on failures of system functions and logins that indicate system probing. Any policy or report that you generate may miss unusual events, so I recommend occasional ad hoc analysis to detect anything hinkey.
  6. Record: Document the audit settings as well as the test results, as you will need them in the future to understand the impact of increased auditing levels. Communicate use of audit to users, and warn them that there will be a performance hit due to regulatory and security requirements. Create a log retention policy. This is necessary – even if the policy simply states that you will collect audit trails and delete them at the end of every week, write it down. Many compliance requirements let you define retention however you choose, so be proactive. You can always change it in the future.

More advanced considerations:

  • Automate: I recommend that you automate as much of the process as you can, once you have done the initial analysis and configuration. Collection of the audit trail, filtering, purging old records, and long-term storage are all tasks that can be automated through simple scripts and cron jobs.
  • Integrate: Reporting services, event management, and change management are all services that help automate security tasks based on audit data.
  • Review: Periodically review log tuning and filtering to determine if the settings are still appropriate. Most organizations collect more data over time rather than less, but who knows? You may want to run some sanity checks on the performance benchmarks every now and again, as vendors make improvements or offer new options.

The standard audit capabilities provided by database vendors provides ample information for compliance reporting, but in cases where SELECT statements are not captured, they are of limited use for security reviews. So in the next post I will go over event analysis to discuss other data collection options and essential events to evaluate.

Remember: if you are a DBA, and people within your company are requiring that you provide them with log files, this is a good thing. It means that they are the ones who will need to review transactions for suspicious activity. They can delve through the reports. You may have more work in setting up the reports and auditing options, but overall this is a good tradeoff.

—Adrian Lane

ESF: Controls: Secure Configurations

By Mike Rothman

Now that we’ve established a process to make sure our software is sparkly new and updated, let’s focus on the configurations of the endpoint devices that connect to our networks. Silly configurations present another path of least resistance for the hackers to compromise your devices. For instance, there is no reason to run FTP on an endpoint device, and your standard configuration should factor that in.

Define Standard Builds

Initially you need to define a standard build, or more likely a few standard builds. Typically for desktops (no sensitive data, and sensitive data), mobile employees, and maybe, kiosks. There probably isn’t a lot of value to going broader than those 4 profiles, but that will depend on your environment.

A good place to start is one of the accepted benchmarks of configurations available in the public domain. Check out the Center for Internet Security, which produces configuration benchmarks for pretty much every operating system and many of the major applications. In order to see your tax dollars at work (if you live in the US anyway), also consult NIST, especially if you are in the government. Its SCAP configuration guides provide a similar type of enumeration of specific setting to lock down your machines.

To be clear, we need to balance security with usability and some of the configurations suggested in the benchmarks clearly impact usability. So it’s about figuring out what will work in your environment, documenting those configurations, getting organizational buy-in, and then implementing.

It also makes sense to put together a list of authorized software as part of the standard builds. You can have this authorized software installed as part of the endpoint build process, but it also provides an opportunity to revisit policies on applications like iTunes, QuickTime, Skype, and others which may not yield a lot of business value and have a history of vulnerability. We’re not saying these applications should not be allowed – you’ve got to figure that out in the context of your organization – but you should take the opportunity to ask the questions.

Anti-Exploitation

As you define your standard builds, at least on Windows, you should turn on anti-exploitation technologies. These technologies make it much harder to gain control of an endpoint through a known vulnerability. I’m referring to DEP (data execution prevention) and ASLR (address space layout randomization), though Apple is also implementing similar capabilities in their software.

To be clear, anti-exploitation technology is not a panacea for protection – as the winners of Pwn2Own at CanSecWest show us every year. Especially for those applications that don’t support it (d’oh!), but the technologies do help make it harder to exploit the vulnerabilities in compatible software.

Other Considerations

  • Running as a standard user – We’ve written a bit on the possibilities of devices running in standard user mode (as opposed to administrator mode), and you should consider this option when designing secure configurations, especially to help enforce authorized software policies.
  • VPN to Corporate – Given the reality that mobile users will do something silly and put your corporate data at risk, one technique to protect them is to run all their Internet traffic through the VPN to your site. Yes, it may add a bit of latency, but at least the traffic will be running through the web gateway and you can both enforce policy and audit what the user is doing. As part of your standard build, you can enforce this network setting.

Implementing Secure Configurations

Once you have the set of secure configurations for your device profiles, how do you start implementing them? First make sure everyone buys into the decisions and understands the ramifications of going down this path. Especially if you plan to stop users from installing certain software or block other device usage patterns. Constantly asking for permission can be dangerously annoying but choosing the right threshold for confirmations is a critical aspect of a designing a policy. If the end users feel they need to go around the security team and their policies to get the job done, everyone loses.

Once the configurations are locked and loaded, you need to figure out how much work is required for implementation. Next you assess the existing endpoints against the configurations. Lots of technologies can do this, ranging from Windows management tools, to vulnerability scanners, to third party configuration management offerings. The scale and complexity of your environment should drive the selection of the tool.

Then plan to bring those non-compliant devices into the fold. Yes, you could just flip the switch and make the changes, but since many of the configuration settings will impact user experience, it makes sense to do a bit of proactive communication to the user community. Of course some folks will be unhappy, but that’s life. More importantly, this should help cut down help desk mayhem when some things (like running that web business from corporate equipment) stop working.

Discussion of actually making the changes brings us to automation. For organizations with more than a couple dozen machines, a pretty significant ROI is available from investing in some type of configuration management. Again, it doesn’t have to be the Escalade of products, and you can even look at things like Group Policy Objects in Windows. The point is making manual changes on devices is idiotic, so apply the level of automation that makes sense in your environment.

Finally, we also want to institutionalize the endpoint configurations, and that means we need to get devices built using the secure configuration. Since you probably have an operations team that builds the machines, they need to get the image and actually use it. But since you’ve gotten buy-in at all steps of this process, that shouldn’t be a big deal, right?

Next up, we’ll discuss the anti-malware space and what makes sense on our endpoints.


Other posts in the Endpoint Security Fundamentals Series

—Mike Rothman

Wednesday, April 07, 2010

Incite 4/7/2010: Everybody Loves the Underdog

By Mike Rothman

Come on, admit it. Unless you have Duke Blue Devil blood running through your veins (and a very expensive diploma on the wall) or had Duke in your tournament bracket with money on the line, you were pulling for the Butler Bulldogs to prevail in Monday night’s NCAA Men’s Basketball final. Of course you were – everyone loves the underdog.

They don't make 'em like they used to... If you think of all the great stories through history, the underdog has always played a major role. Think David taking down Goliath. Moses leading the Israelites out of Egypt. Pretty sure the betting line had long odds on both those scenarios. Think of our movie heroes, like Rocky, Luke Skywalker, Harry Potter, and the list goes on and on. All weren’t supposed to win and we love the fact that they did. We love the underdogs.

Unfortunately reality intruded on our little dream, and on Monday Butler came up a bucket short. But you still felt good about the game and their team, right? I can’t wait for next year’s season to see whether the little team that could can show it wasn’t all just a fluke (remember George Mason from 2006?).

And we love our underdogs in technology, until they aren’t underdogs anymore. No one really felt bad when IBM got railed when mainframes gave way to PCs. Unless you worked at IBM, of course. Those damn blue shirts. And when PCs gave way to the Internet, lots of folks were happy that Microsoft lost their dominance of all things computing. How long is it before we start hating the Google. Or the Apple?

They don't make 'em like they used to... It’ll happen because there will be another upstart taking the high road and showing how our once precious Davids have become evil, profit-driven Goliaths. Yup, it’ll happen. It always does. Just think about it – Apple’s market cap is bigger than Wal-Mart. Not sure how you define underdog, but that ain’t it.

Of course, unlike Rocky and Luke Skywalker, the underdog doesn’t prevail in two hours over a Coke and popcorn. It happens over years, sometimes decades. But before you go out and get that Apple logo tattooed on your forearm to show your fanboi cred, you may want to study history a little. Or you may become as much a laughingstock as the guy who tattooed the Zune logo on his arm. I’m sure that seemed like a good idea at the time, asshat. The mighty always fall, and there is another underdog ready to take its place.

If we learn anything through history, we should now the big dogs will always let us down at some point. So don’t get attached to a brand, a company, or a gadget. You’ll end up as disappointed as the guy who thought The Phantom Menace would be the New Hope of our kids’ generation.

– Mike.

Photo credits: “Underdog Design” originally uploaded by ChrisM70 and “Zune Tattoo Guy” originally uploaded by Photo Giddy


Incite 4 U

  1. What about Ritalin? – Shrdlu has some tips for those of us with an, uh, problem focusing. Yes, the nature of the security managers’ job is particularly acute, but in reality interruption is the way of the world. Just look at CNN or ESPN. There is so much going on I find myself rewinding to catch the headlines flashing across the bottom. Rock on, DVR – I can’t miss that headline about… well whatever it was about. In order to restore any level of productivity, you need to take Shrdlu’s advice and delegate, while removing interruptions – like email notifications, IM and Twitter. Sorry Tweeps, but it’s too hard to focus when you are tempted by links to blending an iPad. It may be counter-intuitive, but you do have to slow down to speed up at times. – MR

  2. Database security is a headless rhicken – As someone who has been involved with database security for a while, it comes as no surprise that this study by the Enterprise Strategy Group shows a lack of coordination is a major issue. Anyone with even cursory experience knows that security folks tend to leave the DBAs alone, and DBAs generally prefer to work without outside influence. In reality, there are usually 4+ stakeholders – the DBA, the application owner/manager/developer, the sysadmin, security, and maybe network administration (or even backup, storage, and…). Everyone views the database differently, each has different roles, and half the time you also have outside contractors/vendors managing parts of it. No wonder DB security is a mess… pretty darn hard when no one is really in charge (but we sure know who gets fired first if things turn south). – RM

  3. Beware of surveys bearing gifts – The PR game has changed dramatically over the past decade. Now (in the security business anyway) it’s about sound bites, statistics, and exploit research. Without either of those three, the 24/7 media circus isn’t going to be interested. Kudos to Bejtlich, who called out BeyondTrust for trumping up a “survey” about the impact of running as a standard user. Now to be clear, I’m a fan of this approach, and Richard acknowledges the benefits of running as a standard user as well. I’m not a fan of doing a half-assed survey, but I guess I shouldn’t be surprised. It’s hard to get folks interested in a technology unless it’s mandated by compliance. – MR

  4. e-Banking and the Basics – When I read Brian Krebs’ article on ‘e-Banking Guidance for Banks & Businesses’, I was happy to see someone offering pragmatic advice on how to detect and mitigate the surge of on-line bank fraud. What shocked me is that the majority of his advice was basic security and anti-fraud steps, and it was geared towards banks! They are not already doing all this stuff? Oh, crap! Does that mean most of these regional banks are about as sophisticated as an average IT shop about security – “not very”? WTF? You don’t monitor for abnormal activity already? You don’t have overlapping controls in place already? You don’t have near-real-time fraud detection in place already? You’re a freaking bank! It’s 2010, and you are not requiring 3rd factor verification for sizable Internet transfers already? I suspect that security will be a form of business Darwinism, and you’ll be out of business soon enough for failing to adapt. Then someone else will worry about your customers. I just hope they don’t get bankrupted before you finish flailing and failing. – AL

  5. If you can’t beat them, OEM – When you have an enterprise firewall that isn’t a market leader in a mature market, what do you do? That’s the challenge facing McAfee. The former Secure Computing offering (Sidewinder) still has a decent presence in the US government, but hasn’t done much in the commercial sector, and isn’t going to displace the market leaders like Cisco, Juniper, or Check Point by hoping some ePO fairy dust changes things. So McAfee is partnering with other folks to integrate firewall capabilities into network devices. A while back they announced a deal with Brocade (the former Foundry switch folks) and this week did a deal with Riverbed to have the firewall built into the WAN optimization box. Clearly security and network stuff need to come together cleanly (something Cisco and Juniper have been pushing) and folks like Foundry and Riverbed had no real security mojo. But the real question is whether this is going to help McAfee capture any share in network security. I’m skeptical because it’s not like the folks using Brocade switches or Riverbed gear aren’t doing security now, and an OEM relationship doesn’t provide the perceived integration that will make a long-term difference. – MR

  6. Compliance owns us – No surprise – yet another survey shows that compliance drives security spending, even though it doesn’t always align with enterprise priorities. Forrester performed a study, commissioned by RSA and Microsoft, on where dollars go compared to the information assets an organization prioritizes. The study did an okay job of constraining the normally fuzzy numbers around losses (limiting costs to hard dollars), but I’m a bit skeptical that organizations are tracking them well in the first place. Some of the conclusions are pretty damn weak, especially considering how they structured the study, but it’s still worth a read to judge attitudes – even if the value numbers are crap. While imperfect, it’s a better methodology than the vast majority of this kind of research. As I’ve said before – I think our compliance obsession is the natural result of the current loss economics, and until we can really measure the costs of IP loss, nothing will change. – RM

  7. If not the FCC, then who? – In a clever move, Comcast was able to successfully argue against net neutrality claims, arguing that because the FCC deregulated the Internet, they have no basis to force compliance with a policy that is not embodied in law. Rather than debate the merits of net neutrality itself, they side-stepped the issue. As there is no other governing body that could enforce the policy at this time, Comcast is getting its way. The corporate equivalent of a cold-blooded murderer getting off on a technicality. But this is a pyrrhic victory, because now we get to see all those clever tools that hide content and protocols from the Chinese government unleashed closer to home, so Verizon, AT&T and Comcast are going to end up having to move the data regardless. Hopefully the public will find a suitable way to avoid broadband providers’ bureaucracy and legislation at the same time. – AL

  8. Are you grin frackking me? – Funny article here on the Business Insider about a former consultant (now a VC) who called bunk on his entire organization, which was basically feeding everyone a load of crap about their capabilities. I’ve been using the term “grin fscker” for years to represent someone who tells you want you want to hear, but has no intention of following through. Sometimes I call them on it, sometimes I don’t – and that’s my bad. The only way to deal with grin fscking is to call it out and shove the grin fscker’s nose in the poop. As the post explains, the buck should stop here. If someone is being disingenuous, it’s everyone’s responsibility to call that out. – MR

—Mike Rothman

Tuesday, April 06, 2010

ESF: Controls: Update and Patch

By Mike Rothman

Running old software is bad. Bad like putting a new iPad in a blender. Bad because all software is vulnerable software, and with old software even unsophisticated bad guys have weaponized exploits to compromise the software. So the first of the Endpoint Security Fundamentals technical controls is to make sure you run updated software.

Does that mean you need to run the latest version of all your software packages? Can you hear the rejoicing across the four corners of the software ecosystem? Actually, it depends. What you do need to do is make sure your endpoint devices are patched within a reasonable timeframe. Like one minute before the weaponized exploit hits the grey market (or shows up in Metasploit).

Assess your (software) assets

Hopefully you have some kind of asset management thing, which can tell you what applications run in your environment. If not, your work gets a bit harder because the first step requires you to inventory software. No, it’s not about license enforcement, it’s about risk assessment. You need to figure out the your software vendors’ track records on producing secure code, and then on patching exploits as they are discovered. You can use sites like US-CERT and Secunia, among others, to figure this out. Your anti-malware vendor also has a research site where you can look at recent attacks by application.

You probably hate the word prioritize already, but that’s what we need to do (again). Based on the initial analysis, stack rank all your applications and categorize into a few buckets.

  • High Risk: These applications are in use by 50M+ users, thus making them high-value targets for the bad guys. Frequent patches are issued. Think Microsoft stuff – all of it, Adobe Acrobat, Firefox, etc.
  • Medium Risk: Anything else that has a periodic patch cycle and is not high-risk. This should be a big bucket.
  • Low Risk: Those apps which aren’t used by many (security by obscurity) and tend to be pretty mature, meaning they aren’t updated frequently.

Before we move on to the updating/patching process, while you assess the software running in your environment, it makes sense to ask whether you really need all that stuff. Even low-risk applications provide attack surface for the bad guys, so eliminating software you just don’t need is a good thing for everyone. Yes, it’s hard to do, but that doesn’t mean we shouldn’t try.

Defining the Update/Patch Process

Next you need to define what your update and patching process is going to be – and yes, you’ll have three different policies for high, medium and low risk applications. The good news is your friends at Securosis have already documented every step of this process, in gory detail, through our Patch Management Quant research.

At a very high level, the cycle is: Monitor for Release/Advisory, Evaluate, Acquire, Prioritize and Schedule, Test and Approve, Create and Test Deployment Package, Deploy, Confirm Deployment, Clean up, and Document/Update Configuration Standards. Within each phase of the cycle, there are multiple steps.

Not every step defined in PM Quant will make sense for your organization, so you can pick and choose what’s required. The requirement is to having a defined, documented, and operational process; and to have answered the following questions for each of your categories:

  • Do you update to the latest version of the application? Within how soon after its release?
  • When a patch is released, how soon should it be applied? What level of testing is required before deployment?

In a perfect world, everything should be patched immediately and all software should be kept at the latest version. Unless you are talking about Microsoft Vista <grin>. But we all know the world isn’t perfect and there are real economic and resource dependencies to tightening the patch window and buying software updates – and discovering more bugs in the patches themselves. So all these factors need to be weighed when defining the process and policies. There is no right or wrong answer – it’s a matter of balancing economic reality against risk tolerance.

Also keep in mind that patching remote and mobile users is a different animal, and you have to factor that into the process. Many of these folks infrequently connect and may not have access to high-bandwidth connections. Specifying a one-day patch window for installing a 400mb patch at a mobile office in the jungle may not be practical.

Tools and Automation

Lots of tools can help you automate your software updating and patching process. They range from full-fledged asset and configuration management offerings to fairly simple patching products. It’s beyond the scope of this series to really dig into the nuances of configuration/patch management, but we’ll just say here that any organization with more than a couple hundred users needs a tool. This is a topic we’ll cover in detail later this year.

The next endpoint control we’ll discuss is Secure Configurations, so stay tuned.

Other posts in the Endpoint Security Fundamentals Series

—Mike Rothman