Login  |  Register  |  Contact
Wednesday, October 20, 2010

Incite 10/20/2010: The Wrongness of Being Right

By Mike Rothman

One of my favorite sayings is “Don’t ask the question if you don’t want the answer.” Of course, when I say answer, what I really mean is opinion. It makes no difference what we are talking about, I probably have an opinion. In fact, a big part of my job is to have opinions and share them with however will listen (and even some who won’t). But to have opinions means you need to judge.

Now those are great words of advice... I like to think I have a finely tuned bullshit detector. I’ve been having vendors lie to me since I got into this business 18 years ago. A lot of end users can be delusional about their true situations as well. So that means I’m judging what’s happening around me at all times, and I tell then what I think. Even if they don’t want to hear my version of the truth. Sometimes I make snap judgements; other times I only take a position after considerable thought and research. I’m trying to determine if something is right or wrong, based on all the information I can gather at that point in time. But I have come to understand that right and wrong is nothing more than another opinion. What is right for you may be wrong for me. Or vice-versa. It took me a long long time to figure that out. Most folks still don’t get this.

I can recall when I was first exposed to the Myers-Briggs test, when I stumbled on a book that included i. Taking that test was very enlightening for me. Turns out I’m an INTJ, which means I build systems, can be blunt and socially awkward (go figure), and tend to judge everything. Folks like me make up about 1% of the population (though probably a bit higher in tech and in the executive suite). I knew I was different ever since my kindergarten teacher tried to hold me back in kindergarten (true story), but I never really understood why.

Even if you buy into the idea there are just 16 personality types, clearly there is a spectrum across each of the 4 characteristics. In my black/white world, there seems to be a lot of color. Who knew? This train of thought was triggered by a tweet by my pal Shack, basically calling BS on one guy’s piece on the value of not trying to be successful. That’s not how Dave rolls.

And that’s fine. Dave is one guy. The dude writing the post is another. What works for that guy clearly would n’t work for Dave. What works for me doesn’t work for you. But what we can’t do is judge it as right or wrong. It’s not my place to tell some guy he needs to strive for success. Nor is it my place to tell the Boss not to get upset about something someone said about something. I would like her not to get upset because when she’s upset it affects me, and it’s all about me. But if she decides to get upset, that’s what she thinks is the right thing to do.

To make this all high concept, a large part of our social problems boil down to one individual’s need to apply their own concept of right to another. Whether it’s religion or politics or parenting or values or anything, everyone thinks they are right. So they ridicule and persecute those who disagree. I’m all for intelligent discord, but at some point you realize you aren’t going to get to common ground. Not without firearms. Trying to shove your right into my backside won’t work very well.

The next time someone does something you think is just wrong, take a step back. Try to put yourself in their shoes and see if there is some way you can rationalize the fact they think it’s right. Maybe you can see it, maybe you can’t. But unless that decision puts you in danger, you just need to let it go. Right? Glad you decided to see it my way (the right way).

– Mike

Photo credits: “wrong way/right way” originally uploaded by undergroundbastard


Recent Securosis Posts

  1. Vaults within Vaults
  2. React Faster and Better: Introduction
  3. Monitoring Up the Stack series
    1. Platform Considerations
    2. Climbing the Stack
  4. Dead or Alive: Pen Testing

Incite 4 U

  1. Verify your plumbing (or end up in brown water) – Daniel Cox of BreakingPoint busts devices for a living. So it’s interesting to read some of his perspectives on what you need to know about your networking gear. Remember, no network, no applications. So if your network is brittle, then your business will be brittle. Spoken by a true plumber, no? There is good stuff there, like understanding what happens during a power cycle and the logging characteristics of the device. The one I like best is #5: Do Not Believe Your Vendor. That’s great advice for any type of purchase. The vendor’s job is to sell you. Your job is to solve a problem. Rarely the twain shall meet, so verify all claims. But only if you want to keep your job, because folks covered in the brown stuff tend to get jettisoned quickly. – MR

  2. It’s new, and it’s old – Adam Shostack’s post Re-architecting the Internet poses a valid question: if we were to build a new Internet from scratch, would it be more secure? I think I have argued both sides of the “need a new Internet” debate at one time or another. Now I am kind of non-plussed on the whole discussion because I believe there won’t be a new Internet, and there won’t be a single Internet. We need to change what we do, but we don’t need a new Internet to do it. There is no reason we cannot continue to use the physical Internet we have and just virtualize the presentation. Much as a virtual server will leverage whatever hardware it has to run different virtual machines, there is no reason we can’t have different virtual Internets running over the same physical infrastructure. We have learned from information centric security that we can encapsulate information into some container, the size of which varies according to the application environment. The playground we play in is defined virtually, and users enter through some application portal with whatever security credentials we care to define. Outside that playground, it’s just encrypted bits of data being routed along with the rest of normal Internet traffic. Of course, the more people you let into your virtual segment of the Internet, the less secure it will be. And just like Adam noted, you can still mess up the new security model, but damage won’t necessarily cascade to everyone’s version of the Internet – just those who screw up. – AL

  3. Ready for the virtual invasion? – If there is a bandwagon, the security marketeers must jump on it. It’s in every product marketing manager’s job description. Of course, the latest bandwagon is virtualization and cloud. The challenge is separating out the folks doing something interesting from those hopping on the bucking bronco. It seems McAfee is taking a different approach to running endpoint AV in virtual machines with their MOVE technology. They have built this not to totally consume 85% of the VM’s processing power and conceptually it’s pretty cool. I haven’t seen it, nor have we spoken to the technical guys to understand more about how it works. But it’s a cool concept. Then you have Fortinet launching a few virtual appliances. Ho hum. Not that they don’t have to do it, but it’s not like this is novel. Though it does give customers the ability to more flexibly provision perimeter protection, and that’s not a bad thing. – MR

  4. Jumpin’ Java threats – Holly Stewart of Microsoft posted Have you checked the Java, an interesting statistical look at the jump in Java exploit attempts over the last 3 quarters. While the malware has focused on three basic exploits, it’s amazing that the number of attacks dwarfed the attempts against PDF vulnerabilities by 10:1. An interesting trend attempting to take advantage of, what Holly aptly calls a blind spot in threat detection systems. We’ll continue to see attackers dig through code looking for more low hanging fruit on what I consider an under-hacked platform before they move on to the next sacrificial goat. You’re looking at one of the reasons we felt the Monitoring up the Stack series was important – to explain why looking at new data sources and leveraging multiple analysis techniques is key. I enjoyed the subtle twist at the end of her post, placing the burden of patching Java squarely at Oracle’s feet. Could a rash of security bugs help Oracle make a political decision to “Free Java”? – AL

  5. Critical infrastructure needs more secure software? Really? – In our Master of the Obvious piece this week, the NERC folks were beating the drum a few weeks back using Stuxnet in an attempt to create urgency for more secure software within critical infrastructure. I’m not disagreeing with the idea, but since lots of control systems are 15-20 years old, do you really think anyone is going to pay to update the software running on those boxes? The code was probably written in hieroglyphics. I should be so cynical (yeah, right), but secure software is a multi-year endeavor (maybe multi-century is more like it, considering their timescales). Won’t the Compliance God come riding in on its white horse to save the day? Just like PCI. Uh no, but you can learn a bit more about NERC CIP to understand that isn’t a panacea either. The real question is what tactically needs to happen to ensure the systems can defend against and recover from the next Stuxnet? Besides us all buying generators and then going all Mad Max on each other. – MR

  6. Get out of the excuses business – Chris Eng nails it here, calling on the carpet all the application developers complaining about how hard it is to get rid of XSS. Chris’s point is that getting rid of XSS isn’t hard (like slaying a dragon), but there are tons of vulnerabilities to stomp out – more like ants. It’s an effective analogy and he even mentions that there will be edge cases that are more akin to slaying the dragon, but they are the exception. It’s been years since I’ve coded much of anything (except some HTML), and Chris acknowledges he’s not a full-time coder either, but threw this thought bomb out there to spur some debate. So debate. What do you think, developers? Oh right, developers don’t read security blogs. They are too busy shipping crappy code. (No offense to any developers who do actually read security blogs. You are unique, but you already knew that.) – MR

  7. Step away from the keyboard… – Normally I’m above throwing mud at my analyst brethren, but I read this totally asinine piece on the Motley Fool and just had to link to it. If only so you can read it too and laugh. Evidently according to Rob Enderle, McAfee is kicking both Symantec and Cisco’s butts. Regardless of whether that is happening or not, it’s Enderle’s ‘reasoning’ that makes me chuckle. Evidently Symantec has all storage sales people who don’t know how to sell security. Hello, Mr. Enderle. It’s 2010, and if anything the SYMC folks can’t sell storage because that part of the business has been dragging them down for the past year. And evidently SideWinder (now called the Enterprise Firewall) is making inroads on Cisco. Uh huh. And in the best bit, Rob goes all thought leader. “Security is in the process of evolving from a reactive technology which, after seeing a threat, responds to it. To a proactive technology that anticipates threats and prepares defenses against them.” Really? Why anyone pays attention to this guy (on security, no less!) is beyond me. Get a frackin’ clue, dude… – MR

—Mike Rothman

Tuesday, October 19, 2010

Vaults within Vaults

By Mike Rothman

My session for the Atlanta BSides conference was about what I expected in 2011. I might as well have thrown a dart at the wall. But the exercise got me thinking about the newest attacks (like Stuxnet) and the realization of how state-sponsored attackers have penetrated our networks with impunity. Clearly we have to shake up the status quo in order to keep up.

This is a point I hit on in last week’s Incite, when discussing Greg Shipley’s post on being outgunned. Obviously what we are doing now isn’t working, and if anything the likelihood of traditional controls such as perimeter defense and anti-malware agents protecting much of anything decreases with every application moved up to the cloud and each trading partner allowed into your corporate network.

The long-term answer is to protect the fundamental element of data. Rich and Adrian (with an assist from Gunnar) are all over that. As what we used to call applications continue to decompose into data, logic, processing, and presentation we have neither control over nor visibility into the data at most points in the cycle. So we are screwed unless we can figure out some way to protect the data regardless of how, where, or by whom it’s going to be used.

But that is going to be a long, long, long, long slog. We don’t even know how to think about tackling the problem, so solutions are probably a decade away, and that’s being optimistic. Unfortunately that’s the wrong answer, because we have the problem now and need to start thinking about what to do. Did I mention we need answers now?

Since I’m the plumber, I took a look into my tool bag and started thinking about what we could do within the constraints of our existing infrastructure, political capital, and knowledge to give us a better chance. This was compounded by the recent disagreement Adrian and I had about how much monitoring is necessary (and feasible) driven by Kindervag’s ideas on Zero Trust.

I always seem to come back to the idea of not a disappearing perimeter, but multiple perimeters. Sorry Jerichonians, but the answer is more effectively segmenting networks with increasingly stringent controls based on the type and sensitivity of data within that domain. Right, this is not a new idea. It’s the idea of trust zones based on type of data. The military has been doing this for years. OK, maybe it isn’t such a great idea… Yes, I’m kidding.

Many folks will say this doesn’t work. It’s just the typical defense in depth rhetoric, which says you need everything you already have, plus this new shiny object to stop the new attack. The problem isn’t with the architecture, it’s with the implementation. We don’t compartmentalize – not even if PCI says to. We run into far too many organizations with flat networks.

From a network ops standpoint, flat networks are certainly a lot easier to deal with than trying to segment networks based on what data can be accessed. But flat networks don’t provide the hierarchy necessary to protect what’s important, and we have to understand that we don’t have the money (or resources) to protect everything. And realize that not everything needs to be protected with the same level of control.

OK Smart Guy, How?

Metaphorically, think about each level of segmented network as a vault. As you climb the stack of data importance, you tighten the controls and make it harder to get to the data (and theoretically harder to compromise), basically implementing another vault within the first. So an attacker going after the crown jewels needs to do more than compromise a vulnerable Windows 2000 Server that someone forgot about to see the targeted assets. Here’s how we do it:

  1. Figure out what’s important: Yes, I’ve been talking about this for years (this is the first step of the Pragmatic CSO).
  2. Find the important data: This is the discover step from all the Pragmatic Data Security research we’ve done.
  3. Assess the data’s importance: This gets back to prioritization and value. How much you can and should spend on protecting the data needs to correlate to how valuable it is, right? You should probably look at 3-4 different levels of data importance/value.
  4. Re-architect your network: This means working with the network guys to figure out how to segment your networks to cordon off each level of increasingly sensitive data.
  5. Add controls: Your existing perimeter defenses are probably fine for the first layer. Then you need to figure out what kind of controls are appropriate for each layer.

More on Controls

Again, the idea of layered controls is old and probably a bit tired. You don’t want a single point of failure. No kidding? But here I’m talking about figuring out what controls are necessary to protect the data, depending on its sensitivity.

For example, maybe you have a call center and those folks have access to private data. Obviously you want that behind more than just the first perimeter, but the reality is that most of the risk is from self-inflicted injury. You know, a rep sending data out inadvertently. Sounds like a situation where DLP would be appropriate.

Next you have some kind of transactional system that drives your business. For the layer, you monitor database and application activity.

Finally you have intellectual property that is the underpinning of your business. This is the most sensitive stuff you have. So it makes sense to lock it down as tightly as possible. Any devices on this network segment are locked down using application whitelisting. You also probably want to implement full network packet capture, so you know exactly what is happening and can watch for naughty behavior.

I’m making this up, but hopefully the idea of implementing different (and more stringent) controls in each network segment makes sense.

None of this is new. As I’m starting to think about my 2011 research agenda, I like this idea of vaults (driven by network segmentation) as a metaphor for infrastructure security. But this isn’t just my show. I’m interested in whether you all think there is merit to this approach, and more importantly the merchandising. Does the concept of a vault make sense? Is it something you’d be able to sell up the stack to your management as an architectural construct?

Let me know (the good, the bad, and the ugly) in the comments.

—Mike Rothman

Monday, October 18, 2010

Incident Response Fundamentals: Introduction

By Mike Rothman

Over the past year, as an industry we have come to realize that we are dealing with different adversaries using different attack techniques with different goals. Yes, the folks looking for financial gain by compromising devices are still out there. But add a well-funded, potentially state-sponsored, persistent and patient adversary to the mix, and we need to draw a new conclusion. Basically, we now must assume our networks and systems are compromised. That is a tough realization, but any other conclusion doesn’t really jive with reality, or at least the reality of everyone we talk to.

For a number of years, we’ve been calling bunk on the concept of “getting ahead of the threat” – most of the things viewed as proactive. Anyone trying to take such action has been disappointed by their ability to stop attacks, regardless of how much money or political capital they expended to drive change. Basing our entire security strategy on the belief that we can stop attacks if we just spend enough, tune enough, or comply enough; is no longer credible – if it ever was. We need to change our definition of success from stopping an attack (which would be nice, but isn’t always practical) to reacting faster and better to attacks, and containing the damage.

We’re not saying you should give up on trying to prevent attacks – but place as much (or more) emphasis on detecting, responding to, and mitigating them. This has been a common theme in Securosis research since the beginning, and now we will document exactly what that means and how to get there.

React Faster

We don’t get a lot of push-back anymore on our position that organizations can’t stop all attacks. From a certain perspective that is progress, and we also believe many security professionals have spent a lot of time managing expectations internally so there is an understanding that perfect security cannot be achieved (or that management is unwilling to fund it and compromise everything else to in favor of security improvements). But following that concept to the next step means we need to get much better at detecting attacks sooner. We have already documented a number of approaches at the network layer in terms of monitoring everything and looking for not normal. They also apply to the application (part 1 & part 2) and database (part 1 & part 2), which we have been talking about in our Monitoring up the Stack series.

So in the first part of this new series, we will talk about the data collection infrastructure you should be thinking about, what kind of organizational model allows you to react faster, and what to do before the attack is detected. If you know you are being attacked, you are already ahead of the vast majority of companies out there. But what then?

And Better

Once you understand you are under attack, then your incident response process needs to kick in. Most organizations do this poorly because they have neither the process nor the skills to figure out what’s happening and do something useful about it. Many organizations have a documented incident response program, but that doesn’t mean it’s effective or that the organization has embraced what it really means to respond to an incident. And this is about much more than just tools and flowcharts. Unless the process is well established and somewhat second nature, it will fail under duress – which is the definition of an incident.

It is also important to remember that this process touches much more than just IT. It must involve other organizations (legal, HR, operational risk, etc.), in order to actually manage or mitigate the organizational risk of any attack. One of the things that Rich’s emergency response experience has shown is that chain of command is critical; and everyone must be in alignment on process, responsibilities, and accountabilities; before the incident happens. Again, a lot of this stuff seems like common sense (and it is!), but we have seen few organizations that do this well, so we’ll walk through what we mean by reacting better throughout the series.

Before, During, and After

The concept we will come back to throughout this series is before, during, and after the attack. This will provide context for the different things that must happen based on where you are within the attack lifecycle.

  • Before: Figure out what data to monitor, how much of it is useful, how to make use of it, and how long to retain it, is key to building the infrastructure for persistent monitoring. This must happen before the attack, because you only get one chance to collect that data, when things are happening. You don’t get to go back and record it after the fact (unless you completely fail to learn from the first attack, and they hit you again – not a good way to get a second chance!).
  • During: How can you contain the damage as quickly as possible? By identifying root cause accurately and remediating effectively. We’ll dig into how to identify the attack, who to work with to provide the data you need, and how to do this in the heat of battle.
  • After: Once the attack has been contained, focus shifts to making sure it doesn’t happen again. In these posts we’ll discuss the forensics process, and necessary tools and skills – as well as how to maintain chain of custody and the post mortem required to learn something from a difficult situation.

We’ll also discuss the current state of threat management tools, including SIEM, IDS/IPS, and network packet capture, to define their place in our approach. Finally we consider how network security is evolving and what kind of architectural constructs you should be thinking about as you revisit your data collection and defensive strategies.

At the end of this series you will have a good overview of how to deal with all sorts of threats and a high level process for identifying the issues, containing the damage, and using the feedback loop to ensure you don’t make the same mistakes again. That’s the plan, anyway.

—Mike Rothman

Monitoring up the Stack: Climbing the Stack

By Adrian Lane

As we have discussed through this series, monitoring additional data types can extend the capabilities of SIEM in a number of different ways. But you have lots of options for which direction to go. So the real question is: where do you start? Clearly you are not going to start monitoring all of these data types at once, particularly because most forms require some integration work on your part – often a great deal. Honestly, there are no hard and fast answers on where to start, or what type of monitoring is most important. Those decisions must be based on your specific requirements and objectives. But we can describe a couple common approaches for climbing the monitoring stack.

Get more from SIEM

The first path we’ll describe involves organizations simply looking to do more with what they have, squeezing additional value from the SIEM system they already own. They start by collecting data on the existing monitoring systems already in place, where they already have the data or the ability to easily get it. From there they add capabilities in order, from easiest to hardest. Usually that means file integrity monitoring first. From the standpoint of additional monitoring capabilities, file integrity is a bit of a standalone feature, but critical because most attacks have some impact on critical system files and so can be detected by monitoring file integrity. Next comes identity monitoring – most SIEM platforms coordinate with server/desktop operations management systems, so this capability is relatively straightforward to add. Why do this? Identity monitoring systems include audit capabilities which provide events to SIEM in order to audit access control system activity, and to map local events back to domain identities.

From there it’s a logical progression to add to user activity monitoring. You leverage the combination of SIEM functions and identity monitoring data against a bunch of new rules and dashboards implemented to track user activity. As sophistication increases, 3rd party web security, endpoint agents, and content analysis tools can provide additional data to fill out a comprehensive view of user activity.

Once those activities are mastered, these organizations tackle database and application monitoring. These two data types overlap less in terms of analysis and data collection techniques, provide more specialized analysis, and address detection of a different class of attack. Their implementations also tend to be the most resource intensive, so without a specific catalyst to drive implementation they tend to fall to the bottom of the list.

Responding to Threats

In the second post in this series, we outlined many of the threats that prompt IT organizations to consider monitoring: malware, SQL injection, and other types of system misuse. If managing these threats is the catalyst to extend your monitoring infrastructure, the progression of what data types to add will depend entirely on which attacks you need address. If you’re interested in stopping web attacks, you’ll likely start with application monitoring, followed by database activity and identity monitoring. Malware detection will drive you towards file integrity monitoring initially, and then probably to identity and user activity monitoring, because bad behavior on behalf of users can indicate a malware outbreak. If you want to detect botnets, user activity monitoring and identity monitoring are a good start.

Your data type priorities will be driven by what you want to detect, based on the greatest risk you perceive to your organization. Though it’s a bit beyond the scope of this research project, we are big fans of threat modeling because it provides structure for what you need to worry about and how to defend against it. With a threat model – even on the back of an envelope – you can map the threats to information your SIEM already provides, and then decide which supplementary add-on functions are necessary to detect attacks.

Privileged Users

One area we tend to forget is the folks who hold the keys to the kingdom. Yes, administrators and other folks who hold privileged access to the resources that drive your organization. This is also a favorite for the auditors out there – perhaps something to do with low hanging fruit – but we see a lot of folks look to advanced monitoring to address an audit deficiency. So to monitor activity on the part of your privileged users, you’ll move towards identity and user activity monitoring first. These data types allow you to identify who is doing what, and where, to detect malfeasance.

From there you add file integrity monitoring – changing system files is an easy way for someone with access to make sure they can maintain it, and also to hide their trail. Database monitoring would then come next, as users changing database access roles can indicate something amiss. The point here is you’ve probably been doing security far too long to trust anyone, and enhanced monitoring can provide the data you need to understand what those insiders are really doing on your key systems.

Political Land Mines

Any time new technologies are introduced, someone has to do the work. Monitoring up the Stack is no different, and perhaps a bit harder because it crosses multiple fiefdoms organizations and requires consensus, which translates roughly to politics. And politics means you can’t get anything done without cooperation from your coworkers. We can’t stress this enough: many good projects die not because of need, budget, or technology, but due to a lack of interdepartmental cooperation. And why not? Most of the time the people who need the data – or even fund the project – are not the folks who have to manage things on a day to day basis.

As an example, DAM installation and maintenance falls on the shoulders of database administrators. All they see is more work. Not only do they have to install the product, but they get blamed for any performance and reliability issues it causes. Pouring more salt into the wound, the DAM system is designed to monitor database administrators! Not only is the DBA’s job now harder because they can’t use their favorite shortcuts, but now someone’s looking over their shoulder and asking annoying questions – and demanding their own help to step up the scrutiny! Very quickly, DBAs tend to find ways to scuttle the project as technically infeasible. This is just one example – application and user activity monitors are usually subverted by being labelled as destabilizing the business or being Big Brother.

How can you address these problems? We recommend paving the way for enhanced monitoring internally before you start. Not only sort out who will pay for it, but lay the groundwork for how automation will make the job easier in the long run. Paint these new capabilities as making everyone’s job easier and increasing security. Explain that if rules and reports are automated, the audit staff won’t be knocking on your cubicle wall asking for a whole bunch of stuff every week. Seriously, regulatory requirements don’t just go away, and making it a team effort and giving each stakeholder a say in the process goes a long way. Ultimately you need to sell it as a win/win. Or watch your monitoring project get pulled under by the weight of organizational politics.

Conclusion

Many organizations have embraced SIEM as a way to understand what is happening in their environments and be alerted to potential attacks before catastrophic loss. Whether the catalyst was compliance or a successful attack, those organizations can clearly improve their ability to deal with the myriad of attacks happening each day with this investment – in software, resources, and process.

SIEM has historically focused on analyzing traditional infrastructure devices, such as network and security devices. But as the technology platforms mature and the attack space evolves, many organizations are now looking to extend the reach of their SIEM beyond infrastructure and focus on where most of the attacks happen: up the Stack.

In this series we have focused on identifying the threats we now face and how advanced monitoring techniques including file integrity monitoring, database activity monitoring (part 1 & part 2), application monitoring (part 1 & part 2), identity monitoring, and user activity monitoring can improve an organization’s security posture. We have also discussed how these additional data types impact the SIEM platform and in this post where to get started.

But all these issues kind hint at the main point of the entire series: the need for context. You get alerts all day, every day, from pretty much all your devices. But you don’t have the time or the resources to really understand which attacks are real, which are imagined, and which present a clear and present danger to critical data. A “monitor everything” approach is far from a panacea, but it does provide you with a foundation of data you can use to find the answer. And with a decent amount of elbow grease invested into alerting rules to detect anomalous behavior in your environment, you can get better utilization of your scarce resources and positively impact your security posture.

And best of all, if you’ve already embraced SIEM for network and security monitoring, you are more than halfway there. Now it’s just a matter of taking that next step and we aren’t religious about which advanced monitoring direction you take. Only that you keep moving. You know the bad guys are – for what that’s worth.

—Adrian Lane

Friday, October 15, 2010

New Blog Series: Incident Response Fundamentals

By Mike Rothman

Our “beat our readers into a content coma” plan is working perfectly. Just when you thought you had enough of NSO Quant, Enterprise Firewall, Monitoring up the Stack, and DLP (just in the last month) – we will be starting another series Monday. Rich and I will begin the “Incident Response Fundamentals: Understanding Threats Before, During, and After the Attack” series. React Faster is something I’ve been talking about for years (literally) and Rich improved it by integrating the importance of incident response to the mix. Now we are going to bring all those aspects together into a very focused view on how you can keep pace with the rapidly evolving attack space.

The general thesis of the series is:

Organizations need to embrace a pervasive monitoring approach to track attacks before, during, and after the threat. Far too many organizations do not capture the proper data at the network layer to detect attacks, find the root cause and remediate, or perform a detailed forensic analysis after the fact. This impairs their ability to protect their environments and ensure they don’t suffer similar breaches over and over again.

We will not only talk about monitoring (as much as Adrian loves that), but also about an incident response plan and what to do before the attack, once you think something is going down, and (from a forensics standpoint) after the fact. We’ll also do a little bit of visioning and take a cut at what network security will look like in 5 years. Overall it will be a great research project and we think the output will be very valuable to practitioners. Which is why we do this stuff.

—Mike Rothman

Monitoring up the Stack: Platform Considerations

By Adrian Lane

So far in the Monitoring up the Stack series, we have focused on a number of additional data types and analysis techniques that extend security monitoring to gain a deeper and better perspective of what’s happening. We have been looking at the added value that is all good, but we all know there is no free lunch. So now let’s look at some of the problems, challenges, and extra work that come along with deeper monitoring goodness. We know most of you who have labored with scalability and configuration challenges with your SIEM product were waiting for the proverbial other shoe to drop. Each new data type and the associated analysis impact the platform. So in this post we will discuss some of these considerations and think a bit about how to work around the potential issues.

To be fair, it’s not all bad news. Some additional data sources are already integrated with the SIEM (as in the case of identity and database activity monitoring), minimizing deployment concerns. However, most options for application, database, user, and file monitoring are not offered as fully integrated features. Monitoring products sometimes need to be set up in parallel – yep, that means another product to deploy, configure, and manage. You’ll configure the separate monitor to feed some combination of events, configuration details, and/or alerts to the SIEM platform – but the integration likely stops there. And each type of monitoring we have discussed has its own idiosyncrasy and/or special deployment requirement, so the blade cuts both ways. To add hard-to-get data and real-time analysis for these additional data sources comes at a cost. But what fun would it be if everything was standardized and worked out of the box? So you know what you’re getting yourself into, the following is a checklist of platform issues to consider when adding these additional data types to your monitoring capabilities.

  • Scalability: When adding monitoring capabilities, integrated or standalone, you need additional processing power. SIEM solutions offer distributed models to leverage multi-tier or multi-platform deployments which may provide the horsepower to process additional data types. You may need to reconfigure your collection and/or analysis architecture to redistribute compute power for these added capabilities. Alternatively, many application and/or database monitoring approaches utilize software agents on the target platform. In some cases this is to access data otherwise not available, or to remove network latency from analysis response times, as well as to distribute the processing load across the organization. Of course there is a downside to agents: overhead and memory consumption could impact the target platform, as well as the normal installation & management headaches. The point is that you need to be aware of the extra work being performed and where, and you will need to absorb that requirement on the target platforms or add horsepower to the SIEM system. Regardless of the deployment model you choose, you will need additional storage to accomodate the extra data collected. You may already be monitoring some application events through syslog, but transaction history can increase event volume per application by an order of magnitude. All monitoring platforms can be set to filter out events by policy, but filtering too much defeats the purpose of monitoring these other sources in the first place.

  • Integration: There are three principle integration points to consider. The first is how to get data into the SIEM and integrated with other event types, and second is how to configure the monitors regarding what to look for. Fully integrated SIEM systems account for both policy management and normalization / correlation of events. While you may need to alter some of your correlation rules and reports to take advantage of these new data types, it can all be performed from a single management console. Standalone monitoring systems can easily be configured to send events, configuration settings, and alerts directly to a SIEM, or drop the data into files for batch processing. SIEM platforms are adept at handling data from heterogenous sources so you just change the correlation, event filtering, and data retention rules to account for the additional data. The second – and most challenging – part of integration is sharing policies & reports between the two systems (SIEM and standalone monitor). Keep in mind that things like configuration analysis, behavioral monitoring, and file integrity monitoring all work by comparing current results against reference values. Unlike hard-coded attribute comparisons in most SIEM platforms, these reference values change over time (by definition). Policies need to be flexible enough to handle these dynamic values so if your SIEM platform can’t you’ll need to use the monitoring platform’s interface for policies, reporting, and data management. We see that with most of the Database Activity Monitoring platforms, where the SIEM is not flexible enough to alert properly. Thus customers need to maintain separate rule bases in the two products. Whenever a rule changes on either side, this disconnection requires manual verification that settings remain consistent between the two platforms. Some monitoring tools have import and export features so you can create a master policy set for all servers, and provide policy reports that detail which rules are active for audit purposes. The third point to consider is that most monitoring systems leverage smart agents, with agent deployment and maintenance managed from the console. Most SIEM platforms leverage a web-based management platform which facilitates central location management, or even the merging of consoles. Many standalone monitoring systems for content, file integrity, and web application monitoring are Windows-specific applications that can’t easily be merged and must be managed as standalone applications.

  • Analysis: Each new data type needs its own set of analysis policies, alerting rules, dashboards, and reports. This is really where the bulk of the effort is spent – to make these broader data sources available and effective. It’s not just that we have new types of data being collected – the flexibility of flat-file event storage used within SIEM products adapts readily enough – but that monitoring tools should leverage more than merely attribute analysis. To detect SQL injection attacks, data exfiltration, or even something as simple as spam, we need to do more with the data we have. Content analysis, behavioral analysis, and contextual analysis – three of the most common options – look at the same events differently. The SIEM platform must have the flexibility to incorporate these analysis techniques, either as part of the remote data collectors, or as add-on functions within the platform. Lower-end platforms won’t have this and probably don’t need to, but leveraging these additional monitoring capabilities within SIEM requires an architecture flexible enough to incorporate different analytics engines. When we refer to the SIEM platform this is what we are talking about. It’s basically an analysis engine, and must be flexible enough to take lots of data and provide multi-variate correlation and alerting.

  • Other considerations: Application monitors are more likely to intercept sensitive data, as they dig around in places built-in SIEM collectors don’t look. You may not care about the privacy of syslog data over the network but that won’t fly for application, database, or identity traffic. You need to secure application requests and database queries because some of this information is private and therefore protected by any number of regulatory hierarchies. SIEM securely stores data once it has collected it, and offers the option to encrypt stored data (with a performance cost). If you need to encrypt the event stream as it is routed to the SIEM platform, you’ll need to set up the platform – or the collector software itself – to secure data transmissions. Collection architecture also needs to account for the intended use case – for instance application and database monitors used to block activity or perform virtual patching must be deployed “in front” of the platform they monitor. And to monitor web applications in the DMZ the collectors must support different network addressing and tunnel data between the collector and SIEM; separately, you might need to alter your network topology.

But as we have discussed throughout this series, there are plenty of reasons to extend monitoring beyond the traditional security and network devices. With the growing popularity of application and database attacks you cannot afford not to monitor these additional data sources. So at least go into the project with your eyes open as to how these additional data types will impact your existing monitoring infrastructure.

That concludes the technical details of this series. If you feel we left anything out that should be discussed within this series, just let us know in the comments. We’ll wrap the series next week by introducing a phased approach to adding these additional data types to address the threats we talked about in the threats post.

—Adrian Lane

Thursday, October 14, 2010

Dead or Alive: Pen Testing

By Mike Rothman

Remember the dead or alive game Howard Stern used to do? I think it was Stern. Not sure if he’s still doing it because I’m too cheap to subscribe to Sirius for the total of 5 minutes I spend in the car driving between coffee shops. Pen testing has been under fire lately. Ranum has been talking for years about how pen testing sucks. Brian Chess also called pen testing dead at the end of 2008.

It’s almost two years later and the death of pen testing has been greatly exaggerated. Pen testing is not dead. Not by a long shot. But it is changing. And we have plenty of folks weighing in on how this evolution is taking place.

First off is the mouth from the South, Dave Maynor. OK, one of the mouths from the South, because I suspect I am another. Dave made some waves regarding whether to use 0-day exploits in a pen test, and then had to respond when everyone started calling him names. Here’s the thing. Dave is right. The bad guys don’t take an oath when they graduate from bad guy school that they won’t use 0-days. They can and do, and you need to know how you’ll respond. Whether it’s part of a pen test or incident response exercise doesn’t matter to me. But if you think you don’t need to understand how you’ll respond under fire, you are wrong.

Second, I got to attend a great session by Dave Kennedy and Eric Smith at BSides Atlanta about strategic pen testing. It was presented from the viewpoint of the pen tester, but you can apply a lot of those lessons to how a practitioner runs a pen test in their organization. First off, a pen test is about learning where you can be exploited. If you think it’s about checking a box (for an audit) or making yourself and your team look good, you’ve missed the point. These guys will break your stuff. The question is what can you learn and how will that change your defensive strategies?

The pen testers need to operate in a reasonable semblance of a real wold scenario. Obviously you don’t want them taking down your production network. But you can’t put them in a box either. The point is to learn and unless their charter is broad enough to make a difference, again you are wasting your time.

Finally, I’ll point to a presentation by Josh Abraham, talking about his “Goal Oriented Pentesting” (PDF) approach. It’s good stuff. Stuff you should know, but probably don’t do.

What do all these things have in common? They talk about the need for pen testing to evolve. But by no means are they talking about its death. Listen – at the end of the day, whether you are surprised by what an attacker does to your network is your business.

I still believe pen testing can provide insights you can’t get any other way. I think those insights are critical to understanding your security posture. Those enlightened organizations whihc don’t pen test do so at their own risk. And the rest of us should thank them – they are the slow gazelles and the lions are hungry.

—Mike Rothman

Wednesday, October 13, 2010

Incite 10/13/2010: the Rise of the Cons

By Mike Rothman

No we aren’t going to talk about jailbreaks or other penal system trials and tribulations. This one is about how the conference circuit is evolving in a really positive way. Most folks attend the big security shows – you know, RSA and BlackHat and maybe some others. Most folks also hate these shows. I hear a lot of complaints about weak content and vendor whoring putting a damper on the experience. Of course, since myself and my ilk tend to speak at most of these shows, we can only point the finger at ourselves. Personally, unless I’m speaking I tend to skip all but the biggest shows, which I attend for networking purposes. But that’s just me.

Now that is a con for you... But nature hates a vacuum, and the vacuum of user-oriented conferences is being filled by the BSides movement and a number of regional hacker cons. If the conference you are attending doesn’t do it for you, get some smart folks together (who are there anyway) and put on an unconference of your own. That’s the general concept for BSides. I attended BSides ATL last week, and it was a really great experience. First shout outs need to be sent to the driving forces bringing BSides to ATL, and they were Eric Smith (@infosecmafia), Nick Owen (@wikidsystems), Marisa Fagan (@dewzi) and MC Petermann (@petermannmc). I know there were tons of other folks who put a lot of blood, sweat, and tears into making BSides ATL happen, and no offense to anyone I didn’t mention. I can’t thank them all enough.

Why is this working? Because it’s about community. I’ve been in Atlanta for over 6 years now, and there isn’t really a cohesive security community. The ISSA meetings are a joke, unless you like vendors to hump your leg for 2 hours every month. We tried to get a CitySec group meeting going (and all three of us who attended enjoyed the beer that I bought), but that fizzled. A new Cloud Security Alliance chapter is forming in the ATL and we are seeing a lot of activity for the NAISG in town as well. Yes, there are other organizations, but it’s generally a small group of folks getting together in an ad hoc fashion.

But what’s been missing has been a more technically oriented conference, where smart folks from the Southeast can get together and share what we are seeing. That happened in spades at BSides ATL. Whether talking Google and Bing hacking with Rob Ragan, exfiltration with Dave Shackleford and Rick Hayes, pen testing with Eric Smith and Dave Kennedy, or having Chris Nickerson show how to bring entire companies down (think attacking robots!) – it was just a flood of information. Good information. And those were just the sessions I attended. There were a bunch of others I had to miss.

The conference organizers even let me play and talk about what I think will happen in 2011. The short answer is I have no idea. But you already knew that. Yet I did get to use a picture of a guinea pig BBQ, which has to set some low bar for depravity.

I’m probably going to get in trouble by talking up BSides because we Securosis folks do a lot of work with the RSA Conference. Next year, we’ll be leading the E10 (CISO-focused) event on Monday at RSA, and Rich is in London and will be in China this month speaking at RSA’s global events. But the writing is on the wall. Content is king, and right now there is a lot of great content being driven through the regional BSides conferences and the other hacker cons.

While I’m talking conferences, I also should mention what seemed to be a rousing success for Hoff and friends at the inaugural HacKid conference in Boston last weekend. It’s such a great concept, to teach kids about security, self-defense and other important topics. I can’t wait to get this going in ATL. And with that, just remember – if you don’t take care of your customers someone else will. Mr. Market told me.

– Mike.

Photo credits: “Pug Shot” originally uploaded by Jerry Reynolds


Recent Securosis Posts

  1. IT Debt: Real or FUD?
  2. FireStarter: Consumer Internet Penalty Box
  3. Friday Summary: October 8, 2010
  4. Monitoring up the Stack:

I should also highlight an article on Application Monitoring in Dark Reading that highlights the Monitoring up the Stack research Adrian and Gunnar are working on right now. I know lots of folks have a hard enough time monitoring their network and security devices, but the application is where the action is, so ignore it at your own peril.


Incite 4 U

  1. Time for the heavy artillery. What heavy artillery? – Greg Shipley makes the point we’ve all come to grips with. We are outgunned. The bad guys have better tools and more motivation, and all we can do is watch it happen and clean up the mess afterwards. This statement kind of says it all: “Recent events suggest that we are at a tipping point, and the need to reassess and adapt has never been greater. That starts with facing some hard truths and a willingness to change the status quo.” Right. So all is not lost, but we need to start thinking differently. But what does that mean? According to Shipley, it’s focusing on the database and maybe things like application white listing. Best of all is the idea to “stop rewarding ineffectiveness and start rewarding innovation.” Bravo. But how do you do that when the checkbox says you need AV? So basically we are in a quandary, but you already knew that. What to do? Basically what we’ve been saying for years. React Faster (and Better), focus on the fundamentals, and if you are targeted, just understand you can’t stop them. And manage expectations accordingly. He closes the article with “If we remain bound to our relentless commitment to mediocrity, we will be worse off moving ahead. We can and must do better. It’s time to change our way of thinking.” Right. – MR

  2. Instructive memory – Ever had something running on your machine and you wanted to know “what the hell this that”? Not for your day to day IT practitioner, but for those of you who wanted to know more about forensic analysis, Lenny Zeltser’s 3 phases of Malware Analysis is a good place to start. Watching application behavior is a good way to get a handle on an applications ‘intentions’ and what the code is trying to do. From there, disassembling the code can help you understand the threat and – hopefully – how to stop it. But some malware can’t be understood through behavioral monitoring, and reverse engineering does not fully disclose code segments. Examining the runtime execution of a program provides a different view – helpful in some cases, and necessary when it comes to memory-resident-only malware. This analysis is a little more difficult – it requires some understanding of operating system functions and memory layout. But for detection of memory-only malware and rootkits, it’s your only option. Lenny’s post mentions some tools and resources so you can try it out for yourself. – AL

  3. Will you end up under the security bus? – Great post here from Adam Ely about who should be making the decisions in your organization. Unfortunately fat, dumb, and/or lazy security folks all too often let vendors dictate what they do and don’t do. Or some analyst quadrant or product review. Adam’s point is that he gets paid to prioritize between all the things he could do and focus on the things he needs to do. I could talk more about what is says, but just go read the post. Now. – MR

  4. Stop. Think. Get Pwned. – Given that it’s cyber-security awareness month, it’s not surprising to see the Department of Homeland Security get into the fray with its awareness campaign called Stop. Think. Connect. Their focus is on getting Americans to take responsibility for what they do online and also tell their friends. How? By putting money into some fora I’ve never heard of which will hold town-hall meetings. That will be pretty effective. Bah, cynicism is getting the better of me. Personally I think DHS should just give a bunch of money to Facebook to educate their users. I know that wouldn’t work, for lots of reasons, but I can tell you the folks who are getting pwned are on Facebook a lot, and probably not going to attend a town hall meeting. Even if they provide coffee and donuts. – MR

  5. Mobile strategy, meet rotary oscillator – I get a chuckle out of articles like Christina Torode’s post on CIOs feverishly working on mobile strategy. Mainly because it implies that there is planning and preparation. Usually “working on a mobile strategy” is not a proactive effort, but is prompted by a dozen executives and half the sales teams getting iPhones and implying the CIO’s job security is suspect if he/she can’t support them. It’s not “How do we harness mobile devices for our business?” but “How can my infrastructure support mobile apps and not break the other applications and my entire security model?” It’s not “How can IT help or hinder the use of mobile devices and applications?” but “Where do I start my investigation?” It’s almost like you need to break other services, applications, or protocols in order to leverage the mobile apps, and it’s these trade-offs that cause CIOs to lose sleep at night. – AL

  6. The return of Free Public WiFi – A story I keep seeing across mass media is the re-emergence of Free Public WiFi. You know, the ad hoc network that shows up on XP machines that haven’t been patched in, I don’t know, maybe 3 years. Right, Microsoft fixed that years ago and I’m not sure what makes this newsworthy now. The point, first of all, is patch your damn machines. If you do a NetStumbler scan and find “Free Public WiFi,” take that person’s machine away. They are clearly not competent to use a computer. Next, teach your damn users not to connect to any old WiFi network. It just goes to show old attacks never die, so old controls can’t die either. Guess that’s why Jaquith called it the Hamster Wheel of Pain. BTW, 5 years later, Andy’s post is still unbelievably on point. – MR

  7. Why SEO is the end of civilization… – Aside from Rob Ragan’s demonstration of how negative SEO can torpedo competition, check out the title of this press release from McAfee. “McAfee, Inc. Security Management Platform Provides Businesses with Proactive Risk Management” What? Clearly this is keyword optimization run amok. Of course, I scoff at the idea of anything proactive in security. Give me a break. And are they talking about security or risk? And don’t bother looking at the release – it’s about as clear as mud. Are they talking about ePO? Or is the Security Management Platform something else? Come to think of it, I don’t care. I just wanted to point out security marketing at pretty close to its absolute worst. – MR

  8. Two more marshmallow deals… – We’ve talked all year about more and more consolidation in the security market. Two more examples hit this week. First off Nitro raised some money and also acquired LogMatrix. Who? They used to be called OpenService and they’ve been the walking dead for the last few years. Guess Nitro got a handful of customers in the deal. And HID Global acquired ActivIdentity. Right – the proximity card, I mean physical security folks. Actually this deal makes sense, since ActivIdentity always did well issuing credentials for universal ID cards. The deal was for $162 million, a 48% premium over the 20-day average, but keep in mind that ActivIdentity had over $90 million in cash. So the deal is for maybe 1.2x sales. Anyone have some marshmallows? I think we are going to see a few more fire sales before we close the books on 2010.

—Mike Rothman

Tuesday, October 12, 2010

IT Debt: Real or FUD?

By Adrian Lane

I just ran across Slashdot’s mention of the Measuring and Monitoring Technical Debt study funded by a research grant. Their basic conclusion is that a failure to modernize software is a form of debt obligation, and companies ultimately must pay off that debt moving forward. And until the modernization process happens, software degrades towards obsolescence or failure.

From Andy Kyte at Gartner:

“The issue is not just that maintenance keeps on getting deferred, it is that the lack of an application inventory and the absence of a structured review process for the application portfolio. This means the IT management team is simply never aware of the true scale of the problem,” Mr. Kyte said. “This problem, hidden from sight, is getting bigger every year and more difficult to deal with every year.”

I am on the fence on the research position – apparently others are as well – and I disagree with many of the assertions because the cost of inaction needs to be weighed against the cost of overhauls. The cost of migration is significant. Retraining users. Retraining IT. New software and maintenance contracts. The necessary testing environments and regression tests. The custom code that needs to be developed in order to work with the software packages. Third party consulting agreements. New workflow and management system integration. Fixing bugs introduced with the new code. And so on.

In 2008, 60% of the clients of my former firm were running on Oracle & IBM versions that were 10 years old – or older. They stayed on those version because the databases and applications worked. The business functions operated exactly as they needed them to – after 2-5 years of tweaking them to get them exactly right. A new migration was considered to be another 2-5 year process. So many firms selected bolt-on, perimeter-based security products because there was no way to build security into a platform in pure maintenance mode. And they were fine with that, as the business application was designed to a specification that did not account for changes to the security landscape, and depended on network and platform isolation. But the primary system function it was designed for worked, so overhaul was a non-starter.

Yes, the cost of new features and bug fixes on very old software, if needed, was steep. But that’s just it … there were very few features and bug fixes needed. The specifications for business processing were static. Configuration and maintenance costs we at a bare minimum. The biggest reason why “The bulk of the budget cut has fallen disproportionately on maintenance activities –” was because they were not paying for new software and maintenance contracts! Added complexity would have come with new software, rather than keeping the status quo. The biggest motivator to upgrade was that older hardware/OS platforms was either too slow, or began failing. A dozen or so financial firms I spoke with performed this cost analysis and felt that every day they did not upgrade saved them money. It was only in segments that required rapid changes to meet changing market – retail and shipping come to mind – that corporations benefitted from modernization and new functionality to improve customer experience.

I’ll be interested to see if this study sways IT organizations to modernize. The “deferred maintenance” message may resonate with some firms, but calling older software a liability is pure FUD. What I hope the study does is prompt firms to compare their current maintenance costs against upgrades and new maintenance – the only meaningful must be performed within a customer environment. That way they can intelligently plan upgrades when appropriate, and be aware of the costs in advance. You can bet every sales organization in the country will be delivering a copy of this research paper to their customers in order to poke and prod them into spending more money.

—Adrian Lane

FireStarter: Consumer Internet Penalty Box

By Mike Rothman

A few weeks back, the fine folks at Microsoft used a healthcare analogy to describe a possible solution to the Internet’s bot infestation. Scott Charney suggested that every PC should have a health certificate which would provide access to the Internet. No health certificate, no access. Kind of like a penalty box for consumer Internet users. It’s an interesting idea, and clearly we need some kind of solution to the reality that Aunt Bessie has no idea her machine has been pwned and is blasting spam and launching DDoS attacks.

Unfortunately it won’t work, unless mandated by some kind of regulation. It’s really an economic thing. Comcast will proactively send devices connected to their network exhibiting bad behavior a message telling them they are likely compromised. They call it their Bot Alert program. Then they point to a nice web page where the consumer can get answers. The consumer is then expected to address the issue. If they can’t (or don’t) Comcast will continue to notify the customer until they do. Here’s the rub: if the consumer knew what they were doing in the first place, they wouldn’t have gotten pwned.

You can’t blame Comcast (or any other ISP) for drawing a line in the sand. They charge maybe $40 a month for Internet service. The minute a customer picks up the phone and calls for help, they lose money for that month. There is no financial incentive for them to try to fix the compromised device. Sure, a bot does bad things. But bad enough to spend staff time trying to fix every one of them? The constant notifications will definitely push a customer to call and force Comcast to help them address the issue. I guess that worked OK in their pilot test, but we’ll see how well it scales as they roll it out nationwide.

And Comcast seems to be out in front on this issue. I’m not familiar with any similar initiatives from the other major ISPs. So let’s tip our hat to Comcast for at least trying to do something. But is it the right approach?

Do we just accept the fact that a percentage of consumer devices will be pwned and will exhibit bad behavior. Is it a cost of doing business for the ISPs? Is there some other kind of technical, procedural, or cultural answer?

I wish I knew. What do you folks think? Can this health certificate thing work? Am I just stuck in a cycle of cynicism that prevents me from seeing any solution to this problem? Or do we just make sure our families aren’t the path of least resistance and forget the rest?

—Mike Rothman

Monday, October 11, 2010

Monitoring up the Stack: User Activity Monitoring

By Gunnar

The previous Monitoring up the Stack post examined Identity Monitoring, which is a set of processes to monitor events around provisioning and managing accounts. The Identity Monitor is typically blind to one very important aspect of accounts: how they are used at runtime. So you know who the user is, but not what they are doing. User Activity Monitoring addresses this gap through reporting not on how the accounts were created and updated in the directory, but by examining user actions on systems and applications, and linking them to assigned roles.

Implementing User Activity Monitoring

User Activity Monitors can be deployed to monitor access patterns and system usage. The collected data regarding how the system is being used, and by who, is then sent to the SIEM/Log Management system. This gives the SIEM/Log Management system data that is particularly helpful for attribution purposes. Implementing User Activity Monitoring rests on four key decisions. First, what constitutes a user? Next, what activities are worth monitoring? Third, what does typical activity look like, and how do we define policies to scope acceptable use? And finally, where and how should the monitor be deployed?

The question about what constitutes a user seems simple, and on one level it is. Most likely a user is an account in the corporate or customer directory, such as Active Directory or LDAP. But sometimes there are accounts for non-human system users, such as service accounts and machine accounts. In many systems service accounts, machine accounts, and other forms of automated batch processing can do just as much damage as any other account/function. After all, these features were programmed and configured by humans, and are subject to misuse like any other accounts, so likely are worth monitoring as well.

Drilling down further into users, how are they identified? To start with, there is probably a username. But remember the data that the User Activity Monitor sends to the SIEM/Log Management system is will be used after the fact. What user data will help a security analyst understand the user’s actions and whether they were malicious or harmful? Several data elements are useful for building a meaningful user record:

  • Username: The basic identifier for a user in the system, including the namespace or other protocol-specific data.
  • Identity Provider: The name of the directory or database that authenticated the user.
  • Group/Role Membership: Any group or role information assigned to the user account, or other data used for authorization purposes.
  • Attributes: Was the user account assigned any privileges or capabilities? Are there time of day or location attributes that are important for verifying user authenticity?
  • Authentication Information: If available, information around how the user was authenticated can be helpful. Was the user dialed in from a remote location? Did they log in from the office? When did they log in? And so on.

A log entry that reads user=rajpatel; is far less useful than one that contains “user=rajpatel; identityprovider=ExternalCORPLDAP; Group=Admin; Authenticated=OTP”. The more detailed the information around the user and their credential, the more precsion the analyst has to work with. Usually this data is easy to get at runtime – it is available in security tokens such as SAML and Kerberos – but the monitor must be configured to collect it.

Now that we see how to identify a user, what activities are of interest to the SIEM/Log Management system? The types of activities mentioned in other Monitoring up the Stack posts can all be enriched through the user data model described above; in addition there are some user-specific events worth tracking, including:

  • User Session Activities: events that create, use, and terminate sessions; such as login and logout events.
  • Security Token Activities: events that issue, validate, exchange and terminate security tokens.
  • System Activities: events based around system exceptions, startups, shutdowns, and availability issues.
  • Platform Activities: events from specific ports or interfaces, such as USB drive access.
  • Inter-Application Activities: events performed by more than one application on behalf of the user, all linked to the same business function.

Now that we know what kind of events that we are looking for, what do we want to do with these events? If we are monitoring we need to specify policies to define appropriate use, and what should be done when an event – or in some cases a series of events – occurs. Policy set up and administration is a giant hurdle with SIEM systems today, and adding user activity monitoring – or any other form of monitoring – will require the same time to set up and adjust over time. Based on an event type listed above, you select the behavior type you want to monitor and define what users can & cannot do. User monitoring systems, at minimum, offer attribute-based analysis. More advanced systems offer heuristics and behavioral analysis; these provide flexibility in how users are monitored, and reduce false positives as the analysis adapts to user actions over time.

The final step is deployment of the User Activity Monitor; and the logical place to start is the Identity repository because repositories can write auditable log events when they issue, validate, and terminate sessions and security tokens; thus the Identity repository can report to the SIEM/Log Management system on what users were issued what sessions and tokens. This location can be made more valuable by adding User Activity Monitors closer to the monitored resources, such as Web Application Firewalls and Web Access Managers. These systems can enhance visibility beyond simply what tokens and sessions were issued (from the Identity repository), adding information on how were they used and what the user accessed.

Correlation: Putting the Data to Work

With monitors situated to report on User Activity, the next step is to use the data. The data and event models described above provide an enriched model that enables the analyst to trace events back upstream. For example, the analyst can set up rules that identify known good and bad behavior patterns to reflect authorized usage and potentially malicious patterns.

Authorized usage patterns generally reflect the use case flows that users follow. In most cases these do not trigger alarms; for example a failed authentication is not necessarily suspicious – many users trigger these multiple times each week. But the stream of events is worth recording because it may be useful later. Consider a case of stock fraud like the Martha Stewart insider trading case several years ago. There was nothing inherently suspicious about her trades at the time, but this evidence was necessary to later press the case on insider trading.

Potentially malicious use cases escalate priority because they contain suspicious data, commands, or sequences. The data is likely not enough to interrupt the application’s processing, but is noteworthy enough for the analyst to review and perhaps investigate further. These signatures are generally not based on use cases, but rather on threat models and attack patterns. The CAPEC community is one source to consider tapping for attack pattern events and signatures.

The collected data can be analyzed using these models to find activity trends. Authorized user activities are kept primarily for evidence purposes – potentially malicious usage is retained as evidence but also flagged for more immediate attention. Rules are typically built into the SIEM/Log Management platform and can correlate the audit records with other sources to provide a more complete picture.

Conclusion

The combination of Identity Monitoring and User Activity Monitoring provides a powerful way for a SIEM/Log Management system to attribute activities to specific user accounts. This enables analysts to tie back to their sessions and tokens, and how they were issued in the first place. When analyzing an incident this evidence can be quite valuable.

—Gunnar

Friday, October 08, 2010

Friday Summary: October 8, 2010

By Adrian Lane

Chris Pepper was kind enough to forward this interview with James Gosling on the Basement Coders blog earlier in the week. I seldom laugh out loud when reading blogs, but his “Java, Just Free It” & “Set Java Free” t-shirts that were pissing off Oracle got me going. And the “Google is kind of a funny company because a lot of them have this peace love and happiness version of evil” quote had me rolling on the floor. In fact I found the entire article entertaining, so I recommend reading it all the way through if you have a chance. James Gosling is an interesting guy, and for someone I have never met, he has had more impact on my career than any other person on the planet.

Around Christmas 1995 I downloaded the Java white paper. At the time I was a porting engineer for Oracle, so my job was to get Oracle and Oracle apps to run on different flavors of Unix. The paper hit me like a ton of bricks. It was the first time I had seen a really good object model, one which could allow good object oriented techniques. But most importantly, being a porting engineer, Java code could run anywhere without the need to be ported. The writing was on the wall that my particular skill set would be decreasing in value every day from then on. As soon as I could, I downloaded the JDK and started programming in Java.

At the first Java One developers conference in 1996 – and seeing the ‘Green Project’ handheld Gosling described in the interview – I was beyond sold. I was more excited about the possibilities in computer science than ever before. I scripted my Oracle porting job, literally, in Perl and Expect scripts, to free up more time to program Java. I spent my days not-so-clandestinely programming whatever Java projects interested me. Within months I left Oracle just so I could go somewhere, anywhere, and program Java. The startup I landed at happened to be a security start-up. But that white paper was the major catalyst in my career and pretty much shaped my professional direction for the next 10 years.

And so it is again – Gosling’s views on NoSQL actually got me to go back and reconsider some of my negative opinions on the movement. I am still not sold, but there are a handful of people I have so much respect for, that their vision is enough to prompt me to reinvestigate my beliefs. I hope Mr. Gosling gets another chance to research new technologies … the last time he set the industry on its ear.

– Adrian

On to the Summary:

Webcasts, Podcasts, Outside Writing, and Conferences

Favorite Securosis Posts

Other Securosis Posts

Favorite Outside Posts

  • Mike Rothman: Why Wesabe Lost to Mint. Not security related, but important nonetheless. The one that makes things easier on the user wins. Sound familiar, Dr. No? If users have to work too hard, they’ll find ways around your controls. Count on it.
  • Adrian Lane: AT&T, Voice Encryption and Trust.
  • Rich: Verizon releases their big PCI compliance report. Seriously good – this actually ties compliance to breaches.
  • Gunnar Peterson: OAuth Bearer Tokens are a Terrible Idea. This is a sad story, because OAuth gained a ton of traction in version 1.0 (many major sites like Twitter & Netflix are using it), and then in the process of moving OAuth to a full-blown IETF standard the primary security protections were dropped!

Project Quant Posts

Research Reports and Presentations

Top News and Posts

—Adrian Lane

Thursday, October 07, 2010

Monitoring up the Stack: Identity Monitoring

By Gunnar

As we continue up the Monitoring stack, we get to Identity Monitoring, which is a distinct set of concerns from User Activity Monitoring (the subject of the next post). In Monitoring Identity, the SIEM/Log Management systems gain visibility into the provisioning and Identity Management processes that enterprise use to identify, store and process user accounts to prepare the user to use the system. Contrast that with User Activity Monitoring, where SIEM/Log Management systems focus on monitoring how the user interacts with the system at runtime and looks for examples of bad behavior. As an example, do you remember when you got your driver’s license? All the processes that you went through at the DMV: Getting your picture taken, verifying your address, and taking the driving tests. All of those activities are related to provisioning an account, getting credentials created; that’s Identity Management. When you are asked to provide your driver’s license, say when checking in at a hotel, or by a police officer for driving too fast; that’s User Activity Monitoring. Identity Monitoring is an important first step because we need to associate a user’s identity with network events and system usage in order to perform User Activity Monitoring. Each requires a different type of Monitoring and different type of report, today we tackle Identity Management (and no, we won’t make you wait in line like the DMV).

To enable Identity Monitoring, the SIEM/Log Management project inventories the relevant Identity Management processes (such as Provisioning), data stores (such as Active Directory and LDAP) and technologies (such as Identity Management suites). The inventory should include the Identity repositories that store accounts used for access to the business’ critical assets. In the old days it was as simple as going to RACF and examining the user accounts and rules for who was allowed to access what. Nowadays, there can be many repositories that store and manage account credentials, so inventorying the critical account stores is the first step.

Process

The next step is to identify the Identity Management processes that govern the Identity repositories. How did the accounts get into LDAP or Active Directory? Who signs off on them? Who updates them? There are many facets to consider in the Identity management lifecycle. The basic Identity Management process includes the following steps:

  • Provisioning: account creation and registration
  • Propagating: synchronizing or replicating the account to the account directory or database
  • Access: accessing the account at runtime
  • Maintenance: changing account data
  • End of Life: Deleting and disabling accounts

The Identity Monitoring system should verify events at each process step, record the events, and write the audit log messages in a way that they can be correlated for security incident response and compliance purposes. This links the event to the account(s) that initiated and authorized the action. For example, who authorized the account that were Provisioned? What manager(s) authorized the account updates? As we saw in the recent Societe Generale case, Jerome Kerviel (the trader who lost billions of the bank’s money) was originally an IT employee who moved over to the trading desk. When he made the move from IT to trading, his account retained his IT privileges and gained new trading privileges. Snowball entitlements enabled him to both execute trades and remove logs and hide evidence. It seems likely there was a process mishap in the account update and maintenance rules that allowed this to happen, and it shows how important the identity management processes are to access control.

In complex systems, the Identity Management process is often automated using an Identity Management suite. These suites generate reports for Compliance and Security purposes, these reports can be published to the SIEM/Log Management system for analysis. Whether automated with a big name suite or not, its important to start Identity Monitoring by understanding the lifecycle that governs the account data for the accounts in your critical systems. To fully close the loop, some processes also reconcile the changes with change requests (and authorizations) to ensure every change is requested and authorized.

Data

In addition to identifying the Identity repositories and the management processes around them, the data itself is useful to inform the auditable messages that are published to the SIEM/Log Management systems. The data aspects for collection typically include the following:

  • User Subject (or entity) which could be a person, an organization, or a host or application.
  • Resource Object which could be a database, a URL, component, queue or a Web Service,
  • Attributes such as Roles, Groups and other information that is used to make authorization decisions.

The identity data should be monitored to record any lifecycle events such as Create, Read, Update, Delete, and Usage events. This is important to give the SIEM/Log Management system an end to end view of the both the account lifecycle and the account data.

Challenges

One challenge in Identity Monitoring is that the systems that are to be monitored (such as authentication systems) sport byzantine protocols and are not easy to get data and reports out of. This may require some extra spelunking to find the optimal protocol to use to communicate with the Identity repository. The good news is this is a one-time effort during implementation. These protocols do not change frequently.

Another challenge is the accuracy of associating the user identity with the activity that a SIEM collects. Simply matching user ID to IP or MAC address is limited, so heuristic and deterministic algorithms are used to help associate users with events. The association can be performed by the collector, but more commonly this feature is integrated within the SIEM engine as an log/event enrichment activity. The de-anonymization occurs as data is normalized, and stored with the events.

Federated identity systems that separate the authentication, authorization and attribution create additional challenges, because the end to end view of the account in both the Identity Provider and in the Relying Party is not usually easy to attain. Granted this is the point of Federation, which resolves the relationship at runtime, but it’s worth pointing out the difficulty this presents to end to end monitoring.

Finally, naming and hierarchies can create challenges in reporting on subjects, objects and attributes because the namespaces and management techniques can create collisions and redundancies.

Conclusion

Monitoring Identity systems benefits both Security and Compliance teams. Monitoring identity process and data events gives Security and Compliance a view into some of the most critical parts of the security architecture: identity repositories. The identity repositories are the source of many access control decisions and this view into how they are populated and managed is fundamental to monitoring the overall security architecture. Identity monitoring is also needed to move to User Activity Monitoring, which is used to provide the linkage between how the accounts were provisioned and how they are used at runtime. We’ll discuss that in the next post.

—Gunnar

Wednesday, October 06, 2010

Incite 10/6/2010: The Answer is 42

By Mike Rothman

One of my favorite passages in literature is when Douglas Adams proclaims the Ultimate Answer to the Ultimate Question of Life, The Universe, and Everything to be 42 in Hitchhiker’s Guide to the Galaxy. Of course, we don’t know the Ultimate Question. Details. This week I plan to discover he was right as I finish my 42nd year on the planet. That seems old. It’s a big number. But I don’t feel old. In fact, I feel like a big kid. Sometimes I look at my own kids and my house and snicker a bit. Can you believe they’ve entrusted any responsibility to me? These kids think I actually know something? Ha, that’s a laugher…

Time well spent... Since I’m trying not to look forward and plan, I figure I should look backward and try to appreciate the journey. As I look back, I can kind of break things up into a couple different phases. My childhood was marked by anger. Yeah, I know you are shocked. But I took everything bad that happened personally, and as a result, I was a pretty angry kid.

College was a blur. I know I drank a lot of beer. I think I studied a bit. When I graduated I entered the unbreakable phase. Right, like the Oracle database. I could do little wrong. I had a pretty quick progression through the corporate ranks. In hindsight it was too quick. I didn’t screw anything up, so I felt invincible. I also didn’t learn a hell of a lot, but thought I did. Sound familiar? Then I started a software company in 1998 to chase the Internet bubble IPO money. I learned pretty quickly that I wasn’t invincible, as I heard the sound of $30 million of someone else’s money being flushed down the toilet. Crash. Big time.

Then I entered the striving stage throughout my 30’s. Striving for more and never being satisfied. From there I proceeded to jump from job to job every 15 months, chasing some shiny object and trying to catch the brass ring. Again, that didn’t work out too well and I found myself getting angry again. Then I started Incite and was a lot happier. I managed to remember what I liked to do and then start to address some of my deeply buried issues. No, I’m not going to bare my soul like Bill Brenner, but we all have demons to face and at that point I started facing my own.

I took a detour back into the vendor world for 15 months, and then sold Rich and Adrian a bill of goods to let me hang my shingle at Securosis. 10 months in, I’m having the time of my life. I’m thinking this is the contented phase. I’ve been working hard, at everything. Physically, I’m in the best shape I’ve been in since my early 20’s. Mentally I’m making progress, working to accept what’s happening and stop looking forward at the expense of being present. I’m happy with what I do and what I have. My family loves me and I love them. What else does a guy need?

I’m still fighting demons, and I probably always will. The hope is that my epic battles will be fewer and farther between over time. I’m still screwing things up, and I’ll probably always do that too. That’s an entrepreneur’s curse. I’m also learning new things almost every day, and when that stops it’s time to move on to the Great Unknown.

As I look back, I figured out what my Ultimate Question is: “When do you realize it’s a game and you should enjoy the ride, both the ups and the downs?” Right. For me, the answer is 42.

– Mike.

Photo credits: “42” originally uploaded by cszar


Recent Securosis Posts

  1. Friday Summary: September 30, 2010
  2. Monitoring up the Stack:
  3. Understanding and Selecting a DLP Solution
  4. NSO Quant Posts

Incite 4 U

  1. Get on the (security incident) cycle – Good summary here by Lenny Zeltser covering a presentation from our hero Richard Bejtlich about how he’s built the Incident Response team at GE to deal with things like well-funded patient attackers (note I didn’t use the a(blank)t acronym). Of course there will always be failures, but the question is about organizational commitment to detecting adversaries and putting the right capabilities in place to protect your organization. And to look at security as a process and – dare I say it – a lifecycle. That means you need to focus on all aspects – before, during, and after the attack. Amazingly enough, Rich and I are starting another blog series on exactly this topic in about a week. – MR

  2. Save the children… with robots – The state of technology education in this country is simply embarrassing. Everyone talks about how kids use a mouse before they can read, but how many of them understand how a computer works? You’d think today’s teenagers would know a hard drive from RAM, but not if they rely on their (standard) school to teach them. However, they are pretty good at putting cats in PowerPoints. Our friend Chris Hoff is trying to change this with a hacking conference dedicated to kids… called, appropriately enough, HacKid. It’s an amazing idea, with everything from Lego robots to online safety covered, and if you have kids of the right age, or just want to support it, I highly recommend attending or getting involved. – RM

  3. No trust for you! – Despite being a big fan of monitoring technologies, I thought the Trust No One, Monitor Everything position was a bit over the top. The “monitor everything” approach fails for exactly the same reasons “encrypt everything” fails: a single technology cannot solve every problem. Monitoring is just another security tool, and before you try to saw wood with a hammer, remember attacks that bypass WAF, IDS, App Monitoring, and DAM are well documented. Don’t get me wrong – we should incorporate this approach as much as possible considering we trust far too much stuff right now. But that’s because the Internet is based on an academic model of trust everything and log nothing important. Adopting a Zero Trust model means not browsing the Internet – the web sites you visit trust people you never would, and they treat your web browser like a public restroom on the information superhighway. Zero Trust means you don’t accept email from the hot chick you met last night because she’s not on your white list. Zero Trust means Grandma is to be considered a hacker until proven otherwise. Kind of difficult to expand your horizons that way. And feel free to monitor everything, but see if you can come up with rules that differentiate good behavior from bad. – AL

  4. Monitor Everything (even if Adrian hates it) – So I was planning to discuss Forrester’s Zero Trust thing, but Adrian beat me to it and once again shows his disdain for monitoring everything. First off, Zero Trust is nothing new. Remember back just two years ago (I know you can), we called that the insider threat and a new category of technology emerged to try to combat this threat. Actually, it was more like trying to spin an existing product into this new category – marketers have been known to do that. I believe you should monitor as much as you can. But collecting every packet that traverses your network (even the ones from Grandma) is probably too much, so you need to consider the point of diminishing returns for monitoring. But most folks don’t monitor much at all, so I’ll keep pushing for monitoring everything, knowing that this is effectively a push for more folks to monitor something. And if that means you need to package it up as Zero Trust, I’m okay with that too. – MR

  5. Reputation is finally ubiquitous – The Big Yellow has been busy, evidently getting ready for their annual user conference, in Spain no less. Sounds like a boondoggle to me. First off they showed off a new logo. Which looks amazingly like the VeriSign check in a yellow circle. To call it awful is being nice. Very nice. Of course, those jokers on Twitter instantly cracked about how SYMC is putting the check in checkbox compliance. LMAO. But they also talked about their new Ubiquity technology, which is basically a fancy name for reputation. It’s good to see Symantec closing the gap with the other anti-malware vendors to what? Maybe two years? But it’s the right thing to do for now. It’s still not enough to save the blacklist approach, but might give them a few more quarters of being able to milk the (cash) cow. – MR

  6. Tomato, Tomaeto – Android apps caught covertly sending GPS data to advertisers. The only shock here would be software that didn’t spy on you. I mean, isn’t that why Google is creating the Android platform? To broaden their ability to monitor activity and collect data to more effectively sell advertising? Heck, most attackers are just building on top of the ‘clever’ techniques pioneered by web intelligence firms for their marketing and merchant applications over the last decade. Cookies, iFrames, scripting … ask yourself, honestly, why that stuff was created. Am I the only one who thought the name Android was a euphemism for an advanced botnet? Isn’t it a little odd to complain that Android apps are “surreptitiously transmitting the user’s phone number …”, GPS coordinates, and other sensitive information when The Google will be using those same hooks to collect the same data from Android platforms? – AL

  7. Oracle bets on authentication – Oracle is at it again. Buying yet another security technology mostly likely never to be heard from again. They acquired PassLogix, which does authentication/SSO stuff. The deal actually makes sense because Oracle already OEMed the technology, and it’s a logical extension of their identity management technology. PassLogix is one of the authentication companies that have been around seemingly forever (like Arcot, recently bought by CA) and now they get an exit. It does beg the question: why now? And what about the others, like Courion? But again, this is just more evidence that security is not a standalone business long-term, but will gradually get lost within some kind of middleware thing. A Fusion of sorts… – MR

  8. 5 problems with security SaaS. Oy. – One of the issues with new technology is overlapping and confusing vernacular. So I see this article on 5 problems with SaaS security and I’m not sure if they are talking about SaaS, PaaS, IaaS, or (blah)aaS. Things like identity is weak in the cloud. Well, let me tell you, identity management not in the cloud is weak too. Other issues include the lack of standards and security by obscurity. Blah blah blah. It’s a new technology, there aren’t going to be standards. And since when do any markets wait for standards? Right, never. Here’s the deal. The cloud (whatever that means) is going to happen. So our choice is whether we (as security folks) start working with the teams internally which are thinking cloudy thoughts and figuring out how to at least get them thinking about security, or give up. Because if we don’t engage and start figuring this stuff out (maybe even influencing it to become a bit more secure over time), we don’t have a fighting chance. – MR

—Mike Rothman

Tuesday, October 05, 2010

Monitoring up the Stack: App Monitoring, Part 2

By Gunnar

In the last post on application monitoring, we looked at why applications are an essential “context provider” and interesting data source for SIEM/Log Management analysis. In this post, we’ll examine how to get started with the application monitoring process, and how to integrate that data into your existing SIEM/Log Management environment.

Getting Started with Application Monitoring

As with any new IT effort, its important to remember that it’s People, Process and Technology – in that order. If your organization has a Build Security in software security regime in place, then you can leverage those resources and tools for building visibility in. If not, application monitoring provides a good entree into the software security process, so here are some basics to get started with Application Monitoring.

Application Monitors can be deployed as off the shelf products (like WAFs), and they can be delivered as custom code. However they are delivered, the design of the Application Monitor must address these issues:

  • Location: Where application monitors may be deployed; what subjects, objects, and events are to be monitored.
  • Audit Log Messages: How the Audit Log Observers collect and report events; these messages must be useful to the human(!) analysts who use them for incident response, event, management and compliance.
  • Publishing: The way the Audit Log Observer publishes data to a SIEM/Log Manager must be robust and implement secure messaging to provide the analyst with high-quality data to review, and to avoid creating YAV (Yet Another Vulnerability).
  • Systems Management: Making sure the monitoring system itself is working and can respond to faults.

Process

The process of integrating your application monitoring data into the SIEM/Log Management platform has two parts. First identify where and what type of Application Monitor to deploy. Similar to the discovery activity required for any data security initiative, you need to figure out what needs to be monitored before you can do anything else. Second, select the way to communicate from the Application Monitor to the SIEM/Log Management platform. This involves tackling data formats and protocols, especially for homegrown applications where the communication infrastructure may not exist.

The most useful Application Monitor provides a source of event data not available elsewhere. Identify key interfaces to high priority assets such as message queues, mainframes, directories, and databases. For those interfaces, the Application Monitor should give visibility into the message exchanges to and from the interfaces, session data, and the relevant metadata and policy information that guides its use. For applications that pass user content, the interception of messages and files provides the visibility you need. In terms of form factor for Application Monitor deployment (in specialized hardware, in the application itself, or in an Access Manager), performance and manageability are key aspects, but less important than what subjects, objects, and events the Application Monitor can access to collect and verify data.

Typically the customer of the Application Monitor is a security incident responder, an auditor, or other operations staff. The Application Monitor domain model described below provides guidance on how to communicate in a way that enables this customer to rely on the information found in the log in a timely way.

Application Monitor Domain Model

The Application Monitor model is fairly simple to understand. The core parts of the Application Monitor include:

  • Observer: A component that listens for events
  • Event Model: Describes the set of events the Observer listens for, such as Session Created and User Account Created
  • Audit Log Record Format: The data model for messages that the Observer writes to the SIEM/Log Manager, based on Event Type
  • Audit Log Publisher: The message exchange patterns, such as publish and subscribe, that are used to communicate the Audit Log Records to the SIEM/Log Manager

These areas should be specified in some detail with the development and operations teams to make sure there is no confusion during the build process (building visibility in), but the same information is needed when selecting off-the-shelf monitoring products. For the Event Model and Audit Log Record, there are several standard log/event formats which can be leveraged, including CEE (from Mitre and ArcSight), XDAS (from Open Group), and PCI DSS (from you-know-who). CEE and XDAS give general purpose frameworks for types of events the observer should listen for and which data should be recorded; the PCI DSS standard is more specific to credit card processing. All these models are worth reviewing to find the most cost-effective way to integrate monitoring into your applications, and to make sure you aren’t reinventing the wheel.

To tailor the standards to your specific deployment, avoid the “drinking from the firehose” effect, where the speed and volume of incoming data make the signal-to-noise ratio unacceptable. As we like to say at Securosis: just because you can doesn’t mean you should. Or think about phasing in the application monitoring process, where you collect the most critical data initially and then expand the scope of monitoring over time to gain a broader view of application activity.

The Event Model and Audit Records should collect and report on the areas described in the previous post (Access Control, Threats, Compliance, and Fraud). However, if your application is smart enough to detect malice or misuse, why wouldn’t you just block it in the application anyway? Ay, there’s the rub. The role of the monitor is to collect and report, not to block. This gets into a philosophical discussion beyond the scope of this research, but for now suffice it to say that figuring out if and what to block is a key next step beyond monitoring.

The Event Model and Audit Records collected should be configureable (not hard-coded) in a rule or other configuration engine. This enables the security team to flexibly turn logging events up and down, data gathering, and other actions as needed without recompiling and redeploying the application.

The two main areas the standards do not address are the Observer and the Audit Log Publisher. The optimal placement of the Observer is often a choke point with visibility into a boundary’s (for example, crossing technical boundaries like Java to .NET or from the web to a mainframe) inputs and outputs. Choke points can be organizational (B2B connection), zone (DMZ/Internal), or state-based (account upgrade, transaction executed). The goal in specifying the location of the Application Monitor is to identify areas where valuable assets need not just protection, but also detection. A choke point in an application provides a centralized location to collect and report on inbound and outbound access. This can mean a WAF at the boundary of web applications, or it can be further down in the stack, but the choke point must have access to the message payload data and be able to parse and make sense of the data to be useful to security analysts.

The Audit Log Publisher must be able to communicate messages to the SIEM/Log Management platform using secure enterprise-class messaging. This means guaranteed delivery and that policies can define when messages are delivered in order, at least once, and at most once. Some examples of enterprise class messaging are JMS and MQ Series. The messages must also be signed and hashed for authentication and integrity purposes.

Where to Go Next

As with many application security efforts, Security must plan an integration strategy. After all, to build security in and monitor applications, the “in” means integration. This can be done at the edge of the application, as through a Web Application Firewall or Filter (where the integration is typically focused on resources like the URI and HTTP streams); or it can be integrated closer to the code through application logging in the application. The book “Enterprise Integration Patterns” by Gregor Hohpe and Bobby Woolf (and companion website: http://www.enterpriseintegrationpatterns.com/) contains plenty of useful real-world guidance on getting started with integration, including patterns for Endpoints (where application monitors may be deployed), message construction and transformation (where and how Audit Log Observers collect and report events), message channels and routing (how publishers send data to a SIEM/Log Manager), and systems management (making sure it works!). Whether delivered as an off-the-shelf product such as a WAF or in custom code, the combination of these patterns makes for an end-to-end integrated system that can report context straight from the authoritative source – the application.

—Gunnar