Research

In Attacks, we discussed both network-based and application-targeting Denial of Service (DoS) attacks. Given the radically different techniques between the types, it’s only logical that we use different defense strategies for each type. But be aware that aspects of both network-based and application-targeting DoS attacks are typically combined for maximum effect. So your DoS defenses need to be comprehensive, protecting against (aspects of) both types. Anti-DoS products and services you will consider defend against both. This post will focus on defending against network-based volumetric attacks. First the obvious: you cannot just throw bandwidth at the problem. Your adversaries likely have an unbounded number of bots at their disposal and are getting smarter at using shared virtual servers and cloud instances to magnify the amount of evil bandwidth at their disposal. So you can’t just hunker down and ride it out. They likely have a bigger cannon than you can handle. You need to figure out how to deal with a massive amount of traffic and separate good traffic from bad, while maintaining availability. Find a way to dump bad traffic before it hoses you somehow without throwing the baby (legitimate application traffic) out with the bathwater. We need to be clear about the volume we are talking about. Recent attacks have blasted upwards of 80-100gbps of network traffic at targets. Unless you run a peering point or some other network-based service, you probably don’t have that kind of inbound bandwidth. Keep in mind that even if you have big enough pipes, the weak link may be the network security devices connected to them. Successful DoS attacks frequently target network security devices and overwhelm their session management capabilities. Your huge expensive IPS might be able to handle 80gbps of traffic in ideal circumstances, but fall over due to session table overflow. Even if you could get a huge check to deploy another network security device in front of your ingress firewall to handle that much traffic, it’s probably not the right device for the job. Before you just call up your favorite anti-DoS service provider, ISP, or content delivery network (CDN) and ask them to scrub your traffic, that approach is no silver bullet either. It’s not like you can just flip a switch and have all your traffic instantly go through a scrubbing center. Redirecting traffic incurs latency, assuming you can even communicate with the scrubbing center (remember, your pipes are overwhelmed with attack traffic). Attackers choose a mix of network and application attacks based on what’s most effective in light of your mitigations. No, we aren’t only going to talk about more problems, but it’s important to keep everything in context. Security is not a problem you can ever solve – it’s about figuring out how much loss you can accept. If a few hours of downtime is fine, then you can do certain things to ensure you are back up within that timeframe. If no downtime is acceptable you will need a different approach. There are no right answers – just a series of trade-offs to manage to the availability requirements of your business, within the constraints of your funding and available expertise. Handling network-based attacks involves mixing and matching a number of different architectural constructs, involving both customer premise devices and network-based service offerings. Many vendors and service providers can mix and match between several offerings, so we don’t have a set of vendors to consider here. But the discussion illustrates how the different defenses play together to blunt an attack. Customer Premise-based Devices The first category of defenses is based around a device on the customer premises. These appliances are purpose-built to deal with DoS attacks. Before you turn your nose up at the idea of installing another box to solve such a specific problem, take another look at your perimeter. There is a reason you have all sorts of different devices. The existing devices already in your perimeter aren’t particularly well-suited to dealing with DoS attacks. As we mentioned, your IPS, firewall, and load balancers aren’t designed to manage an extreme number of sessions, nor are they particularly adept at dealing with obfuscated attack traffic which looks legitimate. Nor can other devices integrate with network providers (to automatically change network routes, which we will discuss later) – or include out-of-the-box DoS mitigation rules, dashboards, or forensics, built specifically to provide the information you need to ensure availability under duress. So a new category of DoS mitigation devices has emerged to deal with these attacks. They tend to include both optimized IPS-like rules to prevent floods and other network anomalies, and simple web application firewall capabilities which we will discuss in the next post. Additionally, we see a number of anti-DoS features such as session scalability, combined with embedded IP reputation capabilities, to discard traffic from known bots without full inspection. To understand the role of IP reputation, let’s recall how email connection management devices enabled anti-spam gateways to scale up to handle spam floods. It’s computationally expensive to fully inspect every inbound email, so dumping messages from known bad senders first enables inspection to focus on email that might be legitimate, and keeps mail flowing. The same methodology applies here. These devices should be as close to the perimeter as possible, to get rid of the maximum amount of traffic before the attack impacts anything else. Some devices can be deployed out-of-band as well, to monitor network traffic and pinpoint attacks. Obviously monitor-and-alert mode is less useful than blocking, which helps maintain availability in real time. And of course you will want a high-availability deployment – an outage due to a failed security device is likely to be even more embarrassing than simply succumbing to a DoS. But anti-DoS devices include their own limitations. First and foremost is the simple fact that if your pipes are overwhelmed, a device on your premises is irrelevant. Additionally, SSL attacks are increasing in frequency. It’s cheap for an army of bots to use SSL to encrypt all their

Gunnar Peterson posted a presentation a while back on how being an investor makes him better at security, and conversely how being in security makes him better at investing. It’s a great concept, and my recent research on different investment techniques has made me realize how amazing his concept is. Gunnar’s presentation gets a handful of the big ideas (including defensive mindset, using data rather than anecdotes to make decisions, and understanding the difference between what is and what should be) right, but actually under-serves his concept – there are many other comparisons that make his point. That crossed my mind when reading An Investor’s Guide to Famous Last Words. Black Swan author Nassim Taleb: “People focus on role models; it is more effective to find antimodels – people you don’t want to resemble when you grow up.” The point in the Fool article is to learn from others’ mistakes. With investing mistakes are often very public, and we share them as examples of what not to do. In security, not so much. Marcus Ranum does a great presentation called the Anatomy of The Security Disaster, pointing out that problem identification is ignored during the pursuit of great ideas, and blame-casting past the point of no return is the norm. I have lived through this sequence of events myself. And I am not arrogant enough to think I always get things right – I know I had to screw things up more than once just to have a decent chance of not screwing up security again in the future. And that’s because I know many things that don’t work, which – theoretically anyway – gives me better odds at success. This is exactly the case with investing, and it took a tech collapse in 2001 to teach me what not to do. We teach famous investment failures but we don’t share security failures. Nobody wants the shame of the blame in security. There is another way investing makes me better at security and it has to do with investment styles, such as meta-trending, day trading, efficient market theory, cyclic investing, hedging, shorting, value investing, and so on. When you employ a specific style you need to collect specific types of data to fuel your model, which in turn helps you make investment choices. You might look at different aspects of a company’s financials, industry trends, market trends, political trends, social trends, cyclic patterns, the management team, or even disasters and social upheaval as information catalysts. Your model defines which data is needed. You quickly realize that mainstream media only caters to certain styles of investing – data for other styles is only a tiny fraction of what the media covers. Under some investment styles all mainstream investment news is misleading BS. The data you don’t want is sprayed at you like a fire hose because those stories interest many people. We hear about simple and sexy investment styles endlessly – boring, safe, and effective investment is ignored. Security practitioners, do you see where I am going with this? It is very hard to filter out the noise. Worse, when noise is all you hear, it’s really easy to fall into the “crap trap”. Getting good data to base decisions on is hard, but bad data is free and easy. The result is to track outside of your model, your style, and your decision process. You react to the BS and slide toward popular or sexy security models or products – that don’t work. It’s frightfully easy to do when all anyone talks about are red herrings. Back to Gunnar’s quote… Know what you don’t want your security model to be. This is a great way to sanity check the controls and processes you put into place to ensure you are not going down the wrong path, or worrying about the wrong threats. On to the Summary: Webcasts, Podcasts, Outside Writing, and Conferences Adrian’s Dark Reading post: What’s the threat? Rich’s Dark Reading post: Security Losses Remain Within Range of Acceptable Adrian’s research paper: Securing Small Databases. Mike’s upcoming webcast: I just got my WAF, Now What? Favorite Securosis Posts Mike Rothman: Securing Big Data: Operational Security Issues. This stuff looks a lot like the issues you face on pretty much everything else. But a little different. That’s the point I take away from this post and the series. Yes it’s a bit different, and a lot of the fundamentals and other disciplines used through the years may not map exactly, but they are still useful. Adrian Lane: Incite: Cash is King. How many startups have I been at that hung on the fax machine at the end of every quarter? How many sales teams have I been with where “the Bell” only rang the last three days of a quarter? Good stuff. Rich I’m picking my Dark Reading post this week. It stirred up a bit of a Twitter debate, and I think I need to write more on this topic because I couldn’t get enough nuance into the initial piece. Other Securosis Posts New Series: Understanding and Selecting Identity Management for Cloud Services. Endpoint Security Management Buyer’s Guide Published (with the Index of Posts). Securing Big Data: Operational Security Issues. Favorite Outside Posts Mike Rothman: DDoS hitmen for hire. You can get anything as a service nowadays. Even a distributed denial of service (DDoS). I guess this is DDoSaaS, eh? Adrian Lane: Think differently about database hacking. Lazlo Toth and Ferenc Spala’s DerbyCon presentation shows how to grab encryption keys and passwords from OCI clients. A bit long, but a look at hacking Oracle databases without SQL injection. Yes, there are non-SQL injection attacks, in case you forgot. Will we see this in the wild? I don’t know. Rich: Antibiotic Resistant security by Valsmith. What he’s really driving at is an expansion of monoculture and our reliance on signature-based AV, combined with a few other factors. It’s a very worthwhile read. The TL;DR version is that we have created

Adrian and Gunnar here, kicking off a new series on Identity Management for Cloud Services. We have been hearing about Federated Identity and Single Sign-On services for the last decade, but demand for these features has only fully blossomed in the last few years, as companies have needed to integrate their internal identity management systems. The meanings of these terms has been actively evolving, under the influence of cloud computing. The ability to manage what resources your users can access outside your corporate network – on third party systems outside your control – is not just a simple change in deployment models; but a fundamental shift in how we handle authentication, authorization, and provisioning. Enterprises want to extend capabilities to their users of low-cost cloud service providers – while maintaining security, policy management, and compliance functions. We want to illuminate these changes in approach and technology. And if you have not been keeping up to date with these changes in the IAM market, you will likely need to unlearn what you know. We are not talking about making your old Active Directory accessible to internal and external users, or running LDAP in your Amazon EC2 constellation. We are talking about the fusion of multiple identity and access management capabilities – possibly across multiple cloud services. We are gaining the ability to authorize users across multiple services, without distributing credentials to each and every service provider. Cloud services – be they SaaS, PaaS, or IaaS – are not just new environments in which to deploy existing IAM tools. They fundamentally shift existing IAM concepts. It’s not just the way IT resources are deployed in the cloud, or the way consumers want to interact with those resources, which have changed, but those changes are driven by economic models of efficiency and scale. For example enterprise IAM is largely about provisioning users and resources into a common directory, say Active Directory or RACF, where the IAM tool enforces access policy. The cloud changes this model to a chain of responsibility, so a single IAM instance cannot completely mediate access policy. A cloud IAM instance has a shared responsibility in – as an example – assertion or validation of identity. Carving up this set of shared access policy responsibilities is a game changer for the enterprise. We need to rethink how we manage trust and identities in order to take advantage of elastic, on-demand, and widely available web services for heterogenous clients. Right now, behind the scenes, new approaches to identity and access management are being deployed – often seamlessly into cloud services we already use. They reduce the risk and complexity of mapping identity to public or semi-public infrastructure, while remaining flexible enough to take full advantage of multiple cloud service and deployment models. Our goal for this series is to illustrate current trends and technologies that support cloud identity, describe the features available today, and help you navigate through the existing choices. The series will cover: The Problem Space: We will introduce the issues that are driving cloud identity – from fully outsourced, hybrid, and proxy cloud services and deployment models. We will discuss how the cloud model is different than traditional in-house IAM, and discuss issues raised by the loss of control and visibility into cloud provider environments. We will consider the goals of IAM services for the cloud – drilling into topics including identity propagation, federation, and roles and responsibilities (around authentication, authorization, provisioning, and auditing). We will wrap up with the security goals we must achieve, and how compliance and risk influence decisions. The Cloud Providers: For each of the cloud service models (SaaS, PaaS, and IaaS) we will delve into the IAM services built into the infrastructure. We will profile IAM offerings from some of the leading independent cloud identity vendors for each of the service models – covering what they offer and how their features are leveraged or integrated. We will illustrate these capabilities with a simple chart that shows what each provides, highlighting the conceptual model each vendor embraces to supply identity services. We will talk about what you will be responsible for as a customer, in terms of integration and management. This will include some of the deficiencies of these services, as well as areas to consider augmenting. Use Cases: We will discuss three of the principal use cases we see today, as organizations move existing applications to the cloud and develop new cloud services. We will cover extending existing IAM systems to cover external SaaS services, developing IAM for new applications deployed on IaaS/PaaS, and adopting Identity as a Service for fully external IAM. Architecture and Design: We will start by describing key concepts, including consumer/service patterns, roles, assertions, tokens, identity providers, relying party applications, and trust. We will discuss the available technologies fors the heavy lifting (such as SAML, XACML, and SCIM) and discuss the problems they are designed to solve. We will finish with an outline of the different architectural models that will frame how you implement cloud identity services, including the integration patterns and tools that support each model. Implementation Roadmap: IAM projects are complex, encompass most IT infrastructure, and may take years to implement. Trying to do everything at once is a recipe for failure. This portion of our discussion will help ensure you don’t bite off more than you can chew. We will discuss how to select an architectural model that meets your requirements, based on the cloud service and deployment models you selected. Then we will create different implementation roadmaps depending on your project goals and critical business requirements. Buyer’s Guide: We will close by examining key decision criteria to help select a platform. We will provide questions to determine with vendors offer solutions that support your architectural model and criteria to measure the appropriateness of a vendor solution against your design goals. We will also help walk you through the evaluation process. As always, we encourage you to ask questions and chime in with comments and suggestions.

Last Friday was the end of the third calendar quarter. For you math majors out there, that’s the 3-month period ending September 30. Inevitably I had meetings and calls canceled at the last minute to deal with “end of quarter” issues. This happens every quarter, so it wasn’t surprising. Just funny. Basically most companies report their revenues and earnings (even the private ones) based on an arbitrary reporting period, usually a calendar quarter. Companies provide significant incentives for sales reps to close deals by the end of each quarter. Buying hardware and software has become a game where purchasing managers sit on large purchase orders (POs) until the end of the quarter to see what extra discounts they can extract in exchange for processing the order on time. I guess other businesses are probably like that too, but I only have direct experience with hardware and software. Even small companies can enjoy the fun. We subscribed to a new SaaS service last week and the rep threw in an extra month on the deal if we signed by Sept 30th. So the last week of the quarter runs something like this: Sales reps pound the voice mails of their contacts to see if and when the PO will be issued. They do this because their sales managers pound their voice mails for status updates. Which happens because VPs of Sales pound the phones of sales managers. It’s a good thing phone service is basically free nowadays. A tweet from Chis Hoff reminded me of the end of Q craziness as he was sweating a really big order coming through. I’ve never had the pleasure (if you can call it that) of waiting for a 9 figure PO to arrive, though I have done my share of hunching over the fax machine thru the years. But the whole end of Q stuff is nonsense. Why are orders any less important if they come in on October 3? Of course they’re not. But tell that to a rep who got his walking papers because the deal didn’t hit by Sept 30th. That’s why I like cash. I can pay my mortgage with cash. We can buy cool Securosis bowling shirts and even upgrade to the iPhone 5, even if AT&T forced us to pay full price since we already upgraded to the 4S and weren’t going to wait until March to upgrade. Cash is king in my book. As the CFO, I don’t have to worry about accruals or any of that other accounting nonsense. It’s liberating. Do work. Bill clients. Get paid. Repeat. Obviously cash accounting doesn’t work for big companies or some smaller businesses. And that’s OK. It works for us. –Mike Photo credits: cash is king originally uploaded by fiveinchpixie Heavy Research We’re back at work on a variety of blog series, so here is a list of the research currently underway. Remember you can get our Heavy Feed via RSS, where you can get all our content in its unabridged glory. And you can get all our research papers too. Defending Against Denial of Service (DoS) Attacks The Attacks Introduction Securing Big Data Recommendations and Open Issues Operational Security Issues Architectural Issues Security Issues with Hadoop Incite 4 U Now this is some funny NMAP: Bloggers know the pain of fending off the Hakin9 folks’ endless attempts to get free contributions to their magazine. I just delete the requests and move on. But a bunch of pissed off (and very funny) security folks decided to write an NMAP article that, well, you have to read to believe. The title is: “Nmap: The Internet Considered Harmful – DARPA Inference Cheking Kludge Scanning.” [sic] Right, they refer to the remediations as DICKS throughout the article. Really. How funny is that? And they used some white paper generator, which spit out mostly nonsensical gibberish. Clearly no one actually read the article before it was published, which would be sad if it wasn’t so damn funny. Just another reminder that you can’t believe everything you read on the Internet. Fyodor provides additional context. – MR Hope is not a DDoS strategy: Looks like Distributed Denial of Service (DDoS) attacks have hit the big time. That happens when a series of attacks take down well-known financial institutions like Wells Fargo. Our timing is impeccable – we are currently writing a series on Defending Against DoS attacks (see the posts linked above). The NWW article says banks can only hope for the best. Uh, WTF? Hope for the best?!?!? Hope doesn’t keep your website up, folks. But these attacks represent brute force. There are many other tactics (including attacking web apps) that can be just as effective as knocking down your site, without melting your pipes. Mike Smith has it right when he says Information, not Hope is key to Surviving DDoS attacks. Mike’s post talks about how Akamai deals with these attacks (at a high level, anyway) for themselves and their customers. Like most security functions nowadays, there is enough data to analyze and draw conclusions. Find the patterns and design mitigations to address the attacks. Or hope for the best, and let me know how that works out for you. – MR Cloudicomplications: Those of you who follow me on Twitter may recall my epic struggles with OpenStack about a year and a half ago. We decided to use it for the private cloud lab in the CCSK training class, and I was stuck with the task of building a self-contained virtual lab that would be resilient to various networks and student systems, given the varied competence of instructors and students. OpenStack was hella-immature at the time and building the lab nearly ended me. Last week the latest version (Folsom) was released and it is supposedly much more mature, especially in networking, which was the part that really complicated the labs. But as Lydia Leong at Gartner reports, open isn’t really open when the project is run by competing vendors operating out

Our previous two posts outlined several security issues inherent to big data architecture, and operational security issues common to big data clusters. With those in mind, how can one go about securing a big data cluster? What tools and techniques should you employ? Before we can answer those questions we need some ground rules, because not all ‘solutions’ are created equally. Many vendors claim to offer big data security, but they are really just selling the same products they offer for other back office systems and relational databases. Those products might work in a big data cluster, but only by compromising the big data model to make it fit the restricted envelope of what they can support. Their constraints on scalability, coverage, management, and deployment are all at odds with the essential big data features we have discussed. Any security product for big data needs a few characteristics: It must not compromise the basic functionality of the cluster It should scale in the same manner as the cluster It should not compromise the essential characteristics of big data It should address – or at least mitigate – a security threat to big data environments or data stored within the cluster. So how can we secure big data repositories today? The following is a list of common challenges, with security measures to address them: User access: We use identity and access management systems to control users, including both regular and administrator access. Separation of duties: We use a combination of authentication, authorization, and encryption to provide separation of duties between administrative personnel. We use application space, namespace, or schemata to logically segregate user access to a subset of the data under management. Indirect access: To close “back doors” – access to data outside permitted interfaces – we use a combination of encryption, access control, and configuration management. User activity: We use logging and user activity monitoring (where available) to alert on suspicious activity and enable forensic analysis. Data protection: Removal of sensitive information prior to insertion and data masking (via tools) are common strategies for reducing risk. But the majority of big data clusters we are aware of already store redundant copies of sensitive data. This means the data stored on disk must be protected against unauthorized access, and data encryption is the de facto method of protecting sensitive data at rest. In keeping with the requirements above, any encryption solution must scale with the cluster, must not interfere with MapReduce capabilities, and must not store keys on hard drives along with the encrypted data – keys must be handled by a secure key manager. Eavesdropping: We use SSL and TLS encryption to protect network communications. Hadoop offers SSL, but its implementation is limited to client connections. Cloudera offers good integration of TLS; otherwise look for third party products to close this gap. Name and data node protection: By default Hadoop HTTP web consoles (JobTracker, NameNode, TaskTrackers, and DataNodes) allow access without any form of authentication. The good news is that Hadoop RPC and HTTP web consoles can be configured to require Kerberos authentication. Bi-directional authentication of nodes is built into Hadoop, and available in some other big data environments as well. Hadoop’s model is built on Kerberos to authenticate applications to nodes, nodes to applications, and client requests for MapReduce and similar functions. Care must be taken to secure granting and storage of Kerberos tickets, but this is a very effective method for controlling what nodes and applications can participate on the cluster. Application protection: Big data clusters are built on web-enabled platforms – which means that remote injection, cross-site scripting, buffer overflows, and logic attacks against and through client applications are all possible avenues of attack for access to the cluster. Countermeasures typically include a mixture of secure code development practices (such as input validation, and address space randomization), network segmentation, and third-party tools (including Web Application Firewalls, IDS, authentication, and authorization). Some platforms offer built-in features to bolster application protection, such as YARN’s web application proxy service. Archive protection: As backups are largely an intractable problem for big data, we don’t need to worry much about traditional backup/archive security. But just because legitimate users cannot perform conventional backups does not mean an attacker would not create at least a partial backup. We need to secure the management plane to keep unwanted copies of data or data nodes from being propagated. Access controls, and possibly network segregation, are effective countermeasures against attackers trying to gain administrative access, and encryption can help protect data in case other protections are defeated. In the end, our big data security recommendations boil down to a handful of standard tools which can be effective in setting a secure baseline for big data environments: Use Kerberos: This is effective method for keeping rogue nodes and applications off your cluster. And it can help protect web console access, making administrative functions harder to compromise. We know Kerberos is a pain to set up, and (re-)validation of new nodes and applications takes work. But without bi-directional trust establishment it is too easy to fool Hadoop into letting malicious applications into the cluster, or into accepting introduce malicious nodes – which can then add, alter, or extract data. Kerberos is one of the most effective security controls at your disposal, and it’s built into the Hadoop infrastructure, so use it. File layer encryption: File encryption addresses two attacker methods for circumventing normal application security controls. Encryption protects in case malicious users or administrators gain access to data nodes and directly inspect files, and it also renders stolen files or disk images unreadable. Encryption protects against two of the most serious threats. Just as importantly, it meets our requirements for big data security tools – it is transparent to both Hadoop and calling applications, and scales out as the cluster grows. Open source products are available for most Linux systems; commercial products additionally offer external key management, trusted binaries, and full support. This is a cost-effective way to address several

We have published the Endpoint Security Management Buyer’s Guide paper, which provides a strategic view of Endpoint Security Management, addressing the complexities caused by malware’s continuing evolution, device sprawl, and mobility/BYOD. The paper focuses on periodic controls that fall under good endpoint hygiene (such as patch and configuration management) and ongoing controls (such as device control and file integrity monitoring) to detect unauthorized activity and prevent it from completing. The crux of our findings involve use of an endpoint security management platform to aggregate the capabilities of these individual controls, providing policy and enforcement leverage to decrease cost of ownership, and increasing the value of endpoint security management. This excerpt says it all: Keeping track of 10,000+ of anything is a management nightmare. With ongoing compliance oversight and evolving security attacks against vulnerable endpoint devices, getting a handle on managing endpoints becomes more important every day. We will not sugarcoat things. Attackers are getting better – and our technologies, processes, and personnel have not kept pace. It is increasingly hard to keep devices protected, so you need to take a different and more creative view of defensive tactics, while ensuring you execute flawlessly – because even the slightest opening provides opportunity for attackers. One of the cool things we ve added to the new Buyer’s Guide format was 10 questions to consider as you evaluate and deploy the technology: What specific controls do you offer for endpoint management? Can the policies for all controls be managed via your console? Does your organization have an in-house research team? How does their work make your endpoint security management product better? What products, devices, and applications are supported by your endpoint security management offerings? What standards and/or benchmarks are offered out of the box for your configuration management offering? What kind of agentry is required by your products? Is the agent persistent or dissolvable? How are updates distributed to managed devices? What do you do to ensure agents are not tampered with? How do you handle remote and disconnected devices? What is your plan to extend your offering to mobile devices and/or virtual desktops (VDI)? Where does your management console run? Do we need a dedicated appliance? What kind of hierarchical management do you support? How customizable is the management interface? What kinds of reports are available out of the box? What is involved in customizing specific reports? What have you done to ensure the security of your endpoint security management platform? Is strong authentication supported? Have you done an application penetration test on your console? Does your engineering team use any kind of secure software development process? You can check out the series of posts we combined into the eventual paper. The Business Impact of Managing Endpoints The ESM Lifecycle Periodic Controls Ongoing Controls – Device Control Ongoing Controls – File Integrity Monitoring Platform Buying Considerations 10 Questions We thank Lumension Security for licensing this research, and enabling us to distribute it at no cost to readers. Check out the full paper in our research library, or download it directly (PDF). Share:

Before I dig into today’s post I want to share a couple observations. First, my new copy of the Harvard Business Review just arrived. The cover story is “Getting Control of Big Data”. It’s telling that HBR thinks big data is a trend important enough to warrant a full spread, and feel business managers need to understand big data and the benefits and risks it poses to business. As soon as I finish this post I intend to dive into these articles. Now that I have just about finished this research effort, I look forward to contrasting what I have discovered with their perspective. Second, when we talk about big data security, we are really referring to both data and infrastructure security. We want to protect the application (or database, if you prefer that term) that manages data, with the end goal of protecting the information under management. If an attacker can access data directly, bypassing the database management system, they will. Barring a direct path to the information, they will look for weaknesses in or ways to subvert the database application. So it’s important to remember that when we talk about database security we mean both data and infrastructure protection. Finally, a point about clarity. Big data security is one of the tougher topics to describe, especially as we here at Securosis prefer to describe things in black and white terms for the sake of clarity. But for just about every rule we establish and every emphatic statement we make, we have to acknowledge exceptions. Given the variety of different big data distributions and add-on capabilities, you can likely find a single instance of every security control described in today’s post. But it’s usually a single security control, like encryption, with the other security controls absent from the various packages. Nothing offers even a partial suite of solutions, much less a comprehensive offering. Today I want to discuss operational security of big data environments. Unlike yesterday’s post that discussed architectural security issues endemic to the platform, it is now time to address security controls of an operational nature. That includes “turning the dials” things like configuration management and access controls, as well as “bolt-on” capabilities such as auditing and security gateways. We see the greatest impact in these areas, and vendors jumping in with security offerings to fill the gaps. Normally when we consider how to secure data repositories, we consider the following major areas: Encryption: The standard for protecting data at rest is encryption to protect data from undesired access. And just because folks don’t use archiving features to back up data does not mean a rogue DBA or cloud service manager won’t. I think two or three of the more obscure NoSQL variants provides encryption for data at rest, but most do not. And the majority of available encryption products offer neither sufficient horizontal scalability nor adequate transparency for use with big data. This is a critical issue. Administrative data access: Each node has an admin, and each admin can read the node’s data if they choose. As with encryption, we need a boundary or facility to provide separation of duties between different administrators. The requirement is the same as on relational platforms – but big data platforms lack their array of built-in facilities, documentation, and third party tools to address requirements. Unwanted direct access to data files or data node processes can be addressed through a combination of access controls, separation of roles, and encryption technologies, but out-of-the box data is only as secure as your least trustworthy administrator. It’s up to the system designer to select controls to close this gap. Configuration and patch management: When managing a cluster of servers, it’s common to have nodes running different configurations and patch levels. And if you’re using dissimilar platforms to support the cluster you need to figure out what how to handle management. Existing configuration management tools work for underlying platforms, and HDFS Federation will help with cluster management, but careful planning is still necessary. I will go more detail about how in the next post, when I make recommendations. The cluster may tolerate nodes cycling without loss of data service interruption, but reboots can still cause serious performance issues, depending on which nodes are affected and how the cluster is configured. The upshot is that people don’t patch, fearing user complaints. Perhaps you have heard that one before. Authentication of applications/clients: Hadoop uses Kerberos to authenticate users and add-on services to the HDFS cluster. But a rogue client can be inserted onto the network if a Kerberos ticket is stolen or duplicated. This is more of a concern when embedding credentials in virtual and cloud environments, where it’s relatively easy to introduce an exact replica of a client app or service. A clone of a node is often all that’s needed to introduce a corrupted node or service into a cluster, it’s easy to impersonate or a service in the cluster, but it requires an attacker to compromise the management plane of your environment, or obtain a backup, of a client. Regardless of it being a pain to set up, strong authentication through Kerberos is one of your principle security tools, it helps solve the critical problem of who can access hadoop services. Audit and logging: One area with a variety of add-on capabilities is logging. Scribe and LogStash are open source tools that integrate into most big data environments, as do a number of commercial products. So you just need to find a compatible tool, install it, integrate it with other systems such as SIEM or log management, and then actually review the results. Without actually looking at the data and developing policies to detect fraud, logging is not useful. Monitoring, filtering, and blocking: There are no built-in monitoring tools to look for misuse or block malicious queries. In fact I don’t believe anyone has ever described what a malicious query might look like in a big data environment – other than crappy MapReduce

Our first post built a case for considering availability as an aspect of security context, rather than only confidentiality and integrity. This has been driven by Denial of Service (DoS) attacks, which are used by attackers in many different ways, including extortion (using the threat of an attack), obfuscation (to hide exfiltration), hacktivism (to draw attention to a particular cause), or even friendly fire (when a promotion goes a little too well). Understanding the adversary and their motivation is one part of the puzzle. Now let’s look at the types of DoS attacks you may face – attackers have many arrows in their quivers, and use them all depending on their objectives and targets. Flooding the Pipes The first kind of Denial of Service attack is really a blunt force object. It’s basically about trying to oversubscribe the bandwidth and computing resources of network (and increasingly server) devices to impact resource availability. These attacks aren’t very sophisticated, but as evidenced by the ongoing popularity of volume-based attacks, fairly effective effective. These tactics have been in use since before the Internet bubble, leveraging largely the same approach. But they have gotten easier with bots to do the heavy lifting. Of course, this kind of blasting must be done somewhat carefully to maintain the usefulness of the bot, so bot masters have developed sophisticated approaches to ensure their bots avoid ISPs penalty boxes. So you will see limited bursts of traffic from each bot and a bunch of IP address spoofing to make it harder to track down where the traffic is coming from, but even short bursts from 100,000+ bots can flood a pipe. Quite a few specific techniques have been developed for volumetric attacks, but most look like some kind of flood. In a network context, the attackers focus on overfilling the pipes. Floods target specific protocols (SYN, ICMP, UDP, etc.), and work by sending requests to a target using the chosen protocol, but not acknowledging the response. Enough of these outstanding requests limit the target’s ability to communicate. But attackers need to stay ahead of Moore’s Law, because targets’ ability to handle floods has improved with processing power. So network-based attacks may include encrypted traffic, forcing the target to devote additional computational resources to process massive amounts of SSL traffic. Given the resource-intensive nature of encryption, this type of attack can melt firewalls and even IPS devices unless they are configured specifically for large-scale SSL support. We also see some malformed protocol attacks, but these aren’t as effective nowadays, as even unsophisticated network security perimeter devices drop bad packets at wire speed. These volume-based attacks are climbing the stack as well, targeting web servers by actually completing connection requests and then making simple GET request and resetting the connection over and over again, with approximately the same impact as a volumetric attack – over-consumption of resources effectively knocking down servers. These attacks may also include a large payload to further consume bandwidth. The now famous Low Orbit Ion Cannon, a favorite tool of the hacktivist crowd, has undertaken a similar evolution, first targeting network resources and proceeding to now target web servers as well. It gets even better – these attacks can be magnified to increase their impact by simultaneously spoofing the target’s IP address and requesting sessions from thousands of other sites, which then bury the target in a deluge of misdirected replies, further consuming bandwidth and resources. Fortunately defending against these network-based tactics isn’t overly complicated, as we will discuss in the next post, but without a sufficiently large network device at the perimeter to block these attacks or an upstream service provider/traffic scrubber to dump offending traffic, devices fall over in short order. Overwhelming the Application But attackers don’t only attack the network – they increasingly attack the applications as well, following the rest of attackers up the stack. Your typical n-tier web application will have some termination point (usually a web server), an application server to handle application logic, and then a database to store the data. Attackers can target all tiers of the stack to impact application availability. So let’s dig into each layer to see how these attacks work. The termination point is usually the first target in application DoS attacks. They started with simple GET floods as described above, but quickly evolved to additional attack vectors. The best known application DoS attack is probably RSnake’s Slowloris, which consumes web server resources by sending partial HTTP requests, effectively opening connections and then leaving the sessions open by sending additional headers at regular intervals. This approach is far more efficient than the GET flood, requiring only hundreds of requests at regular intervals rather than constant thousands, and only requires one device to knock down a large site. These application attacks have evolved over time and now send complete HTTP requests to evade IDS and WAF devices looking for incomplete HTTP requests, but they tamper with payloads to confuse applications and consume resources. As defenders learn the attack vectors and deploy defenses, attackers evolve their attacks. The cycle continues. Web server based attacks can also target weaknesses in the web server platform. For example the Apache Killer attack sends a malformed HTTP range request to take advantage of an Apache vulnerability. The Apache folks quickly patched the code to address this issue, but it shows how attackers target weaknesses in the underlying application stack to knock the server over. And of course unpatched Apache servers are still vulnerable today at many organizations. Similarly, the RefRef attack leverages SQL injection to inject a rogue .js file onto a server, which then hammers a backend database into submission with seemingly legitimate traffic originating from an application server. Again, application and database server patches are available for the underlying infrastructure, but vulnerability remains if either patch is missing. Attackers can also target legitimate application functionality. One example of such an attack targets the search capability within a web site. If an attacker scripts a series of overly broad

Our days just keep getting longer and longer. When the kids were younger afternoons and early evenings were a blur of activities, homework, hygiene, meals, reading, and then bed. Most nights the kids were in bed by 8:30 and the Boss and I could eat in peace, watch a little TV, catch up, and basically take a breath. But since XX1 entered middle school, things have changed. The kids have adapted fine. The Boss and me, not so much. Now it’s all about dividing and conquering. I handle the early shift and get the twins ready for school. They are on the bus by 7:20 and then I usually head over to some coffee shop and start working. The Boss handles XX1 and has her on the bus at 8:10, and then she starts her day of working through all the crap that has to happen to keep the trains running. The twins get off the bus at 3pm or so. Then it’s homework time and shuttling them off to activities. XX2 isn’t home until 4:30; then some days she can get an hour or two of work in, and other days she can’t. Inevitably she gets home from dance and has to start her homework. She usually wraps up around 10, but I usually get enlisted to help with the writing or math. And there are nights when XX1 is up until 11 or even later trying to get everything done. So there is no peace and quiet. Ever. We find ourselves staying up past midnight because those 90 minutes after all the kids go to bed are the only time we have to catch up and figure out the logistics for the next day. Which assumes that I don’t have work I need to get done. I know Rich has it harder right now with his 2 (and soon to be 3) kids under 4. I remember those days, and don’t miss the sleep deprivation. And I’m sure he misses sleeping in on weekends. At least I get to do that – our kids want us to sleep as late a possible, so they can watch more crappy shows on Nick Jr. But I do miss the quiet evenings after the kids were sleeping. Those are likely gone for a little while. For the next 9 years or so, the kids own the night. –Mike Photo credits: We Own The Night originally uploaded by KJGarbutt Heavy Research We’re back at work on a variety of blog series, so here is a list of the research currently underway. Remember you can get our Heavy Feed via RSS, where you can get all our content in its unabridged glory. And you can get all our research papers too. Defending Against Denial of Service (DoS) Attacks Introduction Securing Big Data Architectural Issues Security Issues with Hadoop Incite 4 U Responsible is in the eye of the beholder: My personal views on disclosure have changed a lot over the years. If you haven’t changed your views in the last 10 years you are either a hermit or a religious zealot – the operating environment has changed a lot. And the longer I have watched (and participated) in the debate, the more I realize it seems to be more about egos than the good of the public. And I fully mean this on all sides – researchers, vendors, users (but less), government, and pundits. Take Richard Bejtlich’s latest post on vendors or researchers going public when they find command and control servers. He expresses the legitimate concern that whoever finds and publicizes this information may often be blowing a law enforcement or intelligence operation. On the other hand law enforcement and intelligence agencies sure don’t make it easy to report these findings, and researchers might be sitting there watching people get compromised (including their customers). This is a hard problem to solve – if we even can. Just ask the Stratfor guys who were materially damaged while the FBI was not only watching, but ‘assisting’ the attack via their confidential informant. Better communication and cooperation is probably the answer, but I have absolutely no confidence that can happen at scale, even if some companies (including Richard’s employer) have those ties. No, I don’t have an answer, but we all need open minds, and probably a bit less ego and dogma. – RM The mark of a mature market: You can joke about the SC Magazine reviews operation. How they rarely actually test products, but instead sit through WebEx demos run by experienced SEs who make every product seem totally awesome. And that may be true but it’s not the point. It’s about relative ratings as an indicator of a mature market. If you look at SC Mag’s recent group test on email security devices, you’ll see 9 out of 10 products graded higher than 4 1/4 stars (out of 5). That 10th product must really suck for 3 stars. But even if you deflate the ratings by a star (or two) you’ll see very little outward differentiation. Which means the product category has achieved a lowest common denominator around a base set of features. So how do you decide between largely undifferentiated offerings? Price, of course… – MR Progress, at a glacial pace: I disagree with Mike Mimoso about the Disconnect Between Application Development and Security Getting Wider. We have been talking about this problem for almost a decade with not much improvement, so it certainly can feel that way. But I can say from personal experience that 10 years ago even the companies who developed security software knew nothing about secure code development, while now these is a better than even chance that someone on the team knows a little security. Have their processes changed to embrace security? Only at a handful of firms. The issue, in my opinion, is and has been the invisible boundary around the dev team to shield them from outside influence. Developers are largely isolated to keep

There was a lot of big news this week in the security world, most of it bad. Even if you skip the intro, make sure you read the Top News section. Rich here, Growing up I was – and this might shock some of you – a bit of a nerd. I glommed onto computers and technology pretty much as soon as I had access to them, and when I didn’t I was reading books and watching shows that painted wonderful visions of the future. I was a hacker before I ever heard the word, constantly taking things apart to see how they worked, then building my own versions. Technology is thus very intuitive to me. I never had to learn it in the same way as people coming to computers and electronics later in life. I began programming so early in life that it keyed into the same (maybe) brain pathways that allow children to learn multiple languages with far more facility than adults. While my generational peers are far more comfortable with technology and computers than our parents, I generally still have a leg up due to my early immersion. I naturally assumed that the generations following me would grow up closer to my experiences than my less geeky peers. But much to my surprise, although they are very comfortable with computers, they don’t have the damnedest idea of how they work or how to bend them to their own will. Unless it involves cats and PowerPoint. Lacking teachers who understood tech, they grow up learning how to use Office, not to program or dig into technology beyond the shallowest surface levels. As I have started raising my own kids, I worry about how to get them interested in technology, and algorithmic thinking, in a world where iPads put the entire Disney repository a few taps away. I’m not talking about forcing them to become programmers, but taking advantage of their brain plasticity to reinforce logical thinking and problem solving, and at least convey a sense of deeper exploration. This really did worry me, but over the past few months I have realized that as a parent I have the opportunity to engage my children to degrees my parents couldn’t possibly imagine. It was a big deal when I got my first Radio Shack electronics kit. It was even a bigger deal when I made my first radio. My kids? This past weekend my 3.5 and 2 year old got to play with their first home-built LEGO robot. Yes, I did most of the building and all the programming, but I could see them learning the foundation of how it worked and what we could make it do. Building a robot to play with our cat is a hell of a lot more exciting than putting a picture of a cat in a PowerPoint. This is barely the start. I grew up pushing ASCII pixels on screens. They will grow up programming, and perhaps designing, autonomous flying drones with high-definition video feeds. I grew up making simple electric candles that would turn on in a dark room. They will be able to create wonderful microcontroller-based objects they then embed into 3-D printed housings. There’s no guarantee they will actually be interested in these things, but social engineering isn’t just for pen testing. Hopefully I can manipulate the crap out of them so they at least get the basics. And, if not, it means more stock fab material for me. I’m biased. I think most of my success in life is due to a combination of logical thinking, the exploratory drive of a hacker, and a modest facility with the written word. As a parent I now have tools to teach these skills to my children in ways our parents could only dream about. On to the Summary: Webcasts, Podcasts, Outside Writing, and Conferences Rich quoted on the myth of cyberinsurance. Mike’s Security Intelligence post at Dark Reading. Favorite Securosis Posts Adrian Lane: My Security Fail (and Recovery) for the Week. Gave me a moment of panic. Mike Rothman: Securing Big Data: Architectural Issues. This series is critical for you to learn what’s coming. If it hasn’t already arrived. Rich: David Mortman’s Another Inflection Point. The more we let go of, the more we can do. Other Securosis Posts Defending Against DoS Attacks: The Attacks. Incite 9/27/2012: They Own the Night. New Research Paper: Pragmatic WAF Management. Favorite Outside Posts Adrian Lane: OAuth 2.0 – Google Learns to Crawl. For someone learning just how much I don’t know about authorization, this is a good overview of the high points of the OAuth security discussion. Mike Rothman: 25 Great Quotes from the Princess Bride. 25 YEARS! WTF? I don’t feel that old, but I guess I am. Take a trip down memory lane and remember one of the better movies ever filmed. IMHO, anyway. Rich: Connect with your inner grey hat. The title is a bit misleading, but the content is well stated. You need to change up your thinking constantly. Research Reports and Presentations Pragmatic WAF Management: Giving Web Apps a Fighting Chance. Understanding and Selecting Data Masking Solutions. Evolving Endpoint Malware Detection: Dealing with Advanced and Targeted Attacks. Implementing and Managing a Data Loss Prevention Solution. Defending Data on iOS. Malware Analysis Quant Report. Report: Understanding and Selecting a Database Security Platform. Top News and Posts The big news this week is the compromise and use of an Adobe code signing certificate in targeted attacks. Very serious indeed. Banks still fighting off the Iranian DDoS attacks. OpenBTS on Android. This is the software you use to fake a cell phone base tower. Smart grid control vendor hacked. Yes, they had deep access to their clients, why do you ask? An interview with the author of XKCD. Sudo read this article. PHPMyadmin backdoored. PPTP now really and truly dead. More Java 0day. Seriously, what the hell is going on this week? And to top everything off, a Sophos post