By Adrian Lane
There are several forms of encryption that can encrypt the contents of the database. Each is unique in its level of security, ease of deployment, cost, and performance impact on transaction processing – making the selection process difficult. Further, security and compliance requirements pertaining to encryption are often murky. They key to this process is understanding the requirements and mapping them to the available technologies. The Evaluate phase is commonly the most time consuming if you are working with compliance requirements.
Pay close attention to operations and integration efforts to ensure no hidden are obstacles discovered after deployment. For example, such as finding that tape archiving no longer works, or that user account recovery fails to recover encrypted data. This type of thing is common, so we’ve included it in the process.
- Time to confirm data security & compliance requirements. Gather requirements and have a complete understanding of why you are encrypting the database, and the objectives to be met.
- Time to identify encryption method/tools. Select encryption method (database internal, file/OS, disk, etc.) that fully addresses requirements. Identify the tools or products required.
- Time to identify integration requirements. Understand key management, archiving, and password and disaster recovery requirements; determine what integration work is needed.
- Variable: time to evaluate encryption tools/products. Select vendors, bring in products, and evaluate in terms of requirements.
- Optional: cost to acquire encryption. If the selected encryption solution is not already available, factor in its additional cost.
- Optional: cost to acquire key management. If key management is external to the database and not already purchased, factor in the additional cost of the product.
- Variable: costs for maintenance, licensing, or support services.
Test & Approve
- Time to establish test environment. Verify product in pre-deployment environment.
- Optional: time to archive database and verify. Create system backups and verify.
- Time to install and configure the encryption tool, including (if needed) any key management integration and user accounts for testing.
- Time to test. Time to complete functional testing and operations assurance.
- Optional: time to establish disaster recovery procedures. Encryption based on external key services, or external to the database, requires additional disaster recovery preparation. Verify your disaster recovery process is updated and required resources are allocated.
- Time to collect sign-offs and approval.
Deploy & Integrate
- Time to install encryption engine in production.
- Time to install key management server (if used) and generate keys. Generate master key pairs and database encryption keys, and distribute.
- Time to deploy, encrypt data, and set up user authorization.
- Time to integrate with applications, backups, and authentication. Verify that operational processes are still viable. Perform required application functional tests.
- Time to document. Record requirements and changes to operational policies.
Posted at Friday 29th January 2010 12:00 am
(0) Comments •
Now that we’ve described the Pragmatic Data Security Cycle, it’s time to dig into the phases. As we roll through each of these I’m going to break it into three parts: the process, the technologies, and a case study. For the case study we’re going to follow a fictional organization through the entire process. Instead of showing you every single data protection option at each phase, we’ll focus on a narrow project that better represents what you will likely experience.
Define: The Process
From a process standpoint, this is both the easiest and hardest of the phases. Easy, since there’s only one thing you need to do and it isn’t very technical or complex, hard since it may involve coordination across multiple business units and the quest for executive sponsorship.
- Identify an executive sponsor to support your efforts. Without management support, the rest of the process will be extremely difficult.
- Identify the one piece of information/content/data you want to protect. The definition shouldn’t be too broad. For example, “engineering plans” is too broad, but “engineering plans for project X” is acceptable. Using “PCI/NPI/HIPAA” is acceptable, assuming you narrow it down in the next step.
- Define and model the information you defined in the step above. For totally unstructured content like engineering plans, identify a repository to use for your definition, or any watermarking/labels you are certain will be available to identify and protect the information. For PCI/NPI/HIPAA determine the exact fields/pieces of data to protect. For PCI it might be only the credit card number, for NPI it might be names and addresses, and for HIPAA it might be ICD9 billing codes. If you are protecting data from a database, also identify the source repository.
- Identify key business units with a stake in the information, and contact them to verify the priority, structure, and repositories for this information. It’s no fun if you think you’re going to protect a database of customer data, only to find out halfway through that it’s not really the important one from a business perspective.
That’s it: find a sponsor, identify the category, identify the data/repository, and confirm with the business folks.
None. This is a manual business process and the only technology you need is something to take notes with… or maybe email to communicate.
Define: Case Study
Billy Bob’s Bait Shop and Sushi Outlet is a mid-sized, multi-site retail organization that specializes in “The freshest seafood, for your family or aquatic friends”. Billy Bob’s consists of a corporate headquarters and a few dozen retail outlets in three states. There are about 1,000 employees, and a growing web business due to their capability to ship fresh bait or sushi to any location in the US overnight.
Billy Bob’s is struggling with PCI compliance and wants to avoid a major security breach after seeing the damage caused to their major competitor during a breach (John Boy’s Worms and Grub).
They do not have a dedicated security team, but their CIO designated one of their top network administrators (the former firewall manager) to head up security operations. Frank has a solid history as a network administrator and is familiar with security (including some SANS training and a CISSP class). Due to problems with their first PCI assessment, Frank has the backing of the CIO.
The category of data is PCI. After some research, Frank decides to go with a multilevel definition – at the top is credit card numbers. Since they are (supposedly) not storing them in a database they could feed to any data protection tools, Frank is starting with a regular expression to identify credit card numbers, and then plans on refining it using customer names (which are stored in the database). He is hoping that whatever tools he picks can use a generic credit card number definition for low-priority alerts, and a credit card (generic) tied with a customer name to trigger higher priority alerts. Frank also plans on using violation counts to help find real problems areas.
Frank now has a generic category (PCI), a specific definition (generic regex and customer name from a database) and the repository location (the customer database itself). From the heads of the customer relations and billing, he learned that there are really two databases he needs to worry about: the main transaction processing/records system for the web outlet, and the point of sale transaction processing system for the retail outlets. The web outlet does not store unencrypted credit card numbers, but the retail outlets currently do, and they are working with the transaction processor to fix that. Thus he is adding credit card numbers from the retail database to his list of data sources. Fortunately, they are only stored in the central processing database, and not at the individual retail outlets.
That’s the setup – in our next post we will cover the Discovery process to figure out where the heck all that data is.
Posted at Wednesday 27th January 2010 11:05 pm
(0) Comments •
By Adrian Lane
I have been part of 5 different startups, not including my own, over the last 15 years. Every one of them has sold, or attempted to sell, enterprise software. So it is not surprising that when I provide security advice, by default it is geared toward an enterprise audience. And oddly, when it comes to security, large enterprises are a little further ahead of the curve. They have more resources and people dedicated to the subject than small and medium sized businesses, and their coverage is much more diverse. But security advice does not always transfer well from one audience to the other. The typical SMB IT security team is one person. Or in the case or database security, the DBA and the security practitioner are one and the same. The time they have to spend on learning and performing security tasks are significantly less, and the money they have to spend for security tools and automation is typically minimal.
To remedy that issue I am creating a couple posts for some pragmatic, hands-on tasks for database security. I’ll provide clear and actionable steps to protect your database and the data it stores. This series is geared to small IT shops who just need a straightforward checklist for database security. We’re not covering advanced security here, and we’re not talking about huge database installations with thousands of users, but rather the everyday security stuff you can do in an afternoon. And to keep costs low, I will focus on the built-in database security functions built into the database.
- Access: User and administrative security, and security on the avenues into and out of the database.
- Configuration: Database settings and setup that affect security and protect database functions from subversion or unauthorized alteration. I’ll go into the issue of reliance on the operating system as well.
- Audit: An examination of activity, transactions, and anomalous events.
- Data Protection: In cases where the database cannot protect access to information, we will cover techniques to prevent information from being lost of stolen.
The goal here is to protect the data stored within the database. We often lose sight of this goal as we spend so much time focusing on the container (i.e., the database) and less on the data and how it is used. Of course I will cover database security – much of which will be discussed as part of access control and configuration sections – but I will include security around the data and database functions as well.
Posted at Wednesday 27th January 2010 8:29 pm
(0) Comments •
By Mike Rothman
Maybe it’s the hard-wired pessimist in me, but I never thought I’d live a long life. I know that’s kind of weird to think about, but with my family history of health badness (lots of the Big C), I didn’t give myself much of a chance.
At the time, I must have forgotten that 3 out of my 4 grandparents lived past 85, and my paternal grandma is over 100 now (yes, still alive). But when considering your own mortality, logic doesn’t come into play. I also think my lifestyle made me think about my life expectancy.
3 years ago I decided I needed an attitude adjustment. I was fat and stressed out. Yes, I was running my own business and happy doing that, but it was pretty stressful (because I made it that way) and it definitely took a toll. Then I decided I was tired of being a fat guy. Literally in a second the decision was made. So I joined a gym and actually went. I started eating better and it kind of worked. I’m not where I want to be yet, but I’m getting there.
I’m the kind of guy that needs a goal, so I decided I want to live to 90. I guess 88 would be OK. Or maybe even 92. Much beyond that I think I’ll be intolerably grumpy. I want to be old enough that my kids need to change my adult diapers. Yes, I’m plotting my revenge. Even if it takes 50 years, the tables will be turned.
So how am I going to get there? I stopped eating red meat and chicken. I’m eating mostly plants and I’m exercising consistently and intensely. That’s my plan for now, but I’m also monitoring information sources to figure out what else I can be doing.
That’s when I stumbled upon an interesting video from a TED conference featuring Dan Buettner (the guy from National Geographic) who talked about 9 ways to live to 100, based upon his study of a number of “Blue Zones” around the world where folks have great longevity. It’s interesting stuff and Dan is an engaging speaker. Check it out.
Wish me luck on my journey. It’s a day by day thing, but the idea of depending on my kids to change my diaper in 50 years pretty motivating. And yes, I probably need to talk to my therapist about that.
Photo credit: “and adult diapers” originally uploaded by &y
Incite 4 U
It seems everyone still has APT on the brain. The big debate seems to be whether it’s an apt description of the attack vector. Personally, I think it’s just ridiculous vibrations from folks trying to fathom what the adversary is capable of. Rich did a great FireStarter on Monday that goes into how we are categorizing APT and deflating this ridiculous “cyber-war” mumbo jumbo.
Looking at everything through politically colored glasses – We have a Shrdlu admiration society here at Securosis. If you don’t read her stuff whenever she finds the time to write, you are really missing out. Like this post, which delves into how politics impacts the way we do security. As Rich says, security is about psychology and economics, which means we have to figure out what scares our customers the most. In a lot of cases, it’s auditors and lawyers – not hackers. So we have to act accordingly and “play the game.” I know, you didn’t get into technology to play the game, but too bad. If you want to prosper in any role, you need to understand how to read between the lines, how to build a power base, and how to get things done in your organization. And no, they don’t teach that in CISSP class. – MR
I can haz your cloud in compliance – Even the power of cloud computing can’t evade its cousin, the dark cloud of compliance that ever looms over the security industry. As Chris Hoff notes in Cloud: Security Doesn’t Matter, organizations are far more concerned with compliance than security, and it’s even forcing structural changes in the offerings from cloud providers. Cloud providers are being forced to reduce multi-tenancy to create islands of compliance within their clouds. I spent an hour today talking with a (very very big) company about exactly this problem – how can they adopt public cloud technologies while meeting their compliance needs? Oh sure, security was also on the list – but as on many of these calls, compliance is the opener. The reality is you not only need to either select a cloud solution that meets your compliance needs (good luck), or implement compensating controls on your end, like virtual private storage, and you also need to get your regulator/auditor to sign off on it. – RM
It’s just a wafer thin cookie, Mr. Creosote – Nice job by Michael Coates both on discovering and illustrating a Cookie Forcing attack. In a nutshell, an attacker can alter cookies already set regardless of whether it’s an encrypted cookie or not. By imitating the user in a man-in-the-middle attack, the attacker finds an unsecured HTML conversation, requests an unencrypted meta refresh, and then sends “set cookie” to the browser, which accepts the evil cookie. To be clear, this attack can’t view existing cookies, but can replace them. I was a little shocked by this as I was of the opinion meta refresh had not been considered safe for some time, and because the browser happily conflated encrypted and unencrypted session information. One of the better posts of the last week and worth a read! – AL
IT not as a business, huh? – I read this column on not running IT as a business on infoworld.com and I was astounded. In the mid-90’s running IT as a business was all the rage. And it hasn’t subsided since then. It’s about knowing your customer and treating them like they have a choice in service providers (which they do). In fact, a big part of the Pragmatic CSO is to think about security like a business, with a business plan and everything. So I was a bit disturbed by the premise. Turns out the guy correctly points out that there’s a middle ground. You don’t have to actually price out your services (and do wacky internal chargebacks), but you’d better treat your users as customers. – MR
Trimming the Patch Window – One of the ideas I mentioned in Low Hanging Fruit: Endpoint Security was tightening patch windows. Then I stumbled upon this good article on Dark Reading that goes a layer deeper and provides 4 tips on actually doing that. It’s good stuff, like actually developing a priority list based on criticality of a device, and matching up patch schedules with planned maintenance. Not brain surgery, but good common sense advice. – MR
You like this? I have a bridging VPN to sell you. – I first saw the VPN angle of the Chinese hacker story reported on Dark Reading, much of which was sourced from this post implicating Google’s Virtual Private Network as a medium for the attack. WTF? The thread was later amended with this follow up, where Google officially confirmed the VPN Security review. I am really curious why anyone thinks that VPN security has anything to do with this issue? I still cannot locate a piece of evidence that connects the exploit with VPN security. A medium of conveyance, you know, like the Internet, is a little different than an exploit, like an IE6 0-day. Personally I believe the entire episode was related to coffee. I have strong evidence to support this claim. The Google employee was accidentally served decaf coffee the morning the trojan was dropped onto the machine, and as many Google employees have been seen entering Starbucks since the attack, I am certain coffee played a major factor. That and those little iced lemon cookies. Google did not call me to refute this story, but their silence is telling! These two things could be entirely unrelated, but I doubt it, so I will be the first person to tell you I am not wrong about this. Trust me. – AL
FUD. It tastes like chicken. – Kudos to Russell Thomas for calling out some blatant NetWitness FUD (fear, uncertainty and doubt) mongering, including the obligatory scrunched face guy. The NetWitness folks respond with a treatise on why FUD is OK. I have been on the marketing side a couple of times, and you need to deal with it. Vendors try to create a catalyst for you to return their calls, take their meetings, and hear how their widgets will make your life better. Sometimes trying to scare or confuse you gets thrown into the mix. In fact, sometimes judicious use of FUD internally can help get a project over the finish line. In dealing with vendors it’s another story. I’m a fan of driving the project, as opposed to having a vendor tell me what my problem is, but that’s just me. I think most of those messages are funny and I file them into my marketing buffoonery folder. Try it and you’ll see it’s fun to check those out on a particularly bad day to keep it all in context. At least you don’t have to resort to desperate measures to get a callback. Your customers have a way of finding you just fine. – MR
Shaky Foundations – Every now and then someone sums up pretty much the entire problem with a single paragraph. Gunner nails it when he says, “Here’s the bottom line – basically NONE of the F500 ever designed their systems to run on the Web, they just accreted functionality over time and added layer on top of insecure layer, straw on top of straw, until pretty much everything is connected directly or indirectly to the Web. Now this straw house would not be that big a deal if these enterprises had a half ass dependency on the Web like they did in the early 90s brochure-ware website days, but now the Web runs their businesses.” The truth is, there is only so much security we can continue to layer on top of weak foundations while still achieving results (sort of). Not that most, if any, of you can scrap everything you have and rebuild it from scratch, but as we adopt new technologies (like the cloud) it’s an excellent opportunity to insert security early on in the process and perhaps create a better, stronger, more secure generation of technology. I can dream, can’t I? – RM
Posted at Wednesday 27th January 2010 5:15 am
(1) Comments •
After writing up the Advanced Persistent Threat in this week’s FireStarter, a few people started asking for suggestions on managing the problem.
Before I lay out some suggestions, it’s important to understand what we are dealing with here. APT isn’t some sort of technical term – in this case the threat isn’t a type of attack, but a type of attacker. They are advanced – possessing strong skills and capabilities – and persistent, in that if you are a target they will continue to attempt attacks until they succeed or the costs are greater than the potential rewards.
You don’t just have to block them once so they move on – they will continue to probe and strike until they achieve their goal.
Thus my recommendations will by no means “eliminate” APT. I can make a jazillion recommendations on different technology solutions to block this or that attack technique, but in the end a persistent threat actor will just shift tactics in response. Rather, these suggestions will help detect, contain, and mitigate successful attacks.
I also highly suggest you read Andrew Jaquith’s post, with this quote:
If you fall into the category of companies that might be targeted by a determined adversary, you probably need a counter-espionage strategy – assuming you didn’t have one already. By contrast, thinking just about “APT” in the abstract medicalizes the condition and makes it treatable by charlatans hawking miracle tonics. Customers don’t need that, because it cheapens the threat.
If you believe you are a target, I recommend the following:
- Segregate your networks and information. The more internal barriers an attacker needs to traverse, the greater your chance to detect. Network segregation also improves your ability to tailor security controls (especially monitoring) to the needs of each segment. It may also assist with compartmentalization, but if you allow VPN access across these barriers, segregation won’t help nearly as much. The root cause of many breaches has been a weak endpoint connecting over VPN to a secured network.
- Invest heavily in advanced monitoring. I don’t mean only simple signature-based solutions, although those are part of your arsenal. Emphasize two categories of tools: those that detect unusual behavior/anomalies, and those with extensive collection capabilities to help in investigations once you detect something. Advanced monitoring changes the playing field! We always say the reason you will eventually be hacked is that when you are on defense only, the attacker only needs a single mistake to succeed. Advanced monitoring gives you the same capability – now the attacker needs to execute with greater perfection, over a sustained period of time, or you have a greater chance of detection.
- Upgrade your damn systems. Internet Explorer 6 and Windows XP were released in 2001; these technologies were not designed for today’s operating environment, and are nearly impossible to defend. The anti-exploitation technologies in current operating systems aren’t a panacea, but do raise the barrier to entry significantly. This is costly, and I’ll leave it to you to decide if the price is worth the risk reduction. When possible, select 64 bit options as they include even stronger security capabilities. No, new operating systems won’t solve the problem, but we might as well stop making it so damn easy for the attackers.
Longer term, we also need to pressure our application vendors to update their products to utilize the enhanced security capabilities of modern operating systems. For example, those of you in Windows environments could require all applications you purchase to enable ASLR and DEP (sorry Adobe).
By definition, advanced persistent threats are as advanced as they need to be, and won’t be going away. Compartmentalization and monitoring will help you better detect and contain attacks, and are fairly useful no matter what tactics your opponent deploys. They are also pretty darn hard to implement comprehensively in current operating environments.
But again, nothing can “solve” APT, since we’re talking about determined humans with time and resources, who are out to achieve the specific goal of breaking into your organization.
Posted at Tuesday 26th January 2010 11:02 pm
(0) Comments •
By Adrian Lane
We have three Project Quant for Database Security topics to discuss. The answers to Open Question to the Database Security Community (should we include query analysis as part of the project?), are in. I had exactly three ‘Yes’ responses and three ‘No’ responses. The ‘Yes’ group was consistent, saying this would be helpful. The ‘No’ group was equally consistent, saying “That’s application security and does not belong here.” Which is exactly the internal struggle we had. As the tie breakers, Rich and I are voting to put code review in. It will be brief and we will focus on those tasks in the database realm.
Throughout the series I have differentiated between policies and rules, but it is worth clarifying the distinction, as it may not be obvious.
- Policy: What you want to accomplish, and the outline of a plan for how to go about it. A policy may be comprised of one of more rules.
- Rule: In this context I am talking about the technical component that gets the work done. This is the code, script, or query that performs the task.
As an example, let’s say you want to block SQL injection. That policy might state that you will block queries with specific patterns. If you are aware of a half dozen specific patterns, you might have six specific rules to check against to inbound queries. Or you might have a policy to check databases for buffer overflow attacks. You could have a single rule that checks to see if the database is patched to fix the exploit, or you could use two or three scripts that attempt to exploit the buffer overflow. Tools and platforms such as DAM, VA, or auditing provide a layer of abstraction for you; so you create a policy and the tool builds the rule for you.
Finally, we are looking for input, comments, and suggestions on both the process and metrics we are creating. There is no “industry standard” for database security, and what companies spend varies radically. We could ask “What do you spend today on database security?” but frankly we doubt you know. That’s not intended to be insulting, it’s just that from the enterprise to small single-DBA IT organizations, this spending is rarely tracked. Or the responsibility is shared across multiple people with other duties. If we asked how much time you spend on database security in any given month, would you have an answer? Would it be a guess?
Posted at Tuesday 26th January 2010 4:45 pm
(0) Comments •
By Mike Rothman
To wrap up my low hanging fruit series (I believe Rich and Adrian will be doing their own takes), let’s talk about security management. Yes, there were lots of components of each in the previous LHF posts (network security & endpoint security) that had “management” components, but now let’s talk about the discipline of management, not necessarily the tools.
Think and Be Program
Some folks would rather think and be rich, but if you do security for a living, you need to be thinking about a security program. To be clear, establishing a security program is the single hardest thing any security professional has to do. Period. Nothing else comes close in heartburn, futility, angst, or importance. The folks residing in a hamster wheel of pain (a great term coined by Andy Jaquith, I think) tend to spend most of their time in fire-fighting mode. OK, being honest, they spend all their time fire-fighting.
That also means a program is not really low hanging fruit (it’s more like skyscraper hanging fruit), but I don’t think you’ll make much headway with any kind of security management without having the structure of a program in place. Thus, this is really about context and the importance of that context as you look to other security management techniques.
So why is it so hard to get a program off the ground? Per usual, it gets back to shiny objects and your to-do list. It’s just easier to do something else. Senior management doesn’t have to agree to fixing a firewall rule, re-imaging a machine, or patching a bunch of devices. But they do have to buy into a program. Your peers have to agree to think about security before they do things. Since they don’t like to do that, getting consensus is hard. So most folks just don’t do it – and that’s a big mistake.
Without the program in place, your likelihood of success is small. Best of all, you don’t have to implement a full program to greatly increase your chance of success.
Yet, all is not lost. You can start slowly with the program and do a few things (kind of low hanging) to get you going:
Define success: Without a clear and agreed-upon definition of security success, you may as well give up now. So this really has to be the first step in the process.
Communication: How often do you get face time with senior management? It’s probably not enough. Make sure you get an audience as often as you need. In the initial stages probably once a month (if not more often), later on maybe not as much. But if you don’t have something set in stone, scheduled on the calendar, it won’t happen.
Accountability: In most organizations, the security team is not well liked. In order to have any chance to implement a security program, you need to change that perception. That’s done one step at a time. Tell them what you are going to do and then do it. Yes, it seems pretty easy. But if it was really easy, everyone would be doing it, right?
Just to throw in a shameless plug, I discussed how to implement a security program in The Pragmatic CSO. It goes into a lot of detail on how to structure the program and get acceptance with your business leaders.
No matter what time it is, it’s time to revisit your incident response plan. Hopefully you haven’t had to use it lately, but don’t get lulled into a false sense of security. Before long you’ll be compromised, and whether you live to fight another day has everything to do with how you respond to the incident.
The worst time to learn your IR plan sucks is when you are in the middle of an attack. First make sure senior management understands roles and responsibilities. Who runs point for what? When do the CEO and board need to be notified? When does law enforcement get involved? All of this needs to be documented and agreed upon.
Next run simulations and practice. Lots of my practitioner friends practice using live ammo, but if you aren’t under constant attack, then you’ll need to schedule time to practice. Yes, shiny objects and fires to fight make it hard to carve out the time to practice the IR process, but don’t neglect your preparation.
If there is anything the recent APT (advanced persistent threat) hysteria has shown, it’s that we have little chance against a well-funded and patient attacker. The only chance we have is to figure out they are in the house as soon as possible. I call this Reacting Faster, which of course Rich has to improve by reminding us all to React Faster, and Better.
The point remains that we don’t know where the attacks are coming from (0-day, by definition, means you don’t know about it, so it’s pretty laughable when an IPS vendor says they can protect against a 0-day attack), so we’d better get better at detecting funky behavior.
Anomaly detection is your friend. You need to monitor everything you can, baseline the “normal” course of events, and look for something that is not normal. That gives you something to investigate, as opposed to the literally infinite places where you could be looking for an attack.
Logging: Your regulations say you need to log stuff, so you probably have some rudimentary logging capability in place. Or you are looking at one. That’s a good idea because all security management starts with data, and a good portion of your data is in log files. So having an automated mechanism to gather and parse logs is a critical first step.
Change detection: Malware tends to leave a trail. Well, most malware anyway. To change behavior usually requires some kind of operating system file change. So seeing those changes will usually give you an indication that something is wrong. Look at key network devices and servers, since those are the interesting targets.
Network behavioral analysis: Network flow analysis yields some very interesting perspective on what folks are doing with your corporate IT assets. It’s also very hard to hide an attack (especially exfiltrating data) from the network. So monitoring network activity can provide another treasure trove of information useful for detecting attacks.
There are plenty of commercial tools for logging, change detection, and NBA. There are also open source options as well. What’s important is not how you solve the problem or how much money you spend, but rather that you are thinking about monitoring now, given the reality that you will get pwned, it’s just a matter of when.
Notice I didn’t mention SIEM or correlation in any of these posts. There is nothing low-hanging at all about SIEM, which requires a lot of time and money to get right. If you make the investment, correlating a lot of disparate sources does help. But be clear that it is an investment, and in most cases a rather large one.
Full Packet Capture
You’ll hear a lot about full packet capture over the next few months. This capability involves capturing every packet that traverses your network. A few vendors will try to make the case that full packet capture would have alerted organizations to the APT (and pretty much every other attack) sooner. Maybe that’s true, maybe it isn’t. But that isn’t the point, which is to remember that data is your friend.
Data is essential to incident response, which you’ll learn the first time you “lose” data as the forensics folks are trying to figure out what happened. Since a large part of my security philosophy is based on reacting faster, you need data – lots of it. Thus I’m a fan of full packet capture, where possible. Certainly on sensitive network segments (like those housing CC data) at a minimum.
Yes, it’s a lot of data, but through the wonders of Moore’s Law and some smart math folks, it’s actually possible today. Not necessarily cheap, but definitely possible.
The last of the low hanging fruit I’ll leave you with involves prioritizing your daily activities. Your to-do list isn’t going to be getting any smaller. You aren’t going to be getting more resources, and odds are your budget will remain tight. Which means you have to work smarter, not harder. More efficiently, as opposed to throwing money and resources at things.
That’s why I’m such an advocate for a strong security program, which defines your top priorities and (if done correctly) makes it easy to determine what you need to be working on at any given time. Without that kind of roadmap (agreed upon by the key influencers), all of your activity and decisions are open to interpretation. OK, you’ll be second guessed no matter what you do, but if you have told folks what your priorities are ahead of time, you’ll be able to point back to that.
Posted at Monday 25th January 2010 9:45 pm
(2) Comments •
By Adrian Lane
We have already covered the Monitoring phase, for examining activity and database transactions. In monitoring mode, database activity monitoring (DAM) platforms are deployed “out of band”, collecting activity and generating alerts as a third party observer. DAM can also be used to block dubious queries and enforce proper database use. The typical database activity monitoring customer does not employ blocking and can skip this step. For those who do employ DAM to protect databases, understand that we are differentiating between monitoring and protection for several reasons.
- Blocking is a more advanced DAM feature that can have serious side effects, and is typically employed only after monitoring policies are successfully in place.
- Policies are based on information discovered through monitoring.
- Blocking rules are commonly predicated on comparison to a known behavioral profile, with the profile built over time from monitoring output.
- Blocking warrants more carefully crafted rules to enforce business policies, and on a more practical level additional routine maintenance – as application queries, database structures, and use cases evolve.
Logically it makes sense to include blocking under the protection phase, but we do it this way because it’s much easier to account for the time and resources by splitting blocking into a separate task from normal activity monitoring. The sequence of events is pretty straight forward: you will have something specific you want to address, such as ad-hoc database connections or SQL injection. Identify the databases, create the policy that describes the goal, and then specify the DAM rule or rules that perform the work. DAM tools often provide a level of abstraction, so you set the pre-defined policy, and the rules that form that policy are implemented on your behalf.
- Time to identify what activity to block. This could be specific queries, connection types, users, or simply undefined actions.
- Time to identify databases to protect.
- Time to create rules and polices. Based on the activity you want to block, fully describe what activity you want to guard against, and define the rules to implement the policy.
- Time to specify blocking method. Depending upon the platform, there are options on how to block activity: within the database, dropping connections, interception of query, etc. Account for the time it takes to compare and select the option you want.
- Time to specify incident handling & review.
- Time to deploy blocking rules.
- Optional: time to deploy additional functions. Add installation and configuration costs for features required to block activity. This may include reconfiguration of the network or redeployment of the DAM product.
- Optional: time to build behavioral profiles. If your blocking methodology relies upon user behavior baselines you need to collect activity for comparison.
- Optional: time to integrate with existing systems. If event handling for blocked activity if different than for monitored events, add incremental costs of additional processes or integration work that needs to be performed and was not included in the Monitoring task of the Secure phase.
- Variable: Time to evaluate effectiveness. Evaluate false positive and false negatives and adjustment policies.
- Time to document policies and event handling.
Posted at Monday 25th January 2010 7:16 pm
(0) Comments •
There’s been a lot of talk on the Interwebs recently about the whole Google/China thing. While there are a few bright spots (like anything from the keyboard of Richard Bejtlich), most of it’s pretty bad.
Rather than rehashing the potential attack details, I want to step back and start talking about the bigger picture and its potential implications. The Google hack – Aurora or whatever you want to call it – isn’t the end (or the beginning) of the Advanced Persistent Threat, and it’s important for us to evaluate these incidents in context and use them to prepare for the future.
- As usual, instead of banding together, parts of the industry turned on each other to fight over the bones. On one side are pundits claiming how incredibly new and sophisticated the attack was. The other side insisted it was a stupid basic attack of no technical complexity, and that they had way better zero days which wouldn’t have ever been caught. Few realize that those two statements are not mutually exclusive – some organizations experience these kinds of attacks on a continuing basis (that’s why they’re called “persistent”). For other organizations (most of them) the combination of a zero-day with encrypted channels is way more advanced than what they’re used to or prepared for. It’s all a matter of perspective, and your ability to detect this stuff in the first place.
- The research community pounced on this, with many expressing disdain at the lack of sophistication of the attack. Guess what, folks, the attack was only as sophisticated as it needed to be. Why burn your IE8/Win7 zero day if you don’t have to? I don’t care if an attack isn’t elegant – if it works, it’s something to worry about.
- Do not think, for one instant, that the latest wave of attacks represents the total offensive capacity of our opponents.
- This is espionage, not ‘warfare’ and it is the logical extension of how countries have been spying on each other since the dawn of human history. You do not get to use the word ‘war’ if there aren’t bodies, bombs, and blood involved. You don’t get to tack ‘cyber’ onto something just because someone used a computer.
- There are few to no consequences if you’re caught. When you need a passport to spy you can be sent home or killed. When all you need is an IP address, the worst that can happen is your wife gets pissed because she thinks you’re browsing porn all night.
- There is no motivation for China to stop. They own major portions of our national debt and most of our manufacturing capacity, and are perceived as an essential market for US economic growth. We (the US and much of Europe) are in no position to apply any serious economic sanctions. China knows this, and it allows them great latitude to operate.
- Ever vendor who tells me they can ‘solve’ APT instantly ends up on my snake oil list. There isn’t a tool on the market, or even a collection of tools, that can eliminate these attacks. It’s like the TSA – trying to apply new technologies to stop yesterday’s threats. We can make it a lot harder for the attacker, but when they have all the time in the world and the resources of a country behind them, it’s impossible to build insurmountable walls.
As I said in Yes Virginia, China Is Spying and Stealing Our Stuff, advanced attacks from a patient, persistent, dangerous actor have been going on for a few years, and will only increase over time. As Richard noted, we’ve seen these attacks move from targeting only military systems, to general government, to defense contractors and infrastructure, and now to general enterprise.
Essentially, any organization that produces intellectual property (including trade secrets and processes) is a potential target. Any widely adopted technology services with private information (hello, ISPs, email services, and social networks), any manufacturing (especially chemical/pharma), any infrastructure provider, and any provider of goods to infrastructure providers are on the list.
The vast majority of our security tools and defenses are designed to prevent crimes of opportunity. We’ve been saying for years that you don’t have to outrun the bear, just a fellow hiker. This round of attacks, and the dramatic rise of financial breaches over the past few years, tells us those days are over. More organizations are being deliberately targeted and need to adjust their thinking. On the upside, even our well-resourced opponents are still far from having infinite resources.
Since this is the FireStarter I’ll put my recommendations into a separate post. But to spur discussion, I’ll ask what you would do to defend against a motivated, funded, and trained opponent?
Posted at Monday 25th January 2010 4:08 pm
(6) Comments •
By Mike Rothman
Back when I was the resident security management expert over at TechTarget (a position since occupied by Mort), it was amazing how many questions I got about the value of certifications. Mort confirms nothing has changed.
Alex Hutton’s great posts on the new ISACA CRISC certification (Part 1 & Part 2) got me thinking that it’s probably time to revisit the topic, especially given how the difficult economy has impacted job search techniques. So the question remains for practitioners: are these certifications worth your time and money?
Let’s back up a bit and talk about the fundamental motivators for having any number of certifications.
- Skills: A belief exists that security certifications reflect the competence of the professional. The sponsoring organizations continue to do their job of convincing folks that someone with a CISSP (or any other cert) is better than someone who doesn’t have one.
- Jobs: Lots of folks believe that being certified in certain technologies makes them more appealing to potential employers.
- Money: Certifications also result in higher average salaries and more attractive career paths. According to the folks who sell the certifications, anyway.
- Ego: Let’s be honest here. We all know a professional student or three. These folks give you their business cards and it’s a surprise they have space for their address, with all the acronyms after their name. Certifications make these folks feel important.
So let’s pick apart each of these myths one by one and discuss.
Sorry, but this one is a resounding NFW. Most of the best security professionals I know don’t have a certification. Or they’ve let it lapse. They are simply too busy to stop what they are doing to take the test. That’s not to say that anyone with the cert isn’t good, but I don’t see a strong relationship between skills and certs.
Another issue is that many of the certification curricula get long in the tooth after a few years. Today’s required skills are quite different than a few years ago because the attack vectors have changed. Unfortunately most of the certifications have not.
Finally, to Alex’s point in the links above, lots of new certifications are appearing, especially given the myths described below. Do your homework and make sure the curriculum makes sense based on your skills, interest, and success criteria.
The first justification for going to class and taking the test usually comes down to employment. Folks think that a CISSP, GIAC, or CISM will land them the perfect job. Especially now that there are 100 resumes for every open position, a lot of folks believe the paper will differentiate them.
The sad fact is that far too many organizations do set minimum qualifications for an open position, which then get enforced by the HR automatons. But I’d wonder if that kind of company is somewhere you’d like to work. Can it be a perfect job environment if they won’t talk to you if you don’t have a CISSP?
So getting the paper will not get you the job, but it may disqualify you from interviewing.
The certification bodies go way out of their way to do salary surveys to prove their paper is worth 10-15% over not having it. I’m skeptical of surveys on a good day. If you’re in an existing job, in this kind of economy, your organization has no real need or incentive to give you more money for the certification.
There has also clearly been wage deflation in the security space. Companies believe they can get similar (if not better) talent for less money, so it’s hard for me to see how a certification is going to drive your value up.
There is something to be said for ego. The importance of confidence in a job search cannot be minimized. It’s one of those intangibles that usually swings decisions in your direction. If the paper makes you feel like Superman, go get the paper. Just don’t get into a scrap with an armed dude. You are not bulletproof, I assure you.
The Right Answer: Stop Looking for Jobs
Most of the great performers don’t look for jobs. They know all the headhunters, they network, they are visible in their communities, and they know about all the jobs coming available – usually before they are available. Jobs come and find them.
So how do you do that? Well, show your kung fu on an ongoing basis. Participate in the security community. Go to conferences. Join Twitter and follow the various loudmouths to get involved in the conversation. Start a blog and say something interesting.
That’s right, there is something to this social networking thing. A recommendation from one of the well-known security folks will say a lot more about you than a piece of paper you got from spending a week in a fancy hotel.
The senior security folks you want to work for don’t care about paper. They care about skills. That’s the kind of place I want to work. But hey, that’s just me.
Posted at Friday 22nd January 2010 4:45 pm
(19) Comments •
One of the most common criticisms of analysts is that, since they are no longer practitioners, they lose their technical skills and even sometimes their ability to understand technology.
To be honest, it’s a pretty fair criticism. I’ve encountered plenty of analysts over the years who devalue technical knowledge, thinking they can rely completely on user feedback and business knowledge. I’ve even watched as some of them became wrapped around the little fingers (maybe middle finger) of vendors who took full advantage of the fact they could talk circles around these analysts.
It’s hard to maintain technical skills, even when it’s what you do 10 hours a day. Personally, I make a deliberate effort to play, experiment, and test as much as I can to keep the fundamentals, knowing it’s not the same as being a full time practitioner. I maintain our infrastructure, do most of the programming on our site, and get hands on as often as possible, but I know I’ve lost many of the skills that got me where I am today. Having once been a network administrator, system administrator, DBA, and programmer, I was pretty darn deep, but I can’t remember the last time I set up a database schema or rolled out a group policy object.
I was reading this great article about a food critic spending a week as a waiter in a restaurant she once reviewed (working for a head waiter she was pretty harsh on) and it reminded me of one of my goals this year. It’s always been my thought that every analyst in the company should go out and shadow a security practitioner every year. Spend a week in an organization helping deal with whatever security problems come up. All under a deep NDA, of course. Ideally we’d rotate around to different organizations every year, maybe with an incident management team one year, a mid-size “do it all” team the next, and a web application team after that.
I’m not naive enough to think that one week a year is the same as a regular practitioner job, but I think it will be a heck of a lot more valuable than talking to someone about what they do a few times a year over the phone or at a conference.
Yep – just a crazy idea, but it’s high on my priority list if we can find some willing hosts and work the timing out.
And don’t forget to RSVP for the Securosis and Threatpost Disaster Recovery Breakfast!
On to the Summary:
Webcasts, Podcasts, Outside Writing, and Conferences
Favorite Securosis Posts
Other Securosis Posts
Favorite Outside Posts
- Rich: Brian Krebs’ Top 10 Ways to Get Fired as a Money Mule. It’s awesome to see Brian’s stuff without the editorial filters of a dead-tree publication, and he’s clearly going strong.
- Mike: Bejtlich on APT – Richard had two great posts this week helping us understand the advanced persistent threat. First, What is APT and What Does It Want? and then the follow-up, Is APT After You? Great stuff about a threat we all need to understand.
- Adrian: Oracle TNS Rootkit. Well done.
- Mort: Why I Don’t Like CRISC by Alex Hutton, and his excellent followup, Why I Don’t Like CRISC, Day Two, call out ISACA on why it’s not time for a risk based certification.
- Meier: Tor Project Infrastructure Updates in Response to Security Breach. While the Tor service itself wasn’t compromised, this just goes to show it can happen to anyone. And, well, update your Tor software to get the new authority keys.
Project Quant Posts
Top News and Posts
Blog Comment of the Week
Remember, for every comment selected, Securosis makes a $25 donation to Hackers for Charity. This week’s best comment comes from Fernando Medrano in response to Mike’s FireStarter: Security Endangered Species List.
While I do agree with many of the posts and opinions on this site, I disagree in this case. I believe AV and HIPS are still important to the overall protection in depth architecture. Too many enterprises still run legacy operating systems or unpatched software where upgrading could mean significant time and money. While in a perfect world I would love having all systems on the latest operating system with the latest patches, that just isn’t realistic in every scenario.
I also don’t believe that white listing can function as a complete replacement to AV, just as a compliment. I cannot speak with complete authority to this subject as I have not had experience with many products. However I could envision cases such as the Adobe exploits that might run as part of Adobe (which white listing policy might permit) yet executing embedded malicious code.
HIPS is referred to in this article as signature based, however most of the HIPS products I have used have had little or no use of signatures. HIPS products which I have experience with learn the system calls of an application and map out their logical flow. Any deviations from this flow are then blocked. This is more of a white listing technique than black listing. I may have missed some research done on the effectiveness of this technique, but I see this as a great compliment to AV and white listing on high priority systems.
Posted at Friday 22nd January 2010 5:30 am
(0) Comments •
Back in Part 1 of our series on Pragmatic Data Security we covered some of the guiding concepts of the process, and now it’s time to dig in and show you the process itself.
Before I introduce the process cycle, it’s important to remember that Pragmatic Data Security isn’t about trying to instantly protect everything – it’s a structured, straightforward process to protect a single information type, which you then expand in scope incrementally. It’s designed to answer the question, “How can I protect this specific content at this point in time, in my existing environment?” rather than, “How can I protect all my sensitive data right now?” Once we nail down one type of data, then we can move on to other sensitive information. Why? Because as we mentioned in Part 1, if you start with too broad a scope you dramatically increase your chance of failure.
I previously covered the cycle in another post, but for continuity’s sake here it is, slightly updated:
- Define what information you want to protect (specifically – not general data classification). I suggest something very discrete, such as private customer data (specify which exact fields), or engineering documents for a specific project.
- Discover where it’s located (using any of various tools/techniques, preferably automated, such as DLP, rather than manually).
- Secure the data where it’s stored, and/or eliminate data where it shouldn’t be (access controls, encryption).
- Monitor data usage (various tools, including DLP, DAM, logs, & SIEM).
- Protect the data from exfiltration (DLP, USB control, email security, web gateways, etc.).
For example, if you want to protect credit card numbers you’d define them in step 1, use DLP content discovery in step 2 to locate where they are stored, remove them or lock the repositories down in step 3, use DAM and DLP to monitor where they’re going in step 4, and use blocking technologies to keep them from leaving the organization in step 5.
For the rest of this series we’ll walk through each step, showing what you need to do and tying it all together with a use case.
Posted at Thursday 21st January 2010 7:25 pm
(4) Comments •
By Adrian Lane
Database auditing is the examination of audit or transaction logs to track changes to data or database structure. Databases auditing is not specifically listed as a requirement in most compliance initiatives, but in practice it fills an essential role by providing an accurate and concise history of business processes, data usage, and administrative tasks – all necessary elements for policy enforcement. As such, most audit requirements center on tracking a specific set of users, objects, or data elements within the database. Auditing capabilities are built into all relational database platforms, and most of the major platforms offer more than one way to collect transactional information. You may choose to supplement native database auditing with external data sources, but for the scope of this project, we will stick with the more common built-in auditing.
The metrics gathering for database auditing covers scoping the project to understand which databases need which controls, determining how to configure auditing capabilities to meet your requirements, and then periodically collecting the audit trails generated. Day to day management of the audit trails is often an issue, depending upon how many transaction types you want to track. On high-volume transaction severs the data files grow quickly, requiring archival of the audit files so data is not lost, and instructing the database to truncate logs if necessary to avoid filling disk drives to capacity.
- Time to identify databases.
- Time to identify security goals and compliance requirements. Understand the motivation to audit database events and the needs of external stakeholders.
- Time to select data collection methods. Select audit methods that provide necessary data and meet operational requirements.
- Time to identify users, objects, and transactions of interest. Audits seldom require all activity or transactions to be collected, so specify subset needed.
- Time to specify filtering. To reduce processing and storage requirements, specify audit configuration settings to filter out unneeded events.
- Time to set up and configure auditing.
- Time to integrate with existing systems. If sending data to third party SIEM, log management, or reporting tools, set up data collection.
- Time to implement log file management & clean up.
Document & Report
- Time to document.
- Time to generate reports.
Posted at Thursday 21st January 2010 6:00 pm
(0) Comments •
By Mike Rothman
Getting back to the Low Hanging Fruit series, let’s take a look at the endpoint and see what kinds of stuff we can do to increase security with a minimum of pain and (hopefully) minor expense. To be sure we are consistent from a semantic standpoint, I’m generally considering computing devices used by end users as “endpoints.” They come in desktop and laptop varieties and run some variant of Windows. If we had all Mac endpoints, I’d have a lot less to do, eh?
Yes, that was a joke.
Run Updated Software and Patch
We just learned (the hard way) that running old software is a bad idea. Again. That’s right, the Google hack targeted IE6 on XP. IE6? Really? Yup. A horrifyingly high number of organizations are stuck in a browser/OS time warp.
So, if you need to stick with XP, at least make sure you have SP3 running. It seems Windows 7 finally makes the grade, so it’s time to start planning those upgrades. And yes, maybe MSFT got it right this time. Also make sure to use IE7 or IE8 or Firefox (with NoScript). Yes, browsers will have problems. But old browsers have a lot of problems.
Also make sure your Adobe software remains up to date. The good news is that Adobe realizes they have an issue, and I expect they’ll make big investments to improve their security posture. The bad news is that they are about 5 years behind Microsoft and will emerge as the #1 target of the bad guys this year.
Finally, make sure you tighten patch windows as tightly as possible for the high risk, highly exploitable applications, like browsers and Adobe software. Studies have proven that it’s more important to patch thoroughly, as opposed to quickly. But as seen this past week, it takes one day to turn a proof of concept browser 0-day into a weaponized exploit, so for these high risk apps – all bets are off. As soon as a browser (or Adobe) patch hits, try to get it deployed within days. Not weeks. Not months!
Use Anti-Exploitation Technology
Microsoft got a bad rap on security and some (OK, most) of it was deserved. But they have added some capabilities to the base OS that make sense. Like DEP (Data Execution Prevention – also check out the FAQ) and ASLR (Address Space Layout Randomization). These technologies make it much harder to gain control of an endpoint through a known vulnerability.
So make sure DEP and ASLR are turned on in your standard build. Make sure your endpoint checks confirm these two options remain selected. And most importantly, make sure the apps you deploy actually use DEP and ASLR. IE7 and IE8 do. IE6, not so much. Adobe’s stuff – not so much. And there you have it.
To be clear, anti-exploitation technology is not the cure for cancer. It does help to make it harder to exploit the vulnerabilities in the software you use. But only if you turn it on (and the applications support it). Rich has been writing about this for years.
Enforce Secure Configurations
I have to admit to spending a bit too much time in the Center for Internet Security’s brainwashing course. I actually believe that locking down the configuration of a device will reduce security issues. Those of you in the federal government probably have a bit of SCAP on the brain as well.
You don’t have to follow CIS to the letter. But you do have to shut down non-critical services on your endpoints. And you have to check to make sure those configurations aren’t being messed with. So that configuration management thingy you got through Purchasing last year will come in handy.
Encrypt Your Laptops
How many laptops have to be lost and how many notifications sent out to irate customers because some jackass leaves their laptop on the back seat of their car? Or on the seat of an airplane? Or anywhere else where a laptop with private information will get pinched? Optimally you shouldn’t allow private information on those mobile devices (right, Rich, DLP lives!), but this is the real world and people take stuff with them. Maybe innocently. Maybe not, but all the same – they have stuff on their machines they shouldn’t have.
So you need to encrypt the devices. Bokay?
VPN to Corporate
Let’s stay on this mobile user riff by talking about all the trouble your users can get into. A laptop with a WiFi card is the proverbial loaded gun and quite a few of your users shoot themselves in the foot. They connect on any network. They click on any emails. They navigate to those sites.
You can enforce VPN connections when a user is mobile. So all their traffic gets routed through your network. It goes through your gateway and your policies get enforced. Yes, smart users can get around this – but how many of your users are smart that way? All the same, you probably have a VPN client on there anyway. So it’s worth a try.
Let’s talk about probably the cheapest of all the things you can do to positively impact on your security posture. Yes, you can train your users to not do stupid things. Not to click on those links. Not to visit those sites. And not to leave their laptop bags exposed in cars. Yes, some folks you won’t be able to reach. They’ll still do stupid things and no matter what you say or how many times you teach, you’ll still have to clean up their machines – a lot. Which brings us to the last of the low hanging fruit…
When in doubt, reimage…
Yes, you need to invest in a tool to make a standard image of your desktop. You will use it a lot. Anytime a user comes in with a problem – reimage. If the user stiffs you on lunch, reimage. If someone beats you with a pair of aces in the hole, right – reimage.
Before you go on a reimaging binge, make sure to manage expectations. That means making sure the users realize the importance of backing up their systems and keeping their important files on some shared drive. It’s hard to clean up malware infections – most of the time it doesn’t make sense to even try.
Yummy. That low hanging fruit tastes good, eh?
Posted at Thursday 21st January 2010 3:59 pm
(5) Comments •
By Adrian Lane
I periodically write for Dark Reading, contributing to their Database Security blog. Today I posted What Data Discovery Tools Really Do, introducing how data discovery works within relational database environments. As is the case with many of the posts I write for them, I try not to use the word ‘database’ to preface every description, as it gets repetitive. But sometimes that context is really important.
Ben Tomhave was kind enough to let me know that the post was referenced on the eDiscovery and Digital evidence mailing list. One comment there was, “One recurring issue
has been this: If enterprise search is so advanced and so capable
of excellent granularity (and so touted), why is ESI search still
in the boondocks?” I wanted to add a little color to the post I made on Dark Reading as well as touch on an issue with data discovery for ESI.
Automated data discovery is a relatively new feature for data management, compliance, and security tools. Specifically in regard to relational databases, the limitations of these products have only been an issue in the last couple years due to growing need – particularly in accuracy of analysis. The methodologies for rummaging around and finding stuff are effective, but the analysis methods have a little way to go. That’s why we are beginning to see labeling and content inspection. With growing use of flat file and quasi-relational databases, look for labeling and Google type search to become commonplace.
In my experience, metadata-based data discovery was about 85% effective. Having said that, the number is totally bogus. Why? Most of the stuff I was looking for was easy to find, as the databases were constructed by someone was good at database design, using good naming conventions and accurate column definitions. In reality you can throw the 85% number out, because if a web application developer is naming columns “Col1, Col2, Col3, … Col56”, and defining them as text fields up to 50 characters long, your effectiveness will be 0%. If you do not have labeling or content analysis to support the discovery process, you are wasting your time. Further, with some of the ISAM and flat file databases, the discovery tools do not crawl the database content properly, forcing some vendors to upgrade to support other forms of data management and storage. Given the complexity of environments and the mixture of data and database types, both discovery and analysis components must continue to evolve.
Remember that a relational database is highly structured, with columns and tables being fully defined at the time of creation. Data that is inserted goes through integrity checks, and in some cases, must conform to referential integrity checks as well. Your odds of automated tools finding useful information in such databases is far higher because you have definitive descriptions. In flat files or scanned documents? All bets are off.
As part of a project I conducted in early 2009, I spoke with a bunch of attorneys in California and Arizona regarding issues of legal document discovery and management. In that market, document discovery is a huge business and there is a lot of contention in legal circles regarding its use. In terms of legal document and data discovery, the process and tools are very different from database data discovery. From what I have witnessed and from explanations by people who sit on steering committees for issues pertaining to legal ESI, very little of the data is ever in a relational database. The tools I saw were pure keyword and string pattern matching on flat files. Some of the large firms may have document management software that is a little more sophisticated, but much of it is pure flat file server scanning with reports, because of the sheer volume of data. What surprised me during my discussions was that document management is becoming a huge issue as large legal firms are attempting to win cases by flooding smaller firms with so many documents that they cannot even process the results of the discovery tools. They simply do not have adequate manpower and it undermines their ability to process their casefiles. The fire around this market has to do with politics and not technology. The technology sucks too, but that’s secondary suckage.
Posted at Wednesday 20th January 2010 9:15 pm
(0) Comments •