Anti-Malware Effectiveness: The Truth Is out There

One of the hardest things to do in security is to discover what really works. It’s especially hard on the endpoint, given the explosion of malware and the growth of social-engineering driven attack vectors. Organizations like ICSA Labs,, and VirusBulletin have been testing anti-malware suites for years, though I don’t think most folks put much stock in those results. Why? Most of the tests yield similar findings, which means all the products are equally good. Or more likely, equally bad. I know I declared the product review dead, but every so often you still see comparative reviews – such as Rob Vamosi’s recent work on endpoint security suites in PCWorld. The rankings of the 13 tested are as follows (in order): Top Picks: Norton Internet Security 2010, Kaspersky Internet Security 2010, AVG Internet Security 9.0 Middle Tier: Avast, BitDefender, McAfee, Panda, PC Tools, Trend Micro, and Webroot Laggards: ESET, F-Secure, and ZoneAlarm The PCWorld test was largely driven by a recent study into malware detection. But can one lab produce enough information (especially in a single round of testing) to really understand which product works best? I don’t think so, because my research in this area has shown that 3 testing organizations can produce 10 different results. A case in point is the NSS Labs test from August of last year. Their rankings are as follows, ranked by malware detection rates: Trend Micro, Kaspersky, Norton, McAfee, Norman, F-Secure, AVG, Panda, and ESET. Some similarities, but also a lot of differences. More recently, NSS did an analysis of how well the consumer suites detected the Aurora attacks (PDF), which got so much air play in January. Their results were less than stellar: only McAfee entirely stopped the original attack and a predictable variant two weeks out. ESET and Kaspersky performed admirably as well, but it’s bothersome that most of the products we use to protect our endpoints have awful track records like this. If you look at the av-test ratings and then compare them to the NSS tests, the data shows some inconsistencies – especially with vendors like Trend Micro who are ranked much higher by NSS but close to the bottom by av-test; and AVG which is ranked well by av-test but not by NSS. So what’s the deal here? Your guess is as good as mine. I know the NSS guys and they focus their tests pretty heavily on what they call “social engineering malware,” which are legit downloads with malicious code hidden in the packages. This kind of malware is much harder to detect than your standard malware sample that’s been on the WildList for a month. Reputation and advanced file integrity monitoring capabilities are critical to blocking socially engineered malware, and most folks believe these attacks will continue to proliferate over time. Unfortunately, there isn’t enough detail about the tests to really know what they are digging into. But my spidey sense tingles on the objectivity of their findings when you read this report from December by and commissioned by Trend. It concerns me that had Trend close to the bottom in a number of recent tests, but changed their testing methodology a bit with this test, and shockingly: Trend came out on top. WTF? There is no attempt to reconcile the findings across different sets of tests, but I’d guess it has something to do with green stuff changing hands. Moving forward, it would also be great to see some of the application whitelisting products tested alongside the anti-malware suites – for detection, blocking, and usability. That would be interesting. If I’m an end user trying to decide between these products, I’m justifiably confused. Personally, I favor the NSS tests – if only because they provide a lot more transparency on they did their tests. The inconsistent information being published by is a huge red flag for me. But ultimately you probably can’t trust any of these tests, so you have a choice to make. Do you care about the test scores or not? If not, then you buy based on what you would have bought on anyway: management and price. It probably makes sense to disqualify the bottom performers in each of the tests, since for whatever reason the testers figured out how to beat them, which isn’t a good sign. In the end you will probably kick the tires yourself, pick a short list (2 or 3 packages) and run them side by side though a gauntlet of malware you’ve found in your organization. Or contract with testing labs to do a test on your specific criteria. But that costs money and takes time, neither of which we have a lot of. The Bottom Line The truth may be out there, but Fox Mulder has a better chance of finding it than you. So we focus on the fundamentals of protecting not just the endpoints, but also the networks, servers, applications, and data. Regardless of the effectiveness of the anti-malware suites, your other defenses should help you both block and detect potential breaches. Share:

Read Post

Who to Recruit for Security, How to Get Started, and Career Tracks

Today I read two very different posts on what to look for when hiring, and how to get started in the security field. Each clearly reflects the author’s experiences, and since I get asked both sides of this question a lot, I thought I’d toss my two cents in. First we have Shrdlu’s post over at Layer 8 on Bootstrapping the Next Generation. She discusses the problem of bringing new people into a field that requires a fairly large knowledge base to be effective. Then over at Errata Security, Marisa focuses more on how to get a job through the internship path (with a dollop of self-promotion). As one of our industry’s younger recruits, who successfully built her own internship, she comes from exactly the opposite angle. My advice tends to walk a line slightly in the middle of the two, and varies depending on where in security you want to go. When someone asks me how to get started in security I tend to offer two recommendations: Start with a background as a systems and network administrator… probably starting with the lowly help desk. This is how I got started (yes, I’m thus biased), and I think these experiences build a strong foundation that spans most of the tasks you’ll later deal with. Most importantly, they build experience on how the real world works – even more so than starting as a developer. You are forced to see how systems and applications are really used, learn how users interact with technology, and understand the tradeoffs in keeping things running on a day to day basis. I think even developers should spend some time on the help desk or cleaning up systems – while I was only a mediocre developer from a programming standpoint, I became damn good at understanding user interfaces and workflows from the few years I spent teaching people how to unhide their Start menus and organize those Windows 3.1 folders. Read a crapload of action thriller and spy novels, watch a ton of the same kinds of movies, and express your inner paranoid. This is all about building a security mindset, and it is just as important as any technical skills. It’s easy to say “never assume”, but very hard to put it into practice (and to be prepared for the social consequences). You are building a balanced portfolio of paranoia, cynicism, and skepticism. Go do some police ride-alongs, become an EMT, train in a hard martial art, join the military, or do whatever you need to build some awareness. If you were the kid who liked to break into school or plan your escape routes for when the commies (or yankees) showed up, you’re perfect for the industry. You need to love security. The best security professionals combine their technical skills, a security mindset, and an ability to communicate (Marisa emphasized public speaking skills) with a wrapper of pragmatism and an understanding of how to balance the real world sacrifices inherent to security. These are the kinds of people I look for when hiring (not that I do much of that anymore). I don’t care about a CISSP, but want someone who has worked with users and understands technology from actual experience rather than a library shelf, or a pile of certificates. In terms of entry-level tracks, we are part of a complex profession and thus need to specialize. Even security generalists now need to have at least one deep focus area. I see the general tracks as: Operational Security – The CISO track. Someone responsible for general security in the organization. Usually comes from the systems or network track, although systems integration is another option. Secure Coder – Someone who either programs security software, or is responsible for helping secure general (non-security-specific) code. Needs a programmer’s background, but I’d also suggest some more direct user interaction if they’re used to coding in a closet with pizzas slipped under the door at irregular intervals. Security Assessor (or Pen Tester) – Should ideally come out of the coder or operations track. I know a lot of people are jumping right into pen testing, but the best assessors I know have practical experience on the operational side of IT. That provides much better context for interpreting results and communicating with clients. The vulnerability researcher or penetration tester who speaks in absolutes has probably spent very little time on the defensive or operational side of security. You’ll noticed I skipped a couple options – like the security architect. If you’re a security architect and you didn’t come from a programming or operational background, you likely suck at your job. I also didn’t break out security management – mostly since I hate managers who never worked for a living. To be a manager, start at the bottom and work your way up. In any case, if you’re ready for either of those roles you’re past these beginner’s steps, and if you want to get there, this is how to begin. To wrap this up, when hiring look for someone with experience outside security and mentor them through if they have the right mindset. Yes, this means it’s hard to start directly in security, but I’m okay with that. It only takes a couple years in a foundational role to gain the experience, and if you have a security mindset you’ll be contributing to security no matter your operational role. So if you want to work in security, develop the mindset and jump on every security opportunity that pops up. As either a manager or recruit, also understand the different focus of each career track. Finally, in terms of certifications, focus on the ‘low-level’ technical ones, often from outside security. A CISSP doesn’t teach you a security mindset, and as Shrdlu said it’s insane that something that is supposed to take 5 years of operational experience is a baseline for hiring – and we all know it’s easy to skirt the 5-year rule anyway. I’m sure some of you have

Read Post

ESF: Controls: Update and Patch

Running old software is bad. Bad like putting a new iPad in a blender. Bad because all software is vulnerable software, and with old software even unsophisticated bad guys have weaponized exploits to compromise the software. So the first of the Endpoint Security Fundamentals technical controls is to make sure you run updated software. Does that mean you need to run the latest version of all your software packages? Can you hear the rejoicing across the four corners of the software ecosystem? Actually, it depends. What you do need to do is make sure your endpoint devices are patched within a reasonable timeframe. Like one minute before the weaponized exploit hits the grey market (or shows up in Metasploit). Assess your (software) assets Hopefully you have some kind of asset management thing, which can tell you what applications run in your environment. If not, your work gets a bit harder because the first step requires you to inventory software. No, it’s not about license enforcement, it’s about risk assessment. You need to figure out the your software vendors’ track records on producing secure code, and then on patching exploits as they are discovered. You can use sites like US-CERT and Secunia, among others, to figure this out. Your anti-malware vendor also has a research site where you can look at recent attacks by application. You probably hate the word prioritize already, but that’s what we need to do (again). Based on the initial analysis, stack rank all your applications and categorize into a few buckets. High Risk: These applications are in use by 50M+ users, thus making them high-value targets for the bad guys. Frequent patches are issued. Think Microsoft stuff – all of it, Adobe Acrobat, Firefox, etc. Medium Risk: Anything else that has a periodic patch cycle and is not high-risk. This should be a big bucket. Low Risk: Those apps which aren’t used by many (security by obscurity) and tend to be pretty mature, meaning they aren’t updated frequently. Before we move on to the updating/patching process, while you assess the software running in your environment, it makes sense to ask whether you really need all that stuff. Even low-risk applications provide attack surface for the bad guys, so eliminating software you just don’t need is a good thing for everyone. Yes, it’s hard to do, but that doesn’t mean we shouldn’t try. Defining the Update/Patch Process Next you need to define what your update and patching process is going to be – and yes, you’ll have three different policies for high, medium and low risk applications. The good news is your friends at Securosis have already documented every step of this process, in gory detail, through our Patch Management Quant research. At a very high level, the cycle is: Monitor for Release/Advisory, Evaluate, Acquire, Prioritize and Schedule, Test and Approve, Create and Test Deployment Package, Deploy, Confirm Deployment, Clean up, and Document/Update Configuration Standards. Within each phase of the cycle, there are multiple steps. Not every step defined in PM Quant will make sense for your organization, so you can pick and choose what’s required. The requirement is to having a defined, documented, and operational process; and to have answered the following questions for each of your categories: Do you update to the latest version of the application? Within how soon after its release? When a patch is released, how soon should it be applied? What level of testing is required before deployment? In a perfect world, everything should be patched immediately and all software should be kept at the latest version. Unless you are talking about Microsoft Vista <grin>. But we all know the world isn’t perfect and there are real economic and resource dependencies to tightening the patch window and buying software updates – and discovering more bugs in the patches themselves. So all these factors need to be weighed when defining the process and policies. There is no right or wrong answer – it’s a matter of balancing economic reality against risk tolerance. Also keep in mind that patching remote and mobile users is a different animal, and you have to factor that into the process. Many of these folks infrequently connect and may not have access to high-bandwidth connections. Specifying a one-day patch window for installing a 400mb patch at a mobile office in the jungle may not be practical. Tools and Automation Lots of tools can help you automate your software updating and patching process. They range from full-fledged asset and configuration management offerings to fairly simple patching products. It’s beyond the scope of this series to really dig into the nuances of configuration/patch management, but we’ll just say here that any organization with more than a couple hundred users needs a tool. This is a topic we’ll cover in detail later this year. The next endpoint control we’ll discuss is Secure Configurations, so stay tuned. Other posts in the Endpoint Security Fundamentals Series Introduction Prioritize: Finding the Leaky Buckets Triage: Fixing the Leaky Buckets Share:

Read Post

Totally Transparent Research is the embodiment of how we work at Securosis. It’s our core operating philosophy, our research policy, and a specific process. We initially developed it to help maintain objectivity while producing licensed research, but its benefits extend to all aspects of our business.

Going beyond Open Source Research, and a far cry from the traditional syndicated research model, we think it’s the best way to produce independent, objective, quality research.

Here’s how it works:

  • Content is developed ‘live’ on the blog. Primary research is generally released in pieces, as a series of posts, so we can digest and integrate feedback, making the end results much stronger than traditional “ivory tower” research.
  • Comments are enabled for posts. All comments are kept except for spam, personal insults of a clearly inflammatory nature, and completely off-topic content that distracts from the discussion. We welcome comments critical of the work, even if somewhat insulting to the authors. Really.
  • Anyone can comment, and no registration is required. Vendors or consultants with a relevant product or offering must properly identify themselves. While their comments won’t be deleted, the writer/moderator will “call out”, identify, and possibly ridicule vendors who fail to do so.
  • Vendors considering licensing the content are welcome to provide feedback, but it must be posted in the comments - just like everyone else. There is no back channel influence on the research findings or posts.
    Analysts must reply to comments and defend the research position, or agree to modify the content.
  • At the end of the post series, the analyst compiles the posts into a paper, presentation, or other delivery vehicle. Public comments/input factors into the research, where appropriate.
  • If the research is distributed as a paper, significant commenters/contributors are acknowledged in the opening of the report. If they did not post their real names, handles used for comments are listed. Commenters do not retain any rights to the report, but their contributions will be recognized.
  • All primary research will be released under a Creative Commons license. The current license is Non-Commercial, Attribution. The analyst, at their discretion, may add a Derivative Works or Share Alike condition.
  • Securosis primary research does not discuss specific vendors or specific products/offerings, unless used to provide context, contrast or to make a point (which is very very rare).
    Although quotes from published primary research (and published primary research only) may be used in press releases, said quotes may never mention a specific vendor, even if the vendor is mentioned in the source report. Securosis must approve any quote to appear in any vendor marketing collateral.
  • Final primary research will be posted on the blog with open comments.
  • Research will be updated periodically to reflect market realities, based on the discretion of the primary analyst. Updated research will be dated and given a version number.
    For research that cannot be developed using this model, such as complex principles or models that are unsuited for a series of blog posts, the content will be chunked up and posted at or before release of the paper to solicit public feedback, and provide an open venue for comments and criticisms.
  • In rare cases Securosis may write papers outside of the primary research agenda, but only if the end result can be non-biased and valuable to the user community to supplement industry-wide efforts or advances. A “Radically Transparent Research” process will be followed in developing these papers, where absolutely all materials are public at all stages of development, including communications (email, call notes).
    Only the free primary research released on our site can be licensed. We will not accept licensing fees on research we charge users to access.
  • All licensed research will be clearly labeled with the licensees. No licensed research will be released without indicating the sources of licensing fees. Again, there will be no back channel influence. We’re open and transparent about our revenue sources.

In essence, we develop all of our research out in the open, and not only seek public comments, but keep those comments indefinitely as a record of the research creation process. If you believe we are biased or not doing our homework, you can call us out on it and it will be there in the record. Our philosophy involves cracking open the research process, and using our readers to eliminate bias and enhance the quality of the work.

On the back end, here’s how we handle this approach with licensees:

  • Licensees may propose paper topics. The topic may be accepted if it is consistent with the Securosis research agenda and goals, but only if it can be covered without bias and will be valuable to the end user community.
  • Analysts produce research according to their own research agendas, and may offer licensing under the same objectivity requirements.
  • The potential licensee will be provided an outline of our research positions and the potential research product so they can determine if it is likely to meet their objectives.
  • Once the licensee agrees, development of the primary research content begins, following the Totally Transparent Research process as outlined above. At this point, there is no money exchanged.
  • Upon completion of the paper, the licensee will receive a release candidate to determine whether the final result still meets their needs.
  • If the content does not meet their needs, the licensee is not required to pay, and the research will be released without licensing or with alternate licensees.
  • Licensees may host and reuse the content for the length of the license (typically one year). This includes placing the content behind a registration process, posting on white paper networks, or translation into other languages. The research will always be hosted at Securosis for free without registration.

Here is the language we currently place in our research project agreements:

Content will be created independently of LICENSEE with no obligations for payment. Once content is complete, LICENSEE will have a 3 day review period to determine if the content meets corporate objectives. If the content is unsuitable, LICENSEE will not be obligated for any payment and Securosis is free to distribute the whitepaper without branding or with alternate licensees, and will not complete any associated webcasts for the declining LICENSEE. Content licensing, webcasts and payment are contingent on the content being acceptable to LICENSEE. This maintains objectivity while limiting the risk to LICENSEE. Securosis maintains all rights to the content and to include Securosis branding in addition to any licensee branding.

Even this process itself is open to criticism. If you have questions or comments, you can email us or comment on the blog.