Securosis

Research

The Death of Product Reviews

As a security practitioner, it has always been difficult to select the ‘right’ product. You (kind of) know what problem needs to be solved, yet you often don’t have any idea how any particular product will work and scale in your production environment. Sometimes it is difficult to identify the right vendors to bring in for an evaluation. Even when you do, no number of vendor meetings, SE demos, or proof of concept installations can tell you what you need to know. So it’s really about assembling a number of data points and trying to do your homework to the greatest degree possible. Part of that research process has always been product reviews by ‘independent’ media companies. These folks line up a bunch of competitors, put them through the paces, and document their findings. Again, this doesn’t represent your environment, but it gives you some clues on where to focus during your hands-on tests and can help winnow down the short list. Unfortunately, the patient (the product review) has died. The autopsy is still in process, but I suspect the product review died of starvation. There just hasn’t been enough food around to sustain this legend of media output. And what has been done recently subsists on a diet of suspicion, media bias, and shoddy work. The good news is that tech media has evolved with the times. Who doesn’t love to read tweets about regurgitated press releases? Thankfully Brian Krebs is still out there actually doing something useful. Seeing Larry Suto’s recent update of his web application scanner test (PDF) and the ensuing backlash was the final nail in the coffin for me. But this patient has been sick for a long time. I first recognized the signs years ago when I was in the anti-spam business. NetworkWorld commissioned a bake-off of 40 mail security gateways and published the results. In a nutshell, the test was a fiasco for several reasons: Did not reflect reality: The test design was flawed from the start. The reviewer basically resent his mail to each device. This totally screwed up the headers (by adding another route) and dramatically impacted effectiveness. This isn’t how the real world works. Too many vendors: To really test these products, you have to put them all through their paces. That means at least a day of hands-on time to barely scratch the surface. So to really test 40 devices, it would take 40-60+ man-days of effort. Yeah, I’d be surprised if a third of that was actually spent on testing. Uneven playing field: The reviewers let my company send an engineer to install the product and provide training. We did that with all enterprise sales, so it was standard practice for us, but it also gave us a definite advantage over competitors who didn’t have a resource there. If every review present a choice: a) fly someone to the lab for a day, or b) suffer by comparison to the richer competitors, how fair and comprehensive can reviews really be? Not everyone showed: There is always great risk in doing a product review. If you don’t win and handily, it is a total train wreck internally. Our biggest competitor didn’t show up for that review, so we won, but it didn’t help with in most of our head-to-head battles. Now let’s get back to Suto’s piece to see how things haven’t changed, and why reviews are basically useless nowadays. By the way, this has nothing to do with Larry or his efforts. I applaud him for doing something, especially since evidently he didn’t get compensated for his efforts. In the first wave, the losing vendors take out their machetes and start hacking away at Larry’s methodology and findings. HP wasted no time, nor did a dude who used to work for SPI. Any time you lose a review, you blame the reviewer. It certainly couldn’t be a problem with the product, right? And the ‘winner’ does its own interpretation of the results. So this was a lose-lose for Larry. Unless everyone wins, the methodology will come under fire. Suto tested 7 different offerings, and that probably was too many. These are very complicated products and do different things in different ways. He also used the web applications put forth by the vendors in a “point and shoot” type of methodology for the bulk of the tests. Again, this doesn’t reflect real life or how the product would stand up in a production environment. Unless you actually use the tool for a substantial amount of time in a real application, there is no way around this limitation. I used to love the reviews Network Computing did in their “Real-World Labs.” That was the closest we could get to reality. Too bad there is no money in product reviews these days – that means whoever owns Network Computing and Dark Reading can’t sponsor these kinds of tests anymore, or at least not objective tests. The wall between the editorial and business teams has been gone for years. At the end of the day it gets back to economics. I’m not sure what level of help Larry got from the vendors during the test, but unless it was nothing from nobody, you’re back to the uneven playing field. But even that doesn’t reflect reality, since in most cases (for an enterprise deployment anyway) vendor personnel will be there to help, train, and refine the process. And in most cases, craftily poison the process for other competitors, especially during a proof of concept trial. This also gets back to the complexity issue. Today’s enterprise environments are too complicated to expect a lab test to reflect how things work. Sad, but true. Finally, WhiteHat Security didn’t participate in the test. Jeremiah explained why, and it was probably the right answer. He’s got something to tell his sales guys, and he doesn’t have results that he may have to spin. If we look at other tests, when was someone like Cisco last involved in a product review? Right, it’s been a long time because they don’t have to be. They

Share:
Read Post

Database Security Fundamentals: Database Access Methods

It’s tough to talk about securing database access methods in a series designed to cover database security basics, because the access attacks are not basic. They tend to exploit either communications media or external functions – taking advantage of subtleties or logic flaws – capitalizing on trust relationships, or just being very unconventional and thus hard to anticipate. Still, some of the attacks are right through an open front door, like forgetting to set a TNS Listener password on Oracle. I will cover the basics here, as well as a few more involved things which can be addressed with a few hours and minimal third party tools. Relational platforms are chock full of features and functions, and many have been in use for so long that people simply forget to consider their security ramifications. Or worse, some feature came with the database, and an administrator (who did not fully understand it) was afraid that disabling it would cause side effects in applications. In this post I will cover the communications media and external services provided by the database, and their proper setup to thwart common exploits. Let’s start with the basics: Network Connections: Databases can support multiple network connections over multiple ports. I have two recommendations here. First, to reduce complexity and avoid possible inconsistency with network connection settings, I advise keeping the number of listeners to a minimum: one or two. Second, as many automated database attacks go directly default network ports directly, I recommend moving listeners to a non-standard port numbers. This will annoy application developers and complicate their setup somewhat, but more importantly it will both help stop automated attacks and highlight connection attempts to the default ports, which then indicate either misconfiguration or hardwired attacks. Network Facilities: Some databases use add-on modules to support network connections, and like the database itself are not secure out of the box. Worse, many vulnerability assessment tools omit the network from the scan. Verify that the network facility itself is set up properly, that administrative access requires a password, and that the password is not stored in clear text on the file system. Transport Protocols: Databases support multiple transport protocols. While features such as named pipes are still supported, they are open to spoofing and hijacking. I recommend that you pick a single reliable protocol such as TCP/IP), and disable the rest to prevent insecure connections. Private Communication: Use SSL. If the database contains sensitive data, use SSL. This is especially true for databases in remote or virtual environments. The path between the user or application and the database is not guaranteed to be safe, so use SSL to ensure privacy. If you have never set up SSL before, get some help – otherwise connecting applications can choose to ignore SSL. External Procedure Lockdown: All database platforms have external procedures that are very handy for performing database administration. They enable DBAs to run OS commands, or to run database functions from the OS. These procedures are also a favorite of attackers – once they have hacked either an OS or a database, stored procedures (if enabled) make it trivial to leverage that access into a compromise of the other half. This one is not optional. If you are part of a small IT organization and responsible for both IT and database administration, it will make your day-to-day job just a little harder. Checking these connection methods can be completed in under and hour, and enables you to close off the most commonly used avenues for attack and privilege escalation. A little more advanced: Basic Connection Checks: Many companies, as part of their security policy, do not allow ad hoc connections to production databases. Handy administrative tools like Quest’s Toad are not allowed because they do not enforce change control processes. If you are worried about this issue, you can write a login trigger that detects the application, user, and source location of inbound connections – and then terminates unauthorized sessions. Trusted Connections & Service Accounts: All database platforms offer some form of trusted connections. The intention is to allow the calling application to verify user credentials, and then pass the credentials or verification token through the service account to the database. The problem is that if the calling application or server has been compromised, all the permissions granted to the calling application – and possibly all the permissions assigned to any user of the connection – are available to an attacker. You should review these trust relationships and remove them for high-risk applications. Share:

Share:
Read Post

Friday Summary: February 12, 2010

Chris was kind enough to forward me Game Development in a Post-Agile World this week. What I know about game development could fit on the the head of a pin. Still, one of the software companies I worked for was incubated inside a much larger video game development company. I was always very interested in watching the game team dynamics, and how they differed from the teams I ran. The game developers did not have a lot of overlapping skills and the teams were – whether they knew it or not – built around the classical “surgical team” structure. They was always a single and clear leader of the team, and that person was usually both technically and creatively superior. The teams were small, and if they had a formalized process, I was unaware of it. It appeared that they figured out their task, built the tools they needed to support the game, and then built the game. There was consistency across the teams, and they appeared to be very successful in their execution. Regardless, back to the post. When I saw the title I thought this would be a really cool examination of Agile in a game development environment. After the first 15 pages or so, I realized there is not a damned thing about video game development in the post. What is there, though, is a really well-done examination of the downsides with Agile development. I wrote what I thought to be a pretty fair post on the subject this week, but this post is better! While I was focused on the difficulties of changing an entrenched process, and their impact on developing secure code, this one takes a broader perspective and looks at different Agile methodologies along a continuum of how people-oriented different variations are. The author then looks at how moving along the continuum alters creativity, productivity, and stakeholder involvement. If you are into software development processes, you’re probably a little odd, but you will very much enjoy this post! On to the Summary: Webcasts, Podcasts, Outside Writing, and Conferences It’s the week of Rich Mogull, Media Giant: First, Rich is on the cover of the March issue of Information Security Magazine Then, Rich managed to snag the cover story in this month’s Macworld magazine. It’s all about security issues for consumers, and is only mildly Mac-specific (How cool is it to be at Macworld and have the cover of the magazine at the same time? Congrats Rich! –Adrian) (Are folks there carrying around ‘your’ issue? Has anyone asked you to sign one, or a body part? –Chris) Rich’s cover handily eclipsed the print appearance of Chris’ Google Voice piece (page 56). And, lest he be accused of being an old media lackey, Rich wrote a TidBITS article on iPads in the enterprise. Adrian’s Dark Reading post on Amazon SimpleDB. Favorite Securosis Posts David Mortman: Misconceptions of a DMZ. Mike Rothman: Adrian’s post on People over Process. This may be the best piece written on the blog this year. Just awesome. Rich: Adrian’s post on SDL and Process. Adrian Lane: Mike’s post on The Death of Product Reviews. David Meier: This week’s Incite. Other Securosis Posts Database Security Fundamentals: Database Access Methods Choose Your Own Whitepaper Adventure (and Upcoming Papers) Network Security Fundamentals: Correlation Counterpoint: Correlation Is Useful, but Threat Assessment Is Fundamental Litchfield Discloses Oracle 0-Day at Black Hat FireStarter: Admin access, buh bye Counterpoint: Admin Rights Don’t Matter the Way You Think They Do RSVP for the Securosis and Threatpost Disaster Recovery Breakfast Kill. IE6. Now. Favorite Outside Posts Rich: Gunnar nails the truth on our relationship with China. Before you start touting terms like ‘cyberwar’, you need to understand the economics of the situation. They pwned us, and it has nothing to do with technology. David Mortman: Answering APT Misconceptions. An unmuddying of some of the APT waters. Hallelujah! Mike Rothman: Making Progress Matters Most. Bejtlich gets at the heart of keeping team members engaged and productive. Get the fsck out of their way. Adrian Lane: Martin’s post on PCI Compliance and Public Clouds. Despite the site advertisements, it was my favorite this week. Project Quant Posts Project Quant: Database Security – Masking Project Quant: Database Security – WAF Top News and Posts FEDs want cell phone tracking. And who wouldn’t? Critical Adobe Update. Dave Lewis at LiquidMatrix calls out an anonymous vendor for factually-incorrect FUD. This is a short but important post. Just today I got an email from a vendor who wanted to tell me the “top 3 ways DLP fails”. Successful vendors market on the strength of their products, not the (sometimes fictional) weaknesses of other’s. Hackers Steal $50k; bank says ‘tough’. Microsoft calls for Congress to get involved with cloud computing security. Since they can’t even agree on the best way to run the country into the ground, I’m not expecting any government action. Mudge hits DARPA. This could be exceptionally good, depending on where he starts dropping the cash. Blog Comment of the Week Remember, for every comment selected, Securosis makes a $25 donation to Hackers for Charity. This week’s best comment goes to ds, in response to Rich’s Counterpoint: Admin Rights Don’t Matter the Way You Think They Do: I think that this post is dangerous. While many will understand the difference between removing admin rights from a desktop for the user and restricting/managing admin rights for sysadmins, the distinction isn’t explicitly stated, and some may take this to mean dealing with admin rights isn’t necessary as a blanket statement. Share:

Share:
Read Post

Totally Transparent Research is the embodiment of how we work at Securosis. It’s our core operating philosophy, our research policy, and a specific process. We initially developed it to help maintain objectivity while producing licensed research, but its benefits extend to all aspects of our business.

Going beyond Open Source Research, and a far cry from the traditional syndicated research model, we think it’s the best way to produce independent, objective, quality research.

Here’s how it works:

  • Content is developed ‘live’ on the blog. Primary research is generally released in pieces, as a series of posts, so we can digest and integrate feedback, making the end results much stronger than traditional “ivory tower” research.
  • Comments are enabled for posts. All comments are kept except for spam, personal insults of a clearly inflammatory nature, and completely off-topic content that distracts from the discussion. We welcome comments critical of the work, even if somewhat insulting to the authors. Really.
  • Anyone can comment, and no registration is required. Vendors or consultants with a relevant product or offering must properly identify themselves. While their comments won’t be deleted, the writer/moderator will “call out”, identify, and possibly ridicule vendors who fail to do so.
  • Vendors considering licensing the content are welcome to provide feedback, but it must be posted in the comments - just like everyone else. There is no back channel influence on the research findings or posts.
    Analysts must reply to comments and defend the research position, or agree to modify the content.
  • At the end of the post series, the analyst compiles the posts into a paper, presentation, or other delivery vehicle. Public comments/input factors into the research, where appropriate.
  • If the research is distributed as a paper, significant commenters/contributors are acknowledged in the opening of the report. If they did not post their real names, handles used for comments are listed. Commenters do not retain any rights to the report, but their contributions will be recognized.
  • All primary research will be released under a Creative Commons license. The current license is Non-Commercial, Attribution. The analyst, at their discretion, may add a Derivative Works or Share Alike condition.
  • Securosis primary research does not discuss specific vendors or specific products/offerings, unless used to provide context, contrast or to make a point (which is very very rare).
    Although quotes from published primary research (and published primary research only) may be used in press releases, said quotes may never mention a specific vendor, even if the vendor is mentioned in the source report. Securosis must approve any quote to appear in any vendor marketing collateral.
  • Final primary research will be posted on the blog with open comments.
  • Research will be updated periodically to reflect market realities, based on the discretion of the primary analyst. Updated research will be dated and given a version number.
    For research that cannot be developed using this model, such as complex principles or models that are unsuited for a series of blog posts, the content will be chunked up and posted at or before release of the paper to solicit public feedback, and provide an open venue for comments and criticisms.
  • In rare cases Securosis may write papers outside of the primary research agenda, but only if the end result can be non-biased and valuable to the user community to supplement industry-wide efforts or advances. A “Radically Transparent Research” process will be followed in developing these papers, where absolutely all materials are public at all stages of development, including communications (email, call notes).
    Only the free primary research released on our site can be licensed. We will not accept licensing fees on research we charge users to access.
  • All licensed research will be clearly labeled with the licensees. No licensed research will be released without indicating the sources of licensing fees. Again, there will be no back channel influence. We’re open and transparent about our revenue sources.

In essence, we develop all of our research out in the open, and not only seek public comments, but keep those comments indefinitely as a record of the research creation process. If you believe we are biased or not doing our homework, you can call us out on it and it will be there in the record. Our philosophy involves cracking open the research process, and using our readers to eliminate bias and enhance the quality of the work.

On the back end, here’s how we handle this approach with licensees:

  • Licensees may propose paper topics. The topic may be accepted if it is consistent with the Securosis research agenda and goals, but only if it can be covered without bias and will be valuable to the end user community.
  • Analysts produce research according to their own research agendas, and may offer licensing under the same objectivity requirements.
  • The potential licensee will be provided an outline of our research positions and the potential research product so they can determine if it is likely to meet their objectives.
  • Once the licensee agrees, development of the primary research content begins, following the Totally Transparent Research process as outlined above. At this point, there is no money exchanged.
  • Upon completion of the paper, the licensee will receive a release candidate to determine whether the final result still meets their needs.
  • If the content does not meet their needs, the licensee is not required to pay, and the research will be released without licensing or with alternate licensees.
  • Licensees may host and reuse the content for the length of the license (typically one year). This includes placing the content behind a registration process, posting on white paper networks, or translation into other languages. The research will always be hosted at Securosis for free without registration.

Here is the language we currently place in our research project agreements:

Content will be created independently of LICENSEE with no obligations for payment. Once content is complete, LICENSEE will have a 3 day review period to determine if the content meets corporate objectives. If the content is unsuitable, LICENSEE will not be obligated for any payment and Securosis is free to distribute the whitepaper without branding or with alternate licensees, and will not complete any associated webcasts for the declining LICENSEE. Content licensing, webcasts and payment are contingent on the content being acceptable to LICENSEE. This maintains objectivity while limiting the risk to LICENSEE. Securosis maintains all rights to the content and to include Securosis branding in addition to any licensee branding.

Even this process itself is open to criticism. If you have questions or comments, you can email us or comment on the blog.