Continuing our analysis of the 2014 Open Source Development and Application Security Survey, we can now discuss results as the final version has just been released. Today’s post focuses on application security related facets of the data.

Several questions in the survey focused on security practices within open source development, including vulnerability tracking and who is responsibility for security. I will dive into the results in detail, sharing my perspective on where things are getting better, which results surprised me, and where I believe improvements and attention are still needed. Here we go…

Who’s talking?

When analyzing a survey I always start with this question. It frames many of the survey’s answers. Understanding who is responding also helps illuminate the perspective expressed on the issues and challenges discussed.

When asked “What is your role in the organization?” the respondents were largely developers, at 42.78% of those surveyed. Considering that most architects, DevOps types, and build managers perform some development tasks, it is safe to say that over 50% of respondents have their hands on open source components and projects. A full 79% (include development managers) are in a position to understand the nuances of open source development, judge security, and reflect on policy issues.

Is open source important?

The short answer is “Hell yes, it’s important!” The (Maven) Central Repository – the largest source of open source components for developers – handled thirteen billion download requests last year. That’s more than a billion – with a ‘B’ – every month. This statistic gives you some idea of the scale of open source components used to assemble software applications today.

What’s more, the Sonatype data shows open source component usage on the rise, growing 62% in 2013 over 2012, and more than doubling since 2011.

When asked “What percentage of a typical application in your organization is comprised of open source components?” at least 75% of organizations rely on them in their development practices.

While ‘0-20%’ was an option, I am willing to bet few were really at ‘zero’ because those people would be highly unlikely to participate in this survey. So I believe the number with some involvement (including 1-20%) is closer to 100%.

The survey looked at use of open source components across verticals; they captured responses from most major industries including banks, insurance, technology/ISV, and government. Open source component usage is not relegated to a few target industries – it is widespread.

The survey also asked “How many developers are in your organization?” to which almost 500 participants answered 1,000 or more. Small firms don’t have 1,000 developers, so at least 15% of responses were from large enterprises. That is a strong showing, given that only a few years ago large enterprises did not trust open source and generally refused to officially endorse its use on corporate systems. And with nearly 700 responses from organizations with 26-100 developers, the survey reflects a good balance of organizational size.

Adoption continues to climb because open source have proven its worth – in terms of both quality and getting software built more quickly when you don’t try to build everything from scratch. More software than ever leverages contributions from the open source community, and widespread adoption makes open source software incredibly important.

Are developers worried about security?

Questions around software security were a theme of this year’s audit, which is why the name changed from years past to “Open Source Development and Application Security Survey”.

A central question was “Are open source vulnerabilities a top concern in your position?”, to which 54.16% answered “Yes, we are concerned with open source vulnerabilities.” Concern among more than half of respondents is a good sign – security is seldom part of a product design specification, and has only recently become part of the design and testing phases of development. Respondents’ concerned with vulnerabilities is a positive sign. Viewed another way, 10 years ago that number was about zero, so we see a dramatic change in awareness.

Outside development security practitioners get annoyed that only about 50% responded, “Yes” to this question. They zealously believe that when it comes to software development, everyone from the most senior software architect to the new guy in IT needs to consider security practices a priority. As we have seen in breaches over the last decade, failure only takes one weak link.

Lending support to the argument that software development has a long way to go when it comes to security, 47.29% of respondents said “Developers know it (Security) is important, but they don’t have time to spend on it.” The response “I’m interested in security and the organization is not.” is very common across development organizations. Most developers know security is an open issue. But fixing security typically does not make its way up the list of priorities while there are important features to build – at least not until there is a problem.

Developers’ growing interest in security practices is a good sign; allocation of resources and prioritization remains an issue.

What are they doing about it?

This year’s results offer a mixed impression of what development organizations are actually doing about security.

For example one set of responses showed that developers (40.63%) are responsible for “tracking and resolving newly discovered vulnerabilities in components included in their production applications.” From a developer’s perspective this result looks legitimate. And the 2014 Verizon Data Breach Investigations Report makes clear that the application stack is where the main security issues are being exploited. But application security buying behavior does not jibe with patterns across the rest of the security industry. Understanding that the survey participants were mostly developers with an open source perspective, this number is still surprising because the vast majority of security expenditures are for network and endpoint security devices. Security, including application security, is generally bolted on rather than fixed from within.

Jeremiah Grossman, Gunnar Peterson and others have all discussed the ineffectiveness of gearing security toward the network rather than applications. And the Whitehat Website Security Statistics report shows a long-term cost benefit from fixing problems within applications, but what we see is that network security controls remain the norm.

It is not uncommon for IT or security teams, rather than developers, to decide _how_to fix a specific application vulnerability – it might be a code fix but more likely it’s a non-code workaround, firewall rule, etc. While nearly 41% consider application development responsible for tracking and resolving newly discovered vulnerabilities, about 18% of respondents said this responsibility falls to IT Operations, and another 18% said it is the security team’s burden. Given the choice, most organizations opt for the cheaper, quicker firewall approach rather than code fixes which could destabilize the application stack.

Developers I speak with say they would like to do more for application security but cannot. They break their limitations down into three problem areas: how much latitude they have to fix security flaws, their ability to patch given the complexity of their operating environment, and limited resources to train people on security.

These points are supported by other survey results. For example 29.41% stated they monitor open source components for changes in vulnerabilities – almost one in three are tracking emerging issues.

At the same time only 15.96% said they must prove there are no vulnerable components in their products, and a full 53.28% do not have a policy governing open source vulnerabilities (“How does your open source policy address security vulnerabilities?”). Without a policy it’s not in the development queue, and if it’s not in the queue issues will not be fixed.

Policies and practices are often accompanied by training. Only over the last few years have security testing and practices have worked their way into the software development lifecycle – whether that be Waterfall, Agile, or something else. For organizations which practice secure code development, only a handful of people are trained because security training is expensive – often thousands of dollars per person per class. Security training is generally not budgeted in development shops – and when it is it is generally less specialized on-line tutorials and self-paced study courses – low-cost options. Responses to “What application security training is available to you?” were overwhelmingly e-learning (60.27%), while 26.24% responded that no training was available. Granted, you don’t need to fully train all developers in security. Not every person in development needs to be a security expert, but each team member should have some understanding of security as it pertains to their role.

I believe we are seeing that changes are occurring faster than companies can react – they have yet to understand the disconnect between their security problems and resource expenditures.

It’s not about Heartbleed

Heartbleed was an extremely serious OpenSSL vulnerability that allowed attackers to remotely view portions of server memory, leaking sensitive information. It is incredibly rare for an open source component vulnerability to make national headlines, but given the severity of the issues – and that fact that thousands of very large companies were running known vulnerable versions – I am glad Heartbleed received the attention, which prompted immediate action. Without accelerated action prompted by viral word-of-mouth, catastrophic issues could have resulted.

Coincidentally, when this vulnerability was discovered, Sonatype was a week into the Open Source survey. Roughly 45% of respondents took the survey before the announcement, and 55% after. Sonatype showed me the survey results in three different sets: one comprised of responses before Heartbleed drew international media attention, one after, and the combined results. What amazed me was how consistent the before and after data were. Most before-and-after answers were within 3% standard deviation, with one exception: “Has your company had a breach that can be attributed to a vulnerability in an open source component or dependency in the last 12 months?” 31% responded “Yes” after the news, compared to 19% before.

To be honest, and this is probably a minority opinion, the fact that almost 1 in 3 reported a breach in the last 12 months due to open source is the story here, not Heartbleed. Heartbleed was a big deal – don’t get me wrong. But the fact that 19% – about 1 in 5? – were already reporting a breach is important information. Many firms don’t disclose breaches at all, so we are stuck speculating about what’s really going inside companies without survey data like this. It also underscores the need, discussed above, to track open source vulnerabilities and have a policy or process in place to remediate important security issues in a timely fashion. Those of you who work in Agile or DevOps environments know that without a task card to track and assign issues, work does not get done. That includes fixing vulnerable components.

Open source is incredibly important, but how we handle security is a mixed bag. Overall the trends are positive – awareness, concern, and commitment to security are generally improving. But the lack of tracking and security education, and the insufficient number of people following policies, show development teams are not sufficiently prepared to deal with security as part of the development process.

Our next post will discuss the survey results pertaining to open source development trends. In the meantime you can find the full survey results – sans analysis – at http://www.sonatype.com/about/2014-open-source-software-development-survey.