There is a caste system in technology. It’s an engineering caste system, or at least that’s what I call it. A feeling of superiority developers have over their QA, IT, product management, and release management brethren. Software developers at every firm I have ever worked for – large and small – share a condescending view of their co-workers when it comes to technology. They are at the top of the totem pole, and act as if their efforts are the most important.
It starts in college, where software programs are more competitive to get in and require far more rigorous curricula. It is fostered by the mindset of programmers, who approach their profession more like religion. It’s not a 9-5 day job, and most 20-something developers I have worked with put in longer hours and put in more time into self education than any other profession I have ever seen. They create something from nothing every day; and with software, anything is possible. The mindset is reinforced by pay scales and recognition when products are delivered. Their technical accumen runs far deeper than the other groups and they don’t respect those without it. This relationship between different professions is reinforced when problems arise, as developers are the ones explaining how things work and advising those around them. It’s the engineering team that writes the trickier test cases, and the engineers who comes up with the best product ideas. Heck, of the last four organization I have run, to solve serious IT issues I had to assign members of the engineering team to debug and fix. They are technology rocks stars and prima donnas. Right or wrong, good or bad, this attitude is commonplace.
Why do I bring this up? Reviewing the marketing and sales collateral from several security vendors who are applying their IT marketing angles to software developers, I see a lot of approaches that will not work. When it comes to understanding buying centers, those who have traditionally sold into IT don’t get the developer mindset. They approach sales and marketing as if the two were interchangeable, but they are not. The things developers consider important are not the same things the rest of IT considers important. It is unlikely your “IT Champion” can cross-pollinate your ideas to the development team – both because your champion is likely seen as an outsider by the developers and due to internal tension between different groups. Development sets development requirements.
White box test tools? Web application assessments? WAF? Even pen testing? These all need different buyers, with a different mind set and requirements than the buyers of other IT kit – especially compared to network operations gear. The product and the value proposition needs to work in the development context. Most sales and marketing teams want to target the top – the CIO – and work their way down from there. That works for most of IT, but not with developers who have their own set of requirements over and above business requirements, and often neither fear nor respect upper management. They are far less tolerant of marketing-speak and BS and much more focused on getting things done easily, so you had better show value quickly or you’re wasting time. UI, workflow, integration, and API options need to be more flexible. When it comes to application security, it’s a developer’s world, so adjust or be ignored.
Reader interactions
7 Replies to “Technology Caste System”
@Anonymous: most programmers do not understand the network – just as most network geeks don’t understand what the program is doing.
this myopia is common due to the time it takes to specialize… however, specialization isn’t necessary, a general understanding of the environment can make a huge difference to what a coder writes, and for the network geek to understand what the app is doing (and more importantly: trying to do) can make a world of difference.
this also go to the whole ‘caste’ thing: many either don’t want to cross ‘caste’ lines or are afraid of doing so.
myself, I generalist with specialist knowledge… “Zac of all trades, master of some”. this seems to serve me well and also those that heeded my advice back when I use to be an instructor. (aside: the programmers that took the network course afterwards would have regular epiphanies as they learned about how the network works.)
@ Adrian: “As a pen tester, do you feel you are more closely aligned with QA, or engineering?”
I am not primarily a pen tester. I am an appsec risk management expert and appsec industry analyst—much like yourself. Yes, I do keep my app pen test skills sharp because I consider that a subset of what I do.
I find myself most closely aligned with executive management, especially business owners and potentially application owners. I have worked in dev shops and QA teams, and with QA (dev-test) people inside of dev shops. I’ve worked in dev teams that are aligned with marketing (e.g. CMO), and ones that exist on their own research/engineering efforts (usually CIO or CTO).
“From your experience, do you find more pen testers with a background in QA, IT or engineering?”
I have never had a discussion with anyone, nor have found anyone capable of having a discussion, about basic QA concepts as applied to penetration-testing. I started heavily talking about these points of integration 5 years ago and have received little to no feedback. Yet, there are people in the literature that discuss application security in QA. The only other use of QA tools in application security that I’ve seen have been done by Dinis Cruz in O2. Cigital, iSecPartners, and NGS Software have hinted at other work done, but none (or not much) of it is available to the public—and most of the authors of this work consider themselves more of a QA professional than an appsec professional
The answer is that I find more QA people with a background in penetration testing. They don’t want to do penetration testing full-time. Much like their engineering counterparts, QA people place security importance much lower than all of their other activities—IMO it is simply because they are undisciplined in risk management and advanced information security management theories and practices.
I do not find many penetration testers with one background. Most penetration testers have been through several, and sometimes all major roles that one can have in a technology business. However, if I were to rank them—I would say that most app pen testers have an engineering or QA background; while most net pen testers have an IT background.
@Dre – Thanks for the comment. A couple questions:
As a pen tester, do you feel you are more closely aligned with QA, or engineering?
From your experience, do you find more pen testers with a background in QA, IT or engineering?
I agree it is not a ‘developers world’ in security in the sense that developers have not fully embraced security, and agree the vast majority lack a basic set of secure coding skills. I maintain that security offerings need to be aligned with development processes and tools to gain any traction. OWASP is the principle example of structuring security and security projects to be more palatable by developers, but there are many free (or near free) examples – something most commercial vendors could learn from.
I made no assertion about which domains were most knowledgeable about security, only that developers play by a different set of rules, and security _vendors_ approach them in the wrong way.
@Flibbe – Totally agree. The commoditization of coding has lessened this behavior, but the rock star mentality persists and is often promoted unwittingly.
-Adrian
I do not think it is developer specific, you can map it to any branch you want. I have seen developers who knows only c++/java or something and for them (software development == C++/Java). These are Drone Programmer, Filling the fat in a true bell curve. These people shy away from new technology, avoid learning (I mean producing also) new things. Most large development shops are full of drones because of management. When it is your hobby to draw, you draw with your heart, emotion and intellect but Can I do that? No. If I have to I will buy some imprints :-). So, it is whether that developer is a programmer or a drone ?
Its fine to act as a rock star, but ultimately there is a record label owner who will tire quickly of petulant behaviour and just fire and replace them. Similarly in development, its becoming almost a commodity with large scale outsourcing. Im finding that the guys left are actually those that have made the shift in behaviour to recognise that they are a service provider and they have an end customer to support. What we are left with are not the 3 power chord self confessed rock stars, but the virtuoso Joe Satriani’s and thats a good place to be for all of us.
I find that developers often make common mistakes in their assumptions on how computer networks work. Assuming that packets will never be lost, or that ample bandwidth is always available. This shows up all the time when running applications.
Developers tend to think that they understand the problems of IT and Business Risk because they built an “impenetrable soda machine” as a college project (or because they got free sodas from an easily breakable soda machine in college). Use any equivalent analogy, but developers do not spend significant time understanding the IT and business risks. That’s what people mean when they say, “Developers don’t understand shit about security”.
Application owners, such as product managers, and also the business owners, do learn about IT/business risk in b-school (or wherever). They do hear about the problems and have exposure to them (and potentially even responsibility to them). They are charged with developer responsibility as well.
We do not need to snap the ITSec or InfoSec puzzle piece that doesn’t fit into the developer puzzle piece. We need to fit the app owner puzzle piece with the developer piece. Egos (and your caste system) aside, SOMEBODY does run the business—and can charge the developers and IT people together to solve problems. When they are pitted against each other, they are bound to fail—as James McGovern discusses here—http://duckdown.blogspot.com/2011/03/why-rating-employees-on-curve-is-worst.html
It’s not a developer’s world at all in application security. You see Dan Cornell referencing the work of Eduardo Vela, David Lindsay, Gareth Heyes, Kuza K, Mario Heiderich, etc. Sure, these guys are all “developers”, but they aren’t developers working on a huge platform like a FaceBook or a Zynga or anything. They’re PRIMARILY business owners and strategy consultants—this is the hat that they wear to work everyday. These are the guys that really get stuff done.
Sure, Chris Schmidt and Jim Manico are building the ESAPI. Other classic developers, who wear that hat everyday, also work on hard application security issues. But they are working on them after-the-fact. They are pushing the industry forward—but they are mostly playing catch-up (in a sense).
I don’t predict a future where a Jim Manico (secure developer) can go without a Mike Bailey (app pen-tester). They are never going to be complete in each other’s skills. You said that “most 20-something developers I have worked with put in longer hours and put in more time into self education than any other profession I have ever seen”. That’s offensive to almost everyone I know that’s been involved in what we called “computer security” 20 years ago. Sure, we worked on application and even network security issues back then, too, but computer security in this sense of the term encompasses everything-computers and everything-security.
Your average vulnerability and exploitation researchers are at least ten hundred thousand times as autodidactic as your average developer. I think maybe I am underestimating here… perhaps it’s in the order of hundreds of thousands of times more. To say that developers are not 9-to-5ers makes a mockery out of this whole application security mess.
I am sick of this movement and backlash against penetration testers on behalf of some perhaps very intelligent developers. Yes, the average penetration tester these days is overvalued. But the average vulnerability/exploitation researcher is becoming increasingly undervalued.
Knowledge in the application security space comes down to three major areas: domain knowledge, framework knowledge, and platform knowledge. With regards to frameworks—in many managed languages there is the problem of callbacks, where the code implementation interacts with the VM implementation. Your average application security guru developer (e.g. Chris Schmidt of OWASP ESAPI) knows a lot about the code implementation framework and some domain appsec issues. A more advanced application security guru (e.g. John Steven of Cigital) understands most domain issues coming into a project, as well as some platform (i.e. deployment) issues—but focuses primarily on full framework knowledge. Others such as Andrew Wilson of Trustwave SpiderLabs fall into this same category as John Steven (John is Java and Andrew is .NET). However, David Byrne of Trustwave SpiderLabs carries a strong domain knowledge to appsec problems that is unequaled by any one individual mentioned so far. Someone like Jason Rouse of Cigital has domain knowledge mastery, but perhaps only in mobile and some embedded devices, but has a better understanding of frameworks than David Byrne (although I bet both find extremely interesting issues in any given codebase). I would be willing to bet that Mike Bailey’s platform knowledge exceeds many of these guys mentioned—simply because an advanced penetration tester spends more time on that component of appsec.
I could go on for years with the above examples, but my point here is that we need each other—all of us. We all need to improve in these three areas. Kevin Wall appears to discuss this issue much better than most of the other places I’ve seen lately—http://off-the-wall-security.blogspot.com/2011/03/response-to-mark-curpheys-blog-owasphas.html
Knowledge of the vendor products is only important to a point. Having no knowledge of what Fortify or Cenzic are up to might distance yourself from being an analyst, but it doesn’t solve “big picture” appsec problems either. There is a safe balance here with everything. The key takeaway is that no one individual or type of person is going to be able to deliver appsec-oriented results or understand appsec better than any other.
In my opinion, NEVER buy an appsec product or vendor service without first talking to someone who has already been a customer of that vendor’s specific product that you intend to trial (for your exact purpose) for at least 3 years. IOW: If you want to start using Fortify SCA in your build servers (or in your IDE), or if you want to start forcing your development teams to do this (please never do this), contact a neutral appsec consulting company (i.e. not HP, Fortify, or Cigital) who has been using Fortify SCA in their IDE for 3 years, and who has also been integrating Fortify SCA into build servers for 3 years. If you want a SOC (Security Operations Center) to start populating an AMP portal up with an asset inventory and scanning web applications—make sure you’ve spoken to someone who has been doing this for 3 years first. Once you get access to these people, make sure you go over your plans and trial runs for about a year before acceptance of implementation (i.e. a production-ready rollout)—IOW: Don’t buy until you’ve seen this stuff work in your environment for about a year. Just for safe measure, if we’re talking about WAF—double all numbers I recommend here.
However, for free (or near-free) tools and processes—I say go all out—right away. Burp Pro, WebScarab, and Fiddler are in this category, as is ESAPI and Microsoft’s WPL. Yasca, OWASP CodeCrawler, RIPS, GRAudit, LAPSE+, and CAT.NET fit here as well. Make sure that when you do the appsec commercial tool comparisons, that they stack up to what you’ve built using free tools. This is the approach that many appsec consulting companies have taken—they either built these free tools themselves (Hello, @Stake and Foundstone—and everyone since then!), or they have integrated many free tools into their processes for so long to where a Fortify SCA purchase (more likely a partnership) is the sixth iteration of a Six Sigma process. They know what they need because they’ve built everything else.
The type of commercial products that Enterprises and large-installation, large-organization buyers can focus on (another huge problem here is that much of the Fortune 500 sees that they can make a few key capital investments and “solve” their risk or appsec problems, which never pans out) is on AppSec Enterprise Risk Management portals, such as the SaaS ones available from HoneyApps and Veracode, the custom ones developed by Cigital and Aspect Security, or potentially a home-grown one developed internally (using ESAPI!). I do not believe that Fortify 360 or HP AMP qualify, but perhaps Fortify’s new “Production” product announced at RSA does.
There are plenty of other appsec or appsec-related projects, products, and people to invest into. I suggest aligning with a strategy consultant—someone who really gets it and has a vendor and product neutral view of the appsec landscape. Server/app hardening, especially database hardening, is penultimate to a successful appsec program. There are free resources available at Cisecurity.org that can be put to use as starting points for server/app hardening. There are many Tomcat and IIS hardening tools out there, and perhaps many more should be created (especially home-grown and customized ones for a custom environment). Even in PaaS and IaaS environments, this advice applies. A lot of these web/app/db servers/services need to start shipping in a more appsec-default-secure mode—and somehow prevent easy circumvention. This sometimes needs to be co-ordinated between the developers and users of their frameworks, such as when PHP changed the use of globals in version 4 and again in version 5.
Password management for users is one thing, but it’s entirely a different topic for services and database connectoids. Sentrigo and others are starting to understand the need for large-installation, large-organization rollout of near-production proxies, such as how HyTrust proxies the vSphere infrastructure. This is not obviously an appsec issue, but I believe it is certainly needed to succeed with an appsec program, as with many other peripheral issues. How would one expect a pure developer to understand any of this?