This is Part 5 of our Database Encryption Series. Part 1, Part 2, Part 3, Part 4, and the supporting posts on Database vs. Application Encryption, & Database Encryption: Fact or Fiction are online.

I think key management scares people.

Application developers, IT managers, and database administrators all know effective key management support for encryption is critical, but it remains scary for most practitioners. Despite the incredible mathematical complexity behind the ciphers and the finesse required to implement those ciphers in a secure fashion, they don’t have to understand the gears and cogs inside the machine. Encryption is provided as libraries or fully functional services, so you send out clear text, and you get back encrypted data – easy. Key management worries people because if you don’t get the key management piece right, the whole system fails, and you are the responsible party. To illustrate what I mean, I want to share a couple stories about developers and IT practitioners who manage these systems.

Building database applications from scratch, developers have access to good crypto libraries, but generally little understanding of key management practices and few key management resources. The application developers I know took great pride in securing database fields through encryption, but when I asked them how they stored the key, the answer was usually “in the properties file”. That meant the key was stored on the disk, unencrypted, and in a directory readable by anyone who could access the application. When I pressed the point, I was assured that the key ‘needed’ to be there, otherwise the application would not be able to get the key and thus fail to restart. I have even had developers tell me this is a “chicken vs. egg” conundrum, that if you encrypt the key you cannot access it, therefore a key needed to be kept in clear text somewhere. I kid you not, with my last employer (who, by the way, developed security products), this was the reason the ‘senior’ programmer implemented key management this way, and why he didn’t see a problem with it. The argument always ends the same: a key as a tangible object is fine, but obfuscated and hidden is not. The unspoken reason is something every programmer knows: code has bugs, and a key management bug could be devastating and unrecoverable.

On the IT side, administrators I know have a different, equally frightening, set of problems with key management. Every IT manager I have spoken with has one or more of these questions: What happens if/when I lose keys? How to I back keys up securely? How do I replicate keys across multiple key servers for redundancy? I have 1,000 users reliant on public key cryptography, so how do I share these keys for all these users? If I expire and rotate keys, do I lose access to data archives? If I try to recover data from a tape, how do I get the right key? If I am using specialized key management hardware, how do I recover from fire or other disasters? All are risks in the minds of IT professionals every day.

Lose your key, lose your data. Lose your data, lose your job. And that scares the heck out of people!

Our goal in this section is to discuss key management options for database encryption. The introduction is meant to underscore the state of key management services today, and help illustrate why key management products are deployed the way they are. Yes, you can download excellent encryption tools for free, you can mix and match best of breed features, and you can develop your own operational key management process that is application agnostic, but this approach is becoming a rarity. And that’s a good thing because key management needs to performed by people who know what they are doing. Centralized, automated, embedded, pre-packaged and available as a complete service is the common choice. This removes the complexity and the responsibility of management, and much of the worry about reliability of your developers and IT administrators. Make no mistake, this trade-off comes at a price. Let’s dig into some key management practices for databases and how they are used.

Internally Managed

For database encryption, we define “internal key management” as key services within the database and provided by the database vendor. All of the relational database management platforms provide encryption packages, and included in these packages are key management functions. Typical services include key creation, storage, and retrieval, security; and most systems can handle symmetric and public key encryption options. Usage of the keys can be handled by proxy, as with the transparent encryption options, or through direct API calls to the database package. The keys are stored within the database, usually within a special table that resides in the administrative database or schema. Each vendor’s approach to securing keys varies significantly. Some vendors rely upon simple access controls tied to administrative accounts, some encrypt individuals’ keys with a single master key, while others do not allow any access to keys at all, and perform all key functions within a proxied service.

If you are using transparent encryption options (see Part 2: Selection Process Overview for terminology definitions) provided by the database vendor, all key operations is performed on the users’ behalf. For example, when a query for data within an encrypted column is made, the database performs the typical authorization checks, and when successfully authorized, automatically decrypts the data for the users. Neither the users nor the application need be aware the data is encrypted, needs to make a specific request to decrypt, or needs to supply a decryption key to the database. It is all handled on their behalf, and often performed without their knowledge.

Transparent key management’s greatest asset is just that: its transparency. The storage, management, security, sharing, and backup of the keys is handled by the database. With internally managed encryption keys, there is not a lot for the application to do, or even care about, since all of the use and management of the keys is performed inside the database. The services runs autonomously and is turned on by simple alteration of database parameters, so the DBA need only worry about backing up the system, and running the occasional disaster recovery drill so they are no unexpected ‘gotchas’. The database vendors have worked hard to perfect these operations and made them fast, efficient and reliable. As the functions are bundled with the core database by most vendors, and priced attractively. Finally this flavor has the flexibility to support API calls as well as transparent operation.

On the downside, these keys are typically only as secure as your least trustworthy DBA. In every platform that stores the keys in a database table, they will be accessible to database administrators. Despite vendor advertisements to the contrary, there are attacks that can sniff the key operations within the database, or even scan through database memory structures where the keys reside during use. When you boil it down, transparent encryption and associated internal key management is a very simple and cost effective way to secure data at rest, and not particularly effective at securing keys from database administrators.

Externally Managed

With external key management all key operations are performed outside the database in a dedicated key management server. External key management servers are either dedicated software or hardware appliances (aka Hardware Security Modules, or HSMs), and available through network or application procedure calls. Both provide a full range of functions including key creation, storage, and retrieval; they also support both for symmetric and public key cryptosystems. But they usually have a lot more to offer as well, with administrative support for access management, sharing keys across multiple key managers, and help with policies for key expiration/rotation. They provide additional advantages over internally managed keys in that the platform is dedicated to this function and provides faster cryptographic operations, they produce better results, and they store keys more securely than is possible within the database. They can perform the encryption and decryption operations on the users’ behalf, outside the database, as well.

External key management can be used in two ways: by calling the key management server directly, or by having the database make the requests on behalf of users and applications. All relational database now have their own Cryptographic API support. By calling the database through an established database communication port, application developers get external key management services within the same ‘database’ connection. These API calls are performed in the same context and in the same programmatic language as other database queries. This approach offers the security advantage of enforcing separation of duties, keeping the encryption keys out of the hands of database administrators, and puts them into the hands of the application developers. It allows application developers to choose just how granular they want to get with data encryption, and provides a great deal of flexibility in access control enforcement and key usage. That approach still leverages the database itself to perform the encryption, and can still take advantage of database features to address many structural and systems management challenges.

The other external key management option is calling the key manager directly, and having it perform all of the cryptographic functions on your behalf. In this case you’re outside the realm of database encryption and have moved to external, or application layer, encryption. By performing all encryption outside the database, leveraging none of the built in functions, the database becomes a simple repository for encrypted data. The advantages are that the database does not suffer the performance impact associated with encryption, the network latency of a database API call is not incurred, remote key storage enables separation of admin duties, and these facilities can be used across many different systems. It also means that all of the cryptographic operations are now the responsibility of the application developer, all user and key management must be performed within the application logic, and your database design must be altered to accommodate required changes.

External key management will pick up the key sharing, backup and other services that you lose by not using the database internals. Most customers we speak with now are opting for dedicated hardware to support key management operations. Performance is on advantage of HSMs, as hardware acceleration can generally handle encryption tasks an order of magnitude faster than their software routines on general-purpose (database) servers. Another is high-quality entropy to seed cryptographic ciphers. Additionally, sharing of keys across servers is well secured, and the APIs can be accessed by multiple applications. Finally, HSMs are well protected from physical tampering and electronic eavesdropping, making them the most secure key storage options discussed here.

That is not to say that dedicated software key management does not have its own advantages. Simplicity is major factor; as is support for multiple platforms, which provides some flexibility and conformity in hardware requisition and maintenance; the biggest advantage of dedicated software key management is the flexibility to run anywhere. Running on generic hardware helps to rapidly recover from disasters, and is far easier to use in virtual environments than hardware. Downsides are that the quality of the cryptography is dependent upon its supporting hardware for entropy, and that the keys are not as strongly secured as with dedicated devices.

Hardware based HSMs tend to be very expensive, compared to software (either DBMS features or software encryption applications). With any of these architectures, it’s important to confirm that the key management functions you need are supported by the database and encryption APIs.

One final word about key management database APIs: Each database vendor supports the basic requirements, but not all available functions, so you need to verify what is available through the API is what you need.

In our next post we will discuss common use cases that customers are trying to solve, and how each variant addresses these problems.