Understanding And Selecting A Database Activity Monitoring Solution: Part 2, Technical Architecture

By Rich

In Part 1 of our series we introduced Database Activity Monitoring (DAM) and discussed some of its use cases. In this post we’ll discuss current technical architectures.

Author’s Note: Although I call this product category Database Activity Monitoring, I don’t believe that name sufficiently describes where the market is headed. Over time we will migrate towards Application and Database Monitoring and Protection as products combine application and database monitoring with more options for active blocking, but it’s still too early to rebrand the market with that definition. Some tools do already offer those options, but both product and customer maturity still need to advance before we can migrate the definition.

Base Architecture

One of the key values of DAM is the ability to monitor multiple databases running on multiple database management systems (DBMS) across multiple platforms (Windows vs. Unix vs. …). The DAM tool aggregates collected information from multiple collectors to a central, secure server. In some cases the central server/management console also collects information, while in other cases it serves merely as a repository for collectors to drop data.

This creates three potential options for deployment, depending on the solution you choose:

  1. Single Server/Appliance: A single server or appliance serves as both the sensor/collection point and management console. This mode is typically used for smaller deployments.
  2. Two-tiered Architecture: This option consists of a central management server, and remote collection points/sensors. The central server does no direct monitoring and just aggregates information from remote systems, manages policies, and generates alerts. The remote collectors can use any of the collection techniques and feed data back to the central server.
  3. Hierarchical Architecture: Collection points/sensors aggregate to business or geographical level management servers, which in turn report to an enterprise wide management server. Hierarchical deployments are best suited for large enterprises which may have different business unit or geographic needs. They can also be configured to only pass certain kinds of data between the tiers to manage large volumes of information or maintain unit/geographic privacy and policy needs.

Whatever deployment architecture you choose, the central server aggregates all collected data, performs policy based alerting, and manages reporting and workflow.

I’ve focused this description on typical DAM deployments for database monitoring and alerting; as we delve into the technology we’ll see additional deployment options for more advanced features like blocking.

Collection Techniques At the core of all DAM solutions are the collectors that monitor database traffic and either collect it locally or send it to the central management server. These collectors are, at a minimum, capable of monitoring SQL traffic. This is one of the defining characteristics of DAM and what differentiates it from log management, Security Information and Event Management, or other tools that also offer some level of database monitoring. As usual, I’m going to simplify a bit, but there are three major categories of collection techniques.

  1. Network Monitoring: This technique monitors network traffic for SQL, parses the SQL, and stores in in the collector’s internal database. Most tools monitor bidirectionally, but some early tools only monitored inbound SQL requests. The advantages of network monitoring are that it has zero overhead on the monitored databases, can monitor independent of platform, requires no modification to the databases, and can monitor multiple, heterogenous database management systems at once. The disadvantages are that it has no knowledge of the internal state of the database and will miss any database activity that doesn’t cross the network, such as logging in locally or remote console connections. For this last reason, I only recommend network monitoring when used in conjunction with another monitoring technique that can capture local activity. Network monitoring can still be used if connections to the databases are encrypted via SSL or IPSec by placing a VPN appliance in front of the databases, and positioning the DAM collector between the VPN/SSL appliance and the database, where the traffic is unencrypted.
  2. Audit Log Monitoring: When using this technique, the collector is given administrative access to the target database and native database auditing is turned on. The collector externally monitors the DBMS and collects activity recorded by the native auditing or other internal database features that can output activity data. The overhead on the monitored system is thus the overhead introduced by turning on the native logging/auditing. In some cases this is completely acceptable- e.g., Microsoft SQL Server is designed to provide low-overhead remote monitoring. In other cases, particularly Oracle before version 10g, the overhead is material and may not be acceptable for performance reasons. Advantages include the ability (depending on DBMS platform) to monitor all database activity including local activity, performance equal to the performance of the native logging/monitoring, and monitoring of all database activity, including internal activity, regardless of client connection method. The big disadvantage is potential performance issues depending on the database platform, especially older versions of Oracle. This also requires opening an administrative account on the database and possibly some configuration changes.
  3. Local agent: This technique requires the installation of a software agent on the database server to collect activity. Individual agents vary widely in performance and techniques used, even within a product line, due to the requirements of DBMS and host platform support. Some early agents relied on locally sniffing a network loopback, which misses some types of client connections. The latest round of agents hooks into the host kernel to audit activity without modification to the DBMS and with minimal performance impact. Leading agents typically impact performance no greater than 3-5%, which seems to be the arbitrary limit database administrators are willing to accept. Advantages include collection of all activity without turning on native auditing, ability to monitor internal database activity like stored procedures, and potentially low overhead. Disadvantages include limited platform support (a new agent needs to be built for every platform) and the requirement to install an agent on every monitored database.

The Future is a Hybrid

I’m often asked which collection technique is best, and the answer is, “all of them”. Different collection techniques have different advantages and disadvantages depending on the circumstances. My advice is to select a solution that uses multiple techniques. In some cases, network monitoring involves less database and management overhead, but it misses many types of database connections and activity. Audit log monitoring has lower overhead, but can be problematic on some platforms, especially Oracle. Not all organizations are willing to install agents on critical database servers and current offerings aren’t available on every platform. Thus a combination of techniques can allow for effective monitoring with minimal performance impact.

Long term it wouldn’t surprise me if we dropped network monitoring as agents and log monitoring improve their performance and platform support. All the leading network monitoring vendors already also provide agent or log monitoring capabilities. The exception may be when we want to deploy in-line devices for database intrusion prevention/firewalling.

This covers the basic architecture. In out next post we’ll delve into central management server policies, alerting, workflow, and reporting.

No Related Posts

[...] 1 Part 2 Part 3 Part [...]

By Understanding and Selecting a Database Activity Mo

[...] 1 Part 2 Part 3 Part 4 Part [...]

By Understanding and Selecting a Database Activity Mo

[...] Part 1 Part 2 [...]

By Understanding and Selecting a Database Activity Mo

[...] in the comments to one of my posts on Database Activity Monitoring, Rani asked the question of who should own DAM? I’m going to expand the question to cover all [...]

By Who “Owns” Database Security? | securo

Disclaimer: I work for a company that addresses the entire application data security and compliance lifecycle issues.

Rani, indeed you asked a good question, but IMHO it was already answered by the customers. Take a look at a different, yet similar space; DLP. Where organizations own a fairly complicated technical solution that is used by different stakeholders for different purposes and therefore a different type of management paradigm and UI was required. Now, back to our space: a good solution should address both architectural challenges, provide high performance, any platform, any protocol and application and at the same time provide value to the business unit, security team and compliance group. The way I see it, solutions that will be good for one type of owner will remain in niche position.

By Sharon

@Adrian- excellent point (as usual). I ignored that option since I don’‘t consider it viable.

@Rani- most of the implementations I’‘ve seen of log monitoring make it extremely difficult (or impossible) for the DBA to muck with the auditing. I was very concerned about this in early days of the market, but it hasn’‘t seemed to rise up as a major issue. Something to watch, for sure.

I do need to discuss the owner; excellent suggestion and I may do a post on that today…

By rmogull

There are few other aspects relating to architecture and choice of collection technique that are worth noting:

Network topology is an important factor if you’‘re going for a network based solution. If your network is not organized in a way that creates convenient and few choke points, then network appliances would either cover only some of your databases, or costs start to skyrocket (I know you didn’‘t mention cost in this part of your analysis, but it is a consideration…)

The same is true if you plan on covering not only production databases, but also staging, testing and development.

A longer-term consideration, but one that is beginning to rear its head, is what impact virtualization would have - for instance, in an environment where a virtual application server is communicating with a virtual DBMS server within the same machine or cluster.

As regards audit log monitoring: Two additional disadvantages that this technique has are its inherent lack of prevention capabilities, and lack of proper separation of duties (too easy for DBAs to manipulate DBMS auditing).

As an aside, Rich - do you plan in one of the parts in this series to discuss the issue of who within the organization should own DAM? (or to rephrase - who’s the DAM owner?)
I think it’s an issue unique to this space due to the skill/responsibility gap between security professionals and DBAs.


By Rani

I also wanted to mention, from an architectural perspective, that there have been products on the market that were closely coupled in nature.  Based upon stored procedures and triggers, they performed most of their logical operations within the database.  I no longer see this in production software, and only occasionally from DBA’s who create their own scripts. 
I also expect that you will get some feedback on the ‘‘Local Agent Technique’‘.  There are three or four variants to this model, each with trade-offs in performance, impact to database reliability and the accuracy of the data collected.  There is no ‘‘right’’ answer with local agents, just different flavors, each with something they do well and something they do not. 
Thanks for the ‘‘Authors Note’‘.

By Adrian Lane

If you like to leave comments, and aren’t a spammer, register for the site and email us at and we’ll turn off moderation for your account.