In Part 1 of our series we introduced Database Activity Monitoring (DAM) and discussed some of its use cases. In this post we’ll discuss current technical architectures.

Author’s Note: Although I call this product category Database Activity Monitoring, I don’t believe that name sufficiently describes where the market is headed. Over time we will migrate towards Application and Database Monitoring and Protection as products combine application and database monitoring with more options for active blocking, but it’s still too early to rebrand the market with that definition. Some tools do already offer those options, but both product and customer maturity still need to advance before we can migrate the definition.

Base Architecture

One of the key values of DAM is the ability to monitor multiple databases running on multiple database management systems (DBMS) across multiple platforms (Windows vs. Unix vs. …). The DAM tool aggregates collected information from multiple collectors to a central, secure server. In some cases the central server/management console also collects information, while in other cases it serves merely as a repository for collectors to drop data.

This creates three potential options for deployment, depending on the solution you choose:

1. Single Server/Appliance: A single server or appliance serves as both the sensor/collection point and management console. This mode is typically used for smaller deployments.
2. Two-tiered Architecture: This option consists of a central management server, and remote collection points/sensors. The central server does no direct monitoring and just aggregates information from remote systems, manages policies, and generates alerts. The remote collectors can use any of the collection techniques and feed data back to the central server.
3. Hierarchical Architecture: Collection points/sensors aggregate to business or geographical level management servers, which in turn report to an enterprise wide management server. Hierarchical deployments are best suited for large enterprises which may have different business unit or geographic needs. They can also be configured to only pass certain kinds of data between the tiers to manage large volumes of information or maintain unit/geographic privacy and policy needs.

Whatever deployment architecture you choose, the central server aggregates all collected data, performs policy based alerting, and manages reporting and workflow.

I’ve focused this description on typical DAM deployments for database monitoring and alerting; as we delve into the technology we’ll see additional deployment options for more advanced features like blocking.

Collection Techniques At the core of all DAM solutions are the collectors that monitor database traffic and either collect it locally or send it to the central management server. These collectors are, at a minimum, capable of monitoring SQL traffic. This is one of the defining characteristics of DAM and what differentiates it from log management, Security Information and Event Management, or other tools that also offer some level of database monitoring. As usual, I’m going to simplify a bit, but there are three major categories of collection techniques.

1. Network Monitoring: This technique monitors network traffic for SQL, parses the SQL, and stores in in the collector’s internal database. Most tools monitor bidirectionally, but some early tools only monitored inbound SQL requests. The advantages of network monitoring are that it has zero overhead on the monitored databases, can monitor independent of platform, requires no modification to the databases, and can monitor multiple, heterogenous database management systems at once. The disadvantages are that it has no knowledge of the internal state of the database and will miss any database activity that doesn’t cross the network, such as logging in locally or remote console connections. For this last reason, I only recommend network monitoring when used in conjunction with another monitoring technique that can capture local activity. Network monitoring can still be used if connections to the databases are encrypted via SSL or IPSec by placing a VPN appliance in front of the databases, and positioning the DAM collector between the VPN/SSL appliance and the database, where the traffic is unencrypted.
2. Audit Log Monitoring: When using this technique, the collector is given administrative access to the target database and native database auditing is turned on. The collector externally monitors the DBMS and collects activity recorded by the native auditing or other internal database features that can output activity data. The overhead on the monitored system is thus the overhead introduced by turning on the native logging/auditing. In some cases this is completely acceptable- e.g., Microsoft SQL Server is designed to provide low-overhead remote monitoring. In other cases, particularly Oracle before version 10g, the overhead is material and may not be acceptable for performance reasons. Advantages include the ability (depending on DBMS platform) to monitor all database activity including local activity, performance equal to the performance of the native logging/monitoring, and monitoring of all database activity, including internal activity, regardless of client connection method. The big disadvantage is potential performance issues depending on the database platform, especially older versions of Oracle. This also requires opening an administrative account on the database and possibly some configuration changes.
3. Local agent: This technique requires the installation of a software agent on the database server to collect activity. Individual agents vary widely in performance and techniques used, even within a product line, due to the requirements of DBMS and host platform support. Some early agents relied on locally sniffing a network loopback, which misses some types of client connections. The latest round of agents hooks into the host kernel to audit activity without modification to the DBMS and with minimal performance impact. Leading agents typically impact performance no greater than 3-5%, which seems to be the arbitrary limit database administrators are willing to accept. Advantages include collection of all activity without turning on native auditing, ability to monitor internal database activity like stored procedures, and potentially low overhead. Disadvantages include limited platform support (a new agent needs to be built for every platform) and the requirement to install an agent on every monitored database.

The Future is a Hybrid

I’m often asked which collection technique is best, and the answer is, “all of them”. Different collection techniques have different advantages and disadvantages depending on the circumstances. My advice is to select a solution that uses multiple techniques. In some cases, network monitoring involves less database and management overhead, but it misses many types of database connections and activity. Audit log monitoring has lower overhead, but can be problematic on some platforms, especially Oracle. Not all organizations are willing to install agents on critical database servers and current offerings aren’t available on every platform. Thus a combination of techniques can allow for effective monitoring with minimal performance impact.

Long term it wouldn’t surprise me if we dropped network monitoring as agents and log monitoring improve their performance and platform support. All the leading network monitoring vendors already also provide agent or log monitoring capabilities. The exception may be when we want to deploy in-line devices for database intrusion prevention/firewalling.

This covers the basic architecture. In out next post we’ll delve into central management server policies, alerting, workflow, and reporting.