Today you’d be hard pressed to find a decent sized network that doesn’t have some implementation of Security Event Management (SEM). It’s just a fact of modern regulation that a centralized system to collect all that logolicious information makes sense (and may be mandatory). Part of the problem with architecting and managing these systems is that one runs into the issue of securely collecting the information and subsequently verifying its authenticity.
Almost every network-aware product you might buy today has a logging capability, generally based on syslog – RFC3164. Unfortunately, as defined, syslog doesn’t provide much security. In fact if you need a good laugh I’d suggest reading section 6 of the RFC. You’ll know you’re in the right place when you start to digest information about odors, moths and spiders. It becomes apparent, very quickly, when reading subparagraphs 6.1 through 6.10, that the considerations outlined are there more to tip you off that the authors already know syslog provides minimal security – so don’t complain to them. At this point most sane people question using such a protocol at all because surely there must be something better, right? Yes and no.
First let me clarify: I didn’t set out to create an exhaustive comparison of [enter your favorite alternative to syslog here] for this writeup. Sure RFC5424 obsoletes the originally discussed RFC3164 and yes RFC5425 addresses using TLS as a transport to secure syslog. Or maybe it would be better to configure BEEP on your routers and let’s not forget about the many proprietary and open source agents that you can install on your servers and workstations. I freely admit there are some great technologies to read about in event logging technology. The point though is that since there is considerable immaturity and many options to choose from, most environments fall back to the path of least resistance: good ol’ syslog over UDP.
Unfortunately I’ve never been asked how to do logging right by a client. As long as events are streaming to the SEM and showing up on the glass in the NOC/SOC, it’s not a question that comes up. It may not even be a big deal right now, but I’d be willing to bet you’ll see more on the topic as audits become more scrutinizing. Shouldn’t the integrity of that data be something a little more robust than the unreliable, unauthentic, repudiable and completely insecure protocol you probably have in production? You don’t have to thank me later, but I’d start thinking about it now.
Reader interactions
2 Replies to “Where Art Thou, Security Logging?”
This is a secret of many (most I’d wager) SIM implementation. Besides being able to trivially inject arbitrary messages via source IP spoofing, why doesn’t the attacker just make the SIM a doorstop by bombarding it with spoofed UDP syslog messages? I have seen a SIM that is supposed to scale to over 1 billion messages per day get taken out by a single malfunctioning syslog host.
IT folks (network in particular) have been collecting these logs for decades. The same folks have been sending them to a SIM for years too. I would not expect widespread adoption of “secure syslog” until there is some standard capability that most vendors provide with base functionality (including SIM vendors).
I had never seen the original RFC 3164 document prior to this. That’s not a gag link? Did he really use an Odor analogy?
The classic counter argument I have heard on this topic, from customers and SIM vendors, is that the data collectors grab the log entries within seconds, convey them over a secure SSL channel, and then encrypt contents with the Log Management/SIEM appliance (if configured to do so). Log poisoning is then only possible with a coordinated attack on raw data (database, file, system configuration, etc) and on the log entries to cover your tracks. This needs to be done fast enough to avoid having the data moved to a safer environment. Can be easy to super difficult depending upon the environment and the data you want to alter.
In practice, once you have compromised the host, log poisoning is not that hard. Some of the early SIM and DAM platforms did not validate who was sending data, so you did not even need to touch the host. You could corrupt the stored logs remotely, easily, and insert whatever garbage you wanted to by pretending to be an end point. While it was difficult to erase existing entries without some form of SQL Injection, you could place so much garbage in the event stream with random dates and source identifiers as to make it indistinguishable from real log entries. Sequencing helps, but only on a per platform basis, not multi-source environment when trying to correlate events.
I will leave it to others to discuss the role of immutable log file generation as it pertains to SIEM.
Good post!