[syslog-ng] filtering vs. keeping all logs

[Evan Rempel]

IT teams that I speak with often use filtering to decrease the log volume, but not in the context of archiving or storing the logs. These teams are typically using some analysis product to process their logs and the product has a licensing cost based on the log volume. For example, splunk or loggly fall into this category. So the use case is to filter out log messages that are know to be of no interest, and feed the rest of the stream to the products for processing.

The other use case is where logs are centralized from a large geographical distribution and the cost of the bandwidth to aggregate the logs needs to be minimized.

Larger companies may make enough money that they just pay for the log processing. Perhaps at a capacity where some ceiling or cap on price has been reached. They may have development resources to develop a custom log processing architecture so that the processing is just the cost of the CPU time. Larger companies might have their own networks so they are not affected by costs that are associated to bandwidth.

Just my $0.02

Evan.


On 04/29/2016 01:33 AM, Czanik, Péter wrote:
> Hi,
>
> First of all: thank you for your feedback.
>
> This is very interesting, as it is pretty much the contrary what I hear / read in most discussions. I am often asked how to throw away cron / dhcp / dns / kernel / debug / etc. messages to save bandwidth / disk space and sometimes even to narrow down what is saved from authentication logs (which sounds crazy to my security minded ears...).
>
> I wonder what is the reason of this contradiction. Is it the size of the organization? (assumption: a larger org has more resources to save everything) Or is it compliance? (PCI, etc.) Or both?
>
> Bye,
>
> Peter Czanik (CzP) <peter.czanik at balabit.com <mailto:peter.czanik at balabit.com>>
> Balabit / syslog-ng upstream
> http://czanik.blogs.balabit.com/
> https://twitter.com/PCzanik
>
> On Thu, Apr 28, 2016 at 6:59 PM, Evan Rempel <erempel at uvic.ca <mailto:erempel at uvic.ca>> wrote:
>
>     Logs are used for so many things. Auditing, security, post incident analysis, live alerting (SIEM) and others. It is for this reason that I believe that all raw log data should be saved.
>
>     Adding to the discussion about metadata...
>
>     We add metadata from a variety of sources.
>
>     1. The syslog line itself. We parse EVERY log message to identify specific data and context. For example, a login identifier is often used in an email address, but in the context of an e-mail address, it is NOT a login identifier. This enables data mining on login identifiers without having to further filer out e-mail messages. We populate hundreds of metadata elements this way. tape volumes, database instances, login, uid, gid, disk drive names, logical volume names, FRU components in
>     hardware monitoring. The list is huge.
>
>     2. Incident details. During the parsing of EVERY log message, specific messages are identified as messages that should be alerted on. Metadata is added that contains incident description, URL to resolution documentation, severity of the incident and details on minimizing false positives. For example, a repeating log message may only be an incident if it repeats at a defined rate over a defined duration. All of this data is used to produce alerts to SMS, email, ticketing system.
>
>     3. Inventory management system. We add metadata for tiers of service. We have test, dev, preprod and prod. We also add business application names such as database instance (SID), Facilities management, workflow, MSExchange, listserver etc.
>
>     4. Business responsibility matrix. For each host/application there is a group that is responsible for the service. this metadata is added so that when alerts need to be sent the alerting subsystem can determine where to send the alert. It does this based on this responsibility matrix and data from #2.
>
>
>     All of this metadata gets placed into elasticsearch so we can start to mine the data by asking questions like:
>
>     - show all of the activity by user XXX in service Y in the preproduction tier on linux hosts.
>     - show all of the incidents for host HHH that group GGG is responsible for fixing.
>     - which service is responsible for the large increase in error class syslog lines, and in which tier of service did they occur.
>
>     The metadata is the power that drives this, and without the real time high performance pattern matching it just can't be done.
>
>     Evan.
>
>
>
>     On 04/28/2016 06:23 AM, Scot Needy wrote:
>>     We save all log data and compress/dedup hourly.  For an enterprise of about 5000 servers this averages about 200GB.
>>     Some PCI compartments are special have backup and retention policies for compliance.
>>
>>     Archiving raw log data also gives us data to re-parse should the patterns need to be updated.
>>
>>
>>
>>>     On Apr 28, 2016, at 7:23 AM, Czanik, Péter <peter.czanik at balabit.com <mailto:peter.czanik at balabit.com>> wrote:
>>>
>>>     Hi,
>>>
>>>     I was asking, because up until now I recall a single syslog-ng user, who told me, that he saves all log messages. On the other hand I keep receiving (marketing) e-mails, that no logs should be discarded, everything should be saved. And sometimes I receive the same feedback from the Big Data world: we have enough disk space, why to do any filtering. So I'd be interested to learn from real world experiences, if filtering is really old fashioned or is there any situation (compliance
>>>     requirement, endless storage, etc.) when you really save all log messages.
>>>
>>>     Bye,
>>>
>>>     Peter Czanik (CzP) <peter.czanik at balabit.com <mailto:peter.czanik at balabit.com>>
>>>     Balabit / syslog-ng upstream
>>>     http://czanik.blogs.balabit.com/
>>>     https://twitter.com/PCzanik
>>>
>>>     On Thu, Apr 28, 2016 at 11:11 AM, Fabien Wernli <wernli at in2p3.fr <mailto:wernli at in2p3.fr>> wrote:
>>>
>>>         On Thu, Apr 28, 2016 at 11:06:07AM +0200, Czanik, Péter wrote:
>>>         > One of the major strengths of syslog-ng is message filtering, which
>>>         > facilitates message routing and discarding useless log messages. OTOH I
>>>         > often read, that we have now all the technologies and storage to keep all
>>>         > logs. What do you think?
>>>
>>>         I would go further: we now have the means to add relevant metadata to all the events,
>>>         which in turn allows us to do targeted archiving.
>>>


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20160429/997e8cdd/attachment.htm