[syslog-ng] filtering vs. keeping all logs

[Scot Needy]

We save all log data and compress/dedup hourly.  For an enterprise of about 5000 servers this averages about 200GB. 
Some PCI compartments are special have backup and retention policies for compliance. 

Archiving raw log data also gives us data to re-parse should the patterns need to be updated.  



> On Apr 28, 2016, at 7:23 AM, Czanik, Péter <peter.czanik at balabit.com> wrote:
> 
> Hi,
> 
> I was asking, because up until now I recall a single syslog-ng user, who told me, that he saves all log messages. On the other hand I keep receiving (marketing) e-mails, that no logs should be discarded, everything should be saved. And sometimes I receive the same feedback from the Big Data world: we have enough disk space, why to do any filtering. So I'd be interested to learn from real world experiences, if filtering is really old fashioned or is there any situation (compliance requirement, endless storage, etc.) when you really save all log messages.
> 
> Bye,
> 
> Peter Czanik (CzP) <peter.czanik at balabit.com <mailto:peter.czanik at balabit.com>>
> Balabit / syslog-ng upstream
> http://czanik.blogs.balabit.com/ <http://czanik.blogs.balabit.com/>
> https://twitter.com/PCzanik <https://twitter.com/PCzanik>
> On Thu, Apr 28, 2016 at 11:11 AM, Fabien Wernli <wernli at in2p3.fr <mailto:wernli at in2p3.fr>> wrote:
> On Thu, Apr 28, 2016 at 11:06:07AM +0200, Czanik, Péter wrote:
> > One of the major strengths of syslog-ng is message filtering, which
> > facilitates message routing and discarding useless log messages. OTOH I
> > often read, that we have now all the technologies and storage to keep all
> > logs. What do you think?
> 
> I would go further: we now have the means to add relevant metadata to all the events,
> which in turn allows us to do targeted archiving.
> 
> 
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng>
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
> 
> 
> 
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20160428/5823980e/attachment.htm