<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">We save all log data and compress/dedup hourly. For an enterprise of about 5000 servers this averages about 200GB. </div><div class="">Some PCI compartments are special have backup and retention policies for compliance. </div><div class=""><br class=""></div><div class="">Archiving raw log data also gives us data to re-parse should the patterns need to be updated. </div><div class=""><br class=""></div><div class=""><br class=""></div><br class=""><div><blockquote type="cite" class=""><div class="">On Apr 28, 2016, at 7:23 AM, Czanik, Péter <<a href="mailto:peter.czanik@balabit.com" class="">peter.czanik@balabit.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class=""><div class="">Hi,<br class=""></div><br class="">I was asking, because up until now I recall a single syslog-ng user, who told me, that he saves all log messages. On the other hand I keep receiving (marketing) e-mails, that no logs should be discarded, everything should be saved. And sometimes I receive the same feedback from the Big Data world: we have enough disk space, why to do any filtering. So I'd be interested to learn from real world experiences, if filtering is really old fashioned or is there any situation (compliance requirement, endless storage, etc.) when you really save all log messages.<br class=""><br class=""></div>Bye,<br class=""></div><div class="gmail_extra"><br clear="all" class=""><div class=""><div class="gmail_signature">Peter Czanik (CzP) <<a href="mailto:peter.czanik@balabit.com" target="_blank" class="">peter.czanik@balabit.com</a>><br class="">Balabit / syslog-ng upstream<br class=""><a href="http://czanik.blogs.balabit.com/" target="_blank" class="">http://czanik.blogs.balabit.com/</a><br class=""><a href="https://twitter.com/PCzanik" target="_blank" class="">https://twitter.com/PCzanik</a></div></div>
<br class=""><div class="gmail_quote">On Thu, Apr 28, 2016 at 11:11 AM, Fabien Wernli <span dir="ltr" class=""><<a href="mailto:wernli@in2p3.fr" target="_blank" class="">wernli@in2p3.fr</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Thu, Apr 28, 2016 at 11:06:07AM +0200, Czanik, Péter wrote:<br class="">
> One of the major strengths of syslog-ng is message filtering, which<br class="">
> facilitates message routing and discarding useless log messages. OTOH I<br class="">
> often read, that we have now all the technologies and storage to keep all<br class="">
> logs. What do you think?<br class="">
<br class="">
</span>I would go further: we now have the means to add relevant metadata to all the events,<br class="">
which in turn allows us to do targeted archiving.<br class="">
<br class="">
<br class="">______________________________________________________________________________<br class="">
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank" class="">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br class="">
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank" class="">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br class="">
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank" class="">http://www.balabit.com/wiki/syslog-ng-faq</a><br class="">
<br class="">
<br class=""></blockquote></div><br class=""></div>
______________________________________________________________________________<br class="">Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" class="">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br class="">Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" class="">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br class="">FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" class="">http://www.balabit.com/wiki/syslog-ng-faq</a><br class=""><br class=""></div></blockquote></div><br class=""></body></html>