<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Thanks Andrew, </div><div class=""><br class=""></div><div class="">Version is 3.5. Maybe it would be clearer this way.</div><div class=""><br class=""></div><div class="">We started to send firewall session data to syslog-ng. The end goal is to track firewall sessions to build/update/audit firewall rules. </div><div class="">So our logs increased, not a big deal, I write to $YEAR$MONTH$DAY.$HOST.log but that produced a few 20Gb firewall logs files that are time consuming to compress and parse. </div><div class=""><br class=""></div><div class="">One admin wants to use log-rotate to move logs over $(SIZE) but that could result in many syslog-ng restarts a day and still involves a lot of post processing.</div><div class=""> I could use an $HOUR macro as well but I that still creates some pretty large files. </div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">NEW approach: </div><div class=""><br class=""></div><div class="">Has anyone used the $MSG parsers to accomplish a similar task in line ? </div><div class=""><a href="http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/csv-parser.html" class="">http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/csv-parser.html</a></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">I don’t need to log every session created just unique hits to the highlighted area of this sample in counter format like the stats output has. "SrcRule:IP/PORT DstRule:IP/PORT COUNT</div><div class=""><br class=""></div><div class="">Dec 24 11:02:42 192.168.X.X : %<b class="">FWX_X</b>: Built outbound UDP connection ####### for <b class="">RuleName:SRCIP/PORT</b> (SRCIP/PORT) to <b class="">RuleName:DSTIP/PORT</b> (DSTIP/PORT)</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><br class=""><div><blockquote type="cite" class=""><div class="">On Dec 24, 2014, at 12:08 PM, Andrew J. Caines <<a href="mailto:A.J.Caines@halplant.com" class="">A.J.Caines@halplant.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">Scot,<br class=""><br class="">You fail to mention what version of syslog-ng you are using and on which<br class="">platform.<br class=""><br class=""><blockquote type="cite" class="">If a log file is renamed syslog-ng does not write a new file until<br class="">restarted.<br class=""></blockquote><br class="">Correct. Renaming a file on a unix system is just a change to the parent<br class="">directory. Processes reading from or writing to the file which keep the<br class="">file open will know nothing about the change.<br class=""><br class=""><blockquote type="cite" class="">Is the data received during that time lost<br class=""></blockquote><br class="">No. The process will continue to write to the same file which now has a<br class="">new name.<br class=""><br class=""><blockquote type="cite" class="">and is there a conf option for this.<br class=""></blockquote><br class="">It's not clear what "this" is.<br class=""><br class="">There are lots of log rotation tools and they have various options to<br class="">handle rotation. Two common approaches are<br class=""><br class="">1) Signal (usually HUP) process(es) after rotation<br class="">2) Copy and null<br class=""><br class="">See the documentation and examples for your log rotation tool or better<br class="">yet, use syslog-ng's native log naming capabilities. See 7.2. "Storing<br class="">messages in plain-text files"[1].<br class=""><br class=""><blockquote type="cite" class="">Can syslog-ng rotate based on size ?<br class=""></blockquote><br class="">Not directly in the way rsyslogd does with max-size, for example,<br class="">however many log rotation tools have size parameters if this is a<br class="">requirement.<br class=""><br class=""><blockquote type="cite" class="">What is recommended to manage fast growing files .<br class=""></blockquote><br class="">See e.g. 17.5. "Configuring log rotation"[2].<br class=""><br class="">In general you need to know your log data and your requirements for<br class="">keeping it. Your syslog-ng and/or log rotation tool configuration should<br class="">implement these requirements.<br class=""><br class="">Typically in a two tier environment the clients log only recent data on<br class="">local storage while transmitting some or all log data over the network<br class="">to the loghost(s) for archive, analysis, etc.<br class=""><br class="">Depending on how fast "Fast" is, there may also be performance<br class="">considerations, but start with requirements.<br class=""><br class=""><br class="">[1]<br class=""><a href="http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.6-guides/en/syslog-ng-ose-guide-admin/html/configuring-destinations-file.html" class="">http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.6-guides/en/syslog-ng-ose-guide-admin/html/configuring-destinations-file.html</a><br class="">[2]<br class="">http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.6-guides/en/syslog-ng-ose-guide-admin/html/example-logrotate.html<br class=""><br class=""><blockquote type="cite" class=""><br class=""><br class="">______________________________________________________________________________<br class="">Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<br class="">Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<br class="">FAQ: http://www.balabit.com/wiki/syslog-ng-faq<br class=""><br class=""></blockquote><br class=""><br class=""><br class="">-- <br class="">-Andrew J. Caines- Unix Systems Engineer A.J.Caines@halplant.com<br class=""> "Machines take me by surprise with great frequency" - Alan Turing<br class="">______________________________________________________________________________<br class="">Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<br class="">Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<br class="">FAQ: http://www.balabit.com/wiki/syslog-ng-faq<br class=""><br class=""></div></blockquote></div><br class=""></body></html>