[syslog-ng] 2 Log rotate questions.

[Scot Needy]

Thanks Andrew, 

Version is 3.5. Maybe it would be clearer this way.

We started to send firewall session data to syslog-ng. The end goal is to track firewall sessions to build/update/audit firewall rules.  
So our logs increased, not a big deal, I write to $YEAR$MONTH$DAY.$HOST.log but that produced a few 20Gb firewall logs files that are time consuming to compress and parse. 

One admin wants to use log-rotate to move logs over $(SIZE) but that could result in many syslog-ng restarts a day and still involves a lot of post processing.
 I could use an $HOUR macro as well but I that still creates some pretty large files. 


NEW approach: 

Has anyone used the $MSG parsers to accomplish a similar task in line ? 
http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/csv-parser.html <http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/csv-parser.html>


I don’t need to log every session created just unique hits to the highlighted area of this sample in counter format like the stats output has.    "SrcRule:IP/PORT DstRule:IP/PORT COUNT

Dec 24 11:02:42 192.168.X.X : %FWX_X: Built outbound UDP connection ####### for RuleName:SRCIP/PORT (SRCIP/PORT) to RuleName:DSTIP/PORT (DSTIP/PORT)




> On Dec 24, 2014, at 12:08 PM, Andrew J. Caines <A.J.Caines at halplant.com> wrote:
> 
> Scot,
> 
> You fail to mention what version of syslog-ng you are using and on which
> platform.
> 
>> If a log file is renamed syslog-ng does not write a new file until
>> restarted.
> 
> Correct. Renaming a file on a unix system is just a change to the parent
> directory. Processes reading from or writing to the file which keep the
> file open will know nothing about the change.
> 
>> Is the data received during that time lost
> 
> No. The process will continue to write to the same file which now has a
> new name.
> 
>> and is there a conf option for this.
> 
> It's not clear what "this" is.
> 
> There are lots of log rotation tools and they have various options to
> handle rotation. Two common approaches are
> 
> 1) Signal (usually HUP) process(es) after rotation
> 2) Copy and null
> 
> See the documentation and examples for your log rotation tool or better
> yet, use syslog-ng's native log naming capabilities. See 7.2. "Storing
> messages in plain-text files"[1].
> 
>> Can syslog-ng rotate based on size ?
> 
> Not directly in the way rsyslogd does with max-size, for example,
> however many log rotation tools have size parameters if this is a
> requirement.
> 
>> What is recommended to manage fast growing files .
> 
> See e.g. 17.5. "Configuring log rotation"[2].
> 
> In general you need to know your log data and your requirements for
> keeping it. Your syslog-ng and/or log rotation tool configuration should
> implement these requirements.
> 
> Typically in a two tier environment the clients log only recent data on
> local storage while transmitting some or all log data over the network
> to the loghost(s) for archive, analysis, etc.
> 
> Depending on how fast "Fast" is, there may also be performance
> considerations, but start with requirements.
> 
> 
> [1]
> http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.6-guides/en/syslog-ng-ose-guide-admin/html/configuring-destinations-file.html
> [2]
> http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.6-guides/en/syslog-ng-ose-guide-admin/html/example-logrotate.html
> 
>> 
>> 
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>> 
> 
> 
> 
> -- 
> -Andrew J. Caines-   Unix Systems Engineer   A.J.Caines at halplant.com
>  "Machines take me by surprise with great frequency" - Alan Turing
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20141224/ffa4c996/attachment-0001.htm