Thanks Andrew,
Version is 3.5. Maybe it would be clearer this way.
We started to send firewall session data to syslog-ng. The end goal is to track firewall sessions to build/update/audit firewall rules.So our logs increased, not a big deal, I write to $YEAR$MONTH$DAY.$HOST.log but that produced a few 20Gb firewall logs files that are time consuming to compress and parse.
One admin wants to use log-rotate to move logs over $(SIZE) but that could result in many syslog-ng restarts a day and still involves a lot of post processing.I could use an $HOUR macro as well but I that still creates some pretty large files.
NEW approach:
Has anyone used the $MSG parsers to accomplish a similar task in line ?
I don’t need to log every session created just unique hits to the highlighted area of this sample in counter format like the stats output has. "SrcRule:IP/PORT DstRule:IP/PORT COUNT
Dec 24 11:02:42 192.168.X.X : %FWX_X: Built outbound UDP connection ####### for RuleName:SRCIP/PORT (SRCIP/PORT) to RuleName:DSTIP/PORT (DSTIP/PORT)
On Dec 24, 2014, at 12:08 PM, Andrew J. Caines <A.J.Caines@halplant.com> wrote:
Scot,
You fail to mention what version of syslog-ng you are using and on which
platform.
If a log file is renamed syslog-ng does not write a new file until
restarted.
Correct. Renaming a file on a unix system is just a change to the parent
directory. Processes reading from or writing to the file which keep the
file open will know nothing about the change.
Is the data received during that time lost
No. The process will continue to write to the same file which now has a
new name.
and is there a conf option for this.
It's not clear what "this" is.
There are lots of log rotation tools and they have various options to
handle rotation. Two common approaches are
1) Signal (usually HUP) process(es) after rotation
2) Copy and null
See the documentation and examples for your log rotation tool or better
yet, use syslog-ng's native log naming capabilities. See 7.2. "Storing
messages in plain-text files"[1].
Can syslog-ng rotate based on size ?
Not directly in the way rsyslogd does with max-size, for example,
however many log rotation tools have size parameters if this is a
requirement.
What is recommended to manage fast growing files .
See e.g. 17.5. "Configuring log rotation"[2].
In general you need to know your log data and your requirements for
keeping it. Your syslog-ng and/or log rotation tool configuration should
implement these requirements.
Typically in a two tier environment the clients log only recent data on
local storage while transmitting some or all log data over the network
to the loghost(s) for archive, analysis, etc.
Depending on how fast "Fast" is, there may also be performance
considerations, but start with requirements.
[1]
http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.6-guides/en/syslog-ng-ose-guide-admin/html/configuring-destinations-file.html
[2]
http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.6-guides/en/syslog-ng-ose-guide-admin/html/example-logrotate.html
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq
--
-Andrew J. Caines- Unix Systems Engineer A.J.Caines@halplant.com
"Machines take me by surprise with great frequency" - Alan Turing
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq