[syslog-ng] 2 Log rotate questions.

[Jim Hendrick]

We generate a fair number of firewall logs daily and use the $HOUR macro
to store the flat files.

A nightly cron does a "find -mtime -exec gzip {} \;" to keep older filed
zipped and another deletes them after a suitable period.

As far as parsing - how do you parse the logs? For what? Do you process
them in a SIEM or do you use other programs / scripts?

We use multiple destinations to store logs in local files, send logs to
a SIEM, etc.

I'm sure the list can provide lots of (hopefully) useful suggestions.

Jim


On 12/24/2014 01:45 PM, Scot Needy wrote:
> Thanks Andrew, 
>
> Version is 3.5. Maybe it would be clearer this way.
>
> We started to send firewall session data to syslog-ng. The end goal is
> to track firewall sessions to build/update/audit firewall rules.  
> So our logs increased, not a big deal, I write to
> $YEAR$MONTH$DAY.$HOST.log but that produced a few 20Gb firewall logs
> files that are time consuming to compress and parse. 
>
> One admin wants to use log-rotate to move logs over $(SIZE) but that
> could result in many syslog-ng restarts a day and still involves a lot
> of post processing.
>  I could use an $HOUR macro as well but I that still creates some
> pretty large files. 
>
>
> NEW approach: 
>
> Has anyone used the $MSG parsers to accomplish a similar task in line ? 
> http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/csv-parser.html
>
>
> I don’t need to log every session created just unique hits to the
> highlighted area of this sample in counter format like the stats
> output has.    "SrcRule:IP/PORT DstRule:IP/PORT COUNT
>
> Dec 24 11:02:42 192.168.X.X : %*FWX_X*: Built outbound UDP connection
> ####### for *RuleName:SRCIP/PORT* (SRCIP/PORT) to
> *RuleName:DSTIP/PORT* (DSTIP/PORT)
>
>
>
>
>> On Dec 24, 2014, at 12:08 PM, Andrew J. Caines
>> <A.J.Caines at halplant.com <mailto:A.J.Caines at halplant.com>> wrote:
>>
>> Scot,
>>
>> You fail to mention what version of syslog-ng you are using and on which
>> platform.
>>
>>> If a log file is renamed syslog-ng does not write a new file until
>>> restarted.
>>
>> Correct. Renaming a file on a unix system is just a change to the parent
>> directory. Processes reading from or writing to the file which keep the
>> file open will know nothing about the change.
>>
>>> Is the data received during that time lost
>>
>> No. The process will continue to write to the same file which now has a
>> new name.
>>
>>> and is there a conf option for this.
>>
>> It's not clear what "this" is.
>>
>> There are lots of log rotation tools and they have various options to
>> handle rotation. Two common approaches are
>>
>> 1) Signal (usually HUP) process(es) after rotation
>> 2) Copy and null
>>
>> See the documentation and examples for your log rotation tool or better
>> yet, use syslog-ng's native log naming capabilities. See 7.2. "Storing
>> messages in plain-text files"[1].
>>
>>> Can syslog-ng rotate based on size ?
>>
>> Not directly in the way rsyslogd does with max-size, for example,
>> however many log rotation tools have size parameters if this is a
>> requirement.
>>
>>> What is recommended to manage fast growing files .
>>
>> See e.g. 17.5. "Configuring log rotation"[2].
>>
>> In general you need to know your log data and your requirements for
>> keeping it. Your syslog-ng and/or log rotation tool configuration should
>> implement these requirements.
>>
>> Typically in a two tier environment the clients log only recent data on
>> local storage while transmitting some or all log data over the network
>> to the loghost(s) for archive, analysis, etc.
>>
>> Depending on how fast "Fast" is, there may also be performance
>> considerations, but start with requirements.
>>
>>
>> [1]
>> http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.6-guides/en/syslog-ng-ose-guide-admin/html/configuring-destinations-file.html
>> [2]
>> http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.6-guides/en/syslog-ng-ose-guide-admin/html/example-logrotate.html
>>
>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>
>>
>>
>> -- 
>> -Andrew J. Caines-   Unix Systems Engineer   A.J.Caines at halplant.com
>>  "Machines take me by surprise with great frequency" - Alan Turing
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20141224/f0b047c7/attachment.htm