[syslog-ng] 2 Log rotate questions.

[Jim Hendrick]

I just noticed the part about tracking firewall sessions.

Do you (or would you consider) collecting netflows ?

Much more space efficient and designed specifically for that kind of
analysis.

Take a look at the SiLK tools for an excellent suite that might give you
some good ideas.

Jim

On 12/24/2014 04:57 PM, Jim Hendrick wrote:
> We generate a fair number of firewall logs daily and use the $HOUR
> macro to store the flat files.
>
> A nightly cron does a "find -mtime -exec gzip {} \;" to keep older
> filed zipped and another deletes them after a suitable period.
>
> As far as parsing - how do you parse the logs? For what? Do you
> process them in a SIEM or do you use other programs / scripts?
>
> We use multiple destinations to store logs in local files, send logs
> to a SIEM, etc.
>
> I'm sure the list can provide lots of (hopefully) useful suggestions.
>
> Jim
>
>
> On 12/24/2014 01:45 PM, Scot Needy wrote:
>> Thanks Andrew, 
>>
>> Version is 3.5. Maybe it would be clearer this way.
>>
>> We started to send firewall session data to syslog-ng. The end goal
>> is to track firewall sessions to build/update/audit firewall rules.  
>> So our logs increased, not a big deal, I write to
>> $YEAR$MONTH$DAY.$HOST.log but that produced a few 20Gb firewall logs
>> files that are time consuming to compress and parse. 
>>
>> One admin wants to use log-rotate to move logs over $(SIZE) but that
>> could result in many syslog-ng restarts a day and still involves a
>> lot of post processing.
>>  I could use an $HOUR macro as well but I that still creates some
>> pretty large files. 
>>
>>
>> NEW approach: 
>>
>> Has anyone used the $MSG parsers to accomplish a similar task in line ? 
>> http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/csv-parser.html
>>
>>
>> I don’t need to log every session created just unique hits to the
>> highlighted area of this sample in counter format like the stats
>> output has.    "SrcRule:IP/PORT DstRule:IP/PORT COUNT
>>
>> Dec 24 11:02:42 192.168.X.X : %*FWX_X*: Built outbound UDP connection
>> ####### for *RuleName:SRCIP/PORT* (SRCIP/PORT) to
>> *RuleName:DSTIP/PORT* (DSTIP/PORT)
>>
>>
>>
>>
>>> On Dec 24, 2014, at 12:08 PM, Andrew J. Caines
>>> <A.J.Caines at halplant.com <mailto:A.J.Caines at halplant.com>> wrote:
>>>
>>> Scot,
>>>
>>> You fail to mention what version of syslog-ng you are using and on which
>>> platform.
>>>
>>>> If a log file is renamed syslog-ng does not write a new file until
>>>> restarted.
>>>
>>> Correct. Renaming a file on a unix system is just a change to the parent
>>> directory. Processes reading from or writing to the file which keep the
>>> file open will know nothing about the change.
>>>
>>>> Is the data received during that time lost
>>>
>>> No. The process will continue to write to the same file which now has a
>>> new name.
>>>
>>>> and is there a conf option for this.
>>>
>>> It's not clear what "this" is.
>>>
>>> There are lots of log rotation tools and they have various options to
>>> handle rotation. Two common approaches are
>>>
>>> 1) Signal (usually HUP) process(es) after rotation
>>> 2) Copy and null
>>>
>>> See the documentation and examples for your log rotation tool or better
>>> yet, use syslog-ng's native log naming capabilities. See 7.2. "Storing
>>> messages in plain-text files"[1].
>>>
>>>> Can syslog-ng rotate based on size ?
>>>
>>> Not directly in the way rsyslogd does with max-size, for example,
>>> however many log rotation tools have size parameters if this is a
>>> requirement.
>>>
>>>> What is recommended to manage fast growing files .
>>>
>>> See e.g. 17.5. "Configuring log rotation"[2].
>>>
>>> In general you need to know your log data and your requirements for
>>> keeping it. Your syslog-ng and/or log rotation tool configuration should
>>> implement these requirements.
>>>
>>> Typically in a two tier environment the clients log only recent data on
>>> local storage while transmitting some or all log data over the network
>>> to the loghost(s) for archive, analysis, etc.
>>>
>>> Depending on how fast "Fast" is, there may also be performance
>>> considerations, but start with requirements.
>>>
>>>
>>> [1]
>>> http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.6-guides/en/syslog-ng-ose-guide-admin/html/configuring-destinations-file.html
>>> [2]
>>> http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.6-guides/en/syslog-ng-ose-guide-admin/html/example-logrotate.html
>>>
>>>>
>>>>
>>>> ______________________________________________________________________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation:
>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>
>>>
>>>
>>> -- 
>>> -Andrew J. Caines-   Unix Systems Engineer   A.J.Caines at halplant.com
>>>  "Machines take me by surprise with great frequency" - Alan Turing
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20141224/7b51dd1c/attachment-0001.htm