<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
We generate a fair number of firewall logs daily and use the $HOUR
macro to store the flat files.<br>
<br>
A nightly cron does a "find -mtime -exec gzip {} \;" to keep older
filed zipped and another deletes them after a suitable period.<br>
<br>
As far as parsing - how do you parse the logs? For what? Do you
process them in a SIEM or do you use other programs / scripts?<br>
<br>
We use multiple destinations to store logs in local files, send logs
to a SIEM, etc.<br>
<br>
I'm sure the list can provide lots of (hopefully) useful
suggestions.<br>
<br>
Jim<br>
<br>
<br>
<div class="moz-cite-prefix">On 12/24/2014 01:45 PM, Scot Needy
wrote:<br>
</div>
<blockquote
cite="mid:250D6B82-EF03-42D4-8111-ABDE8072C399@gmail.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<div class="">Thanks Andrew, </div>
<div class=""><br class="">
</div>
<div class="">Version is 3.5. Maybe it would be clearer this way.</div>
<div class=""><br class="">
</div>
<div class="">We started to send firewall session data to
syslog-ng. The end goal is to track firewall sessions to
build/update/audit firewall rules. </div>
<div class="">So our logs increased, not a big deal, I write to
$YEAR$MONTH$DAY.$HOST.log but that produced a few 20Gb firewall
logs files that are time consuming to compress and parse. </div>
<div class=""><br class="">
</div>
<div class="">One admin wants to use log-rotate to move logs over
$(SIZE) but that could result in many syslog-ng restarts a day
and still involves a lot of post processing.</div>
<div class=""> I could use an $HOUR macro as well but I that still
creates some pretty large files. </div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">NEW approach: </div>
<div class=""><br class="">
</div>
<div class="">Has anyone used the $MSG parsers to accomplish a
similar task in line ? </div>
<div class=""><a moz-do-not-send="true"
href="http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/csv-parser.html"
class="">http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/csv-parser.html</a></div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">I don’t need to log every session created just
unique hits to the highlighted area of this sample in counter
format like the stats output has. "SrcRule:IP/PORT
DstRule:IP/PORT COUNT</div>
<div class=""><br class="">
</div>
<div class="">Dec 24 11:02:42 192.168.X.X : %<b class="">FWX_X</b>:
Built outbound UDP connection ####### for <b class="">RuleName:SRCIP/PORT</b>
(SRCIP/PORT) to <b class="">RuleName:DSTIP/PORT</b>
(DSTIP/PORT)</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Dec 24, 2014, at 12:08 PM, Andrew J. Caines
<<a moz-do-not-send="true"
href="mailto:A.J.Caines@halplant.com" class="">A.J.Caines@halplant.com</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">Scot,<br class="">
<br class="">
You fail to mention what version of syslog-ng you are using
and on which<br class="">
platform.<br class="">
<br class="">
<blockquote type="cite" class="">If a log file is renamed
syslog-ng does not write a new file until<br class="">
restarted.<br class="">
</blockquote>
<br class="">
Correct. Renaming a file on a unix system is just a change
to the parent<br class="">
directory. Processes reading from or writing to the file
which keep the<br class="">
file open will know nothing about the change.<br class="">
<br class="">
<blockquote type="cite" class="">Is the data received during
that time lost<br class="">
</blockquote>
<br class="">
No. The process will continue to write to the same file
which now has a<br class="">
new name.<br class="">
<br class="">
<blockquote type="cite" class="">and is there a conf option
for this.<br class="">
</blockquote>
<br class="">
It's not clear what "this" is.<br class="">
<br class="">
There are lots of log rotation tools and they have various
options to<br class="">
handle rotation. Two common approaches are<br class="">
<br class="">
1) Signal (usually HUP) process(es) after rotation<br
class="">
2) Copy and null<br class="">
<br class="">
See the documentation and examples for your log rotation
tool or better<br class="">
yet, use syslog-ng's native log naming capabilities. See
7.2. "Storing<br class="">
messages in plain-text files"[1].<br class="">
<br class="">
<blockquote type="cite" class="">Can syslog-ng rotate based
on size ?<br class="">
</blockquote>
<br class="">
Not directly in the way rsyslogd does with max-size, for
example,<br class="">
however many log rotation tools have size parameters if this
is a<br class="">
requirement.<br class="">
<br class="">
<blockquote type="cite" class="">What is recommended to
manage fast growing files .<br class="">
</blockquote>
<br class="">
See e.g. 17.5. "Configuring log rotation"[2].<br class="">
<br class="">
In general you need to know your log data and your
requirements for<br class="">
keeping it. Your syslog-ng and/or log rotation tool
configuration should<br class="">
implement these requirements.<br class="">
<br class="">
Typically in a two tier environment the clients log only
recent data on<br class="">
local storage while transmitting some or all log data over
the network<br class="">
to the loghost(s) for archive, analysis, etc.<br class="">
<br class="">
Depending on how fast "Fast" is, there may also be
performance<br class="">
considerations, but start with requirements.<br class="">
<br class="">
<br class="">
[1]<br class="">
<a moz-do-not-send="true"
href="http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.6-guides/en/syslog-ng-ose-guide-admin/html/configuring-destinations-file.html"
class="">http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.6-guides/en/syslog-ng-ose-guide-admin/html/configuring-destinations-file.html</a><br
class="">
[2]<br class="">
<a class="moz-txt-link-freetext" href="http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.6-guides/en/syslog-ng-ose-guide-admin/html/example-logrotate.html">http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.6-guides/en/syslog-ng-ose-guide-admin/html/example-logrotate.html</a><br
class="">
<br class="">
<blockquote type="cite" class=""><br class="">
<br class="">
______________________________________________________________________________<br
class="">
Member info:
<a class="moz-txt-link-freetext" href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br
class="">
Documentation:
<a class="moz-txt-link-freetext" href="http://www.balabit.com/support/documentation/?product=syslog-ng">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br
class="">
FAQ: <a class="moz-txt-link-freetext" href="http://www.balabit.com/wiki/syslog-ng-faq">http://www.balabit.com/wiki/syslog-ng-faq</a><br class="">
<br class="">
</blockquote>
<br class="">
<br class="">
<br class="">
-- <br class="">
-Andrew J. Caines- Unix Systems Engineer
<a class="moz-txt-link-abbreviated" href="mailto:A.J.Caines@halplant.com">A.J.Caines@halplant.com</a><br class="">
"Machines take me by surprise with great frequency" - Alan
Turing<br class="">
______________________________________________________________________________<br
class="">
Member info:
<a class="moz-txt-link-freetext" href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br
class="">
Documentation:
<a class="moz-txt-link-freetext" href="http://www.balabit.com/support/documentation/?product=syslog-ng">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br
class="">
FAQ: <a class="moz-txt-link-freetext" href="http://www.balabit.com/wiki/syslog-ng-faq">http://www.balabit.com/wiki/syslog-ng-faq</a><br class="">
<br class="">
</div>
</blockquote>
</div>
<br class="">
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">______________________________________________________________________________
Member info: <a class="moz-txt-link-freetext" href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a>
Documentation: <a class="moz-txt-link-freetext" href="http://www.balabit.com/support/documentation/?product=syslog-ng">http://www.balabit.com/support/documentation/?product=syslog-ng</a>
FAQ: <a class="moz-txt-link-freetext" href="http://www.balabit.com/wiki/syslog-ng-faq">http://www.balabit.com/wiki/syslog-ng-faq</a>
</pre>
</blockquote>
<br>
</body>
</html>