<div dir="ltr"><div><div>What is your syslog-ng version? Pretty old versions (I think before 3.2?), syslog-ng buffered output messages until we've reached the flush-lines() setting, unless the time specified in flush-timeout() has elapsed.<br><br></div>This is not the current behavior anymore. Maybe you have an old version? You say, that you have 400 events per second, so flush-lines(100) should be 4 times every second.<br><br></div>Anyhow, the version of syslog-ng you are using would be pretty important to have.<br></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">-- <br>Bazsi<br></div></div></div>
<br><div class="gmail_quote">On Wed, Apr 18, 2018 at 10:06 AM, Kavita Mohite (kmohite) <span dir="ltr"><<a href="mailto:kmohite@cisco.com" target="_blank">kmohite@cisco.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="white" link="#0563C1" vlink="#954F72" lang="EN-US">
<div class="m_666602182814586015WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;color:#222222;background:white">Hello Support Team,</span><span style="font-size:11.0pt"><u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;color:#222222"><u></u> <u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;color:#222222">In the syslog-ng configuration, we have 2 forward rules as follows:<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;color:#222222"><u></u> <u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;color:#222222">1. One rule reads from source (port 514) and writes syslogs to the file for all the syslogs greater than debug.<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;color:#222222">2. Second rule reads from source <span style="background:white">(port 514) and forwards the the syslog to the destination if a set of filter rules match. These set of
filter rules are called whitelist filter rules are filter rules containing match filter and priority filter in each rule. As there are multiple filter rules, they are ORed to create a main filter rule.</span><u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;color:#222222;background:white"><br>
<br>
</span><span style="font-size:11.0pt;color:#222222"><u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;color:#222222;background:white">Now we have syslogs coming in at a rate of 400 eps. Following is our observation:</span><span style="font-size:11.0pt;color:#222222"><u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;color:#222222;background:white"><br>
<br>
</span><span style="font-size:11.0pt;color:#222222"><u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;color:#222222;background:white">A. If the syslog matches the filter in point 2 above, then following happens</span><span style="font-size:11.0pt;color:#222222"><u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:47.25pt;background:white">
<u></u><span style="font-size:10.0pt;font-family:Symbol;color:#222222"><span>·<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11.0pt;color:#222222">It is forwarded to the destination provided in point 2.<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:47.25pt;background:white">
<u></u><span style="font-size:10.0pt;font-family:Symbol;color:#222222"><span>·<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11.0pt;color:#222222"> Also at the same time it is written to a file as per rule in point 1.<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;color:#222222"> <u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;color:#222222;background:white">B. If the syslog does not match the filter in point 2 above, then following happens</span><span style="font-size:11.0pt;color:#222222"><u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:47.25pt;background:white">
<u></u><span style="font-size:10.0pt;font-family:Symbol;color:#222222"><span>·<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11.0pt;color:#222222">It is NOT forwarded to the destination provided in point 2 as there is no match.<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:47.25pt;background:white">
<u></u><span style="font-size:10.0pt;font-family:Symbol;color:#222222"><span>·<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11.0pt;color:#222222">It is suppose to still write to file as the rule in 1 matches. <b>But this does not happen.</b><u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;color:#222222"><u></u> <u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;color:#222222">We are seeing that if the filter in point 2 matches then both the forward rules get executed immediately and we see the log written to file also immediately. But in case
of B, if the syslog does not match we are seeing that the write to file is happening g in bulk for 30 minutes. <b>Why is this ? Why does it not write to file immediately in case of (B) but does it in case of A ?</b><u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;color:#222222"><u></u> <u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;color:#222222">Below is the snap of the config values we are using in the syslog-ng.conf<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;color:#222222"><u></u> <u></u></span></p>
<p class="MsoNormal" style="margin-bottom:9.0pt;background:white"><span style="font-size:11.0pt;color:#333333">options {<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-bottom:9.0pt;background:white"><span style="font-size:11.0pt;color:#333333"> flush_lines (100);<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-bottom:9.0pt;background:white"><span style="font-size:11.0pt;color:#333333"> log_fetch_limit(1000);<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-bottom:9.0pt;background:white"><span style="font-size:11.0pt;color:#333333"> log_iw_size(10000);<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-bottom:9.0pt;background:white"><span style="font-size:11.0pt;color:#333333"> time_reopen (10);<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-bottom:9.0pt;background:white"><span style="font-size:11.0pt;color:#333333"> log_fifo_size (10000);<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-bottom:9.0pt;background:white"><span style="font-size:11.0pt;color:#333333"> chain_hostnames (off);<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-bottom:9.0pt;background:white"><span style="font-size:11.0pt;color:#333333"> use_dns (no);<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-bottom:9.0pt;background:white"><span style="font-size:11.0pt;color:#333333"> use_fqdn (no);<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-bottom:9.0pt;background:white"><span style="font-size:11.0pt;color:#333333"> create_dirs (yes);<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-bottom:9.0pt;background:white"><span style="font-size:11.0pt;color:#333333"> keep_hostname (yes);<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-bottom:9.0pt;background:white"><span style="font-size:11.0pt;color:#333333"> threaded(yes);<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-bottom:9.0pt;background:white"><span style="font-size:11.0pt;color:#333333">};<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks<span class="HOEnZb"><font color="#888888"><u></u><u></u></font></span></span></p><span class="HOEnZb"><font color="#888888">
<p class="MsoNormal"><span style="font-size:11.0pt">Kavita<u></u><u></u></span></p>
</font></span></div>
</div>
<br>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/<wbr>mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/<wbr>support/documentation/?<wbr>product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/<wbr>syslog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div>