<div dir="ltr"><div><div><div>Well, some very basic information is missing from your report, like syslog-ng and OS versions.<br><br></div>The 750 filter rules seems like a lot, however 300 EPS is not that much. If you are using expensive regexps for filtering that can potentially explain why so much is dropped on the floor. Also, please look at the UDP statistics and double-check that we are dropping messages before reception into the syslog-ng process.<br><br></div>If this is a critical production deployment however, I would seriously consider the paid for services. The Open Source version of syslog-ng is not supported by Balabit's
professional support team, community is usually helping on a best effort
basis. Results are available as patches against the latest version, which you would have to compile and deploy, that is usually not enough to address P1 type of problems.<br><br></div><div>Anyhow, if you post some more details, we would look into this, as time allows.<br></div><div><br></div><div>Cheers,<br></div>Bazsi<br></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">-- <br>Bazsi<br></div></div></div>
<br><div class="gmail_quote">On Fri, Apr 13, 2018 at 10:50 PM, Schoonover, Mark E HHHH <span dir="ltr"><<a href="mailto:Mark.Schoonover@cigna.com" target="_blank">Mark.Schoonover@cigna.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div link="#0563C1" vlink="#954F72" lang="EN-US">
<div class="m_913163747494737891WordSection1">
<p class="MsoNormal"><span style="color:#1f497d">Naveen,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d">Need some additional information to help. How many messages per sec are arriving at your NG server? Can you post the section of the syslog-ng.conf file showing your sources?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d"><u></u> <u></u></span></p>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:#1f497d">Regards,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas;color:#1f497d">Mark Schoonover – KA6WKE - Infrastructure Engineering Manager
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas;color:#1f497d">ENE : Tools, Instrumentation and Common Services Team<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas;color:#1f497d">Office: 32.8697° N, 116.9711° W - Phone : 770-261-7934 - Email :
<a href="mailto:mark.schoonover@cigna.com" target="_blank"><span style="color:#0563c1">mark.schoonover@cigna.com</span></a><u></u><u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1f497d">HPSM Team: ENE NMS Engineering<u></u><u></u></span></b></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1f497d"> </span></b><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1f497d"><img id="m_913163747494737891Picture_x0020_1" src="cid:image002.png@01D3D32E.58B06D30" alt="vet" width="96" border="0" height="61"><u></u><u></u></span></b></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1f497d"><u></u> <u></u></span></b></p>
<p class="MsoNormal"><i><span style="font-size:10.0pt;font-family:"Segoe UI",sans-serif;color:#444444">Confidential, unpublished property of Cigna. Do not duplicate or distribute. Use and distribution limited solely to authorized personnel. © Copyright 2018
Cigna.</span></i><span style="font-family:Consolas;color:#1f497d"><u></u><u></u></span></p>
</div>
<p class="MsoNormal"><span style="color:#1f497d"><u></u> <u></u></span></p>
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> syslog-ng [mailto:<a href="mailto:syslog-ng-bounces@lists.balabit.hu" target="_blank">syslog-ng-bounces@<wbr>lists.balabit.hu</a>]
<b>On Behalf Of </b>Naveen Bhalla (nbhalla)<br>
<b>Sent:</b> Friday, April 13, 2018 10:43 AM<br>
<b>To:</b> <a href="mailto:syslog-ng@lists.balabit.hu" target="_blank">syslog-ng@lists.balabit.hu</a><br>
<b>Subject:</b> Re: [syslog-ng] Support for Open Source Syslog-ng<u></u><u></u></p>
</div>
</div><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:#1f497d">Team,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:#1f497d"> Could you pls help us on below P1 situation for us?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:#1f497d"><u></u> <u></u></span></p>
<div>
<p class="MsoNormal"><span style="font-size:13.0pt;color:#1f3864"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:#1f3864"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:#1f3864">Regards,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:#1f497d"><img id="m_913163747494737891_x0000_i1025" src="cid:image003.png@01D3D32E.58B06D30" alt="http://wwwin.cisco.com/c/dam/cec/organizations/gmcc/services-tools/signaturetool/images/logo/logo_gradient.png" width="110" border="0" height="73"><u></u><u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:#1f3864">Naveen Bhalla</span></b><span style="font-size:12.0pt;color:#1f3864"> |
</span><span style="font-size:10.0pt;color:#1f3864">Manager.Technical Support</span><span style="font-size:12.0pt;color:#1f3864"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:#1f3864">CMS Platform Operations<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:#1f3864"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:#1f3864">Cell: +91-9880362157<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:#1f3864">Desk: +91-80-44260795<u></u><u></u></span></p>
</div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:#1f497d"><u></u> <u></u></span></p>
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Naveen Bhalla (nbhalla) <br>
<b>Sent:</b> 13 April 2018 09:14 PM<br>
<b>To:</b> '<a href="mailto:support@balabit.com" target="_blank">support@balabit.com</a>' <<a href="mailto:support@balabit.com" target="_blank">support@balabit.com</a>><br>
<b>Subject:</b> Support for Open Source Syslog-ng<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:#1f4e79">Hello Support Team,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:#1f4e79"> We have a situation in our platform where syslog-ng is dropping part of syslog traffic coming into our server. The syslog-ng has around 750 match rules in its configuration. Based on these
rules the syslogs are forwarded to the destinations. Also, there is one rule to write all the received syslogs to disk. We are receiving syslogs at the rate of 300 eps.
<u></u><u></u></span></p>
<p class="m_913163747494737891p1"><span style="font-size:12.0pt;font-family:"Calibri",sans-serif;color:#1f4e79"><u></u> <u></u></span></p>
<p class="m_913163747494737891p1"><span style="font-size:12.0pt;font-family:"Calibri",sans-serif;color:#1f4e79">The issue is that we are seeing that syslog-ng is not able to process the syslogs and forward them to the destinations. It is not writing to the disk also. We are seeing
that there is a big delay after which some syslogs are getting written to the disk. We are seeing loss of UDP packets. The UDP buffer size is big enough.<u></u><u></u></span></p>
<p class="m_913163747494737891p1"><u></u> <u></u></p>
<p class="MsoNormal" style="margin-bottom:9.0pt;background:#f1f1f1"><span style="font-size:10.5pt;font-family:"Helvetica Neue";color:#333333">net.ipv4.tcp_rmem = 4096 4194304 16777216<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-bottom:9.0pt;background:#f1f1f1"><span style="font-size:10.5pt;font-family:"Helvetica Neue";color:#333333">net.ipv4.tcp_wmem = 98304 4194304 16777216<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-bottom:9.0pt;background:#f1f1f1"><span style="font-size:10.5pt;font-family:"Helvetica Neue";color:#333333">net.core.rmem_default = 234217728<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-bottom:9.0pt;background:#f1f1f1"><span style="font-size:10.5pt;font-family:"Helvetica Neue";color:#333333">net.core.wmem_default = 234217728<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-bottom:9.0pt;background:#f1f1f1"><span style="font-size:10.5pt;font-family:"Helvetica Neue";color:#333333">net.core.rmem_max = 234217728<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-bottom:9.0pt;background:#f1f1f1"><span style="font-size:10.5pt;font-family:"Helvetica Neue";color:#333333">net.core.wmem_max = 234217728<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-bottom:9.0pt;background:#f1f1f1"><span style="font-size:10.5pt;font-family:"Helvetica Neue";color:#333333">net.ipv4.tcp_window_scaling = 1<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-bottom:9.0pt;background:#f1f1f1"><span style="font-size:10.5pt;font-family:"Helvetica Neue";color:#333333">net.ipv4.ip_local_port_range = 32768 61000<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-bottom:9.0pt;background:#f1f1f1"><span style="font-size:10.5pt;font-family:"Helvetica Neue";color:#333333">fs.file-max = 2097152<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-bottom:9.0pt;background:#f1f1f1"><span style="font-size:10.5pt;font-family:"Helvetica Neue";color:#333333">net.core.optmem_max = 40960<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-bottom:9.0pt;background:#f1f1f1"><span style="font-size:10.5pt;font-family:"Helvetica Neue";color:#333333">net.core.netdev_max_backlog = 50000<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-bottom:9.0pt;background:#f1f1f1"><span style="font-size:10.5pt;font-family:"Helvetica Neue";color:#333333">net.ipv4.udp_rmem_min = 8192<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-bottom:9.0pt;background:#f1f1f1"><span style="font-size:10.5pt;font-family:"Helvetica Neue";color:#333333">net.ipv4.udp_wmem_min = 8192<u></u><u></u></span></p>
<p class="m_913163747494737891p1"><u></u> <u></u></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:#1f4e79">We need help to resolve this issue.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:#1f4e79"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:#1f4e79"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:#1f4e79">We are using open-source syslog-ng in our setup.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:#1f4e79"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:13.0pt;color:#1f3864"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:#1f3864"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:#1f3864">Regards,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:#1f497d"><img id="m_913163747494737891Picture_x0020_4" src="cid:image003.png@01D3D32E.58B06D30" alt="http://wwwin.cisco.com/c/dam/cec/organizations/gmcc/services-tools/signaturetool/images/logo/logo_gradient.png" width="110" border="0" height="73"><u></u><u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:#1f3864">Naveen Bhalla</span></b><span style="font-size:12.0pt;color:#1f3864"> |
</span><span style="font-size:10.0pt;color:#1f3864">Manager.Technical Support</span><span style="font-size:12.0pt;color:#1f3864"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:#1f3864">CMS Platform Operations<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:#1f3864"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:#1f3864">Cell: +91-9880362157<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:#1f3864">Desk: +91-80-44260795<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
</div></div></div>
<p>------------------------------<wbr>------------------------------<wbr>------------------<br>CONFIDENTIALITY NOTICE: If you have received this email in error,<br>please immediately notify the sender by e-mail at the address shown. <br>This email transmission may contain confidential information. This <br>information is intended only for the use of the individual(s) or entity to <br>whom it is intended even if addressed incorrectly. Please delete it from <br>your files if you are not the intended recipient. Thank you for your <br>compliance. Copyright (c) 2018 Cigna<br>==============================<wbr>==============================<wbr>==================</p></div>
<br>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/<wbr>mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/<wbr>support/documentation/?<wbr>product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/<wbr>syslog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div>