<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:11pt; color:rgb(0,0,0)">
Hi Mark,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:11pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:11pt; color:rgb(0,0,0)">
I have some tips:</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:11pt; color:rgb(0,0,0)">
<ul>
<li>disable flow-control for UPD sources. Flow-control will not prevent message loss in case of UDP protocol, hence the only thing it can cause is that your syslog-ng source will not read messages from the kernel buffer and then packet drops can happen.</li><li>max-connections() does not scale UDP sources, there is no need to set it.</li><li>flush-lines() are not affecting network destinations <span style="font-family:Calibri,Arial,Helvetica,sans-serif; background-color:rgb(255,255,255); display:inline!important">(network(), syslog(), unix-*(), drivers)</span> performance, however the admin
guide is not clear about this. We will fix this.</li><li>log_iw_size() option is related to flow-control, no need to set it.</li><li><span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt;">set log-fetch-limit() to higher so you source can read more messages in a poll. The default value is 10, so trying out with e.g. 100 is a good start.</span><br>
</li><li>you also need to adjust destination buffer's size with log-fifo-size().<br>
<span style="margin: 0px; background-color: rgb(255, 255, 255); display: inline !important">You need to make sure that you log-fifo-size can hold enough messages without packet drop, e.g. if you can close to the max kernel buffer you configured. </span></li></ul>
</div>
<div id="appendonsend"></div>
<div><span style="font-family: calibri, arial, helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">We have a chapter about configuring syslog-ng parameters, it mostly focuses on flow-control:</span></div>
<div><span style="font-family: calibri, arial, helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);"><a href="https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.22/administration-guide/54#TOPIC-1209277" id="LPlnk800497">https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.22/administration-guide/54#TOPIC-1209277</a><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Regards,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Gabor</div>
<hr tabindex="-1" style="display: inline-block; width: 98%; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Faine, Mark R. (MSFC-IS40)[NICS] <mark.faine@nasa.gov><br>
<b>Sent:</b> Monday, July 29, 2019 16:59<br>
<b>To:</b> Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu><br>
<b>Subject:</b> [syslog-ng] Tips for handling large message load</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt">
<div class="PlainText">CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.<br>
<br>
<br>
I have several Splunk log aggregators that gets thousands of messages per second but we are seeing issues with dropping messages from UDP sources.<br>
<br>
I've read the section in the docs about handling large message load and we've made many of those changes. Do you have any other suggestions to improve performance?<br>
<br>
We are using flow control. We have made the following sysctl changes:<br>
- net.core.rmem_max = 268435456<br>
- net.core.netdev_max_backlog = 2000<br>
<br>
We have increased flush-lines to 100<br>
<br>
We are also looking to increase the initial window size below and max connections as well as the so_rcvbuf.<br>
<br>
network(port(514) transport("tcp") max-connections(100) log_iw_size(10000) flags(syslog-protocol));<br>
network(port(514) transport("udp") max-connections(100) log_iw_size(10000) flags(syslog-protocol));<br>
network(ip(0.0.0.0) transport("tls") port(10514) max-connections(100) log_iw_size(10000) flags(syslog-protocol)<br>
<br>
I'm currently showing about 50 TCP connections and almost 1000 UDP connections, though it's early on Monday morning so it is likely to increase as the day goes on. Most of the log messages are coming from the VPNs (hence so many UDP connections).<br>
<br>
The servers are quite beefy with 64GB of RAM and 24 Xeon cores @ 2.4GHz (Dell PowerEdge R530). We are using two separate network interfaces one for TCP and the other for UDP.<br>
<br>
I'd appreciate any suggestions on how to further increase performance, also, any general rules or calculations I can use to determine optimal values for these parameters from available system metrics would be very helpful.<br>
<br>
Thanks,<br>
-Mark<br>
______________________________________________________________________________<br>
Member info: <a href="https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7Cgabor.nagy%40oneidentity.com%7C0906048877534ba0012a08d7143562cd%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637000091887077907&sdata=AWkyZjqIhSW2OU7%2F2gFUL3DBfIAmaJ5Rb8sWC87MOho%3D&reserved=0">
https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7Cgabor.nagy%40oneidentity.com%7C0906048877534ba0012a08d7143562cd%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637000091887077907&sdata=AWkyZjqIhSW2OU7%2F2gFUL3DBfIAmaJ5Rb8sWC87MOho%3D&reserved=0</a><br>
Documentation: <a href="https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7Cgabor.nagy%40oneidentity.com%7C0906048877534ba0012a08d7143562cd%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637000091887077907&sdata=khjr%2FWBVx1V4lR9bB67i5KX%2BxTGWYinQIa4utv8LNO8%3D&reserved=0">
https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7Cgabor.nagy%40oneidentity.com%7C0906048877534ba0012a08d7143562cd%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637000091887077907&sdata=khjr%2FWBVx1V4lR9bB67i5KX%2BxTGWYinQIa4utv8LNO8%3D&reserved=0</a><br>
FAQ: <a href="https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7Cgabor.nagy%40oneidentity.com%7C0906048877534ba0012a08d7143562cd%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637000091887077907&sdata=7mevZAo06Ry0el0BAstkUbo3ql7zP2FV7vw5gO2inl8%3D&reserved=0">
https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7Cgabor.nagy%40oneidentity.com%7C0906048877534ba0012a08d7143562cd%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637000091887077907&sdata=7mevZAo06Ry0el0BAstkUbo3ql7zP2FV7vw5gO2inl8%3D&reserved=0</a><br>
<br>
</div>
</span></font></div>
</body>
</html>