<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hello Daniel,</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
3 to 500 events per second is definitely in the manageable range for Syslog-ng. However, in case of UDP there is no guarantee for delivery and it also lacks the traffic shaping mechanism of TCP. Thus, while your events/sec seems to be low, there could be peaks
in your traffic, which can cause packet drops on several levels.</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
At a first glance I would check the output of the ifconfig command to see if there is any packet drop on the interface. If there is no loss on the interface, you can check the statistics of Syslog-ng. <a href="https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.26/administration-guide/89#TOPIC-1431250">https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.26/administration-guide/89#TOPIC-1431250</a></div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Regarding to the "multi-threaded" topic. I would recommend Peter Czanik's blog post for a start.
<a href="https://www.syslog-ng.com/community/b/blog/posts/improved-log-collection-over-udp">
https://www.syslog-ng.com/community/b/blog/posts/improved-log-collection-over-udp</a> However tuning and optimizing your system is a trial-and-error process, and there is no "one fit for all" solution.</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Note: From the information you provided, at this point those logs can be missing because of a simple filter rule, which drops them. This is not necessarily a transmission issue. This is the reason why we need much more precise information about your setup.</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Br,</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Laci</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Daniel Ehrlich <Daniel.Ehrlich@usq.edu.au><br>
<b>Sent:</b> Tuesday, June 15, 2021 07:48<br>
<b>To:</b> Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu><br>
<b>Subject:</b> Re: [syslog-ng] Tips to diagnose missing syslog messages?</font>
<div> </div>
</div>
<style type="text/css" style="display:none">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div dir="ltr">
<div style="background-color:#FFEB9C; width:100%; border-style:solid; border-color:#9C6500; border-width:1pt; padding:2pt; font-size:10pt; line-height:12pt; font-family:'Calibri'; color:Black; text-align:left">
<span style="color:#9C6500; font-weight:bold">CAUTION:</span> This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.</div>
<br>
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Thanks Laci,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Yes this is syslog-ng as a receiver, from a network source of UDP 514.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
The destination is files on the local drive.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
The network path is allowed by firewall as we receive several messages, though some go missing.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
For example, some hosts send every minute and only 5 of the 15 might make it to the dest file.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I may be overloading the listener with 3-500 events per second?</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I think I am running syslog-ng 3.5 from RedHat repos.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I was sure if the multi-threaded option would help?</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Thanks</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Daniel</div>
<div>
<div id="x_appendonsend"></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Laszlo Szemere (lszemere) <Laszlo.Szemere@oneidentity.com><br>
<b>Sent:</b> Tuesday, 15 June 2021 12:34 AM<br>
<b>To:</b> Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu><br>
<b>Subject:</b> Re: [syslog-ng] Tips to diagnose missing syslog messages?</font>
<div> </div>
</div>
<div dir="ltr">
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Hello Daniel,</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
If I understand you correctly: you have a problem that one of your Syslog-ng server is not receiving a log message. (Not that you do not receive a message FROM one of your Syslog-ng server.)</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
From my experience: In those cases when someone tries to diagnose an issue where Syslog-ng do not receive a message, it is always a good first step to determine if the message actually reaches Syslog-ng or not.</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
i.e.: in case of a network source, try to receive the message with a simple netcat command. (Maybe the message was dropped by a firewall before reaching your machine, and Syslog-ng has nothing to do with it.) At this point we do not really care about the format
of the message.</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
If you made sure that those messages are reaching the application, we should try to diagnose Syslog-ng itself.<span style="color:rgb(0,0,0); font-family:Arial,Helvetica,sans-serif; font-size:12pt"> For that we will need some information about your setup.</span></div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
i.e.:</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
- your platform</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
- version of Syslog-ng (Where it is obtained from? i.e. local build)</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
- Related config parts. (including the source driver which is expected to receive the logs)</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
- How do you start Syslog-ng? (i.e.: as a service)</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Best regards,</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Laci</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div id="x_x_appendonsend"></div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Daniel Ehrlich <Daniel.Ehrlich@usq.edu.au><br>
<b>Sent:</b> Wednesday, June 9, 2021 07:24<br>
<b>To:</b> Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu><br>
<b>Subject:</b> [syslog-ng] Tips to diagnose missing syslog messages?</font>
<div> </div>
</div>
<div dir="ltr">
<div style="background-color:#FFEB9C; width:100%; border-style:solid; border-color:#9C6500; border-width:1pt; padding:2pt; font-size:10pt; line-height:12pt; font-family:'Calibri'; color:Black; text-align:left">
<span style="color:#9C6500; font-weight:bold">CAUTION:</span> This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.</div>
<br>
<div>
<div class="x_x_x_x_gmail_quote">
<div>Hi Everyone,</div>
<div><br>
</div>
<div>Does anyone have some diagnostic tips to offer to diagnose why syslog messages are not being received?</div>
<div>I have syslog-ng on a few servers but one is losing messages (others syslog-ng servers might be dropping that I am unaware off).</div>
<div>Not sure if there is some options I should add to the conf or diag commands or load specifications I should check?</div>
<div><br>
</div>
<div>Options:<br>
options {
<div> chain_hostnames(no);</div>
<div> create_dirs (yes);</div>
<div> dir_perm(0755);</div>
<div> dns_cache(yes);</div>
<div> keep_hostname(yes);</div>
<div> log_fifo_size(2048);</div>
<div> log_msg_size(8192);</div>
<div> perm(0644);</div>
<div> time_reopen (10);</div>
<div> use_dns(yes);</div>
<div> use_fqdn(yes);</div>
<div> flush_lines(100);</div>
<span>};</span><br>
</div>
<div><span><br>
</span></div>
<div><span>Also adding the flag-control flag to the log stanza.</span></div>
<div><span><br>
</span></div>
<div>Thanks</div>
<div>Daniel</div>
</div>
<div>__________________________________________________________________
<div>This email (including any attached files) is confidential and is
<div>for the intended recipient(s) only. If you received this email by
<div>mistake, please, as a courtesy, tell the sender, then delete this
<div>email.<br>
<div>The views and opinions are the originator's and do not necessarily
<div>reflect those of the University of Southern Queensland. Although
<div>all reasonable precautions were taken to ensure that this email
<div>contained no viruses at the time it was sent we accept no
<div>liability for any losses arising from its receipt.<br>
<div>The University of Southern Queensland is a registered provider
<div>of education with the Australian Government.
<div>(CRICOS Institution Code QLD 00244B / NSW 02225M, TEQSA PRV12081) </div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div>__________________________________________________________________
<div>This email (including any attached files) is confidential and is
<div>for the intended recipient(s) only. If you received this email by
<div>mistake, please, as a courtesy, tell the sender, then delete this
<div>email.<br>
<div>The views and opinions are the originator's and do not necessarily
<div>reflect those of the University of Southern Queensland. Although
<div>all reasonable precautions were taken to ensure that this email
<div>contained no viruses at the time it was sent we accept no
<div>liability for any losses arising from its receipt.<br>
<div>The University of Southern Queensland is a registered provider
<div>of education with the Australian Government.
<div>(CRICOS Institution Code QLD 00244B / NSW 02225M, TEQSA PRV12081) </div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>