[syslog-ng] Tips to diagnose missing syslog messages?

Tue Jun 15 07:25:08 UTC 2021

Hello Daniel,

3 to 500 events per second is definitely in the manageable range for Syslog-ng. However, in case of UDP there is no guarantee for delivery and it also lacks the traffic shaping mechanism of TCP. Thus, while your events/sec seems to be low, there could be peaks in your traffic, which can cause packet drops on several levels.

At a first glance I would check the output of the ifconfig command to see if there is any packet drop on the interface. If there is no loss on the interface, you can check the statistics of Syslog-ng. https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.26/administration-guide/89#TOPIC-1431250

Regarding to the "multi-threaded" topic. I would recommend Peter Czanik's blog post for a start. https://www.syslog-ng.com/community/b/blog/posts/improved-log-collection-over-udp However tuning and optimizing your system is a trial-and-error process, and there is no "one fit for all" solution.

Note: From the information you provided, at this point those logs can be missing because of a simple filter rule, which drops them. This is not necessarily a transmission issue. This is the reason why we need much more precise information about your setup.

Br,
Laci

________________________________
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of Daniel Ehrlich <Daniel.Ehrlich at usq.edu.au>
Sent: Tuesday, June 15, 2021 07:48
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu>
Subject: Re: [syslog-ng] Tips to diagnose missing syslog messages?

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

Thanks Laci,

Yes this is syslog-ng as a receiver, from a network source of UDP 514.
The destination is files on the local drive.
The network path is allowed by firewall as we receive several messages, though some go missing.
For example, some hosts send every minute and only 5 of the 15 might make it to the dest file.
I may be overloading the listener with 3-500 events per second?
I think I am running syslog-ng 3.5 from RedHat repos.
I was sure if the multi-threaded option would help?

Thanks
Daniel

________________________________
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of Laszlo Szemere (lszemere) <Laszlo.Szemere at oneidentity.com>
Sent: Tuesday, 15 June 2021 12:34 AM
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu>
Subject: Re: [syslog-ng] Tips to diagnose missing syslog messages?

Hello Daniel,

 If I understand you correctly: you have a problem that one of your Syslog-ng server is not receiving a log message. (Not that you do not receive a message FROM one of your Syslog-ng server.)

 From my experience: In those cases when someone tries to diagnose an issue where Syslog-ng do not receive a message, it is always a good first step to determine if the message actually reaches Syslog-ng or not.
 i.e.: in case of a network source, try to receive the message with a simple netcat command. (Maybe the message was dropped by a firewall before reaching your machine, and Syslog-ng has nothing to do with it.) At this point we do not really care about the format of the message.

 If you made sure that those messages are reaching the application, we should try to diagnose Syslog-ng itself. For that we will need some information about your setup.
 i.e.:
  - your platform
  - version of Syslog-ng (Where it is obtained from? i.e. local build)
  - Related config parts. (including the source driver which is expected to receive the logs)
  - How do you start Syslog-ng? (i.e.: as a service)

Best regards,
Laci

________________________________
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of Daniel Ehrlich <Daniel.Ehrlich at usq.edu.au>
Sent: Wednesday, June 9, 2021 07:24
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu>
Subject: [syslog-ng] Tips to diagnose missing syslog messages?

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

Hi Everyone,

Does anyone have some diagnostic tips to offer to diagnose why syslog messages are not being received?
I have syslog-ng on a few servers but one is losing messages (others syslog-ng servers might be dropping that I am unaware off).
Not sure if there is some options I should add to the conf or diag commands or load specifications I should check?

Options:
options {
   chain_hostnames(no);
   create_dirs (yes);
   dir_perm(0755);
   dns_cache(yes);
   keep_hostname(yes);
   log_fifo_size(2048);
   log_msg_size(8192);
   perm(0644);
   time_reopen (10);
   use_dns(yes);
   use_fqdn(yes);
   flush_lines(100);
};

Also adding the flag-control flag to the log stanza.

Thanks
Daniel
__________________________________________________________________
This email (including any attached files) is confidential and is
for the intended recipient(s) only. If you received this email by
mistake, please, as a courtesy, tell the sender, then delete this
email.
The views and opinions are the originator's and do not necessarily
reflect those of the University of Southern Queensland. Although
all reasonable precautions were taken to ensure that this email
contained no viruses at the time it was sent we accept no
liability for any losses arising from its receipt.
The University of Southern Queensland is a registered provider
of education with the Australian Government.
(CRICOS Institution Code QLD 00244B / NSW 02225M, TEQSA PRV12081)
__________________________________________________________________
This email (including any attached files) is confidential and is
for the intended recipient(s) only. If you received this email by
mistake, please, as a courtesy, tell the sender, then delete this
email.
The views and opinions are the originator's and do not necessarily
reflect those of the University of Southern Queensland. Although
all reasonable precautions were taken to ensure that this email
contained no viruses at the time it was sent we accept no
liability for any losses arising from its receipt.
The University of Southern Queensland is a registered provider
of education with the Australian Government.
(CRICOS Institution Code QLD 00244B / NSW 02225M, TEQSA PRV12081)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20210615/e2670b73/attachment-0001.html>