[syslog-ng] Missing messages

Tue Jul 16 12:32:19 UTC 2019

My “missing messages” problem seems to be resolved. To close the loop on this issue I thought I would document what I had done. The first step was to update my syslog-ng.conf file with these 2 parameters on my ‘source’ statement.

so-reuseport(1) so_rcvbuf(70000000)

Next, I needed to adjust some kernel sysctl settings:
net.core.rmem_max = 88777666
net.core.rmem_default = 77666555
net.ipv4.udp_mem = 500000  1000000  2000000

I also found a good way to monitor the issue of “dropped UDP packets” is to use ‘netstat –us’ and look at the ‘receive buffer errors’ value. Normally, this should be zero. If it is not zero, it means you have dropped some UDP packets.

Bryan

From: syslog-ng [mailto:syslog-ng-bounces at lists.balabit.hu] On Behalf Of Attila Szakacs (aszakacs)
Sent: Monday, July 01, 2019 6:18 AM
To: 'Syslog-ng users' and developers' mailing list'
Subject: [EXTERNAL] Re: [syslog-ng] Missing messages

Hi Bryan,

Yes, you can use the same port for UDP and TCP.

It would look something like this, in your config:

source s_net {
  syslog(ip(0.0.0.0) transport("udp") port(514));
  syslog(ip(0.0.0.0) transport("tcp") port(514));
};
Syslog-ng also has an option, which might be interesting for you as an alternative/addition to your architecture change: so-reuseport().
It enables SO_REUSEPORT on systems that support it. When enabled, the kernel allows multiple UDP sockets to be bound to the same port, and the kernel load-balances incoming UDP datagrams to the sockets.

You can read more about it here, under the so-reuseport() section: https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.22/administration-guide/25#TOPIC-1209161

Best regards,
Attila
________________________________
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of Klimek, Bryan J. <bklimek at mayo.edu>
Sent: Saturday, June 29, 2019 9:01 PM
To: 'Syslog-ng users' and developers' mailing list'
Subject: Re: [syslog-ng] Missing messages

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

Attila,

Thank you for that reminder that UDP is not reliable. As a quick test I added a line to the /etc/rsyslog.conf file the server that has the missing messages and sent it to my test syslog-ng server that sees much less traffic. Now with the message traffic going to both my normal centralized syslog-ng server and my basically idle test syslog-ng server, I can see a difference. I seem to be dropping quite a few messages. Not just the one message that was found to be missing.

What are the ramifications of changing my "s_net" source from UPD to TCP? Do I need to reconfigure the thousands of hosts that are sending to my centralized syslog server to send using TCP.

Can I configure both transports (UDP and TCP) on the same “s_net” source? If this were possible, this would seem to be an easy way to continue to support UDP and transition over time to the TCP transport.

Bryan Klimek

Mayo Clinic

From: syslog-ng [mailto:syslog-ng-bounces at lists.balabit.hu] On Behalf Of Attila Szakacs (aszakacs)
Sent: Thursday, June 27, 2019 3:25 AM
To: 'syslog-ng at lists.balabit.hu'
Subject: [EXTERNAL] Re: [syslog-ng] Missing messages

Hi Bryan,

If I understand correctly, the message, which got lost, was expected to come from the "s_net" source.

If that is the case, it is normal to lose logs there, as UDP does not provide reliable data transmission.

Logging services have different solutions to this problem, syslog-ng Premium Edition has Advanced Log Transfer Protocol (ALTP).

You can read more about it here: https://support.oneidentity.com/technical-documents/syslog-ng-premium-edition/7.0.14/administration-guide/58#TOPIC-1187363

If I misunderstood the problem, please correct me.

Have a nice day! 🙂

Best regards,

Attila

________________________________

From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of Klimek, Bryan J. <bklimek at mayo.edu>
Sent: Wednesday, June 26, 2019 3:50 PM
To: 'syslog-ng at lists.balabit.hu'
Subject: Re: [syslog-ng] Missing messages

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

Attila,

Thank you for your response. I enabled the "stats-level(1)" so I can see the statistics. But the “dropped” counters are all zero.

I’ve attached the syslog-ng.conf file for review.

This syslog-ng instance receives syslog from 2000+ hosts. We create a separate file for every server for every day. We also fork the data to one consolidated file (all traffic from all hosts into 1 file). We also fork a 3rd copy to our SEIM (Security Event Information Management) solution.

The missing message does not appear in the individual host file, nor does it appear in the consolidate file of all hosts. I’ve never bothered to check with our SEIM people to see if they are getting the missing message. I just assume they are not.

Bryan Klimek
Phone: 507-284-9396
E-mail: klimek.bryan at mayo.edu<mailto:klimek.bryan at mayo.edu>

From: syslog-ng [mailto:syslog-ng-bounces at lists.balabit.hu] On Behalf Of Attila Szakacs (aszakacs)
Sent: Monday, June 24, 2019 8:40 AM
To: 'syslog-ng at lists.balabit.hu'
Subject: [EXTERNAL] Re: [syslog-ng] Missing messages

Hi Bryan,

Thank you for using the syslog-ng mailing list! 🙂

Message drop could happen for several reasons. I couple reasons, that suddenly comes up to me:

  1.  The log path has a filter or parser, which does not match for the message.
  2.  The log path has a source, which has a built-in parser (syslog for example), and the message does not match the protocol.
  3.  Flow-control is not enabled and the destination is not alive for a longer period of time.
  4.  Flow-control is configured incorrectly.
  5.  ...

Saying so, it is hard to come up with one general way to investigate this.

However, I can give you some tips:

  1.  You can set "stats-level(1)" in the global options and use "sbin/syslog-ng-ctl stats", then look for the "dropped" counters.
  2.  You can start syslog-ng in debug mode (./sbin/syslog-ng -Fedtv) and look for the following logs: "Destination queue full, dropping message;" or "UNMATCHED".
  3.  Check if flow-control is enabled and configured properly.
  4.  If you do not want to use flow-control, you can use disk-queue alternatively.

If you could share some parts of your setup and config, where the problem happens, we could give you more insight.

Best regards,

Attila

________________________________

From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of Klimek, Bryan J. <bklimek at mayo.edu>
Sent: Monday, June 24, 2019 3:05 PM
To: 'syslog-ng at lists.balabit.hu'
Subject: [syslog-ng] Missing messages

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

First time poster, so be gentle.

We run syslog-ng in our environment as a centralized syslog manager for the archiving of syslog data. We have over 2000+ Unix/Linux systems sending their syslog data with a daily ingest rate of about 10GB per day.

It was recently pointed out to me that one particular message from one specific host is not getting persisted to the files on our syslog-ng server all the time. That is to say, it is intermittent.

If one person can find one message that is not making into our syslog-ng archive, I can only assume that we are dropping other messages as well.

How can I debug if and when messages are being lost and not making into the files on my centralized syslog server?

Bryan Klimek

Mayo Clinic

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20190716/652c078e/attachment-0001.html>