July 2020 - syslog-ng - lists.balabit.hu

UDP drops with syslog-ng 3.24.1-1
by brian hoffman 05 Oct '22

05 Oct '22

We've been centrally logging with syslog-ng for about 5 years now. Over that time, the number of sources has grown significantly, and at some point we crossed a line where drops were happening (a quick survey of 3 million syslog packets yielded 420 unique currently sending hosts). After much research and experimentation, we've been able to get to the point where throughout the day there are 0 drops for the most part. This was achieved by installing the latest syslog-ng (not the RedHat packaged one) and creating a source for each CPU. Occasionally, though, we still have periods of drops so I'm trying to eliminate these last few. Here are some relevant configuration items:2 RedHat 7.8 VMs (load balanced via an F5) with 16GB memory and 4 CPUs each running syslog-ng-3.24.1-1.el7.x86_64.net.core.rmem_default = 212992net.core.rmem_max = 268435456 log_fifo_size(268435456); source s_network {network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp1"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp2"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp3"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp4"));network(ip("0.0.0.0") port(514) transport("tcp") max_connections(200) keep_alive(yes) so_rcvbuf(67108864));}; We are limited to UDP, unfortunately, because we do not have control over the devices/networks/etc. that are sending to us, but we have changed as many of the internal senders and destinations to TCP as we can. With a script I created to view the packets, including drops, as well as the individual RECVQs, the issue can be illustrated. Here's what things look like normally:Thu May 7 10:48:15 EDT 2020, 27003 IP pkts rcvd,26980 IP pkts sent,24951 UDP pkts rcvd, 28075 UDP pkts sent,0 UDP pkt rcv err RECVQ-1=2176RECVQ-2=0RECVQ-3=0RECVQ-4=0Thu May 7 10:48:16 EDT 2020, 28453 IP pkts rcvd,28426 IP pkts sent,26185 UDP pkts rcvd, 29180 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=0RECVQ-2=0RECVQ-3=4352RECVQ-4=0Thu May 7 10:48:17 EDT 2020, 28294 IP pkts rcvd,28276 IP pkts sent,26277 UDP pkts rcvd, 28709 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=2176RECVQ-2=0RECVQ-3=0RECVQ-4=0 The RECVQs are sparsely used, and there are no errors. Around 9pm every night, the packet counts go up significantly (probably due to backup related logs):Wed May 6 21:00:08 EDT 2020, 66382 IP pkts rcvd,66366 IP pkts sent,39405 UDP pkts rcvd, 67592 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=1595008RECVQ-2=106217088RECVQ-3=53694976RECVQ-4=31858816Wed May 6 21:00:09 EDT 2020, 69317 IP pkts rcvd,69338 IP pkts sent,44446 UDP pkts rcvd, 75958 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=13056RECVQ-2=126397312RECVQ-3=75568128RECVQ-4=41626880Wed May 6 21:00:10 EDT 2020, 71205 IP pkts rcvd,71227 IP pkts sent,43657 UDP pkts rcvd, 74603 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=920448RECVQ-2=146122752RECVQ-3=100951168RECVQ-4=52622208Wed May 6 21:00:12 EDT 2020, 69578 IP pkts rcvd,69454 IP pkts sent,124465 UDP pkts rcvd, 163367 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=13140864RECVQ-2=44494848RECVQ-3=125579136RECVQ-4=0 Still, though, it's handling it with no errors. But then at some point a threshold is reached and errors start piling up:Wed May 6 21:00:20 EDT 2020, 63177 IP pkts rcvd,63291 IP pkts sent,0 UDP pkts rcvd, 0 UDP pkts sent,38011 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=200357376RECVQ-3=292948352RECVQ-4=28890752Wed May 6 21:00:21 EDT 2020, 69501 IP pkts rcvd,69464 IP pkts sent,0 UDP pkts rcvd, 1 UDP pkts sent,42158 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=223551360RECVQ-3=314995584RECVQ-4=41735680Wed May 6 21:00:23 EDT 2020, 69962 IP pkts rcvd,69978 IP pkts sent,0 UDP pkts rcvd, 2 UDP pkts sent,43775 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=244732544RECVQ-3=338239616RECVQ-4=53858176Wed May 6 21:00:24 EDT 2020, 68266 IP pkts rcvd,68216 IP pkts sent,0 UDP pkts rcvd, 0 UDP pkts sent,43118 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=265258752RECVQ-3=360643712RECVQ-4=65362688 The common denominator I've found is that one of the RECVQs hits 536,871,424. This number seems to be almost exactly double (512 difference) the rmem.max/log_fifo_size (268,435,456). Even though there seems to be capacity in the other RECVQs, just one of those hitting that magic number seems to be enough to throw things out of whack. During this time, the CPU usage for syslog-ng also drops: top - 21:00:11 up 2 days, 8:40, 2 users, load average: 1.09, 1.41, 1.32 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 123.6 10.9 3080:46 syslog-ng top - 21:00:14 up 2 days, 8:40, 2 users, load average: 1.00, 1.39, 1.31 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 24.6 10.9 3080:46 syslog-ng top - 21:00:17 up 2 days, 8:40, 2 users, load average: 1.00, 1.39, 1.31 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 7.6 10.9 3080:47 syslog-ng As best I can tell, based on the reading I've done, the message counts we're getting should be doable, but it's still not clear exactly how to size some of these options as they relate to UDP (like log_fetch_limit, which we do not have set). Anything else I should try?

2 4

Re: [syslog-ng] syslog-ng multiple VRF
by Terry Slattery 10 Aug '20

10 Aug '20

Alex, The simplest solution is to configure the network to share routes between the two VRFs. Linux now supports VRF, however, my quick read on the subject suggests that its configuration is complex. Applications like syslog-ng are rarely VRF-aware. It would have to bind to two different interfaces, one in each VRF. -tcs On 7/22/20 8:00 AM, syslog-ng-request(a)lists.balabit.hu wrote: > Date: Tue, 21 Jul 2020 15:18:46 +0100 > From: Alexandre Santos <alexandre.rosas.santos(a)gmail.com> > To: "Syslog-ng users' and developers' mailing list" > <syslog-ng(a)lists.balabit.hu> > Subject: [syslog-ng] syslog-ng multiple VRF > Message-ID: > <CAHxG_72hhkfCBu3j9nPxr_MVwKevJtFcpcRhVJLYPPYx4Mys1g(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi All, > > I have a question regarding syslog-ng and VRF. > I want to read from a syslog source, which interface is in the default VRF, > and send the logs to a syslog/network destination interface which is in a > MGMT VRF. > > Can syslog-ng support this? > If yes, what are the aspects I should be careful about? > > Thanks and regards, > Alex -- Terry Slattery CCIE #1026 NetCraftsmen www.netcraftsmen.com

5 7

journald prefix
by Fabien Wernli 05 Aug '20

05 Aug '20

Hi, I'm in the process of simplifying my structured logging config. Most of my systems now have journald shipping local logs to syslog-ng via the `system()` auto-expanding source. While this is convenient, as it makes it possible to use the same config for all my systems (systemd and rest of world), it also makes it impossible to configure the underlying driver's defaults. The task at hand is shipping the journald additional fields, which would be easy to do using the following excerpt from the documentation: source s_journald { systemd-journal(prefix(".SDATA.journald.")); }; This is not possible of course using `system()`. I've got a couple of ideas on how to handle this, but I'd like to hear other (probably less insane than my own) ways first :-) Cheers

2 2

Error message in syslog-ng (LOGMSG_REFCACHE_VALUE_TO_REF(old_value) >= 1)
by Ahmed Soyef 29 Jul '20

29 Jul '20

Hi All, We're seeing some instances of this error message ERROR:lib/logmsg/logmsg.c:1521:log_msg_ref: assertion failed: (LOGMSG_REFCACHE_VALUE_TO_REF(old_value) >= 1) I've had a quick search about and can't figure out what it actually means. Can anyone shed some light and possibly some guidance on how to avoid it happening? Regards, Soyef ********************************************************************** This email and any files transmitted with it are private and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please return it to the address it came from telling them it is not for you and then delete it from your system. This email message has been swept for computer viruses. **********************************************************************

2 1

File destination is not being written, when it can not bind remote host address
by Alexandre Santos 29 Jul '20

29 Jul '20

Hi, I do not know if this a seen issue or not. I am using syslog-ng version 3.19 on a Debian 10. I have the configuration which is attached in this email: syslog-ng.conf, that has one local file destination and one remote host destination. 1. Start the syslog-ng The localip of syslog destination is not available and therefore I have these errors, as expected: syslog-ng[1482]: [2020-07-27T12:58:00.069422] Error binding socket; addr='AF_INET(10.0.2.6:0)', error='Cannot assign requested address (99)' syslog-ng[1482]: [2020-07-27T12:58:00.069439] Initiating connection failed, reconnecting; time_reopen='60' 2. After some time running syslog-ng stops sending the logs to the local file. Why? 3. If I reload the syslog-ng configuration, without the remote destination, the old logs are immediately flushed to the local file. Jul 27 *15:05:55* localhost syslog-ng[29680]: [*2020-07-27T15:05:55.378*019] Incoming log entry from journal; message='[*2020-07-27T13:44:06.890*952] Outgoing message; message=\'<30>1 2020-07-27T13:44:06.772+00:00 localhost syslog-ng 29680 - - [2020-07-27T13:44:06.772722] Processing the time zone file (32bit part); filename=\\'/usr/share/zoneinfo/UTC\\'\x0a\'' Jul 27 15:05:55 localhost syslog-ng[29680]: [2020-07-27T15:05:55.378032] json-parser(): no marker at the beginning of the message, skipping JSON parsing ; input='[2020-07-27T13:44:06.890952] Outgoing message; message=\'<30>1 2020-07-27T13:44:06.772+00:00 localhost syslog-ng 29680 - - [2020-07-27T13:44:06.772722] Processing the time zone file (32bit part); filename=\\'/usr/share/zoneinfo/UTC\\'\x0a\'', marker='@cim:' Is this a new issue? Is there any configuration to prevent this? In attachment also the journalctl of syslog-ng in debug mode (syslog-ng -Fvde). Thanks and regards, Alex

2 1

setting sequenceId in forwarded log messages read from journald reader
by Peter Vollmer 29 Jul '20

29 Jul '20

Hi, I am currently trying to find a way to set meta.sequenceId of log messages that have been read from the locally running systemd-journal to forward them to a remote syslog server that expects the logs to contain a sequenceId according to RFC 5424 section 7.3.1. I found that a sequence number could be taken from the __CURSOR field "i=..." of the journald log: # journalctl -o json-pretty -f ... "__CURSOR" : "s=02a7b30ba17b4a43846f265706bd3a70;i=f01;b=ba633698f20848e480bca4e72476e4d3;m=1a355c1d5;t=5ab670340c8ea;x=33389988ef680e7e", ... My problem is that the journal reader does not seem to parse the __CURSOR string when reading from journald logs. Is there a way to get this information into meta.sequenceId of the forwarded log without modifying the systemd-journal module in syslog-ng ? Thank you for any ideas and best regards Peter Vollmer

3 2

Certificate ip authentication
by Alexandre Damas 27 Jul '20

27 Jul '20

Hi, As my objective on the utilization of syslog-ng is the authentication and certification of received security and auditing events, I implemented an internal CA which generates a certificate per syslog-ng client. Once there was someone that misconfigured a client and got a certificate for a different client. The funny part of it is that no one observed any problem as the certificate, which was generated for a client having an ip (different from the one configured on the machine), was working and communications (using ALTP with TLS) went up for the exchange of messages. Has anyone experienced this? Does someone have a clue on how to prevent certificate reutilization on the client side? On Linux side I did not find any way of preventing the utilization of a certificate for one machine, that was issued for a different machine, having a different ip. If there is no cross checking on the server for the ip addresses on the certificate with the ip address on the received event, I don't have a way of non-repudiate a received event and the client can reuse the certificate for other machines, allowing the events to be received on the server. Kind Regards Alexandre Damas

2 2

Configuration validation Issue with snmptrap module
by Luca Da Col 23 Jul '20

23 Jul '20

Running syslog-ng 3.27.1 on Centos 7. Configuration validation returns an error with snmptrap command. Configuration example has been taken directly from the OSE documentation. Error error was reproducible with version available on epel repository (3.5), though documentation indicates the snmptrap is available from 3.10 Configuration file includes statements: @version:3.22 @include "scl.conf" Any assistance is appreciated, Thank you, Luca [root@adev syslog-ng]# syslog-ng --syntax-only Error parsing source statement, source plugin snmptrap not found in /etc/syslog-ng/syslog-ng.conf:92:5-92:13: 90 log { 91 source { 92----> snmptrap(filename("/var/log/snmptrapd.log")); 92----> ^^^^^^^^ 93 }; 94 destination { 95 file("/var/log/example.log"); 96 }; 97 }; [root@dev syslog-ng]# syslog-ng -V syslog-ng 3 (3.27.1) Config version: 3.22 Installer-Version: 3.27.1 Revision: Compile-Date: Jun 8 2020 07:54:17 Module-Directory: /usr/lib64/syslog-ng Module-Path: /usr/lib64/syslog-ng Include-Path: /usr/share/syslog-ng/include Available-Modules: affile,afprog,afsocket,afstomp,afuser,basicfuncs,confgen,cryptofuncs,csvparser,dbparser,linux-kmsg-format,syslogformat,system-source,add-contextual-data,appmodel,cef,disk-buffer,examples,graphite,hook-commands,json-plugin,kvformat,map-value-pairs,pseudofile,sdjournal,stardate,tags-parser,tfgetent,timestamp,xml Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Systemd: on [root@dev syslog-ng]#

2 1

syslog-ng multiple VRF
by Alexandre Santos 22 Jul '20

22 Jul '20

Hi All, I have a question regarding syslog-ng and VRF. I want to read from a syslog source, which interface is in the default VRF, and send the logs to a syslog/network destination interface which is in a MGMT VRF. Can syslog-ng support this? If yes, what are the aspects I should be careful about? Thanks and regards, Alex

2 2

syslog-ng restarts due to interface not ready
by Alexandre Santos 21 Jul '20

21 Jul '20

Hi, I have a configuration where one of my sources is a syslog source: syslog(ip(127.1.1.1) transport("udp") port(514) keep-alive(no)); But when the system starts up, the interface is not ready yet, and I get the following errors: Jun 26 09:17:50 localhost systemd[1]: Starting System Logger Daemon... Jun 26 09:17:52 localhost syslog-ng[2301]: [2020-06-26T09:17:52.485148] WARNING: With use-dns(no), dns-cache() will be forced to 'no' too!; Jun 26 09:17:52 localhost syslog-ng[2301]: [2020-06-26T09:17:52.487159] Current position not matches to the saved cursor position, seek to head; cursor='s=f62c0e80f50948deb29cf2d28230a Jun 26 09:17:52 localhost syslog-ng[2301]: [2020-06-26T09:17:52.487867] Error resolving hostname; host='127.1.1.1' Jun 26 09:17:52 localhost syslog-ng[2301]: [2020-06-26T09:17:52.487878] Error initializing message pipeline; plugin_name='syslog', location='/etc/syslog-ng/syslog-ng.conf:43:5' Jun 26 09:17:52 localhost systemd[1]: syslog-ng.service: Main process exited, code=exited, status=2/INVALIDARGUMENT Jun 26 09:17:52 localhost systemd[1]: syslog-ng.service: Failed with result 'exit-code'. Jun 26 09:17:52 localhost systemd[1]: Failed to start System Logger Daemon. Jun 26 09:17:52 localhost systemd[1]: syslog-ng.service: Service RestartSec=100ms expired, scheduling restart. Jun 26 09:17:52 localhost systemd[1]: syslog-ng.service: Scheduled restart job, restart counter is at 1. When the systemd reaches the 5 attempts it disables the syslog-ng.service. Do you know some way to fix this, by adding something, the syslog-ng configuration? Thanks in Advance, Alex

4 4