January 2021 - syslog-ng - lists.balabit.hu

UDP drops with syslog-ng 3.24.1-1
by brian hoffman 05 Oct '22

05 Oct '22

We've been centrally logging with syslog-ng for about 5 years now. Over that time, the number of sources has grown significantly, and at some point we crossed a line where drops were happening (a quick survey of 3 million syslog packets yielded 420 unique currently sending hosts). After much research and experimentation, we've been able to get to the point where throughout the day there are 0 drops for the most part. This was achieved by installing the latest syslog-ng (not the RedHat packaged one) and creating a source for each CPU. Occasionally, though, we still have periods of drops so I'm trying to eliminate these last few. Here are some relevant configuration items:2 RedHat 7.8 VMs (load balanced via an F5) with 16GB memory and 4 CPUs each running syslog-ng-3.24.1-1.el7.x86_64.net.core.rmem_default = 212992net.core.rmem_max = 268435456 log_fifo_size(268435456); source s_network {network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp1"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp2"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp3"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp4"));network(ip("0.0.0.0") port(514) transport("tcp") max_connections(200) keep_alive(yes) so_rcvbuf(67108864));}; We are limited to UDP, unfortunately, because we do not have control over the devices/networks/etc. that are sending to us, but we have changed as many of the internal senders and destinations to TCP as we can. With a script I created to view the packets, including drops, as well as the individual RECVQs, the issue can be illustrated. Here's what things look like normally:Thu May 7 10:48:15 EDT 2020, 27003 IP pkts rcvd,26980 IP pkts sent,24951 UDP pkts rcvd, 28075 UDP pkts sent,0 UDP pkt rcv err RECVQ-1=2176RECVQ-2=0RECVQ-3=0RECVQ-4=0Thu May 7 10:48:16 EDT 2020, 28453 IP pkts rcvd,28426 IP pkts sent,26185 UDP pkts rcvd, 29180 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=0RECVQ-2=0RECVQ-3=4352RECVQ-4=0Thu May 7 10:48:17 EDT 2020, 28294 IP pkts rcvd,28276 IP pkts sent,26277 UDP pkts rcvd, 28709 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=2176RECVQ-2=0RECVQ-3=0RECVQ-4=0 The RECVQs are sparsely used, and there are no errors. Around 9pm every night, the packet counts go up significantly (probably due to backup related logs):Wed May 6 21:00:08 EDT 2020, 66382 IP pkts rcvd,66366 IP pkts sent,39405 UDP pkts rcvd, 67592 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=1595008RECVQ-2=106217088RECVQ-3=53694976RECVQ-4=31858816Wed May 6 21:00:09 EDT 2020, 69317 IP pkts rcvd,69338 IP pkts sent,44446 UDP pkts rcvd, 75958 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=13056RECVQ-2=126397312RECVQ-3=75568128RECVQ-4=41626880Wed May 6 21:00:10 EDT 2020, 71205 IP pkts rcvd,71227 IP pkts sent,43657 UDP pkts rcvd, 74603 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=920448RECVQ-2=146122752RECVQ-3=100951168RECVQ-4=52622208Wed May 6 21:00:12 EDT 2020, 69578 IP pkts rcvd,69454 IP pkts sent,124465 UDP pkts rcvd, 163367 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=13140864RECVQ-2=44494848RECVQ-3=125579136RECVQ-4=0 Still, though, it's handling it with no errors. But then at some point a threshold is reached and errors start piling up:Wed May 6 21:00:20 EDT 2020, 63177 IP pkts rcvd,63291 IP pkts sent,0 UDP pkts rcvd, 0 UDP pkts sent,38011 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=200357376RECVQ-3=292948352RECVQ-4=28890752Wed May 6 21:00:21 EDT 2020, 69501 IP pkts rcvd,69464 IP pkts sent,0 UDP pkts rcvd, 1 UDP pkts sent,42158 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=223551360RECVQ-3=314995584RECVQ-4=41735680Wed May 6 21:00:23 EDT 2020, 69962 IP pkts rcvd,69978 IP pkts sent,0 UDP pkts rcvd, 2 UDP pkts sent,43775 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=244732544RECVQ-3=338239616RECVQ-4=53858176Wed May 6 21:00:24 EDT 2020, 68266 IP pkts rcvd,68216 IP pkts sent,0 UDP pkts rcvd, 0 UDP pkts sent,43118 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=265258752RECVQ-3=360643712RECVQ-4=65362688 The common denominator I've found is that one of the RECVQs hits 536,871,424. This number seems to be almost exactly double (512 difference) the rmem.max/log_fifo_size (268,435,456). Even though there seems to be capacity in the other RECVQs, just one of those hitting that magic number seems to be enough to throw things out of whack. During this time, the CPU usage for syslog-ng also drops: top - 21:00:11 up 2 days, 8:40, 2 users, load average: 1.09, 1.41, 1.32 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 123.6 10.9 3080:46 syslog-ng top - 21:00:14 up 2 days, 8:40, 2 users, load average: 1.00, 1.39, 1.31 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 24.6 10.9 3080:46 syslog-ng top - 21:00:17 up 2 days, 8:40, 2 users, load average: 1.00, 1.39, 1.31 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 7.6 10.9 3080:47 syslog-ng As best I can tell, based on the reading I've done, the message counts we're getting should be doable, but it's still not clear exactly how to size some of these options as they relate to UDP (like log_fetch_limit, which we do not have set). Anything else I should try?

2 4

Help with filtering at destination based on custom macro value match
by Akshay Joshi 30 Jan '21

30 Jan '21

My client is sending logs and it has the following config : *template t_global {template("<${PRI}>${LOGHOST} ${MSG_TAG}${MSGHDR}${MSG}\n"); };* *# Global logging remote destination:#-----------------------------------destination d_global_remote { tcp("proxy.dc.nuagedemo.net <http://proxy.dc.nuagedemo.net>" port(10514) template(t_global) tls(peer-verify(required-untrusted) ca-dir('/etc/default/bootstrap/keys') cert_file('/etc/default/bootstrap/keys/cert.pem') key-file('/etc/default/bootstrap/keys/key.pem') ) flags("threaded") );};source s_nuageDiag { file("/home/user/nuage/nuage_diagnostics_daemon.log" follow-freq(10) default-facility(local1) default-priority(info) tags("nuageDiag"));};rewrite w_nuageDiag { set("nuage-diag: ", value("MSG_TAG") condition(tags("nuageDiag"))); };* On the destination, I have this : *source s_network { tcp( port(10514) max-connections(1000) tls( peer-verify(required-untrusted) key-file("/opt/proxy/config/keys/proxy-Key.pem") cert_file("/opt/proxy/config/keys/proxyCert.pem") ca-dir("/opt/proxy/config/keys/proxy-CA.pem") ) );};* *filter nsg_diag { match("nuage-diag: " value("MSG_TAG"));};filter f_messages { (level(info..warn) and filter (nsg_diag)); };* *destination d_logs { file( "/var/log/syslog-ng/logs.txt" owner("root") group("root") perm(0777) );};log { source(s_sys); source(s_network); filter(f_messages); destination(d_logs); };* I can write logs locally without the filtering. But with filtering, it does not match "nuage-diag: " macro. This "MSG_TAG" does not seem to be a standard header but a custom one. I couldn't find many straightforward examples on forums etc.. as well. I am missing a trick or two config-wise for sure. Any pointers / help will be much appreciated. -- Regards, Akshay Joshi

3 2

syslog-ng won't talk to systemd?
by Dan Egli 26 Jan '21

26 Jan '21

I'm trying to setup syslog-ng on a systemd system. But even though I have several things writing to syslog, syslog-ng doesn't write ANYTHING to the /var/log/messages except kernel messages and it's own info. So I searched around, and found the systemd-journal() source. So I tried it. And I get an error that says syslog-ng can't initialize the logger plugin: # syslog-ng -F [2021-01-23T23:25:36.525097] The configuration must not contain more than one systemd-journal() source; [2021-01-23T23:25:36.525157] Error initializing journal_reader; [2021-01-23T23:25:36.525173] Error initializing message pipeline; plugin_name='systemd-journal', location='/etc/syslog-ng/syslog-ng.conf:26:5' Here's the config, basically copied from the syslog-ng.com page: @version: 3.26 # # Syslog-ng default configuration file for Gentoo Linux # https://bugs.gentoo.org/426814 @include "scl.conf" options { threaded(yes); chain_hostnames(no); stats_freq(43200); mark_freq(3600); }; source s_journald { systemd-journal(prefix(".SDATA.journald.")); }; source src { system(); internal(); file("/proc/kmsg"); }; destination messages { file("/var/log/messages"); }; destination console_all { file("/dev/tty12"); }; log { source(src); destination(messages); }; log { source(s_journald); destination(messages); }; log { source(src); destination(console_all); }; What am I doing wrong? I am POSITIVE things are sending to the syslog sub-system. I have ensured several programs have sent something. But without the systemd-journal line syslog-ng loads fine, recording /proc/kmsg and syslog-ng messages but nothing else. I am completely stumped here.

2 3

Polling additional machines
by Dan Egli 23 Jan '21

23 Jan '21

I have an interesting setup that was proposed to me, and while I _THINK_ syslog-ng can handle it I don't know for sure, let alone HOW. There are multiple service-specific machines in the network, behind the firewall. I'll just use the services they provide as their hostnames in this example. So firewall fw is a small linux machine running syslog-ng and fail2ban. Now fail2ban works off of the log files, but it also wants to apply changes to the firewall rules locally. So either www has its firewall rules while smtp has its rules and imap has a third set (ugly) or fail2ban can run on fw and modify the rules there. Trick is, in order to do that, all the logs have to be accessible in real time so that fail2ban can see them. My thought was to have syslog-ng on each sub machine somehow report to the syslog-ng on fw. Then fw's syslog-ng can write the files that fail2ban wants to read from. What would be the best way to go about this? I am still learning syslog-ng so if this is something simple, I'm sorry.

2 1

syslog-ng on CentOS 7 Aarch64
by Peter Czanik (pczanik) 20 Jan '21

20 Jan '21

Hi, I just learned that RHEL 7 on Aarch64 (64bit Arm) is end of life. Copr, where I build syslog-ng packages, removes RHEL/CentOS 7 Aarch64 packages in about half a year and it is not possible to build new packages there any more. CentOS 7 on the other hand keeps going on Aarch64 (using CentOS 8 kernel and CentOS 7 userland). There might be a workaround for the situation (using a different build service), but it's quite a lot extra effort. So, before investing any time and effort into keep building CentOS 7 Aarch64, I want to know, if there is a significant number of syslog-ng users, who keep using CentOS 7 on Aarch64. You can respond here or to my Twitter poll at: https://twitter.com/PCzanik/status/1351816930345291776 Thanks, Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Signed Syslog Messages RFC 5848
by Alexandre Santos 20 Jan '21

20 Jan '21

Hi, Are there any plans to add Signed Syslog Messages RFC 5848 to syslog-ng? Best Regards, Alex

3 4

Signed Syslog Messages RFC 5848
by Secure-Logging, Generic 19 Jan '21

19 Jan '21

Hi Alexandre, as a replacement for signed messages you might have a look at the secure logging module of syslog-ng. This module authentically encrypts each message, so you get an integrity guarantee which is stronger than a pure signature. You can then forward the message to a backend for verification and further processing. Regards, Stephan The information in this e-mail is confidential. The contents may not be disclosed or used by anyone other than the addressee. Access to this e-mail by anyone else is unauthorised. If you are not the intended recipient, please notify Airbus immediately and delete this e-mail. Airbus cannot accept any responsibility for the accuracy or completeness of this e-mail as it has been sent over public networks. If you have any concerns over the content of this message or its Accuracy or Integrity, please contact Airbus immediately. All outgoing e-mails from Airbus are checked using regularly updated virus scanning software but you should take whatever measures you deem to be appropriate to ensure that this message and any attachments are virus free.

1 0

Failover/Hi Availability
by Faine, Mark R. (MSFC-IS40)[NICS] 18 Jan '21

18 Jan '21

If I have two Syslog-ng servers receiving logs from the same source and writing to the exact same file system destination, is there a way to do this without running into issues with the two instances clobbering the files or file locking issues? Thanks, -Mark

2 4

Filtering Destination by Source
by Peter Griggs 12 Jan '21

12 Jan '21

Hello, We have a lot of network logs all being pointed to a central syslog however this is a mix of vendors (Cisco / Juniper / Checkpoint) etc. is there a way of splitting the destination file by vendor type / or source IP address? We ingest this data into Splunk so want to get the source typing right however I am unable to get the sources to point to various listeners and I would prefer. Thanks Peter.

3 3