April 2021 - syslog-ng - lists.balabit.hu

UDP drops with syslog-ng 3.24.1-1
by brian hoffman 05 Oct '22

05 Oct '22

We've been centrally logging with syslog-ng for about 5 years now. Over that time, the number of sources has grown significantly, and at some point we crossed a line where drops were happening (a quick survey of 3 million syslog packets yielded 420 unique currently sending hosts). After much research and experimentation, we've been able to get to the point where throughout the day there are 0 drops for the most part. This was achieved by installing the latest syslog-ng (not the RedHat packaged one) and creating a source for each CPU. Occasionally, though, we still have periods of drops so I'm trying to eliminate these last few. Here are some relevant configuration items:2 RedHat 7.8 VMs (load balanced via an F5) with 16GB memory and 4 CPUs each running syslog-ng-3.24.1-1.el7.x86_64.net.core.rmem_default = 212992net.core.rmem_max = 268435456 log_fifo_size(268435456); source s_network {network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp1"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp2"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp3"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp4"));network(ip("0.0.0.0") port(514) transport("tcp") max_connections(200) keep_alive(yes) so_rcvbuf(67108864));}; We are limited to UDP, unfortunately, because we do not have control over the devices/networks/etc. that are sending to us, but we have changed as many of the internal senders and destinations to TCP as we can. With a script I created to view the packets, including drops, as well as the individual RECVQs, the issue can be illustrated. Here's what things look like normally:Thu May 7 10:48:15 EDT 2020, 27003 IP pkts rcvd,26980 IP pkts sent,24951 UDP pkts rcvd, 28075 UDP pkts sent,0 UDP pkt rcv err RECVQ-1=2176RECVQ-2=0RECVQ-3=0RECVQ-4=0Thu May 7 10:48:16 EDT 2020, 28453 IP pkts rcvd,28426 IP pkts sent,26185 UDP pkts rcvd, 29180 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=0RECVQ-2=0RECVQ-3=4352RECVQ-4=0Thu May 7 10:48:17 EDT 2020, 28294 IP pkts rcvd,28276 IP pkts sent,26277 UDP pkts rcvd, 28709 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=2176RECVQ-2=0RECVQ-3=0RECVQ-4=0 The RECVQs are sparsely used, and there are no errors. Around 9pm every night, the packet counts go up significantly (probably due to backup related logs):Wed May 6 21:00:08 EDT 2020, 66382 IP pkts rcvd,66366 IP pkts sent,39405 UDP pkts rcvd, 67592 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=1595008RECVQ-2=106217088RECVQ-3=53694976RECVQ-4=31858816Wed May 6 21:00:09 EDT 2020, 69317 IP pkts rcvd,69338 IP pkts sent,44446 UDP pkts rcvd, 75958 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=13056RECVQ-2=126397312RECVQ-3=75568128RECVQ-4=41626880Wed May 6 21:00:10 EDT 2020, 71205 IP pkts rcvd,71227 IP pkts sent,43657 UDP pkts rcvd, 74603 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=920448RECVQ-2=146122752RECVQ-3=100951168RECVQ-4=52622208Wed May 6 21:00:12 EDT 2020, 69578 IP pkts rcvd,69454 IP pkts sent,124465 UDP pkts rcvd, 163367 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=13140864RECVQ-2=44494848RECVQ-3=125579136RECVQ-4=0 Still, though, it's handling it with no errors. But then at some point a threshold is reached and errors start piling up:Wed May 6 21:00:20 EDT 2020, 63177 IP pkts rcvd,63291 IP pkts sent,0 UDP pkts rcvd, 0 UDP pkts sent,38011 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=200357376RECVQ-3=292948352RECVQ-4=28890752Wed May 6 21:00:21 EDT 2020, 69501 IP pkts rcvd,69464 IP pkts sent,0 UDP pkts rcvd, 1 UDP pkts sent,42158 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=223551360RECVQ-3=314995584RECVQ-4=41735680Wed May 6 21:00:23 EDT 2020, 69962 IP pkts rcvd,69978 IP pkts sent,0 UDP pkts rcvd, 2 UDP pkts sent,43775 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=244732544RECVQ-3=338239616RECVQ-4=53858176Wed May 6 21:00:24 EDT 2020, 68266 IP pkts rcvd,68216 IP pkts sent,0 UDP pkts rcvd, 0 UDP pkts sent,43118 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=265258752RECVQ-3=360643712RECVQ-4=65362688 The common denominator I've found is that one of the RECVQs hits 536,871,424. This number seems to be almost exactly double (512 difference) the rmem.max/log_fifo_size (268,435,456). Even though there seems to be capacity in the other RECVQs, just one of those hitting that magic number seems to be enough to throw things out of whack. During this time, the CPU usage for syslog-ng also drops: top - 21:00:11 up 2 days, 8:40, 2 users, load average: 1.09, 1.41, 1.32 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 123.6 10.9 3080:46 syslog-ng top - 21:00:14 up 2 days, 8:40, 2 users, load average: 1.00, 1.39, 1.31 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 24.6 10.9 3080:46 syslog-ng top - 21:00:17 up 2 days, 8:40, 2 users, load average: 1.00, 1.39, 1.31 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 7.6 10.9 3080:47 syslog-ng As best I can tell, based on the reading I've done, the message counts we're getting should be doable, but it's still not clear exactly how to size some of these options as they relate to UDP (like log_fetch_limit, which we do not have set). Anything else I should try?

2 4

Re: [syslog-ng] Request for config help
by Pradeep Gotaparthi 15 Jun '21

15 Jun '21

Hi, Could anybody help us out with the below query? Thanks. On Fri, Mar 26, 2021 at 12:06 AM Pradeep Gotaparthi <pradeep.gp(a)gmail.com> wrote: > Hi, > > We have been using syslog-ng for over a year now and it is working great, > thanks! > > One issue we face is that when we stop the process when there is some > buffer, the threshold is ignored and a bunch of logs get pushed out (as > much as 10 times the threshold value). This is causing problems on the > receiving end as it can't handle the load. We thought the log-fifo-size > parameter can help as syslog uses this parameter to store logs in memory > but it is not helping. > > It would be great if you can guide us. > > > Thanks. >

5 8

Syslog-ng + Consuming syslog alerts via FIFO: Alarms being lost, no logs related anywhere
by Richter Vitali Guedes 16 Apr '21

16 Apr '21

This is my first time posting here. I hope you can help me. We have 8 collector servers running syslog probes. They also run a collector application. Syslog-ng receives all alarms through UDP Ports, filter them, and write a fifo. It also forwards these alarms through UDP to another server. We are using this another server that receives these forwarded messages as a validation point. There are some filter rules for the pipe delivery, and no filters for UDP delivery. But we could see several not filtered alarms being received at the UDP receiver server, but not being received by the collector application, that reads from the fifo (/opt/solution/src1.pipe). Collector logs have no error lines. I tried rewrititng syslog-ng configuration file, in order to use flow-control, that, according to syslog-ng documentation, is necessary for high volume environments (as this is the case here, 500-1000 alarms per second). I applied it yesterday, together with net.core.rmem_max=1024000, and the scenario got even worst. We noticed missing messages on the UDP destination server and on the fifo. Dropped packets counter on ifconfig was increasing quickly. We rolled back to the previous syslog-ng configuration. Below, you will find the first version of the syslog-ng.conf file, without flow-control (several names , filters and addresses were changed in sake of privacy): @version:3.19 @include "scl.conf" # syslog-ng configuration file. # # This should behave pretty much like the original syslog on RedHat. But # it could be configured a lot smarter. # # See syslog-ng(8) and syslog-ng.conf(5) for more information. # # Note: it also sources additional configuration files (*.conf) # located in /etc/syslog-ng/conf.d/ options { flush_lines (0); time_reopen (10); log_fifo_size (1000); chain_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; source s_sys { system(); internal(); # udp(ip(0.0.0.0) port(514)); }; template t_customer { template("${DATE} ${SOURCEIP} ${HOST} ${MSGHDR} ${MESSAGE}\n"); }; # UDP Source destination d_cons { file("/dev/console"); }; destination d_mesg { file("/var/log/messages"); }; destination d_auth { file("/var/log/secure"); }; destination d_mail { file("/var/log/maillog" flush_lines(10)); }; destination d_spol { file("/var/log/spooler"); }; destination d_boot { file("/var/log/boot.log"); }; destination d_cron { file("/var/log/cron"); }; destination d_kern { file("/var/log/kern"); }; destination d_mlal { usertty("*"); }; filter f_kernel { facility(kern); }; filter f_default { level(info..emerg) and not (facility(mail) or facility(authpriv) or facility(cron)); }; filter f_auth { facility(authpriv); }; filter f_mail { facility(mail); }; filter f_emergency { level(emerg); }; filter f_news { facility(uucp) or (facility(news) and level(crit..emerg)); }; filter f_boot { facility(local7); }; filter f_cron { facility(cron); }; #log { source(s_sys); filter(f_kernel); destination(d_cons); }; #log { source(s_sys); filter(f_kernel); destination(d_kern); }; log { source(s_sys); filter(f_default); destination(d_mesg); }; log { source(s_sys); filter(f_auth); destination(d_auth); }; log { source(s_sys); filter(f_mail); destination(d_mail); }; log { source(s_sys); filter(f_emergency); destination(d_mlal); }; log { source(s_sys); filter(f_news); destination(d_spol); }; log { source(s_sys); filter(f_boot); destination(d_boot); }; log { source(s_sys); filter(f_cron); destination(d_cron); }; #Debug filter f_debu_1 { not level(debug); }; #Information filter f_info_1 { not match("PROGRAM-" value("MESSAGE")) }; filter f_info_2 { not match("Multicast Client" value("MESSAGE")); }; filter f_info_3 { not match("executeCommand" value("MESSAGE")); }; filter f_info_4 { not match("has the privilege to execute" value("MESSAGE")); }; filter f_info_5 { not match("Telnet " value("MESSAGE")); }; #Warning filter f_warn_1 { not match("LSP went Up" value("MESSAGE")); }; filter f_warn_2 { not match("LSP went Down" value("MESSAGE")); }; filter f_warn_3 { not match("Subscriber created" value("MESSAGE")); }; filter f_warn_4 { not match("Subscriber deleted" value("MESSAGE")); }; filter f_warn_5 { not match("BGP-WARN" value("MESSAGE")); }; filter f_warn_6 { not match("DHCPS-WARNING" value("MESSAGE")); }; filter f_warn_7 { not match("DHCP-WARNING" value("MESSAGE")); }; #Notice filter f_noti_1 { not match("iCDR\[[0-9]+\]:"); }; #Error filter f_erro_1 { not match("FAN started working" value("MESSAGE")); }; filter f_erro_2 { not match("FAN is not working" value("MESSAGE")); }; #Alert filter f_aler_1 { not match("LDP virtual tunnel went" value("MESSAGE")); }; #Critical filter f_crit_1 { not match("traps dropped" value("MESSAGE")); }; filter f_crit_2 { not match("upstream E1 signals" value("MESSAGE")); }; filter f_crit_3 { not match(",ABCDE," value("MESSAGE")); }; filter f_crit_4 { not match(",XYZW," value("MESSAGE")); }; filter f_crit_5 { not match(" for ABCDE-[0-9]" value("MESSAGE")); }; source s_dummy{ udp(ip(0.0.0.0) port(5140)); }; source src1{ udp(ip(0.0.0.0) port(514)); }; source s_10514{ udp(ip(0.0.0.0) port(10514)); }; source src2{ udp(ip(0.0.0.0) port(515)); }; destination d_src1 { pipe("/opt/solution/src1.pipe" template(t_customer)); }; destination d_src2 { pipe("/opt/solution/src2.pipe" template(t_customer)); }; destination d_syslogCustom{ pipe("/opt/solution/syslog_custom.pipe" template(t_customer)); }; destination d_udp_dst{ syslog("192.168.1.151" transport("udp") port(514) spoof_source(no)); }; filter f_src2 { host("r(core|distribution|management).*") or host("^abcd.*") or host("^d2") or host("src2") or host("xyz1") or host("^jkl") or host("asddf") or host("zxcvb"); }; log { source(src1); source(s_10514); filter(f_debu_1); filter(f_info_1); filter(f_info_2); filter(f_info_3); filter(f_info_4); filter(f_info_5); filter(f_warn_1); filter(f_warn_2); filter(f_warn_3); filter(f_warn_4); filter(f_warn_5); filter(f_warn_6); filter(f_warn_7); filter(f_noti_1); filter(f_erro_1); filter(f_erro_2); filter(f_aler_1); filter(f_crit_1); filter(f_crit_2); filter(f_crit_3); filter(f_crit_4); filter(f_crit_5); destination(d_src1); }; log { source(src2); filter(f_src2); destination(d_src2); }; log { source(src1); destination(d_udp_dst); }; That's all. I Also tried using flow-control. The main achanges are below: source src1{ # udp( network( #new driver syntax transport("udp") #new driver syntax ip(0.0.0.0) port(514) max-connections(3000) ); }; source s_10514{ # udp( network( #new driver syntax transport("udp") #new driver syntax ip(0.0.0.0) port(10514) max_connections(3000) ); }; source src2{ # udp( network( #new driver syntax transport("udp") #new driver syntax ip(0.0.0.0) port(515) max_connections(3000) ); }; destination d_src1 { pipe( "/opt/solution/src1.pipe" template(t_customer) log_fifo_size(600000) ); }; destination d_src2 { pipe( "/opt/solution/src2.pipe" template(t_customer) log_fifo_size(300000) ); }; destination d_udp_dst{ syslog( "192.168.1.151" transport("udp") port(514) spoof_source(no) log_fifo_size(300000) ); }; And the situation got even worst. So, I would like to ask you: How can I fix it? I need to be sure that all data received is forwarded to their destinations, respecting the filters, and without any information loss. I'm quite sure all data is being received, I've checked it using tcpdump, no udp packet loss during transmission. The most important sessions are: log { source(src1); source(s_10514); filter(f_debu_1); filter(f_info_1); filter(f_info_2); filter(f_info_3); filter(f_info_4); filter(f_info_5); filter(f_warn_1); filter(f_warn_2); filter(f_warn_3); filter(f_warn_4); filter(f_warn_5); filter(f_warn_6); filter(f_warn_7); filter(f_noti_1); filter(f_erro_1); filter(f_erro_2); filter(f_aler_1); filter(f_crit_1); filter(f_crit_2); filter(f_crit_3); filter(f_crit_4); filter(f_crit_5); destination(d_src1); }; log { source(src1); destination(d_udp_dst); }; Thanks a lot, and sorry for the huge message, Richter <https://www.linkedin.com/company/icarotech>

2 3

Insider 2021-04: Grafana; Windows agent; BSD;
by Peter Czanik (pczanik) 15 Apr '21

15 Apr '21

Dear syslog-ng users, This is the 90th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Grafana, Loki, syslog-ng: jump-starting a new logging stack ----------------------------------------------------------- Talking to syslog-ng users, I found that many of them plan to take a closer look at Grafana, due to the upheaval around the change of licensing terms for Elastic. Luckily, it is now possible to jump-start the complete, new logging stack – including Grafana, Loki, syslog-ng and tools to monitor this stack – with a single command. All you need to do is to point a couple of syslog clients at the included syslog-ng server and open Grafana in your browser. Of course, this setup is far from being production-ready, but it can speed up preparing a test environment for you. From this blog, you can learn how to install Grafana, Loki, syslog-ng stack, how to forward your log messages there, and how to check the results in Grafana. https://www.syslog-ng.com/community/b/blog/posts/grafana-loki-syslog-ng-jum… When to use the syslog-ng agent for Windows? -------------------------------------------- You can collect log messages from a Windows host in multiple ways using syslog-ng. For large scale installations the easiest is to use the Windows Event Collector (WEC) component of syslog-ng Premium Edition (PE). This way you don’t have to install any new client software on the Windows side, just point the WEC to the destionation to send their log messages. Please note that WEC only works for Windows EventLog. If you need to collect log messages from text files, you need to install the syslog-ng agent for Windows on your hosts. For example, web servers often log to files instead of Windows EventLog. Let’s review how to do a standalone installation of syslog-ng agent for Windows and then see the differences between using the legacy (RFC3164) and the new (RFC5424) syslog protocol. https://www.syslog-ng.com/community/b/blog/posts/when-to-use-the-syslog-ng-… Syslog-ng on BSDs ----------------- My FOSDEM presentation in the BSD devroom showcased what is new in sudo and syslog-ng and explained how to install or compile these software yourself on FreeBSD. Not only am I a long time FreeBSD user (started with version 1.0 in 1994) I also work on keeping the syslog-ng port in FreeBSD up to date. But soon after my presentation I was asked what I knew about other BSDs. And – while I knew that all BSDs have syslog-ng in their ports system – I realized I had no idea about the shape of those ports. For this article I installed OpenBSD, DragonFlyBSD and NetBSD to check syslog-ng on them. Admittedly, they are not in the best shape: they contain old versions, some do not even start or are unable to collect local log messages. https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-on-bsds WEBINARS * Scale log collection from Windows endpoints with WEC clustering in syslog-ng https://www.syslog-ng.com/event/scale-log-collection-from-windows-endpoints… * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Try again: What's wrong with my config?
by Dan Egli 13 Apr '21

13 Apr '21

Okay. I'm completely stumped on this, and just before the list went down I was hoping someone could help me with this. I'm trying to break everything out of the monolithic /var/log/messages and place each service in it's own log file. To that extent, I created the following config file: syslog-ng config: @version: 3.30 @include "scl.conf" options { threaded(yes); chain_hostnames(no); stats_freq(43200); mark_freq(3600); }; source src { system(); internal(); }; filter samba { program("samba") or program("nmbd") or program("smbd"); }; filter sshd { program("ssdhd"); }; filter syslog { not filter(sshd) and not filter(samba); }; destination console { file("/dev/tty12"); }; destination messages { file("/var/log/messages"); }; destination sshd_log { file("/var/log/sshd/sshd.log"); }; log { source(src); filter(sshd); destination(sshd_log); flags(final); }; log { source(src); filter(syslog); destination(console); }; log { source(src); filter(syslog); destination(messages); }; so, as I understand the logic. The three log { } lines do this: log { source(src); filter(sshd); destination(sshd_log); flags(final); }; Anything from sshd gets written to the /var/log/sshd/sshd.log. Nothing else goes here. log { source(src); filter(syslog); destination(console); }; Anything that is not from sshd, not from smbd, not from sabma and not from nmbd goes to the /dev/tty12 device log { source(src); filter(syslog); destination(messages); }; likewises for /var/log/messages. Is my understanding correct? If so, WHY do I see ssh log entries in /var/log/messages? And how do I stop it!? sshd messages should ONLY show up in /etc/sshd/sshd.log. jupiter ~ # grep sshd /var/log/messages | head -n 2 Apr 13 00:00:50 jupiter sshd[14721]: Received disconnect from <IP> port 18726:11: [preauth] Apr 13 00:00:50 jupiter sshd[14721]: Disconnected from <IP> port 18726 [preauth] Thanks!

3 3

Re: [syslog-ng] Try again: What's wrong with my config?
by Dan Egli 13 Apr '21

13 Apr '21

I wish I knew how is was happening. On 4/13/2021 12:26 AM, Dan Egli wrote: > > Okay. I'm completely stumped on this, and just before the list went > down I was hoping someone could help me with this. > > I'm trying to break everything out of the monolithic /var/log/messages > and place each service in it's own log file. To that extent, I created > the following config file: > >

1 0

Re: [syslog-ng] Try again: What's wrong with my config?
by Dan Egli 13 Apr '21

13 Apr '21

crap. Not that bug again. Ignore this, and watch for my completed message. Sorry. On 4/13/2021 12:12 AM, Dan Egli wrote: > > Okay. I'm completely stumped on this, and just before the list went > down I was hoping someone could help me with this. >

1 0

mail list works again
by Peter Kokai (pkokai) 13 Apr '21

13 Apr '21

Hello all, The previous week there were some issue with the mail list, and it was not possible to send mail to it (mailman likely reported dns unable to resolve). This issue have been resolved, and you should be able to send mail now. -- kokan

1 0

Socket too many open files (24)
by Faine, Mark R. (MSFC-IS40)[NICS] 09 Apr '21

09 Apr '21

syslog-ng 3 (3.19.1) Error Message: syslog-ng[50418]: Error accepting control socket connection; error='Too many open files (24)' I'm seeing this in the syslog-ng logs. However, I've done everything I can think of to increase the file handles and the number of open file handles does not seem to be anywhere close to the max. What else could be causing this? Thanks, -Mark

2 1

Syslog-ng not honoring negative flag
by Dan Egli 09 Apr '21

09 Apr '21

I'm having a bit of a problem and hope someone here can help. I'm trying to separate individual items into specific logs, i.e. ssh events in sshd.log, samba messages in samba.log, etc... I managed to come up with filters that pull out the events I started with, and they are going into the correct log files. But they are ALSO going into /var/log/messages even though I specifically have a filter on that one that says not to include samba or sshd events. I'll copy my config file here. Hopefully someone can tell me what I did wrong. Thanks! --------------------------------------------- @version: 3.30 @include "scl.conf" options { threaded(yes); chain_hostnames(no); stats_freq(43200); mark_freq(3600); }; source src { system(); internal(); }; filter samba { program("samba"); }; filter ssh_messages { facility("AUTH") and level("INFO"); }; filter syslog { not filter("ssh_messages") and not filter("samba"); }; destination console { file("/dev/tty12"); }; destination messages { file("/var/log/messages"); }; destination sshd_log { file("/var/log/sshd/sshd.log"); }; destination smb_logs { file("/var/log/samba/samba.log"); }; log { source(src); destination(smb_logs); filter(samba); flags(final); ); log { source(src); destination(sshd_log); filter(ssh_messages); flags(final); }; log { source(src); destination(console); filter(syslog); }; log { source(src); destination(messages); filter(syslog); };

6 15