March 2022 - syslog-ng - lists.balabit.hu

UDP drops with syslog-ng 3.24.1-1
by brian hoffman 05 Oct '22

05 Oct '22

We've been centrally logging with syslog-ng for about 5 years now. Over that time, the number of sources has grown significantly, and at some point we crossed a line where drops were happening (a quick survey of 3 million syslog packets yielded 420 unique currently sending hosts). After much research and experimentation, we've been able to get to the point where throughout the day there are 0 drops for the most part. This was achieved by installing the latest syslog-ng (not the RedHat packaged one) and creating a source for each CPU. Occasionally, though, we still have periods of drops so I'm trying to eliminate these last few. Here are some relevant configuration items:2 RedHat 7.8 VMs (load balanced via an F5) with 16GB memory and 4 CPUs each running syslog-ng-3.24.1-1.el7.x86_64.net.core.rmem_default = 212992net.core.rmem_max = 268435456 log_fifo_size(268435456); source s_network {network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp1"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp2"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp3"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp4"));network(ip("0.0.0.0") port(514) transport("tcp") max_connections(200) keep_alive(yes) so_rcvbuf(67108864));}; We are limited to UDP, unfortunately, because we do not have control over the devices/networks/etc. that are sending to us, but we have changed as many of the internal senders and destinations to TCP as we can. With a script I created to view the packets, including drops, as well as the individual RECVQs, the issue can be illustrated. Here's what things look like normally:Thu May 7 10:48:15 EDT 2020, 27003 IP pkts rcvd,26980 IP pkts sent,24951 UDP pkts rcvd, 28075 UDP pkts sent,0 UDP pkt rcv err RECVQ-1=2176RECVQ-2=0RECVQ-3=0RECVQ-4=0Thu May 7 10:48:16 EDT 2020, 28453 IP pkts rcvd,28426 IP pkts sent,26185 UDP pkts rcvd, 29180 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=0RECVQ-2=0RECVQ-3=4352RECVQ-4=0Thu May 7 10:48:17 EDT 2020, 28294 IP pkts rcvd,28276 IP pkts sent,26277 UDP pkts rcvd, 28709 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=2176RECVQ-2=0RECVQ-3=0RECVQ-4=0 The RECVQs are sparsely used, and there are no errors. Around 9pm every night, the packet counts go up significantly (probably due to backup related logs):Wed May 6 21:00:08 EDT 2020, 66382 IP pkts rcvd,66366 IP pkts sent,39405 UDP pkts rcvd, 67592 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=1595008RECVQ-2=106217088RECVQ-3=53694976RECVQ-4=31858816Wed May 6 21:00:09 EDT 2020, 69317 IP pkts rcvd,69338 IP pkts sent,44446 UDP pkts rcvd, 75958 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=13056RECVQ-2=126397312RECVQ-3=75568128RECVQ-4=41626880Wed May 6 21:00:10 EDT 2020, 71205 IP pkts rcvd,71227 IP pkts sent,43657 UDP pkts rcvd, 74603 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=920448RECVQ-2=146122752RECVQ-3=100951168RECVQ-4=52622208Wed May 6 21:00:12 EDT 2020, 69578 IP pkts rcvd,69454 IP pkts sent,124465 UDP pkts rcvd, 163367 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=13140864RECVQ-2=44494848RECVQ-3=125579136RECVQ-4=0 Still, though, it's handling it with no errors. But then at some point a threshold is reached and errors start piling up:Wed May 6 21:00:20 EDT 2020, 63177 IP pkts rcvd,63291 IP pkts sent,0 UDP pkts rcvd, 0 UDP pkts sent,38011 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=200357376RECVQ-3=292948352RECVQ-4=28890752Wed May 6 21:00:21 EDT 2020, 69501 IP pkts rcvd,69464 IP pkts sent,0 UDP pkts rcvd, 1 UDP pkts sent,42158 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=223551360RECVQ-3=314995584RECVQ-4=41735680Wed May 6 21:00:23 EDT 2020, 69962 IP pkts rcvd,69978 IP pkts sent,0 UDP pkts rcvd, 2 UDP pkts sent,43775 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=244732544RECVQ-3=338239616RECVQ-4=53858176Wed May 6 21:00:24 EDT 2020, 68266 IP pkts rcvd,68216 IP pkts sent,0 UDP pkts rcvd, 0 UDP pkts sent,43118 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=265258752RECVQ-3=360643712RECVQ-4=65362688 The common denominator I've found is that one of the RECVQs hits 536,871,424. This number seems to be almost exactly double (512 difference) the rmem.max/log_fifo_size (268,435,456). Even though there seems to be capacity in the other RECVQs, just one of those hitting that magic number seems to be enough to throw things out of whack. During this time, the CPU usage for syslog-ng also drops: top - 21:00:11 up 2 days, 8:40, 2 users, load average: 1.09, 1.41, 1.32 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 123.6 10.9 3080:46 syslog-ng top - 21:00:14 up 2 days, 8:40, 2 users, load average: 1.00, 1.39, 1.31 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 24.6 10.9 3080:46 syslog-ng top - 21:00:17 up 2 days, 8:40, 2 users, load average: 1.00, 1.39, 1.31 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 7.6 10.9 3080:47 syslog-ng As best I can tell, based on the reading I've done, the message counts we're getting should be doable, but it's still not clear exactly how to size some of these options as they relate to UDP (like log_fetch_limit, which we do not have set). Anything else I should try?

2 4

Local sources seem not to be working
by Alexandre Santos 28 Apr '22

28 Apr '22

Hi, I have syslog-ng 3.32.1 running in a Debian GNU/Linux 10 (buster) with the configuration in the attachement. After sometime running, syslog-ng seems be unable to read from system() and internal() sources. Log messages from syslog(ip(10.20.30.40) transport("udp") port(514) keep-alive(no)); are seen in the output folders. Also journald logs are working fine. After a reload of configuration in which what changes is this line: rewrite r_host { set("MACHINE-${HOST}", value("HOST")); }; logging is resumed. Here is the time gap for logs: <43>1 2022-03-11T11:55:23.802+00:00 xmm4-1-1 syslog-ng 8283 - [meta sequenceId="767"] Last message 'Destination reliable' repeated 8933 times, suppressed by syslog-ng on xmm4-1-1 <46>1 2022-03-14T07:19:01.817+00:00 xmm4-1-1 syslog-ng 8283 - [meta sequenceId="1"] Module loaded and initialized successfully; module='syslogformat' Do you know why this is happening? Thanks & Regards, Alex

2 13

Permissions are not set correctly on created directories
by Faine, Mark R. (MSFC-IS40)[NICS] 28 Mar '22

28 Mar '22

I have the following in my syslog-ng.conf for options: options { stats_freq(60); flush_lines(500); #flush_timeout(10000); time_reopen(10); time_reap(5); log_fifo_size(536870912); chain_hostnames(off); log-msg-size(65536); use_dns(no); keep_hostname(yes); use_fqdn(yes); dns_cache(off); dns-cache-size(2000); dns-cache-expire(87600); create_dirs(yes); perm(0640); owner(root); group(splunk); dir-perm(2750); dir-owner(root); dir-group(splunk); threaded(yes); }; However, instead of 2750 for the directory permissions, I'm seeing d-wSrwxrwT. 10 root splunk 4096 Mar 28 14:07 ise [root@molecule8-mfaine data]# stat -c '%a' ise 5276 I would expect: drwxr-s---. 2 root splunk 4096 Mar 28 14:11 ise Where are these permissions coming from? Thanks, Mark

2 2

Value is dropped or unset in resolved destination template
by Faine, Mark R. (MSFC-IS40)[NICS] 25 Mar '22

25 Mar '22

I have a destination template in my destination (d_default) that uses $location destination d_default { file("`BASEPATH`/$location/$(lowercase ${HOST})/$app/$(lowercase ${HOST})_$app.log" create_dirs(yes) flags("threaded", "no-multi-line")); }; That is set by a series of filters that look like this log { if { filter... rewrite { set("foo" value("location")); }; }; if { filter... rewrite { set("bar" value("location")); }; }; ... }; This appears to be working, I can see it in the logs: Setting value; name='location', value='unknown', msg='0x7f190c001e00' # Unknown is a default value that would be appropriate on this testing server Later additional log paths are applied like so: log { source(default); filter(f_asa); rewrite(r_asa); destination(d_default); flags(final ); }; ... log { source(default); filter(f_fw); rewrite(r_fw); destination(d_default); flags(final ); }; ... However, when I go to the destination on the filesystem the location is empty so the files are being written as though location was this: ("`BASEPATH`//$(lowercase ${HOST})/$app/$(lowercase ${HOST})_$app.log" Any ideas what could be causing this? Thanks, Mark

4 5

Help with embedded log paths
by Faine, Mark R. (MSFC-IS40)[NICS] 25 Mar '22

25 Mar '22

I need this configuration to do as the comments say but I can't figure out how to make it happen. Any pointers would be appreciated. Thanks, Mark log { source(default); # One of these first four should always match, if not $location=unknown log { filter(f_arc); rewrite(r_arc); }; log { filter(f_gsfc); rewrite(r_gsfc); }; log { filter(f_jsc); rewrite(r_jsc); }; log { filter(f_msfc); rewrite(r_msfc); }; ## The first one of these to match writes to disk and stops processing further log paths ## I don't know how this can be done since I can't add flags(final) inside of an embedded log path # Log Path for asa log { filter(f_asa); destination { file("/var/log/remote/data/${location:-unknown}/$(lowercase asa/${HOST}/${HOST}-asa.${UNIXTIME}.log)" create_dirs(yes) flags("threaded", "no-multi-line")); }; }; # Log Path for fmg log { filter(f_fmg); destination { file("/var/log/remote/data/${location:-unknown}/$(lowercase fmg/${HOST}/${HOST}-fmg.${UNIXTIME}.log)" create_dirs(yes) flags("threaded", "no-multi-line")); }; }; # Log Path for faz log { filter(f_faz); destination { file("/var/log/remote/data/${location:-unknown}/$(lowercase faz/${HOST}/${HOST}-faz.${UNIXTIME}.log)" create_dirs(yes) flags("threaded", "no-multi-line")); }; }; # Log Path for fw log { filter(f_fw); destination { file("/var/log/remote/data/${location:-unknown}/$(lowercase fw/${HOST}/${HOST}-fw.${UNIXTIME}.log)" create_dirs(yes) flags("threaded", "no-multi-line")); }; }; # Log Path for fw_block log { filter(f_fw_block); destination { file("/var/log/remote/data/${location:-unknown}/$(lowercase fw_block/${HOST}/${HOST}-fw_block.${UNIXTIME}.log)" create_dirs(yes) flags("threaded", "no-multi-line")); }; }; # Log Path for fortigate log { filter(f_fortigate); destination { file("/var/log/remote/data/${location:-unknown}/$(lowercase fortigate/${HOST}/${HOST}-fortigate.${UNIXTIME}.log)" create_dirs(yes) flags("threaded", "no-multi-line")); }; }; # Log Path for ips log { filter(f_ips); destination { file("/var/log/remote/data/${location:-unknown}/$(lowercase ips/${HOST}/${HOST}-ips.${UNIXTIME}.log)" create_dirs(yes) flags("threaded", "no-multi-line")); }; }; # Log Path for psa log { filter(f_psa); destination { file("/var/log/remote/data/${location:-unknown}/$(lowercase psa/${HOST}/${HOST}-psa.${UNIXTIME}.log)" create_dirs(yes) flags("threaded", "no-multi-line")); }; }; # Log Path for pubfw log { filter(f_pubfw); destination { file("/var/log/remote/data/${location:-unknown}/$(lowercase pubfw/${HOST}/${HOST}-pubfw.${UNIXTIME}.log)" create_dirs(yes) flags("threaded", "no-multi-line")); }; }; # Log Path for utm_other log { filter(f_utm_other); destination { file("/var/log/remote/data/${location:-unknown}/$(lowercase utm_other/${HOST}/${HOST}-utm_other.${UNIXTIME}.log)" create_dirs(yes) flags("threaded", "no-multi-line")); }; }; # Log Path for waf log { filter(f_waf); destination { file("/var/log/remote/data/${location:-unknown}/$(lowercase waf/${HOST}/${HOST}-waf.${UNIXTIME}.log)" create_dirs(yes) flags("threaded", "no-multi-line")); }; }; # Log Path for webfilter log { filter(f_webfilter); destination { file("/var/log/remote/data/${location:-unknown}/$(lowercase webfilter/${HOST}/${HOST}-webfilter.${UNIXTIME}.log)" create_dirs(yes) flags("threaded", "no-multi-line")); }; }; flags(final); };

3 4

Customizing syslog-ng snmp() destination option
by Maurya, Shivani 23 Mar '22

23 Mar '22

Hi All, I am using syslog-ng v3.30 and forwarding snmp traps from this server using snmp() destination option - [cid:image001.png@01D839F6.A0D8F220] I wanted to add a field "agent-addr" as part of the trap & not in the variable-binding. This field "agent-addr" will contain the IP address of the device which is sending the syslog message to the syslog server. How can I add a customized filed as part of the trap? Regards, Shivani Maurya

4 9

Gsoc preparation
by Kavinda Shehan Rajapakshe 22 Mar '22

22 Mar '22

Hello, I’m Kavinda Rajapakse, an undergraduate at the University of Moratuwa, Sri Lanka. Currently, I'm studying Computer Science and Engineering. This is my first time contributing to open source. I have some experience working in Rsyslog and Elastic stack. I would like to contribute to the "Support Elastic's beats protocol” project in GSOC 2022. I would be highly obliged if someone could provide me with an introduction as to how to get started. Thank you.

2 1

Insider 2022-03: syslog-ng 4; MQTT source; Zinc; Elastic Cloud; 3.36;
by Peter Czanik (pczanik) 17 Mar '22

17 Mar '22

Dear syslog-ng users, This is the 99th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS syslog-ng future: the path to syslog-ng 4 ----------------------------------------- Balázs Scheidler, founder of the syslog-ng project, describes the path to syslog-ng version 4.0: “It is very important to state that syslog-ng 4 is not the revolutionary, break-everything kind of release that we see too often in the software world. Rather it is an evolutionary change that will be produced similarly to previous releases” Read the rest of the blog at https://syslog-ng-future.blog/syslog-ng-future-the-path-to-syslog-ng-4/ MQTT source ----------- Version 3.35.1 of syslog-ng introduced an MQTT source. Just for some fun in the last syslog-ng blog post of the year, I created an endless loop using syslog-ng and the Mosquitto MQTT broker. Of course, it does not have much practical value other than possibly a bit of stress testing, but hopefully provides a fun introduction to MQTT-related technologies in syslog-ng. https://www.syslog-ng.com/community/b/blog/posts/creating-an-endless-loop-u… Another use for the syslog-ng elasticsearch-http destination: Zinc --------------------------- There is a new drop-in replacement for Elasticsearch, at least if you don’t mind the limitations and the alpha status. However, it definitely lives up to the promise that it provides an Elasticsearch-compatible API for data ingestion. I tested it with the elasticsearch-http() destination of syslog-ng, and it worked perfectly after I modified the URL in the configuration example I found. https://www.syslog-ng.com/community/b/blog/posts/another-use-for-the-syslog… Sending logs to Elastic Cloud using syslog-ng --------------------------------------------- The Elastic Cloud is a service by Elastic providing Elasticsearch and related services in an easy-to-use package. Last year someone reported an issue that it does not work properly with syslog-ng. I did not have time to investigate at that time. Now I started a free trial and soon my log messages from syslog-ng started to appear in Kibana in Elastic Cloud. >From this blog you can learn how to configure syslog-ng for the Elastic Cloud. I go with the most basic settings: exploring Elastic Cloud and syslog-ng Elasticsearch features are both out of scope, as both are very well documented on their respective websites. NEW RELEASES * syslog-ng 3.36: https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-3-36-news-better… WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

my syslog-ng-stable repos are updated to 3.36.1
by Peter Czanik (pczanik) 16 Mar '22

16 Mar '22

Hi, Version 3.36.1 of syslog-ng was released almost two weeks ago. Hundreds of people already installed it, nobody reported any problems, so my syslog-ng-stable RPM repos for openSUSE / SLES and Fedora / RHEL and compatibles are now updated to syslog-ng 3.36.1 See https://www.syslog-ng.com/community/b/blog/posts/introducing-the-syslog-ng-… Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

where did you switch from CentOS 8 Linux?
by Peter Czanik (pczanik) 16 Mar '22

16 Mar '22

Hi, By now I hope anyone who used CentOS 8 Linux proper previously switched to something else. If the question in the subject of my e-mail surprises you, please read the announcement about making CentOS 8 Linux End of Life by the end of 2021: https://blog.centos.org/2020/12/future-is-centos-stream/ By now all repositories are removed as well. Question: Where did you switch? RHEL or a compatible release by Rocky Linux, AlmaLinux? CentOS 8 Stream, the rolling pre-release of RHEL? Or something completely different, like openSUSE or FreeBSD? You can answer here or on my Twitter poll: https://twitter.com/PCzanik/status/1504034771944493056 Background: For years roughly 80% of syslog-ng users ran RHEL / CentOS & compatibles. This is expected to change now. Some stay on traditional RHEL (& compatible) releases, some move to Stream, some to something completely different. Your answer helps me to decide where to focus my effort as a syslog-ng packager. Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0