November 2021 - syslog-ng - lists.balabit.hu

UDP drops with syslog-ng 3.24.1-1
by brian hoffman 05 Oct '22

05 Oct '22

We've been centrally logging with syslog-ng for about 5 years now. Over that time, the number of sources has grown significantly, and at some point we crossed a line where drops were happening (a quick survey of 3 million syslog packets yielded 420 unique currently sending hosts). After much research and experimentation, we've been able to get to the point where throughout the day there are 0 drops for the most part. This was achieved by installing the latest syslog-ng (not the RedHat packaged one) and creating a source for each CPU. Occasionally, though, we still have periods of drops so I'm trying to eliminate these last few. Here are some relevant configuration items:2 RedHat 7.8 VMs (load balanced via an F5) with 16GB memory and 4 CPUs each running syslog-ng-3.24.1-1.el7.x86_64.net.core.rmem_default = 212992net.core.rmem_max = 268435456 log_fifo_size(268435456); source s_network {network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp1"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp2"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp3"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp4"));network(ip("0.0.0.0") port(514) transport("tcp") max_connections(200) keep_alive(yes) so_rcvbuf(67108864));}; We are limited to UDP, unfortunately, because we do not have control over the devices/networks/etc. that are sending to us, but we have changed as many of the internal senders and destinations to TCP as we can. With a script I created to view the packets, including drops, as well as the individual RECVQs, the issue can be illustrated. Here's what things look like normally:Thu May 7 10:48:15 EDT 2020, 27003 IP pkts rcvd,26980 IP pkts sent,24951 UDP pkts rcvd, 28075 UDP pkts sent,0 UDP pkt rcv err RECVQ-1=2176RECVQ-2=0RECVQ-3=0RECVQ-4=0Thu May 7 10:48:16 EDT 2020, 28453 IP pkts rcvd,28426 IP pkts sent,26185 UDP pkts rcvd, 29180 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=0RECVQ-2=0RECVQ-3=4352RECVQ-4=0Thu May 7 10:48:17 EDT 2020, 28294 IP pkts rcvd,28276 IP pkts sent,26277 UDP pkts rcvd, 28709 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=2176RECVQ-2=0RECVQ-3=0RECVQ-4=0 The RECVQs are sparsely used, and there are no errors. Around 9pm every night, the packet counts go up significantly (probably due to backup related logs):Wed May 6 21:00:08 EDT 2020, 66382 IP pkts rcvd,66366 IP pkts sent,39405 UDP pkts rcvd, 67592 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=1595008RECVQ-2=106217088RECVQ-3=53694976RECVQ-4=31858816Wed May 6 21:00:09 EDT 2020, 69317 IP pkts rcvd,69338 IP pkts sent,44446 UDP pkts rcvd, 75958 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=13056RECVQ-2=126397312RECVQ-3=75568128RECVQ-4=41626880Wed May 6 21:00:10 EDT 2020, 71205 IP pkts rcvd,71227 IP pkts sent,43657 UDP pkts rcvd, 74603 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=920448RECVQ-2=146122752RECVQ-3=100951168RECVQ-4=52622208Wed May 6 21:00:12 EDT 2020, 69578 IP pkts rcvd,69454 IP pkts sent,124465 UDP pkts rcvd, 163367 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=13140864RECVQ-2=44494848RECVQ-3=125579136RECVQ-4=0 Still, though, it's handling it with no errors. But then at some point a threshold is reached and errors start piling up:Wed May 6 21:00:20 EDT 2020, 63177 IP pkts rcvd,63291 IP pkts sent,0 UDP pkts rcvd, 0 UDP pkts sent,38011 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=200357376RECVQ-3=292948352RECVQ-4=28890752Wed May 6 21:00:21 EDT 2020, 69501 IP pkts rcvd,69464 IP pkts sent,0 UDP pkts rcvd, 1 UDP pkts sent,42158 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=223551360RECVQ-3=314995584RECVQ-4=41735680Wed May 6 21:00:23 EDT 2020, 69962 IP pkts rcvd,69978 IP pkts sent,0 UDP pkts rcvd, 2 UDP pkts sent,43775 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=244732544RECVQ-3=338239616RECVQ-4=53858176Wed May 6 21:00:24 EDT 2020, 68266 IP pkts rcvd,68216 IP pkts sent,0 UDP pkts rcvd, 0 UDP pkts sent,43118 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=265258752RECVQ-3=360643712RECVQ-4=65362688 The common denominator I've found is that one of the RECVQs hits 536,871,424. This number seems to be almost exactly double (512 difference) the rmem.max/log_fifo_size (268,435,456). Even though there seems to be capacity in the other RECVQs, just one of those hitting that magic number seems to be enough to throw things out of whack. During this time, the CPU usage for syslog-ng also drops: top - 21:00:11 up 2 days, 8:40, 2 users, load average: 1.09, 1.41, 1.32 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 123.6 10.9 3080:46 syslog-ng top - 21:00:14 up 2 days, 8:40, 2 users, load average: 1.00, 1.39, 1.31 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 24.6 10.9 3080:46 syslog-ng top - 21:00:17 up 2 days, 8:40, 2 users, load average: 1.00, 1.39, 1.31 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 7.6 10.9 3080:47 syslog-ng As best I can tell, based on the reading I've done, the message counts we're getting should be doable, but it's still not clear exactly how to size some of these options as they relate to UDP (like log_fetch_limit, which we do not have set). Anything else I should try?

2 4

Selective application of keep-hostname()?
by Steve Bernacki 29 Nov '21

29 Nov '21

I have a use case where I need to selectively apply keep-hostname(yes) to messages from certain source IPs, while defaulting to keep-hostname(no) for the rest of my sources. I know I can apply this option on a per-source basis, but in this case I want to be able to selectively apply this option based on the source IP of the message. I'm trying to avoid having to set up an alternate port if I can. Thanks Steve

2 1

Insider 2021-11: 3.35; MacOS; mqtt() destination updates;
by Peter Czanik (pczanik) 24 Nov '21

24 Nov '21

Dear syslog-ng users, This is the 96th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS syslog-ng 3.35.1 available -------------------------- Version 3.35.1 of syslog-ng is released. New features include: - APT repo for x86 Debian and Ubuntu - Throttle filter - MQTT source - PROXY header before TLS payload You can read more details and download the syslog-ng source code at https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-3.35.1 Sending logs from syslog-ng store box to Splunk ----------------------------------------------- One of the most popular applications to feed Splunk with syslog messages is syslog-ng. However not everyone is happy to work on the command line anymore. This is where syslog-ng store box (SSB), an appliance built around syslog-ng, can help. The SSB GUI provides you not only with an easyto-use interface to configure most syslog-ng features, but also a search interface and complete log life cycle management. It can forward log messages to several destinations, recently also to Splunk’s HTTP Event Collector (HEC). From this blog you can learn about how SSB fits into your logging infrastructure and how to configure SSB for Splunk. https://www.syslog-ng.com/community/b/blog/posts/sending-logs-from-syslog-n… MacOS support ------------- For the past couple of months, Yash Mathne has been working on testing syslog-ng on MacOS as a GSoC (Google Summer of Code) student. He worked both on x86 and on the freshly released ARM hardware. And we have some good news here to share: while there is still room for improvement, most of syslog-ng works perfectly well on MacOS. You can read the two parts of his report at: Overview: https://www.syslog-ng.com/community/b/blog/posts/gsoc-report-syslog-ng-maco… Technical details: https://www.syslog-ng.com/community/b/blog/posts/status-of-syslog-ng-on-mac… Syslog-ng 3.34: MQTT destination with TLS and WebSocket support --------------------------------------------------------------- Version 3.33 of syslog-ng arrived with basic MQTT support. Version 3.34 has added many important features to it: user authentication, TLS support and WebSocket support. These features give you both security and flexibility while sending log messages to an MQTT broker. This blog helps you to make your first steps securing your MQTT connection. https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-3-34-mqtt-destin… WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

syslog-ng-stable rpm repos
by Peter Czanik (pczanik) 22 Nov '21

22 Nov '21

Hi, I just updated my syslog-ng-stable rpm repos to syslog-ng version 3.35. It was released a week ago, quite a few people already use it and nobody reported any problems. So, it should be a safe update. For new features and changes of 3.35, check: https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-3.35.1 For an overview of my unofficial RPM repositories for openSUSE / SLES, Fedora / RHEL (and compatibles), check: https://www.syslog-ng.com/community/b/blog/posts/overview-of-syslog-ng-rpm-… Bye, CzP Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 1

CentOS EoL is near
by Peter Czanik (pczanik) 16 Nov '21

16 Nov '21

Hi, For many years CentOS Linux was the most popular platform to run syslog-ng. Unfortunately, CentOS 8 will be end of life by the end of the year: https://www.centos.org/centos-linux-eol/ I wonder if you already made the switch to something else, RHEL, one of the old or new RHEL clones, CentOS Stream or something completely different. Or are you still in the planning phase? Or completely unaware of the CentOS Linux EoL? I put together a poll on Twitter: https://twitter.com/PCzanik/status/1460610601848299525 and I would appreciate voting and retweets. And of course, I'm very interested in your experiences: where did you switch, why, how did it go. Either here on the mailing list, or directly to me if you do not want to share it publicly. Your answers help me to decide where should I put more effort while packaging and testing syslog-ng. And of course, also what OS should I use more often in my blogs. Previously the focus was CentOS with everything else appearing a lot less, now BSD variants and openSUSE also appear frequently. Thanks, Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

syslog-ng-3.35.1
by Attila Szakacs (aszakacs) 15 Nov '21

15 Nov '21

3.35.1 syslog-ng OSE APT repository >From now on, Ubuntu and Debian packages will be published with every syslog-ng release in the form of an APT repository. We, syslog-ng developers, provide these packages and the APT repository "as is" without warranty of any kind, on a best-effort level. Currently, syslog-ng packages are released for the following distribution versions (x86-64): * Debian: bullseye, buster, stretch, sid, testing * Ubuntu: Impish, Focal, Bionic, Xenial For instructions on how to install syslog-ng on Debian/Ubuntu distributions, see the README<https://github.com/syslog-ng/syslog-ng/blob/master/README.md#debianubuntu>. Highlights * throttle(): added a new filter that allows rate limiting messages based on arbitrary keys in each message. Note: messages over the rate limit are dropped (just like in any other filter). filter f_throttle { throttle( template("$HOST") rate(5000) ); }; (#3781<https://github.com/syslog-ng/syslog-ng/pull/3781>) * mqtt(): added a new source that can be used to receive messages using the MQTT protocol. Supported transports: tcp, ws, ssl, wss Example config: source { mqtt{ topic("sub1"), address("tcp://localhost:4445") }; }; (#3809<https://github.com/syslog-ng/syslog-ng/pull/3809>) Features * afsocket: Socket options, such as ip-ttl() or tcp-keepalive-time(), are traditionally named by their identifier defined in socket(7) and unix(7) man pages. This was not the case with the pass-unix-credentials() option, which - unlike other similar options - was also possible to set globally. A new option called so-passcred() is now introduced, which works similarly how other socket related options do, which also made possible a nice code cleanup in the related code sections. Of course the old name remains supported in compatibility modes. The PR also implements a new source flag ignore-aux-data, which causes syslog-ng not to propagate transport-level auxiliary information to log messages. Auxiliary information includes for example the pid/uid of the sending process in the case of UNIX based transports, OR the X.509 certificate information in case of SSL/TLS encrypted data streams. By setting flags(ignore-aux-data) one can improve performance at the cost of making this information unavailable in the log messages received through affected sources. (#3670<https://github.com/syslog-ng/syslog-ng/pull/3670>) * network: add support for PROXY header before TLS payload This new transport method called proxied-tls-passthrough is capable of detecting the PROXY header<http://www.haproxy.org/download/1.8/doc/proxy-protocol.txt> before the TLS payload. Loggen has been updated with the--proxied-tls-passthrough option for testing purposes. source s_proxied_tls_passthrough{ network( port(1234) transport("proxied-tls-passthrough"), tls( key-file("/path/to/server_key.pem"), cert-file("/path/to/server_cert.pem"), ca-dir("/path/to/ca/") ) ); }; (#3770<https://github.com/syslog-ng/syslog-ng/pull/3770>) * mqtt() destination: added client-id option. It specifies the unique client ID sent to the broker. (#3809<https://github.com/syslog-ng/syslog-ng/pull/3809>) Bugfixes * unset(), groupunset(): fix unwanted removal of values on different log paths Due to a copy-on-write bug, unset() and groupunset() not only removed values from the appropriate log paths, but from all the others where the same message went through. This has been fixed. (#3803<https://github.com/syslog-ng/syslog-ng/pull/3803>) * regexp-parser(): fix storing unnamed capture groups under prefix() (#3810<https://github.com/syslog-ng/syslog-ng/pull/3810>) * loggen: cannot detect plugins on platforms with non .so shared libs (osx) (#3832<https://github.com/syslog-ng/syslog-ng/pull/3832>) Packaging * debian/control: Added libcriterion-dev as a build dependency, where it is available from APT. (debian-bullseye, debian-testing, debian-sid) (#3794<https://github.com/syslog-ng/syslog-ng/pull/3794>) * centos-7: kafka and mqtt modules are now packaged. The following packages are used as dependencies: * librdkafka-devel from EPEL 7 * paho-c-devel from copr:copr.fedorainfracloud.org:czanik:syslog-ng-githead (#3797<https://github.com/syslog-ng/syslog-ng/pull/3797>) * debian: Added bullseye support. (#3794<https://github.com/syslog-ng/syslog-ng/pull/3794>) * bison: support build with bison 3.8 (#3784<https://github.com/syslog-ng/syslog-ng/pull/3784>) Notes to developers * dbld: As new distributions use python3 by default it makes sense to explicitly state older platforms which use python2 instead of the other way around, so it is not necessary to add that new platform to the python3 case. (#3780<https://github.com/syslog-ng/syslog-ng/pull/3780>) * dbld: move dbld image cache from DockerHub to GitHub In 2021, GitHub introduced the GitHub Packages service. Among other repositories - it provides a standard Docker registry. DBLD uses this registry, to avoid unnecessary rebuilding of the images. (#3782<https://github.com/syslog-ng/syslog-ng/pull/3782>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler, Balázs Barkó, Benedek Cserhati, Colin Douch, Gabor Nagy, Laszlo Szemere, László Várady, Norbert Takacs, Parrag Szilárd, Peter Czanik (CzP), Peter Kokai, Robert Paschedag, Ryan Faircloth, Szilárd Parrag, Thomas Klausner, Zoltan Pallagi

1 0

Sequence – making PatternDB creation for syslog-ng easier
by Peter Czanik (pczanik) 09 Nov '21

09 Nov '21

Hi, I try not to spam the mailing list about each of my blogs separately. However, with all its problems I love sysog-ng's PatternDB technology for message parsing. Thanks to the hard work of Fabien Wernli and his colleagues, creating patterns for PatternDB just became a whole lot more easier! "We are well into the 21st century, but most of the log messages still arrive in an unstructured format. For well over a decade, syslog-ng had a solution to turn unstructured messages into name-value pairs, called PatternDB. However, creating a pattern database for PatternDB from scratch is a source of major pain. Or rather, it was: sequence-rtg – a fork of the sequence log analyzer – provides a new hope! It can easily create ready-to-use patterns for your most frequent log messages. Sequence-rtg is still in beta phase, and therefore is a bit rough around the edges. However, once you deal with the initial struggles of creating the database, it works just fine. Especially if you have lots of log messages. My experience was that the more log messages and larger batch sizes I had, the better quality patterns were generated." Read my blog at https://www.syslog-ng.com/community/b/blog/posts/sequence-making-patterndb-… Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0