June 2021 - syslog-ng - lists.balabit.hu

UDP drops with syslog-ng 3.24.1-1
by brian hoffman 05 Oct '22

05 Oct '22

We've been centrally logging with syslog-ng for about 5 years now. Over that time, the number of sources has grown significantly, and at some point we crossed a line where drops were happening (a quick survey of 3 million syslog packets yielded 420 unique currently sending hosts). After much research and experimentation, we've been able to get to the point where throughout the day there are 0 drops for the most part. This was achieved by installing the latest syslog-ng (not the RedHat packaged one) and creating a source for each CPU. Occasionally, though, we still have periods of drops so I'm trying to eliminate these last few. Here are some relevant configuration items:2 RedHat 7.8 VMs (load balanced via an F5) with 16GB memory and 4 CPUs each running syslog-ng-3.24.1-1.el7.x86_64.net.core.rmem_default = 212992net.core.rmem_max = 268435456 log_fifo_size(268435456); source s_network {network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp1"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp2"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp3"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp4"));network(ip("0.0.0.0") port(514) transport("tcp") max_connections(200) keep_alive(yes) so_rcvbuf(67108864));}; We are limited to UDP, unfortunately, because we do not have control over the devices/networks/etc. that are sending to us, but we have changed as many of the internal senders and destinations to TCP as we can. With a script I created to view the packets, including drops, as well as the individual RECVQs, the issue can be illustrated. Here's what things look like normally:Thu May 7 10:48:15 EDT 2020, 27003 IP pkts rcvd,26980 IP pkts sent,24951 UDP pkts rcvd, 28075 UDP pkts sent,0 UDP pkt rcv err RECVQ-1=2176RECVQ-2=0RECVQ-3=0RECVQ-4=0Thu May 7 10:48:16 EDT 2020, 28453 IP pkts rcvd,28426 IP pkts sent,26185 UDP pkts rcvd, 29180 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=0RECVQ-2=0RECVQ-3=4352RECVQ-4=0Thu May 7 10:48:17 EDT 2020, 28294 IP pkts rcvd,28276 IP pkts sent,26277 UDP pkts rcvd, 28709 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=2176RECVQ-2=0RECVQ-3=0RECVQ-4=0 The RECVQs are sparsely used, and there are no errors. Around 9pm every night, the packet counts go up significantly (probably due to backup related logs):Wed May 6 21:00:08 EDT 2020, 66382 IP pkts rcvd,66366 IP pkts sent,39405 UDP pkts rcvd, 67592 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=1595008RECVQ-2=106217088RECVQ-3=53694976RECVQ-4=31858816Wed May 6 21:00:09 EDT 2020, 69317 IP pkts rcvd,69338 IP pkts sent,44446 UDP pkts rcvd, 75958 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=13056RECVQ-2=126397312RECVQ-3=75568128RECVQ-4=41626880Wed May 6 21:00:10 EDT 2020, 71205 IP pkts rcvd,71227 IP pkts sent,43657 UDP pkts rcvd, 74603 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=920448RECVQ-2=146122752RECVQ-3=100951168RECVQ-4=52622208Wed May 6 21:00:12 EDT 2020, 69578 IP pkts rcvd,69454 IP pkts sent,124465 UDP pkts rcvd, 163367 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=13140864RECVQ-2=44494848RECVQ-3=125579136RECVQ-4=0 Still, though, it's handling it with no errors. But then at some point a threshold is reached and errors start piling up:Wed May 6 21:00:20 EDT 2020, 63177 IP pkts rcvd,63291 IP pkts sent,0 UDP pkts rcvd, 0 UDP pkts sent,38011 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=200357376RECVQ-3=292948352RECVQ-4=28890752Wed May 6 21:00:21 EDT 2020, 69501 IP pkts rcvd,69464 IP pkts sent,0 UDP pkts rcvd, 1 UDP pkts sent,42158 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=223551360RECVQ-3=314995584RECVQ-4=41735680Wed May 6 21:00:23 EDT 2020, 69962 IP pkts rcvd,69978 IP pkts sent,0 UDP pkts rcvd, 2 UDP pkts sent,43775 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=244732544RECVQ-3=338239616RECVQ-4=53858176Wed May 6 21:00:24 EDT 2020, 68266 IP pkts rcvd,68216 IP pkts sent,0 UDP pkts rcvd, 0 UDP pkts sent,43118 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=265258752RECVQ-3=360643712RECVQ-4=65362688 The common denominator I've found is that one of the RECVQs hits 536,871,424. This number seems to be almost exactly double (512 difference) the rmem.max/log_fifo_size (268,435,456). Even though there seems to be capacity in the other RECVQs, just one of those hitting that magic number seems to be enough to throw things out of whack. During this time, the CPU usage for syslog-ng also drops: top - 21:00:11 up 2 days, 8:40, 2 users, load average: 1.09, 1.41, 1.32 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 123.6 10.9 3080:46 syslog-ng top - 21:00:14 up 2 days, 8:40, 2 users, load average: 1.00, 1.39, 1.31 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 24.6 10.9 3080:46 syslog-ng top - 21:00:17 up 2 days, 8:40, 2 users, load average: 1.00, 1.39, 1.31 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 7.6 10.9 3080:47 syslog-ng As best I can tell, based on the reading I've done, the message counts we're getting should be doable, but it's still not clear exactly how to size some of these options as they relate to UDP (like log_fetch_limit, which we do not have set). Anything else I should try?

2 4

Dynamic setting value out of message?
by Matthias Gruber 01 Jul '21

01 Jul '21

Hi! I hope it is simple and my thoughts and seeks about it were to complicated :-), simply I didnt know how to do that, perhaps someone has a clue for me I am getting e.g. messages like "MESSAGE": "UIMUC4Maintenance.py: \"== deactivate_uc4_monitoring = ENDE ==\"", (thats out of an JSON-formatted syslog-ng output), What I would like to do is, to extract the 'UIMUC4Maintenance.py:' and put it into a SDATA-Custom-Variable or PROG but based on a regex so some sort of rewrite-rule like (no not a correct syntax, only to describe it) rewrite r_fill_program { set(match("^\w*\.py:" value("MESSAGE")) value("PROG")); }; As far as I understand it, set requires a "string" as first parameter, I could use a lots of rewrites with a condition, but I am in "lack of a static string", this should be some sort of variable :-) or I could do that static with a filter for every "^\w*\.py:"-Text, but I hope I could do that dynamic, every time a match of my regex syslog-ng inserts that part into a variable and so on... Is that possible? cheers Matthias ------------------------------------------------------------------------------------ METZLER Informationstechnologie Matthias Gruber IT-Infrastruktur & -Betrieb B. Metzler seel. Sohn & Co. Aktiengesellschaft Untermainanlage 1 60329 Frankfurt am Main Telefon (0 69) 21 04 - 43 30 Telefax (0 69) 21 04 - 40 40 MGruber(a)metzler.com www.metzler.com Vorstand: Harald Illy, Emmerich Müller, Gerhard Wiesheu Vorsitzender des Aufsichtsrats: Dr. Christoph Schücking Sitz der Gesellschaft: Frankfurt am Main, Handelsregister-Nr. HRB 123 365 Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfänger sein, so bitten wir Sie höflich, dies unverzüglich dem Absender mitzuteilen und die Nachricht zu löschen. Es ist unzulässig, die Nachricht unbefugt weiterzuleiten oder zu kopieren. Da wir nicht die Echtheit oder Vollständigkeit der in dieser Nachricht enthaltenen Informationen garantieren oder zusichern können, sind die vorstehenden Ausführungen rechtlich nicht bindend. Eine Haftung hierfür wird ausgeschlossen. This message is confidential. If you are not the intended recipient, we kindly ask you to inform the sender and delete the information. Any unauthorised dissemination or copying hereof is prohibited. As we cannot guarantee or assure the genuineness or completeness of the information contained in this message, the statements set forth above are not legally binding. Accordingly we cannot accept any liability for their contents.

3 4

Is there any compatibility between ver3.2 and ver3.25 about syslog-ng.conf?
by leis 25 Jun '21

25 Jun '21

Hello, I am looking to migrate my environment from CentOS5 syslog-ng ver3.2 to ubnutu20.02 syslog-ng ver3.25. But when I use syslog-ng.conf of ver3.2 to syslog-ng.conf of ver3.25, syslog-ng cannot start. Is there any compatibility between ver3.2's syslog-ng.conf and ver3.25's syslog-ng.conf? Thanks

2 1

Re: [syslog-ng] Tips to diagnose missing syslog messages?
by brian hoffman 15 Jun '21

15 Jun '21

We've spent a lot of time troubleshooting UDP drops, and the so_reuseport option did help quite a bit. There's a max number that the RECVQ can reach (which can be seen with: /usr/bin/netstat -tupln|egrep "(^udp.*syslog-ng)" ) that correlates to the so_rcvbuf. Having more queues won't keep drops from happening, since the hashing in the OSE is related to source IP, so a talkative host can still fill up one of those queues and cause drops. The PE uses eBPF which is apparently better at preventing this, but we found that by using the updated version at "copr:copr.fedorainfracloud.org:czanik:syslog-ng324/x86_64" instead of the stock RH one and bumping up the so_rcvbuf, it did perform much better (and I believe you may need that version for the so_reuseport).-Brian

1 0

Re: [syslog-ng] Request for config help
by Pradeep Gotaparthi 15 Jun '21

15 Jun '21

Hi, Could anybody help us out with the below query? Thanks. On Fri, Mar 26, 2021 at 12:06 AM Pradeep Gotaparthi <pradeep.gp(a)gmail.com> wrote: > Hi, > > We have been using syslog-ng for over a year now and it is working great, > thanks! > > One issue we face is that when we stop the process when there is some > buffer, the threshold is ignored and a bunch of logs get pushed out (as > much as 10 times the threshold value). This is causing problems on the > receiving end as it can't handle the load. We thought the log-fifo-size > parameter can help as syslog uses this parameter to store logs in memory > but it is not helping. > > It would be great if you can guide us. > > > Thanks. >

5 8

Insider 2021-06: Alerting; EoL technologies; Google Summer of Code;
by Peter Czanik (pczanik) 10 Jun '21

10 Jun '21

Dear syslog-ng users, This is the 92nd issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS First steps of sending alerts to Discord and others from syslog-ng: http() and Apprise ------------------------------- A returning question I get is: “I see, that you can send alerts from syslog-ng to Slack and Telegram, but do you happen to support XYZ?” Replace XYZ with Discord and countless others. Up until recently, my regular answer has been: “Take a look at the Slack destination of syslog-ng, and based on that, you can add support for your favorite service”. Then I learned about Apprise, a notification library for Python, supporting dozens of different services. This blog is the first part of a series. It covers how to send log messages to Discord using the http() destination of syslog-ng and an initial try at using Apprise for alerting. https://www.syslog-ng.com/community/b/blog/posts/first-steps-of-sending-ale… Changes in technologies supported by syslog-ng: Python 2, CentOS 6 & Co. ---------------------------------------------------- Technology is continuously evolving. There are regular changes in platforms running syslog-ng: old technologies disappear, and new technologies are introduced. While we try to provide stability and continuity to our users, we also need to adapt. Python 2 reached its end of life a year ago, CentOS 6 in November 2020. Using Java-based drivers has been problematic for many, so they were mostly replaced with native implementations. From this blog you can learn about recent changes affecting syslog-ng development and packaging. https://www.syslog-ng.com/community/b/blog/posts/changes-in-technologies-su… Google Summer of Code 2021 -------------------------- This year, the syslog-ng team participates in Google Summer of Code (GSoC) again as a mentoring organization. Two students paid by GSoC work on syslog-ng under the mentoring of syslog-ng developers. One of the students works on MacOS support, including the new ARM-based systems, while the other one is working developing on a new regular expression parser: https://summerofcode.withgoogle.com/organizations/5548293561516032/ WEBINARS * syslog-ng, Proxy Protocol Support: https://www.syslog-ng.com/event/syslogng-proxy-protocol-support-8148877/ * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Debian/Ubuntu OBS packages: discontinued
by Laszlo Budai 01 Jun '21

01 Jun '21

Hi, unofficial syslog-ng Debian/Ubuntu OBS[1] packages are discontinued. The previously built syslog-ng packages are still available (up to version 3.31.2). Some of you might know that the origin of the repository's name was a kind of joke : at that time I was told that 'we need syslog-ng binary packages', others said: 'we cannot ship binary packages for OSE'. I resolved this conflict by creating an OBS repository under my personal name in my free time... problem solved... at least until now. This was my personal project, under my personal name, working on it in my free time - and unfortunately I don't have the time for maintaining it anymore. I'd suggest to migrate to DBLD[2] packages from my OBS repository (if not possible for some reasons, just contact to me and I'm trying to help you). [1] https://build.opensuse.org/project/show/home:laszlo_budai:syslog-ng [2] https://www.syslog-ng.com/community/b/blog/posts/dbld-a-syslog-ng-developer… kind regards, Laszlo Budai [Twitter]<https://twitter.com/Budai_L> [LinkedIn] <https://www.linkedin.com/in/laszlo-budai-55243a3/> [Github] <https://github.com/lbudai>

1 0