April 2022 - syslog-ng - lists.balabit.hu

UDP drops with syslog-ng 3.24.1-1
by brian hoffman 05 Oct '22

05 Oct '22

We've been centrally logging with syslog-ng for about 5 years now. Over that time, the number of sources has grown significantly, and at some point we crossed a line where drops were happening (a quick survey of 3 million syslog packets yielded 420 unique currently sending hosts). After much research and experimentation, we've been able to get to the point where throughout the day there are 0 drops for the most part. This was achieved by installing the latest syslog-ng (not the RedHat packaged one) and creating a source for each CPU. Occasionally, though, we still have periods of drops so I'm trying to eliminate these last few. Here are some relevant configuration items:2 RedHat 7.8 VMs (load balanced via an F5) with 16GB memory and 4 CPUs each running syslog-ng-3.24.1-1.el7.x86_64.net.core.rmem_default = 212992net.core.rmem_max = 268435456 log_fifo_size(268435456); source s_network {network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp1"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp2"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp3"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp4"));network(ip("0.0.0.0") port(514) transport("tcp") max_connections(200) keep_alive(yes) so_rcvbuf(67108864));}; We are limited to UDP, unfortunately, because we do not have control over the devices/networks/etc. that are sending to us, but we have changed as many of the internal senders and destinations to TCP as we can. With a script I created to view the packets, including drops, as well as the individual RECVQs, the issue can be illustrated. Here's what things look like normally:Thu May 7 10:48:15 EDT 2020, 27003 IP pkts rcvd,26980 IP pkts sent,24951 UDP pkts rcvd, 28075 UDP pkts sent,0 UDP pkt rcv err RECVQ-1=2176RECVQ-2=0RECVQ-3=0RECVQ-4=0Thu May 7 10:48:16 EDT 2020, 28453 IP pkts rcvd,28426 IP pkts sent,26185 UDP pkts rcvd, 29180 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=0RECVQ-2=0RECVQ-3=4352RECVQ-4=0Thu May 7 10:48:17 EDT 2020, 28294 IP pkts rcvd,28276 IP pkts sent,26277 UDP pkts rcvd, 28709 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=2176RECVQ-2=0RECVQ-3=0RECVQ-4=0 The RECVQs are sparsely used, and there are no errors. Around 9pm every night, the packet counts go up significantly (probably due to backup related logs):Wed May 6 21:00:08 EDT 2020, 66382 IP pkts rcvd,66366 IP pkts sent,39405 UDP pkts rcvd, 67592 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=1595008RECVQ-2=106217088RECVQ-3=53694976RECVQ-4=31858816Wed May 6 21:00:09 EDT 2020, 69317 IP pkts rcvd,69338 IP pkts sent,44446 UDP pkts rcvd, 75958 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=13056RECVQ-2=126397312RECVQ-3=75568128RECVQ-4=41626880Wed May 6 21:00:10 EDT 2020, 71205 IP pkts rcvd,71227 IP pkts sent,43657 UDP pkts rcvd, 74603 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=920448RECVQ-2=146122752RECVQ-3=100951168RECVQ-4=52622208Wed May 6 21:00:12 EDT 2020, 69578 IP pkts rcvd,69454 IP pkts sent,124465 UDP pkts rcvd, 163367 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=13140864RECVQ-2=44494848RECVQ-3=125579136RECVQ-4=0 Still, though, it's handling it with no errors. But then at some point a threshold is reached and errors start piling up:Wed May 6 21:00:20 EDT 2020, 63177 IP pkts rcvd,63291 IP pkts sent,0 UDP pkts rcvd, 0 UDP pkts sent,38011 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=200357376RECVQ-3=292948352RECVQ-4=28890752Wed May 6 21:00:21 EDT 2020, 69501 IP pkts rcvd,69464 IP pkts sent,0 UDP pkts rcvd, 1 UDP pkts sent,42158 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=223551360RECVQ-3=314995584RECVQ-4=41735680Wed May 6 21:00:23 EDT 2020, 69962 IP pkts rcvd,69978 IP pkts sent,0 UDP pkts rcvd, 2 UDP pkts sent,43775 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=244732544RECVQ-3=338239616RECVQ-4=53858176Wed May 6 21:00:24 EDT 2020, 68266 IP pkts rcvd,68216 IP pkts sent,0 UDP pkts rcvd, 0 UDP pkts sent,43118 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=265258752RECVQ-3=360643712RECVQ-4=65362688 The common denominator I've found is that one of the RECVQs hits 536,871,424. This number seems to be almost exactly double (512 difference) the rmem.max/log_fifo_size (268,435,456). Even though there seems to be capacity in the other RECVQs, just one of those hitting that magic number seems to be enough to throw things out of whack. During this time, the CPU usage for syslog-ng also drops: top - 21:00:11 up 2 days, 8:40, 2 users, load average: 1.09, 1.41, 1.32 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 123.6 10.9 3080:46 syslog-ng top - 21:00:14 up 2 days, 8:40, 2 users, load average: 1.00, 1.39, 1.31 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 24.6 10.9 3080:46 syslog-ng top - 21:00:17 up 2 days, 8:40, 2 users, load average: 1.00, 1.39, 1.31 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 7.6 10.9 3080:47 syslog-ng As best I can tell, based on the reading I've done, the message counts we're getting should be doable, but it's still not clear exactly how to size some of these options as they relate to UDP (like log_fetch_limit, which we do not have set). Anything else I should try?

2 4

Filtering all systemd user service logs
by Thomas Preisner 02 May '22

02 May '22

Hello all, I've been using syslog-ng for quite a while now to centrally collect logs of a few computers running debian and it worked great. However, a few weeks ago I enabled systemd user services on those machines and since then, my 'daemon' logs have been basically completely flooded with (in my opinion) useless log messages created by the aforementioned user services. So far, I've been using the following source configuration: source src { internal(); system(); udp(); udp6(); }; Until now, I've been filtering out the user service logs with my log-analysis tool using some regexes but due to the logs being 'user-controlled' (users can create arbitrary logs by simply creating and starting custom systemd user services) this is obviously no permanent solution. Even worse, the user-created logs end up in the 'daemon'-facility and not in the 'user'-facility as mentioned before. Therefore, here my question: Is there any clean way to detect/filter out these user service logs? I couldn't find anything relevant in the documentation so far. After all, these logs are collected separately by systemd-journald and then merged with the remaining system logs by syslog-ng (at least as far as I see it). Thank you for your response in advance! Best regards, Thomas Preisner

2 1

Local sources seem not to be working
by Alexandre Santos 28 Apr '22

28 Apr '22

Hi, I have syslog-ng 3.32.1 running in a Debian GNU/Linux 10 (buster) with the configuration in the attachement. After sometime running, syslog-ng seems be unable to read from system() and internal() sources. Log messages from syslog(ip(10.20.30.40) transport("udp") port(514) keep-alive(no)); are seen in the output folders. Also journald logs are working fine. After a reload of configuration in which what changes is this line: rewrite r_host { set("MACHINE-${HOST}", value("HOST")); }; logging is resumed. Here is the time gap for logs: <43>1 2022-03-11T11:55:23.802+00:00 xmm4-1-1 syslog-ng 8283 - [meta sequenceId="767"] Last message 'Destination reliable' repeated 8933 times, suppressed by syslog-ng on xmm4-1-1 <46>1 2022-03-14T07:19:01.817+00:00 xmm4-1-1 syslog-ng 8283 - [meta sequenceId="1"] Module loaded and initialized successfully; module='syslogformat' Do you know why this is happening? Thanks & Regards, Alex

2 13

Insider 2022-04: typing; sudo; Zinc; Elastic Cloud; 3.36;
by Peter Czanik (pczanik) 21 Apr '22

21 Apr '22

Dear syslog-ng users, This is the 100th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS syslog-ng 4 theme: typing ------------------------- Balázs Scheidler, founder of the syslog-ng project, describes a major new syslog-ng version 4.0 feature: typing. “syslog-ng traditionally assumes that log data, even if it comes in a structured form (like RFC5424 structured data or JSON) is primarily textual in nature. For this reason, name-value pairs in syslog-ng are text values just as the log message as a whole.” Read the rest of the blog to learn how typing helps not just when forwarding logs to Elasticsearch and others, but also when doing routing decisions: https://syslog-ng-future.blog/syslog-ng-4-theme-typing/ Working with JSON logs from sudo in syslog-ng --------------------------------------------- >From this blog you will learn how to work with JSON formatted logs in syslog-ng and also about new sudo features along the way: JSON logging in sudo, chroot support, logging sub-commands, and how to work with these logs in syslog-ng. https://www.syslog-ng.com/community/b/blog/posts/working-with-json-logs-fro… Elasticsearch 8 and syslog-ng ----------------------------- General availability of Elasticsearch 8 was announced recently. There were quite a few rumors that it will break compatibility with third party tools. I tested it as soon as I had a little time: I am happy to share that anything I tested with the elasticsearch-http() destination of syslog-ng still seems to work perfectly well with the latest version of Elasticsearch. https://www.syslog-ng.com/community/b/blog/posts/elasticsearch-8-and-syslog… Contacting the syslog-ng team: reporting problems, asking questions --------------------------------------------- Recently I got some complaints that it is difficult to figure out how to contact the syslog-ng team to get help or report problems. Most of this information is available both on the syslog-ng website and at the syslog-ng repository on GitHub, but collecting here all information might be still useful for some people. https://www.syslog-ng.com/community/b/blog/posts/contacting-the-syslog-ng-t… WEBINARS * Register for our latest webinars at https://www.syslog-ng.com/event/syslogng-webcast-series-2022/ * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

[GSoC] Rotate log files based on size
by Devharsh Trivedi 13 Apr '22

13 Apr '22

Hi @all, I am a PhD student and would like to apply to the Google Summer of Code. The proposal "Rotate log files based on size” seems interesting to me. Please advise. Regards, Devharsh.

3 2

3.36.1 new log format of some internal messages
by Evan Rempel 06 Apr '22

06 Apr '22

syslog-ng version 3.36.1 has introduced an additional time stamp in some of its internally generated log messages. [2022-04-05T07:10:27.574110] WARNING: Configuration file format is too old, syslog-ng is running in compatibility mode. Please update it to use the syslog-ng 3.36 format at your time of convenience. To upgrade the configuration, please review the warnings about incompatible changes printed by syslog-ng, and once completed change the @version header at the top of the configuration file; config-version='3.34' In the past this log message would have been formatted as WARNING: Configuration file format is too old, syslog-ng is running in compatibility mode. Please update it to use the syslog-ng 3.36 format at your time of convenience. To upgrade the configuration, please review the warnings about incompatible changes printed by syslog-ng, and once completed change the @version header at the top of the configuration file; config-version='3.34' Is this intentional? Why is it not consistent with other internal messages such as syslog-ng starting up; version='3.36.1' Another thing of note is that the configuration warning is actually logged via journald while all other internal messages are only available in the syslog-ng log streams sourcing "internal()" -- Evan

2 2

syslog-ng and ELK integration
by Park, Ethan 04 Apr '22

04 Apr '22

Greetings, I don't know this is the right place to ask about this question but I will give it a shot. We are running a syslog-ng server and we want to use Kibana as a visualization tool. I got stuck in at this point when configuring conf.d file. source s_tcp { tcp(ip("0.0.0.0") port("514")); udp(ip("0.0.0.0") port("514")); }; destination d_el { elasticsearch2( client-mode("http") index("syslog-ng") type("test") cluster-url(http://10.1.3.222:9200) ); #file("/var/log/networks/$HOST/$YEAR/$MONTH/$HOST-$YEAR-$MONTH-$DAY.log"); }; log { source(s_tcp); destination(d_el); flags(flow-control); }; If I start syslog-ng daemon, I got the below error. Apr 04 11:54:42 syslog-ng syslog-ng[22294]: [2022-04-04T11:54:42.443764] Plugin module not found in 'module-path'; module-path='/usr/lib/syslog-ng/3.13', module='mod-java' Apr 04 11:54:42 syslog-ng syslog-ng[22294]: Error parsing destination, destination plugin elasticsearch2 not found in /etc/syslog-ng/conf.d/firewals.conf at line 10, column Apr 04 11:54:42 syslog-ng syslog-ng[22294]: included from /etc/syslog-ng/syslog-ng.conf line 163, column 1 Apr 04 11:54:42 syslog-ng syslog-ng[22294]: elasticsearch2( Apr 04 11:54:42 syslog-ng syslog-ng[22294]: ^^^^^^^^^^^^^^ Apr 04 11:54:42 syslog-ng syslog-ng[22294]: syslog-ng documentation: https://www.balabit.com/support/documentation?product=syslog-ng-ose Apr 04 11:54:42 syslog-ng syslog-ng[22294]: contact: https://lists.balabit.hu/mailman/listinfo/syslog-ng Apr 04 11:54:42 syslog-ng systemd[1]: syslog-ng.service: Main process exited, code=exited, status=1/FAILURE Apr 04 11:54:42 syslog-ng systemd[1]: syslog-ng.service: Failed with result 'exit-code'. Apr 04 11:54:42 syslog-ng systemd[1]: Failed to start System Logger Daemon. -- Subject: Unit syslog-ng.service has failed I am running 3.13 (syslog-ng) and what did I miss? PRIVILEGED AND CONFIDENTIAL COMMUNICATION This message is privileged and confidential under state and federal law. It is intended for the exclusive use of the named recipient. If the reader of this message is not the named or intended recipient, or the employee or agent responsible for delivering the message to its intended recipient, you are hereby notified that any use, copying, disclosure or dissemination of this message is strictly prohibited. If you are not the named or intended recipient and have received this communication in error, please notify us immediately by replying to the sender of this e-mail or at the address and phone number listed below, and take all steps necessary to delete this communication. Exactech, Inc. 2320 NW 66th Ct. Gainesville, FL. 32653 352-377-1140 An Equal Opportunity Employer

2 1

Multiple options blocks
by Faine, Mark R. (MSFC-IS40)[NICS] 02 Apr '22

02 Apr '22

Can I have an options block in syslog-ng.conf that is for default configuration such as for /var/log and another options block in an include file that is specific to those destinations?

2 1