August 2022 - syslog-ng - lists.balabit.hu

Re: [syslog-ng] Local sources seem not to be working
by Szilard Parrag (sparrag) 11 Jan '23

11 Jan '23

Hi Alex, We've checked it too and syslog-ng does not release the file descriptor of journald even with flow-control enabled. Also, your internal logs seem rather terse, maybe there is a filter which filters out the important parts. Could you please check it? Szilard

4 12

UDP drops with syslog-ng 3.24.1-1
by brian hoffman 05 Oct '22

05 Oct '22

We've been centrally logging with syslog-ng for about 5 years now. Over that time, the number of sources has grown significantly, and at some point we crossed a line where drops were happening (a quick survey of 3 million syslog packets yielded 420 unique currently sending hosts). After much research and experimentation, we've been able to get to the point where throughout the day there are 0 drops for the most part. This was achieved by installing the latest syslog-ng (not the RedHat packaged one) and creating a source for each CPU. Occasionally, though, we still have periods of drops so I'm trying to eliminate these last few. Here are some relevant configuration items:2 RedHat 7.8 VMs (load balanced via an F5) with 16GB memory and 4 CPUs each running syslog-ng-3.24.1-1.el7.x86_64.net.core.rmem_default = 212992net.core.rmem_max = 268435456 log_fifo_size(268435456); source s_network {network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp1"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp2"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp3"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp4"));network(ip("0.0.0.0") port(514) transport("tcp") max_connections(200) keep_alive(yes) so_rcvbuf(67108864));}; We are limited to UDP, unfortunately, because we do not have control over the devices/networks/etc. that are sending to us, but we have changed as many of the internal senders and destinations to TCP as we can. With a script I created to view the packets, including drops, as well as the individual RECVQs, the issue can be illustrated. Here's what things look like normally:Thu May 7 10:48:15 EDT 2020, 27003 IP pkts rcvd,26980 IP pkts sent,24951 UDP pkts rcvd, 28075 UDP pkts sent,0 UDP pkt rcv err RECVQ-1=2176RECVQ-2=0RECVQ-3=0RECVQ-4=0Thu May 7 10:48:16 EDT 2020, 28453 IP pkts rcvd,28426 IP pkts sent,26185 UDP pkts rcvd, 29180 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=0RECVQ-2=0RECVQ-3=4352RECVQ-4=0Thu May 7 10:48:17 EDT 2020, 28294 IP pkts rcvd,28276 IP pkts sent,26277 UDP pkts rcvd, 28709 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=2176RECVQ-2=0RECVQ-3=0RECVQ-4=0 The RECVQs are sparsely used, and there are no errors. Around 9pm every night, the packet counts go up significantly (probably due to backup related logs):Wed May 6 21:00:08 EDT 2020, 66382 IP pkts rcvd,66366 IP pkts sent,39405 UDP pkts rcvd, 67592 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=1595008RECVQ-2=106217088RECVQ-3=53694976RECVQ-4=31858816Wed May 6 21:00:09 EDT 2020, 69317 IP pkts rcvd,69338 IP pkts sent,44446 UDP pkts rcvd, 75958 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=13056RECVQ-2=126397312RECVQ-3=75568128RECVQ-4=41626880Wed May 6 21:00:10 EDT 2020, 71205 IP pkts rcvd,71227 IP pkts sent,43657 UDP pkts rcvd, 74603 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=920448RECVQ-2=146122752RECVQ-3=100951168RECVQ-4=52622208Wed May 6 21:00:12 EDT 2020, 69578 IP pkts rcvd,69454 IP pkts sent,124465 UDP pkts rcvd, 163367 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=13140864RECVQ-2=44494848RECVQ-3=125579136RECVQ-4=0 Still, though, it's handling it with no errors. But then at some point a threshold is reached and errors start piling up:Wed May 6 21:00:20 EDT 2020, 63177 IP pkts rcvd,63291 IP pkts sent,0 UDP pkts rcvd, 0 UDP pkts sent,38011 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=200357376RECVQ-3=292948352RECVQ-4=28890752Wed May 6 21:00:21 EDT 2020, 69501 IP pkts rcvd,69464 IP pkts sent,0 UDP pkts rcvd, 1 UDP pkts sent,42158 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=223551360RECVQ-3=314995584RECVQ-4=41735680Wed May 6 21:00:23 EDT 2020, 69962 IP pkts rcvd,69978 IP pkts sent,0 UDP pkts rcvd, 2 UDP pkts sent,43775 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=244732544RECVQ-3=338239616RECVQ-4=53858176Wed May 6 21:00:24 EDT 2020, 68266 IP pkts rcvd,68216 IP pkts sent,0 UDP pkts rcvd, 0 UDP pkts sent,43118 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=265258752RECVQ-3=360643712RECVQ-4=65362688 The common denominator I've found is that one of the RECVQs hits 536,871,424. This number seems to be almost exactly double (512 difference) the rmem.max/log_fifo_size (268,435,456). Even though there seems to be capacity in the other RECVQs, just one of those hitting that magic number seems to be enough to throw things out of whack. During this time, the CPU usage for syslog-ng also drops: top - 21:00:11 up 2 days, 8:40, 2 users, load average: 1.09, 1.41, 1.32 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 123.6 10.9 3080:46 syslog-ng top - 21:00:14 up 2 days, 8:40, 2 users, load average: 1.00, 1.39, 1.31 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 24.6 10.9 3080:46 syslog-ng top - 21:00:17 up 2 days, 8:40, 2 users, load average: 1.00, 1.39, 1.31 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 7.6 10.9 3080:47 syslog-ng As best I can tell, based on the reading I've done, the message counts we're getting should be doable, but it's still not clear exactly how to size some of these options as they relate to UDP (like log_fetch_limit, which we do not have set). Anything else I should try?

2 4

syslog-ng-3.38.1
by Laszlo Varady (lvarady) 29 Aug '22

29 Aug '22

3.38.1 <https://github.com/syslog-ng/syslog-ng/blob/master/NEWS.md#highlights> Highlights <https://github.com/syslog-ng/syslog-ng/blob/master/NEWS.md#sneak-peek-into-…> Sneak peek into syslog-ng v4.0 syslog-ng v4.0 is right around the corner. This release (v3.38.1) contains all major changes, however, they are currently all hidden behind a feature flag. To enable and try those features, you need to specify @version: 4.0 at the top of the configuration file. You can find out more about the 4.0 changes and features here<https://github.com/syslog-ng/syslog-ng/blob/master/NEWS-4.0.md>. Read our practical introduction to typing at syslog-ng-future.blog<https://syslog-ng-future.blog/syslog-ng-4-progress-3-38-1-release/>. <https://github.com/syslog-ng/syslog-ng/blob/master/NEWS.md#features> Features * grouping-by(): added inject-mode(aggregate-only) This inject mode will drop individual messages that make up the correlation context (key() groups) and would only yield the aggregate messages (e.g. the results of the correlation). (#3998<https://github.com/syslog-ng/syslog-ng/pull/3998>) * add-contextual-data(): add support for type propagation, e.g. set the type of name-value pairs as they are created/updated to the value returned by the template expression that we use to set the value. The 3rd column in the CSV file (e.g. the template expression) now supports specifying a type-hint, in the format of "type-hint(template-expr)". Example line in the CSV database: selector-value,name-value-pair-to-be-created,list(foo,bar,baz) (#4051<https://github.com/syslog-ng/syslog-ng/pull/4051>) * $(format-json): add --key-delimiter option to reconstruct JSON objects using an alternative structure separator, that was created using the key-delimiter() option of json-parser(). (#4093<https://github.com/syslog-ng/syslog-ng/pull/4093>) * json-parser(): add key-delimiter() option to extract JSON structure members into name-value pairs, so that the names are flattened using the character specified, instead of dot. Example: Input: {"foo":{"key":"value"}} Using json-parser() without key-delimiter() this is extracted to: foo.key="value" Using json-parser(key-delimiter("~")) this is extracted to: foo~key="value" This feature is useful in case the JSON keys contain dots themselves, in those cases the syslog-ng representation is ambigious. (#4093<https://github.com/syslog-ng/syslog-ng/pull/4093>) <https://github.com/syslog-ng/syslog-ng/blob/master/NEWS.md#bugfixes> Bugfixes * Fixed buffer handling of syslog and timestamp parsers Multiple buffer out-of-bounds issues have been fixed, which could cause hangs, high CPU usage, or other undefined behavior. (#4110<https://github.com/syslog-ng/syslog-ng/pull/4110>) * Fixed building with LibreSSL (#4081<https://github.com/syslog-ng/syslog-ng/pull/4081>) * network(): Fixed a bug, where syslog-ng halted the input instead of skipping a character in case of a character conversion error. (#4084<https://github.com/syslog-ng/syslog-ng/pull/4084>) * redis(): Fixed bug where using redis driver without the batch-lines option caused program crash. (#4114<https://github.com/syslog-ng/syslog-ng/pull/4114>) * pdbtool: fix a SIGABRT on FreeBSD that was triggered right before pdbtool exits. Apart from being an ugly crash that produces a core file, functionally the tool behaved correctly and this case does not affect syslog-ng itself. (#4037<https://github.com/syslog-ng/syslog-ng/pull/4037>) * regexp-parser(): due to a change introduced in 3.37, named capture groups are stored indirectly in the LogMessage to avoid copying of the value. In this case the name-value pair created with the regexp is only stored as a reference (name + length of the original value), which improves performance and makes such name-value pairs use less memory. One omission in the original change in 3.37 is that syslog-ng does not allow builtin values to be stored indirectly (e.g. $MESSAGE and a few of others) and this case causes an assertion to fail and syslog-ng to crash with a SIGABRT. This abort is now fixed. Here's a sample config that reproduces the issue: regexp-parser(patterns('(?<MESSAGE>.*)')); (#4043<https://github.com/syslog-ng/syslog-ng/pull/4043>) * set-tag: fix cloning issue when string literal were used (see #4062) (#4065<https://github.com/syslog-ng/syslog-ng/pull/4065>) * add-contextual-data(): fix high memory usage when using large CSV files (#4067<https://github.com/syslog-ng/syslog-ng/pull/4067>) <https://github.com/syslog-ng/syslog-ng/blob/master/NEWS.md#other-changes> Other changes * The json-c library is no longer bundled in the syslog-ng source tarball Since all known OS package managers provide json-c packages nowadays, the json-c submodule has been removed from the source tarball. The --with-jsonc=internal option of the configure script has been removed accordingly, system libraries will be used instead. For special cases, the JSON support can be disabled by specifying --with-jsonc=no. (#4078<https://github.com/syslog-ng/syslog-ng/pull/4078>) * platforms: Dropped support for ubuntu-impish as it became EOL (#4088<https://github.com/syslog-ng/syslog-ng/pull/4088>) <https://github.com/syslog-ng/syslog-ng/blob/master/NEWS.md#credits> Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Alvin Šipraga, Andras Mitzki, Attila Szakacs, Balazs Scheidler, Bálint Horváth, Daniel Klauer, Fabrice Fontaine, Gabor Nagy, HenryTheSir, László Várady, Parrag Szilárd, Peter Kokai, Shikhar Vashistha, Szilárd Parrag, Vivin Peris Installation from binaries: https://github.com/syslog-ng/syslog-ng#installation-from-binaries<https://github.com/syslog-ng/syslog-ng#installation-from-binaries<https://github.com/syslog-ng/syslog-ng#installation-from-binaries>

1 0

Potential issue with line breaking not working correctly when writing logs to disk
by John Law 23 Aug '22

23 Aug '22

Hi, We have a number of network devices sending syslog traffic to syslog-ng OSE 3.25 installed on RHEL 7.9. We are having an issue where multiple events are being written as the same line. The config file is template-function adm1 "${HOST} message-length=$(length \"${MSG}\") ${MESSAGE}\n"; source s_adm1 { udp(ip(0.0.0.0) port(5527) ); tcp(ip(0.0.0.0) port(5527) ); }; destination d_adm1 { file( "/Data/syslog_data/$LOGHOST/$R_YEAR-$R_MONTH-$R_DAY/adm1/$HOST_FROM/$HOST/$FACILITY.local" template("$(adm1-function)\n") ); }; log { source(s_adm1); destination(d_adm1); flags(final); }; A small extract of one of the files is, naturally I have changed the IPs 192.1.1.1/31181->192.1.3.1/135 0x0 source rule r12 N/A N/A 6 ACME-CNB010 ACME CNB 1727847 N/A(N/A) reth3.860 UNKNOWN UNKNOWN UNKNOWN275 <14>Aug 19 11:38:50 sunny-fwl29 RT_FLOW: RT_FLOW_SESSION_CREATE: session created 192.6.20.1/47250->192.104.20.1/443 0x0 junos-https 192.6.20.1/47250->192.104.20.1/443 0x0 N/A N/A N/A N/A 6 ACME-BWC002 ACME BWC_EDESIX_AWS 917879 N/A(N/A) reth3.860 UNKNOWN UNKNOWN UNKNOWN275 <14>Aug 19 11:38:50 sunny-fwl29 RT_FLOW: RT_FLOW_SESSION_CREATE: session created 192.6.20.1/47252->192.104.20.1/443 0x0 junos-https 192.6.20.1/47252->192.104.20.1/443 0x0 N/A N/A N/A N/A 6 ACME-BWC002 ACME BWC_EDESIX_AWS 595069 N/A(N/A) reth3.860 UNKNOWN UNKNOWN UNKNOWN285 <14>Aug 19 11:38:50 sunny-fwl29 RT_FLOW: RT_FLOW_SESSION_CREATE: session created 192.1.57.1/54205->192.1.7.1/137 0x0 junos-nbname 192.1.57.1/54205->192.1.7.1/137 0x0 N/A N/A N/A N/A 17 ACME-COMP007 ACME CCA 1735324 N/A(N/A) reth3.860 UNKNOWN UNKNOWN UNKNOWN288 <14>Aug 19 11:38:50 sunny-fwl29 RT_FLOW: RT_FLOW_SESSION_CREATE: session created 192.1.57.1/62486->192.1.7.1/135 0x0 junos-ms-rpc-tcp 192.1.57.1/62486->192.1.7.1/135 0x0 N/A N/A N/A N/A 6 ACME-COMP007 ACME CCA 1027474 N/A(N/A) reth3.860 UNKNOWN UNKNOWN UNKNOWN249 I suspect it might be something to do with the amount of logs that are received in very quick succession, but I have no idea how to address this. One thing I have noticed is that it always appears to be the same device that is causing the issue. When I look at other devices coming in on the same port, they appear to write out correctly. Any suggestion would be really welcome. Thanks John This message has been scanned for malware by Forcepoint. www.forcepoint.com

2 4

Multiple timezone seen on messages originating from local machine
by Francois Marier 16 Aug '22

16 Aug '22

I'm trying to understand timezones in syslog-ng, but I think there's something I'm missing. If I do the following on my OpenWRT machine: /etc/init.d/syslog-ng restart logger TestA I see the following in /var/log/messages: Aug 14 20:39:35 hostname syslog-ng[9860]: syslog-ng shutting down; version='3.37.1' Aug 14 20:39:36 hostname syslog-ng[10024]: syslog-ng starting up; version='3.37.1' Aug 14 20:39:36 hostname syslog-ng[10024]: Syslog connection established; fd='15', server='AF_INET(192.168.1.10:514)', local='AF_INET(0.0.0.0:0)' Aug 15 03:39:49 hostname root: TestA The correct timezone is the one in the first three lines. Other daemon messages are displayed using the incorrect timezone like `logger`. I've attached my syslog-ng.conf file. So my question is why is the time information correct in the first three messages and then incorrect from there on? Francois -- https://fmarier.org/

3 4

drop log lines having keyword "INTERNET"
by Sachchidanand Upadhyay 04 Aug '22

04 Aug '22

Hi, I want to drop all log lines having keyword "INTERNET", I tried following scenario: scenario-1 filter f_log { facility(local3); }; filter f_nointernet { not message("INTERNET"); }; log { source(s_sys); filter(f_nointernet); filter(f_log); destination(d_log); }; scenario-2 filter f_internet { message("INTERNET"); }; filter f_log { facility(local3); and not filter(f_internet); }; log { source(s_sys); filter(f_log); destination(d_log); }; scenario-3 filter f_log { facility(local3); }; filter f_internet { message("INTERNET"); }; filter f_nointernet { not filter(f_internet); }; log { source(s_sys); filter(f_nointernet); filter(f_log); destination(d_log); }; But none of them is working. Please help. Regards, Sachchidanand

1 0