May 2021 - syslog-ng - lists.balabit.hu

UDP drops with syslog-ng 3.24.1-1
by brian hoffman 05 Oct '22

05 Oct '22

We've been centrally logging with syslog-ng for about 5 years now. Over that time, the number of sources has grown significantly, and at some point we crossed a line where drops were happening (a quick survey of 3 million syslog packets yielded 420 unique currently sending hosts). After much research and experimentation, we've been able to get to the point where throughout the day there are 0 drops for the most part. This was achieved by installing the latest syslog-ng (not the RedHat packaged one) and creating a source for each CPU. Occasionally, though, we still have periods of drops so I'm trying to eliminate these last few. Here are some relevant configuration items:2 RedHat 7.8 VMs (load balanced via an F5) with 16GB memory and 4 CPUs each running syslog-ng-3.24.1-1.el7.x86_64.net.core.rmem_default = 212992net.core.rmem_max = 268435456 log_fifo_size(268435456); source s_network {network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp1"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp2"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp3"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp4"));network(ip("0.0.0.0") port(514) transport("tcp") max_connections(200) keep_alive(yes) so_rcvbuf(67108864));}; We are limited to UDP, unfortunately, because we do not have control over the devices/networks/etc. that are sending to us, but we have changed as many of the internal senders and destinations to TCP as we can. With a script I created to view the packets, including drops, as well as the individual RECVQs, the issue can be illustrated. Here's what things look like normally:Thu May 7 10:48:15 EDT 2020, 27003 IP pkts rcvd,26980 IP pkts sent,24951 UDP pkts rcvd, 28075 UDP pkts sent,0 UDP pkt rcv err RECVQ-1=2176RECVQ-2=0RECVQ-3=0RECVQ-4=0Thu May 7 10:48:16 EDT 2020, 28453 IP pkts rcvd,28426 IP pkts sent,26185 UDP pkts rcvd, 29180 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=0RECVQ-2=0RECVQ-3=4352RECVQ-4=0Thu May 7 10:48:17 EDT 2020, 28294 IP pkts rcvd,28276 IP pkts sent,26277 UDP pkts rcvd, 28709 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=2176RECVQ-2=0RECVQ-3=0RECVQ-4=0 The RECVQs are sparsely used, and there are no errors. Around 9pm every night, the packet counts go up significantly (probably due to backup related logs):Wed May 6 21:00:08 EDT 2020, 66382 IP pkts rcvd,66366 IP pkts sent,39405 UDP pkts rcvd, 67592 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=1595008RECVQ-2=106217088RECVQ-3=53694976RECVQ-4=31858816Wed May 6 21:00:09 EDT 2020, 69317 IP pkts rcvd,69338 IP pkts sent,44446 UDP pkts rcvd, 75958 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=13056RECVQ-2=126397312RECVQ-3=75568128RECVQ-4=41626880Wed May 6 21:00:10 EDT 2020, 71205 IP pkts rcvd,71227 IP pkts sent,43657 UDP pkts rcvd, 74603 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=920448RECVQ-2=146122752RECVQ-3=100951168RECVQ-4=52622208Wed May 6 21:00:12 EDT 2020, 69578 IP pkts rcvd,69454 IP pkts sent,124465 UDP pkts rcvd, 163367 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=13140864RECVQ-2=44494848RECVQ-3=125579136RECVQ-4=0 Still, though, it's handling it with no errors. But then at some point a threshold is reached and errors start piling up:Wed May 6 21:00:20 EDT 2020, 63177 IP pkts rcvd,63291 IP pkts sent,0 UDP pkts rcvd, 0 UDP pkts sent,38011 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=200357376RECVQ-3=292948352RECVQ-4=28890752Wed May 6 21:00:21 EDT 2020, 69501 IP pkts rcvd,69464 IP pkts sent,0 UDP pkts rcvd, 1 UDP pkts sent,42158 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=223551360RECVQ-3=314995584RECVQ-4=41735680Wed May 6 21:00:23 EDT 2020, 69962 IP pkts rcvd,69978 IP pkts sent,0 UDP pkts rcvd, 2 UDP pkts sent,43775 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=244732544RECVQ-3=338239616RECVQ-4=53858176Wed May 6 21:00:24 EDT 2020, 68266 IP pkts rcvd,68216 IP pkts sent,0 UDP pkts rcvd, 0 UDP pkts sent,43118 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=265258752RECVQ-3=360643712RECVQ-4=65362688 The common denominator I've found is that one of the RECVQs hits 536,871,424. This number seems to be almost exactly double (512 difference) the rmem.max/log_fifo_size (268,435,456). Even though there seems to be capacity in the other RECVQs, just one of those hitting that magic number seems to be enough to throw things out of whack. During this time, the CPU usage for syslog-ng also drops: top - 21:00:11 up 2 days, 8:40, 2 users, load average: 1.09, 1.41, 1.32 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 123.6 10.9 3080:46 syslog-ng top - 21:00:14 up 2 days, 8:40, 2 users, load average: 1.00, 1.39, 1.31 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 24.6 10.9 3080:46 syslog-ng top - 21:00:17 up 2 days, 8:40, 2 users, load average: 1.00, 1.39, 1.31 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 7.6 10.9 3080:47 syslog-ng As best I can tell, based on the reading I've done, the message counts we're getting should be doable, but it's still not clear exactly how to size some of these options as they relate to UDP (like log_fetch_limit, which we do not have set). Anything else I should try?

2 4

Re: [syslog-ng] Request for config help
by Pradeep Gotaparthi 15 Jun '21

15 Jun '21

Hi, Could anybody help us out with the below query? Thanks. On Fri, Mar 26, 2021 at 12:06 AM Pradeep Gotaparthi <pradeep.gp(a)gmail.com> wrote: > Hi, > > We have been using syslog-ng for over a year now and it is working great, > thanks! > > One issue we face is that when we stop the process when there is some > buffer, the threshold is ignored and a bunch of logs get pushed out (as > much as 10 times the threshold value). This is causing problems on the > receiving end as it can't handle the load. We thought the log-fifo-size > parameter can help as syslog uses this parameter to store logs in memory > but it is not helping. > > It would be great if you can guide us. > > > Thanks. >

5 8

Using custom parser with only a subset of udp traffic
by Faine, Mark R. (MSFC-IS40)[NICS] 27 May '21

27 May '21

Hopefully one of you can help me figure this out. All of my udp traffic is coming in through one source (network driver), some of it is firewall data but I have to filter it out based on where it is coming from and other fields in the log data. I want to use the csv_parser here since it would make it easier for me to get the data I need from the firewall data. Does this require splitting my one source into two sources, since I would need to use flags('no-parse') on a firewall data source? I also think this means that I can't use a single log path to process both sources? Also, how to prevent potential duplication since the non-firewall source will still receive the firewall traffic and try to process it with syslog-parser. I think the flow is something like this: firewall source -> custom parser -> filters ->rewrites / \ all_data -> \ -> destination \ / all udp source -> filters -> rewrites / I understand this is a bit complex to answer simply, I'm mostly looking for answers to the above questions as well as high level guidance for how to proceed. Thanks, -Mark

3 4

Slack or other
by Faine, Mark R. (MSFC-IS40)[NICS] 21 May '21

21 May '21

Have you considered or would you consider a slack or other real time based communication environment for syslog-ng? The mailing list is great but it's kind of slow asynchronous communication. It would be awesome to have a community where I could just fire off a quick question and get a quick answer from anyone who might be around. Thanks, -Mark

3 3

using VARARGS correctly
by Faine, Mark R. (MSFC-IS40)[NICS] 21 May '21

21 May '21

Syslog-ng 3.19 block destination d_default(basepath('/var/log/remote/backup') location("") app("") name("")) { file("`basepath`/`location`/`app`/${HOST}/${HOST}_`app``__VARARGS__`.log" persist-name(`name`) create_dirs(yes) flags("threaded", "no-multi-line")); }; However, when I call it like so: d_default(basepath(`BASEPATH`) location("$location") app("$app") name('d_default_udp') "$session"); I get a syntax error when checking with --syntax-only Error parsing block reference, syntax error, unexpected LL_STRING, expecting ')' in /etc/syslog-ng/conf.d/splunk.conf: 23 categorize_loc(); 24 categorize_app(); 25 }; 26 }; 27 destination { 28----> d_default(basepath(`BASEPATH`) location("$location") app("$app") name('d_default_udp') "$session"); 28----> ^^^^^^^^^^ >From looking at the documentation, it looks like it's a valid way to use it. $session here is just a number from 0-9 that is set from rewriting a user defined macro from the message's sessionid field. It's only going to exist for a certain kind of message. Thanks, -Mark

3 5

ACLs with syslog-ng
by Faine, Mark R. (MSFC-IS40)[NICS] 19 May '21

19 May '21

RHEL 7 and 8 syslog-ng 3.19 and 3.31 We have always set permissions on directories that we want the Splunk universal forwarder to be able to read as root:splunk 640, but now security doesn't like this and wants everything under /var/log to always be root:root except for some specific exceptions. We had tried to solve this with an ACL in the past, however, syslog-ng always seems to clobber the ACL, even when it's the default ACL on the folder. Is this a known issue, is there a way to get syslog-ng to play nice with ACLs. Thanks, -Mark Mark Faine System Administrator SAIC/NICS 215 Wynn Dr. 5065 Huntsville, AL 35805 256-961-1295 (Desk) 256-617-4861 (Work Cell)

2 4

Insider 2021-05: 3.32; Fortigate; Windows event collector; FreeBSD; Pi;
by Peter Czanik (pczanik) 13 May '21

13 May '21

Dear syslog-ng users, This is the 91st issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS syslog-ng 3.32 is now available ------------------------------- Highlights include multi-threaded connection pooling to MongoDB servers and template support for its collection() option. OpenBSD support is now fixed. For a complete list of changes, check the release notes at: https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-3.32.1 Parsing Fortigate logs and other syslog-ng 3.31 news ---------------------------------------------------- Version 3.31 of syslog-ng has been released recently. One of its most user-visible features is the parser for Fortigate logs, yet another networking vendor that produces log messages not conforming to syslog specifications. Parsing Fortigate logs builds upon the new no-header flag of syslog-ng combined with the key-value and date parsers. Other features include a new silent message option for the Telegram destination and automatic directory creation for disk-buffer files. https://www.syslog-ng.com/community/b/blog/posts/parsing-fortigate-logs-and… Collecting Windows EventLog without installing extra software on Windows -------------------------------------------- Talking to users at both IT security and operations related conferences, they confessed the same: the less software the better. Less software to learn, maintain and follow in bug trackers. The Windows Event Collector (WEC) component of syslog-ng Premium Edition (PE) brings this to the extremes: it enables Windows machines to act as log sources without installing any software. Just enable the WEC component of syslog-ng PE on your Linux server and enjoy the logs flowing in. From this blog you can learn how to set up a test environment for WEC. I will also show you an easier way to prepare the certificates used, help you avoid some of the failures, and point you to alternative ways, where necessary. https://www.syslog-ng.com/community/b/blog/posts/windows-event-collector-fo… syslog-ng on the FreeBSD wiki ----------------------------- My original blog about getting started with syslog-ng on FreeBSD became outdated. The content was updated and extended a bit, and with some help from the FreeBSD community, it is now available on the FreeBSD wiki at https://wiki.freebsd.org/Ports/sysutils/syslog-ng Syslog-ng & Pi day ------------------ We had March 14th recently, or as many geeks refer to it: Pi day. On this occasion, I would like to show you a syslog-ng configuration, which prints a huge π on the screen, and two recent articles that feature syslog-ng on the Raspberry Pi. https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-pi-day WEBINARS * syslog-ng, Swiss Army Knife of Log Management: https://www.syslog-ng.com/event/syslogng-swiss-army-knife-of-log-management… * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Changing message transfer for rfc6587
by Hartmann Benedict - Maschinenfabrik Reinhausen 12 May '21

12 May '21

Hello, at the moment I am sending messages via destination d_syslog_visu_conf_IETF { syslog("`IPADRESS`" transport(TCP) port(`PORT`) `TLS_CERT`) ; }; This approach will lead to the rfc6587 standard. The messages are collected via visual syslog. Is it possible to change the message transfer mode from "octet counting" (leading message length) to "non-transparent-framing" (no leading message length) via config? Best regards Benedict Hartmann Maschinenfabrik Reinhausen GmbH General Manager / Geschäftsführer: Dr. Nicolas Maier-Scheubeck, Wilfried Breuer, Holger Michalka Chairman of the Board / Vors. des Aufsichtsrats: Hans-Jürgen Thaus Registered Office / Sitz der Gesellschaft: Regensburg Local Court Regensburg / Amtsgericht Regensburg HRB 3687 This e-mail and any attachments contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender and delete this message. Thank you. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. This e-mail was scanned for viruses, vandals and malicious content.

2 1

syslog-ng mail list is fixed
by Attila Szakacs (aszakacs) 10 May '21

10 May '21

Hello fellow syslog-ng users, Apologies for the past few weeks' syslog-ng mail list outage. We have had some issues with our mail server, but everything seems to be resolved now. If you still cannot send mails to syslog-ng(a)lists.balabit.hu, please write a mail to me directly: attila.szakacs(a)oneidentity.com. Cheers, Attila

1 0

syslog-ng-3.32.1
by Peter Kokai (pkokai) 10 May '21

10 May '21

3.32.1 ====== ## Highlights * `mongodb()`: add `workers()` support (multi-threaded connection pooling) The MongoDB driver now supports the `workers()` option, which specifies the number of parallel workers to be used. Workers are based on the connection pooling feature of the MongoDB C library. This increases the throughput of the MongoDB destination driver. Example: ``` destination { mongodb( uri("mongodb://hostA,hostB/syslog?replicaSet=my_rs&wtimeoutMS=10000&socketTimeoutMS=10000&connectTimeoutMS=10000&serverSelectionTimeoutMS=5000") collection("messages") workers(8) ); }; ``` ([#3621](https://github.com/syslog-ng/syslog-ng/pull/3621)) * `mongodb()`: template support for the `collection()` option The `collection()` option of the MongoDB destination driver now accepts templates, for example: ``` destination { mongodb( uri("mongodb://host/syslog") collection("${HOST}_messages") ); }; ``` ([#3621](https://github.com/syslog-ng/syslog-ng/pull/3621)) ## Features * `time-reopen`: Support the `time-reopen()` option on the driver level for the following drivers: * sources: `example-diskq-source`, `python-fetcher` * destinations: `amqp`, `example-destination`, `file`, `http`, `mongodb`, `network`, `pipe`, `program`, `pseudofile`, `python`, `redis`, `riemann`, `smtp`, `sql`, `stomp`, `syslog`, `tcp`, `tcp6`, `udp`, `udp6`, `unix-dgram`, `unix-stream`, `usertty` ([#3585](https://github.com/syslog-ng/syslog-ng/pull/3585)) * `csv-parser()`: add drop-invalid() option along with the already existing flag with the same name. This is to improve the consistency of the configuration language. ([#3547](https://github.com/syslog-ng/syslog-ng/pull/3547)) * `usertty() destination`: Support changing the terminal disable timeout with the `time-reopen()` option. Default timeout change to 60 from 600. If you wish to use the old 600 timeout, add `time-reopen(600)` to your config in the `usertty()` driver. ([#3585](https://github.com/syslog-ng/syslog-ng/pull/3585)) * `syslog-parser()`: add a new drop-invalid() option that allows the use of syslog-parser() in if statements. Normally a syslog-parser() injects an error message instead of failing. ([#3565](https://github.com/syslog-ng/syslog-ng/pull/3565)) ## Bugfixes * date-parser: if the timestamp pattern did not covered a field (for example seconds) that field had undefined value The missing fields are initialized according to the following rules: 1) missing all fields -> use current date 2) only miss year -> guess year based on current year and month (current year, last year or next year) 3) the rest of the cases don't make much sense, so zero initialization of the missing field makes sense. And the year is initialized to the current one. ([#3615](https://github.com/syslog-ng/syslog-ng/pull/3615)) * Fix compilation issues on OpenBSD syslog-ng can now be compiled on OpenBSD. ([#3661](https://github.com/syslog-ng/syslog-ng/pull/3661)) * loggen: debug message printed wrong plugin name (ssl-plugin instead of socket_plugin) ([#3624](https://github.com/syslog-ng/syslog-ng/pull/3624)) * tls: fixup EOF detection issue in tls (before 3.0 version) syslog-ng error message: "I/O error occurred while reading; fd='13', error='Success (0)'" ([#3618](https://github.com/syslog-ng/syslog-ng/pull/3618)) * kafka: the config() block couldn't contain option that is already a keyword in syslog-ng (example: retries) ([#3658](https://github.com/syslog-ng/syslog-ng/pull/3658)) * templates: fixed error reporting when invalid templates were specified The `amqp()`, `file()` destination, `sql()`, `stomp()`, `pdbtool`, and `graphite()` plugins had template options that did not report errors at startup when invalid values were specified. ([#3660](https://github.com/syslog-ng/syslog-ng/pull/3660)) ## Packaging * bison: minimum version of bison is bumped to 3.7.6 ([#3547](https://github.com/syslog-ng/syslog-ng/pull/3547)) * java-modules: the minimum version of gradle changed from 2.2 to 3.4 ([#3645](https://github.com/syslog-ng/syslog-ng/pull/3645)) * light: add to the release tarball ([#3613](https://github.com/syslog-ng/syslog-ng/pull/3613)) ## Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Attila Szakacs, Balazs Scheidler, Gabor Nagy, Janos SZIGETVARI, Laszlo Budai, Laszlo Szemere, LittleFish33, László Várady, Ming Liu, Norbert Takacs, Peter Kokai, Todd C. Miller, Yi Fan Yu, Zoltan Pallagi

1 0