March 2021 - syslog-ng - lists.balabit.hu

UDP drops with syslog-ng 3.24.1-1
by brian hoffman 05 Oct '22

05 Oct '22

We've been centrally logging with syslog-ng for about 5 years now. Over that time, the number of sources has grown significantly, and at some point we crossed a line where drops were happening (a quick survey of 3 million syslog packets yielded 420 unique currently sending hosts). After much research and experimentation, we've been able to get to the point where throughout the day there are 0 drops for the most part. This was achieved by installing the latest syslog-ng (not the RedHat packaged one) and creating a source for each CPU. Occasionally, though, we still have periods of drops so I'm trying to eliminate these last few. Here are some relevant configuration items:2 RedHat 7.8 VMs (load balanced via an F5) with 16GB memory and 4 CPUs each running syslog-ng-3.24.1-1.el7.x86_64.net.core.rmem_default = 212992net.core.rmem_max = 268435456 log_fifo_size(268435456); source s_network {network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp1"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp2"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp3"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp4"));network(ip("0.0.0.0") port(514) transport("tcp") max_connections(200) keep_alive(yes) so_rcvbuf(67108864));}; We are limited to UDP, unfortunately, because we do not have control over the devices/networks/etc. that are sending to us, but we have changed as many of the internal senders and destinations to TCP as we can. With a script I created to view the packets, including drops, as well as the individual RECVQs, the issue can be illustrated. Here's what things look like normally:Thu May 7 10:48:15 EDT 2020, 27003 IP pkts rcvd,26980 IP pkts sent,24951 UDP pkts rcvd, 28075 UDP pkts sent,0 UDP pkt rcv err RECVQ-1=2176RECVQ-2=0RECVQ-3=0RECVQ-4=0Thu May 7 10:48:16 EDT 2020, 28453 IP pkts rcvd,28426 IP pkts sent,26185 UDP pkts rcvd, 29180 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=0RECVQ-2=0RECVQ-3=4352RECVQ-4=0Thu May 7 10:48:17 EDT 2020, 28294 IP pkts rcvd,28276 IP pkts sent,26277 UDP pkts rcvd, 28709 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=2176RECVQ-2=0RECVQ-3=0RECVQ-4=0 The RECVQs are sparsely used, and there are no errors. Around 9pm every night, the packet counts go up significantly (probably due to backup related logs):Wed May 6 21:00:08 EDT 2020, 66382 IP pkts rcvd,66366 IP pkts sent,39405 UDP pkts rcvd, 67592 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=1595008RECVQ-2=106217088RECVQ-3=53694976RECVQ-4=31858816Wed May 6 21:00:09 EDT 2020, 69317 IP pkts rcvd,69338 IP pkts sent,44446 UDP pkts rcvd, 75958 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=13056RECVQ-2=126397312RECVQ-3=75568128RECVQ-4=41626880Wed May 6 21:00:10 EDT 2020, 71205 IP pkts rcvd,71227 IP pkts sent,43657 UDP pkts rcvd, 74603 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=920448RECVQ-2=146122752RECVQ-3=100951168RECVQ-4=52622208Wed May 6 21:00:12 EDT 2020, 69578 IP pkts rcvd,69454 IP pkts sent,124465 UDP pkts rcvd, 163367 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=13140864RECVQ-2=44494848RECVQ-3=125579136RECVQ-4=0 Still, though, it's handling it with no errors. But then at some point a threshold is reached and errors start piling up:Wed May 6 21:00:20 EDT 2020, 63177 IP pkts rcvd,63291 IP pkts sent,0 UDP pkts rcvd, 0 UDP pkts sent,38011 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=200357376RECVQ-3=292948352RECVQ-4=28890752Wed May 6 21:00:21 EDT 2020, 69501 IP pkts rcvd,69464 IP pkts sent,0 UDP pkts rcvd, 1 UDP pkts sent,42158 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=223551360RECVQ-3=314995584RECVQ-4=41735680Wed May 6 21:00:23 EDT 2020, 69962 IP pkts rcvd,69978 IP pkts sent,0 UDP pkts rcvd, 2 UDP pkts sent,43775 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=244732544RECVQ-3=338239616RECVQ-4=53858176Wed May 6 21:00:24 EDT 2020, 68266 IP pkts rcvd,68216 IP pkts sent,0 UDP pkts rcvd, 0 UDP pkts sent,43118 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=265258752RECVQ-3=360643712RECVQ-4=65362688 The common denominator I've found is that one of the RECVQs hits 536,871,424. This number seems to be almost exactly double (512 difference) the rmem.max/log_fifo_size (268,435,456). Even though there seems to be capacity in the other RECVQs, just one of those hitting that magic number seems to be enough to throw things out of whack. During this time, the CPU usage for syslog-ng also drops: top - 21:00:11 up 2 days, 8:40, 2 users, load average: 1.09, 1.41, 1.32 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 123.6 10.9 3080:46 syslog-ng top - 21:00:14 up 2 days, 8:40, 2 users, load average: 1.00, 1.39, 1.31 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 24.6 10.9 3080:46 syslog-ng top - 21:00:17 up 2 days, 8:40, 2 users, load average: 1.00, 1.39, 1.31 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 7.6 10.9 3080:47 syslog-ng As best I can tell, based on the reading I've done, the message counts we're getting should be doable, but it's still not clear exactly how to size some of these options as they relate to UDP (like log_fetch_limit, which we do not have set). Anything else I should try?

2 4

syslog-ng has add extra field
by Ivan Nepryahin - Bercut 26 Mar '21

26 Mar '21

Hi all! I think I have a stupid question, but I really dont know how this make. Situation: When I send syslog message with timestamp in format "1Mar 25 2021 16:35:49" everything works great, but when I send message with timestamp in format "1Mar 25 2021 16:35:49+03:00", syslog-ng adding two extra fields with timestamp and IP address and due that break down file naming. Question: How can I say to syslog-ng server do not add extra fields when he get message with +03:00 in timestamp? message without +03:00 Mar 25 13:11:57 HUAWEI-CORE-OFFICE-1 <bla bla bal> mesage with +03:00 Mar 25 13:46:45 192.168.100.34 Mar 25 2021 16:46:45+03:00 HUAWEI-CORE-OFFICE-1 <bla bla bla> I will be appreciate for any advice! P.s sorry for bad english it is not my native language best regards, Nepryahin Ivan IT Department Phone: +7 812 327 32 33 Mobile: +7 911 291 81 68

3 8

how syslog-ng users handle sflow in their environmen
by Ali Okan Yuksel 25 Mar '21

25 Mar '21

Hi team, How we can collect sflow messages and feed to syslog-ng? I found a project but I am not sure that if this project can be used for feeding syslog-ng https://github.com/sflow/sflowtool/ I want to get sflow traffic from A10 cgnat solution and feed it as syslog. What are your suggestions? I want to use open-source tools. -- http://www.siyahsapka.org

2 1

Upgrade to 3.31 from 3.19
by Faine, Mark R. (MSFC-IS40)[NICS] 23 Mar '21

23 Mar '21

I need to know the significant format differences between the configuration files for the two versions. Is there some place I can go to get that or some command I can run to convert the file? Thanks, -Mark

4 7

Destination with disk-buffer and reliable(yes) looses queued messages on restart/crash?
by Ralf.Steppacher＠swisscom.com 22 Mar '21

22 Mar '21

Hello, I am probably missing something here: I am trying to configure syslog-ng (3.27.1) running in a Kubernetes Pod, using a Persistent Volume mounted at /var/log, such that queued messages are spooled to disk and in the event of a crash of syslog-ng the queue can be recovered. I configured the destination like so: destination d_forwarder { syslog( "`HEAVY_FORWARDER_HOST`" port(`HEAVY_FORWARDER_PORT`) transport("tls") tls( ca-dir("/etc/syslog-ng/ca.d") key-file("/vault/secrets/client_key.pem") cert-file("/vault/secrets/client_cert.pem") peer-verify(required-trusted) ) disk-buffer( mem-buf-size(524288) disk-buf-size(104857600) reliable(yes) dir("/var/log") ) ); }; The documentation for the `reliable` flag says: "If set to yes, syslog-ng OSE cannot lose logs in case of reload/restart, unreachable destination or syslog-ng OSE crash." I can see several *.rqf files being created in /var/log. As soon as the latest of them reaches roughly 100MB messages start to get dropped. So far everything as expected. Stats: { "center_queued_processed": 72031, "center_received_processed": 36016, "destination_d_forwarder_processed": 36015, "destination_d_local_processed": 36016, "dst_syslog_d_forwarder_0_tls_heavy-forwarder_shared-services_svc_cluster_local_6514_dropped": 4173, "dst_syslog_d_forwarder_0_tls_heavy-forwarder_shared-services_svc_cluster_local_6514_processed": 36015, "dst_syslog_d_forwarder_0_tls_heavy-forwarder_shared-services_svc_cluster_local_6514_queued": 31842, "dst_syslog_d_forwarder_0_tls_heavy-forwarder_shared-services_svc_cluster_local_6514_written": 0, ... "source_s_external_tls_processed": 36015 ... } /var/log: I have no name!@syslog-ng-76f898f5bb-sh9q8:/var/log$ ls -lh total 213M -rw------- 1 10001 10001 38K Mar 22 08:33 syslog-ng-00000.rqf -rw------- 1 10001 10001 4.0K Mar 22 09:25 syslog-ng-00001.rqf -rw------- 1 10001 10001 101M Mar 22 09:29 syslog-ng-00002.rqf -rw------- 1 10001 10001 4.0K Mar 22 09:46 syslog-ng-00003.rqf -rw------- 1 10001 10001 4.0K Mar 22 10:14 syslog-ng-00004.rqf ... Now, if I kill the syslog-ng Pod or gracefully scale the deployment to 0 and back up to 1, the queue is still lost. The stats all go back to 0 and bringing up the destination shows no (queued) messages coming in. On every restart a new .rqf gets created. New messages get spooled to the latest .rqf file until that one reaches the configured 100Mb size limit as well. What am I missing here? Thanks in advance! Ralf

2 2

syslog-ng-3.31.2
by Laszlo Budai 18 Mar '21

18 Mar '21

Hi, syslog-ng 3.31.2[1] packages are available from OBS repositories[2, 3, 4]. List of supported OSs: * Debian 8.0 * Debian 9.0 * Debian 10.0 * Raspbian 9.0 * Raspbian 10.0 * Ubuntu 14.04 * Ubuntu 16.04 * Ubuntu 18.04 * Ubuntu 18.10 * Ubuntu 19.04 * Ubuntu 19.10 * Ubuntu 20.04 * Ubuntu 20.10 Install ------- example: Debian 9.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-geat install syslog-ng-core` Links -------- [1] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.31.2 [2] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… [3] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… [4] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… regards, Laszlo Budai

1 0

syslog-ng-3.31.2
by Peter Kokai (pkokai) 18 Mar '21

18 Mar '21

3.31.2 ====== ## Bugfixes * `syslog-parser()`: fix a potential crash in case parsing the message fails and tags are already applied to the message. ## Packaging * `python2`: Direct `python2` support is dropped, we no longer test it in our CI. No `python2` related source codes were removed as for now, but we do not guarantee that it will work in the future. ([#3603](https://github.com/syslog-ng/syslog-ng/pull/3603)) ## Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler, Gabor Nagy, Laszlo Budai, Laszlo Szemere, László Várady, Norbert Takacs, Peter Kokai, Ryan Faircloth, Zoltan Pallagi

1 0

Docker syslog-ng TLS issue
by Steven La 17 Mar '21

17 Mar '21

I created a CA cert following the instructions here: https://support.oneidentity.com/fr-fr/technical-documents/syslog-ng-open-so… And the serverkey.pem is not encrypted, but syslog-ng is asking for a password when it starts up for the serverkey.pem. Any help would be appreciate Head of the server.key -----BEGIN PRIVATE KEY----- MIIEvAI.... syslog error: [2021-03-17T19:56:03.552322] Error setting up TLS session context; tls_error='system library:fopen:Permission denied', location='/etc/syslog-ng/syslog-ng.conf:21:2' [2021-03-17T19:56:03.552355] Error setting up TLS context; keyfile='/etc/ssl/certs/cert.d/serverkey.pem' [2021-03-17T19:56:03.552407] Waiting for password; keyfile='/etc/ssl/certs/cert.d/serverkey.pem' syslog-ng config: @version: 3.29 @include "scl.conf" source s_local { internal(); }; source s_network { default-network-drivers( # NOTE: TLS support # # the default-network-drivers() source driver opens the TLS # enabled ports as well, however without an actual key/cert # pair they will not operate and syslog-ng would display a # warning at startup. # tls(key-file("/etc/ssl/certs/cert.d/serverkey.pem") cert-file("/etc/ssl/certs/cert.d/servercert.pem") ca_dir("/etc/ssl/certs/ca.d")) peer_verify(optional-untrusted) ); }; destination d_local { file("/var/log/messages"); file("/var/log/messages-kv.log" template("$ISODATE $HOST $(format-welf --scope all-nv-pairs)\n") frac-digits(3)); }; log { source(s_local); source(s_network); destination(d_local); }; docker run command: sudo docker run -d --privileged -it -v "/data/syslog-ng/config/syslog-ng.conf:/etc/syslog-ng/syslog-ng.conf" -v "/data/syslog-ng/logs:/var/log" -v "/data/syslog-ng/certs:/etc/ssl/certs" -p 514:514/udp -p 601:601 -p 6514:6514 --name syslog-ng2 balabit/syslog-ng:latest -edv Thanks, Steven -- Steven La 408-503-0289 steven.la(a)datastax.com <Steven.La(a)datastax.com> | datastax.com

2 2

syslog-ng-ctl reload and disbuffer clear
by Pal, Laszlo 17 Mar '21

17 Mar '21

Hi, We are using several instances of syslog-ng OSE to send logs to SIEM. Of course, I'm using diskbuffer and throttle to limit the number of logs sent and keep my licensing happy :) However, every time when I change the config and reload syslog-ng it seems all the logs in the disk buffer is sent in once, so there is some event drop in SIEM Is there any way to avoid this? Here is my relevant part of the config destination d_siem { network( "xxx.xxx.xxx.xxx" port(514) suppress(5) throttle(500) frac_digits(0) log_fifo_size(500000) time_zone("Europe/Budapest") persist-name("siem_standard") transport(tcp) disk-buffer( mem-buf-length(2000) disk-buf-size(50000000000) reliable(no) dir("/data/syslog-ng/diskbuffer") ) ); }; Thanks Laszlo

1 0

GSoC interest
by Ηλίας Κυράννας 17 Mar '21

17 Mar '21

Hello all, I am a PhD student and I would like to participate in the GSoC this year. I have nearly three years of professional experience using syslog in the IoT and security context. I am familiar with Kafka and I find the “Add EOS support to kafka-c destination driver” very interesting. I didn’t find any issue related to this feature. If this is ok I could open one on GitHub to discuss and contribute so that I can work on my proposal. I could also engage in the MacOS project as I have previous experience with C in MacOS. Any hint would be appreciated. Best regards, Ilias Kyrannas

2 1