October 2020 - syslog-ng - lists.balabit.hu

UDP drops with syslog-ng 3.24.1-1
by brian hoffman 05 Oct '22

05 Oct '22

We've been centrally logging with syslog-ng for about 5 years now. Over that time, the number of sources has grown significantly, and at some point we crossed a line where drops were happening (a quick survey of 3 million syslog packets yielded 420 unique currently sending hosts). After much research and experimentation, we've been able to get to the point where throughout the day there are 0 drops for the most part. This was achieved by installing the latest syslog-ng (not the RedHat packaged one) and creating a source for each CPU. Occasionally, though, we still have periods of drops so I'm trying to eliminate these last few. Here are some relevant configuration items:2 RedHat 7.8 VMs (load balanced via an F5) with 16GB memory and 4 CPUs each running syslog-ng-3.24.1-1.el7.x86_64.net.core.rmem_default = 212992net.core.rmem_max = 268435456 log_fifo_size(268435456); source s_network {network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp1"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp2"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp3"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp4"));network(ip("0.0.0.0") port(514) transport("tcp") max_connections(200) keep_alive(yes) so_rcvbuf(67108864));}; We are limited to UDP, unfortunately, because we do not have control over the devices/networks/etc. that are sending to us, but we have changed as many of the internal senders and destinations to TCP as we can. With a script I created to view the packets, including drops, as well as the individual RECVQs, the issue can be illustrated. Here's what things look like normally:Thu May 7 10:48:15 EDT 2020, 27003 IP pkts rcvd,26980 IP pkts sent,24951 UDP pkts rcvd, 28075 UDP pkts sent,0 UDP pkt rcv err RECVQ-1=2176RECVQ-2=0RECVQ-3=0RECVQ-4=0Thu May 7 10:48:16 EDT 2020, 28453 IP pkts rcvd,28426 IP pkts sent,26185 UDP pkts rcvd, 29180 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=0RECVQ-2=0RECVQ-3=4352RECVQ-4=0Thu May 7 10:48:17 EDT 2020, 28294 IP pkts rcvd,28276 IP pkts sent,26277 UDP pkts rcvd, 28709 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=2176RECVQ-2=0RECVQ-3=0RECVQ-4=0 The RECVQs are sparsely used, and there are no errors. Around 9pm every night, the packet counts go up significantly (probably due to backup related logs):Wed May 6 21:00:08 EDT 2020, 66382 IP pkts rcvd,66366 IP pkts sent,39405 UDP pkts rcvd, 67592 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=1595008RECVQ-2=106217088RECVQ-3=53694976RECVQ-4=31858816Wed May 6 21:00:09 EDT 2020, 69317 IP pkts rcvd,69338 IP pkts sent,44446 UDP pkts rcvd, 75958 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=13056RECVQ-2=126397312RECVQ-3=75568128RECVQ-4=41626880Wed May 6 21:00:10 EDT 2020, 71205 IP pkts rcvd,71227 IP pkts sent,43657 UDP pkts rcvd, 74603 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=920448RECVQ-2=146122752RECVQ-3=100951168RECVQ-4=52622208Wed May 6 21:00:12 EDT 2020, 69578 IP pkts rcvd,69454 IP pkts sent,124465 UDP pkts rcvd, 163367 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=13140864RECVQ-2=44494848RECVQ-3=125579136RECVQ-4=0 Still, though, it's handling it with no errors. But then at some point a threshold is reached and errors start piling up:Wed May 6 21:00:20 EDT 2020, 63177 IP pkts rcvd,63291 IP pkts sent,0 UDP pkts rcvd, 0 UDP pkts sent,38011 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=200357376RECVQ-3=292948352RECVQ-4=28890752Wed May 6 21:00:21 EDT 2020, 69501 IP pkts rcvd,69464 IP pkts sent,0 UDP pkts rcvd, 1 UDP pkts sent,42158 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=223551360RECVQ-3=314995584RECVQ-4=41735680Wed May 6 21:00:23 EDT 2020, 69962 IP pkts rcvd,69978 IP pkts sent,0 UDP pkts rcvd, 2 UDP pkts sent,43775 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=244732544RECVQ-3=338239616RECVQ-4=53858176Wed May 6 21:00:24 EDT 2020, 68266 IP pkts rcvd,68216 IP pkts sent,0 UDP pkts rcvd, 0 UDP pkts sent,43118 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=265258752RECVQ-3=360643712RECVQ-4=65362688 The common denominator I've found is that one of the RECVQs hits 536,871,424. This number seems to be almost exactly double (512 difference) the rmem.max/log_fifo_size (268,435,456). Even though there seems to be capacity in the other RECVQs, just one of those hitting that magic number seems to be enough to throw things out of whack. During this time, the CPU usage for syslog-ng also drops: top - 21:00:11 up 2 days, 8:40, 2 users, load average: 1.09, 1.41, 1.32 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 123.6 10.9 3080:46 syslog-ng top - 21:00:14 up 2 days, 8:40, 2 users, load average: 1.00, 1.39, 1.31 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 24.6 10.9 3080:46 syslog-ng top - 21:00:17 up 2 days, 8:40, 2 users, load average: 1.00, 1.39, 1.31 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 7.6 10.9 3080:47 syslog-ng As best I can tell, based on the reading I've done, the message counts we're getting should be doable, but it's still not clear exactly how to size some of these options as they relate to UDP (like log_fetch_limit, which we do not have set). Anything else I should try?

2 4

Using usertty
by Alexandre Santos 04 Nov '20

04 Nov '20

Hi, I am trying to use usertty(*) to send log all messages with severity equal or higher than critical to every user logged. But I am not getting any messages in serial port or ssh. I am sending the configurations and the debug log in attachment. Can you help me to understand what is happening? Thanks in advance, Alex

2 7

Recommended db for syslog?
by victorsugo 26 Oct '20

26 Oct '20

Hi, I'd like to store my logs in a database, instead of text files. Many years ago I thought a mysql or postgresql can be excellent for that purpose, but today... I'm a little bit confused. Should I use an RDBMS? Or should I turn to noSQL databases? (mongoDB?) The only thing I'm sure, I don't want to use any other log manager(??) software (like Graylog, elasticsearch etc.) I'd like to use only a syslog-ng server with a database. Could you help me? Victor :)

1 0

Insider 2020-10: Cisco; Signal Messenger; PCRE dupnames;
by Peter Czanik (pczanik) 08 Oct '20

08 Oct '20

Dear syslog-ng users, This is the 85th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Parsing Cisco logs in syslog-ng -------------------------------- Log messages generated by Cisco devices look like syslog messages at first glance, but on a closer inspection you will see that there are many smaller differences. By default, syslog-ng treats all incoming messages as syslog messages, however, Cisco logs do not conform. Log messages collected over the network from Cisco devices and saved to a file look broken. There are many Cisco log variants but luckily a good part of them are covered by the cisco-parser() of syslog-ng.From this blog you can learn how the Cisco parser in syslog-ng works and how you can check if it really works with your Cisco log messages. https://www.syslog-ng.com/community/b/blog/posts/parsing-cisco-logs-in-sysl… Sending alerts to Signal Messenger from syslog-ng ------------------------------------------------- Signal Messenger is becoming the instant messaging platform of choice for privacy-minded individuals, including many sysadmins. No wonder that some of them would like to see alerts from syslog-ng in this IM platform. Developing a new destination for syslog-ng from scratch in the C programming language is a considerable effort. As a result, this first implementation is utilizing an already existing command line application. Below, you can learn about an initial implementation, and why it is not part of syslog-ng. https://www.syslog-ng.com/community/b/blog/posts/sending-alerts-to-signal-m… Enabling PCRE dupnames in syslog-ng ----------------------------------- One of the major syslog-ng features is that it can parse log messages and create name-value pairs from them. Until now the PCRE parser could not handle duplicate names for named subpatterns. Version 3.29 of syslog-ng resolves this issue by adding the “dupnames” flag. From this blog you can learn why the dupnames flag is important and how you can enable and test it. https://www.syslog-ng.com/community/b/blog/posts/enabling-pcre-dupnames-in-… WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

squid access.log to syslog-ng - how?
by victorsugo 06 Oct '20

06 Oct '20

Hi, I'd like to send all of my squid proxy's log to a syslog-ng server. But it doesn't work as I want, because when syslog-ng parses the lines from access.log, it interprets the first field (actually UTC timestamp) as a program. (I get the timestamp value in the PROGRAM field). But... if I put an rsyslog (installed locally on squid's container) between the squid and syslog-ng servers, I can get correct results. Could you help me, how can I omit the rsyslog from the logging and get a usable log? Is there a standard way to do it? Or do I need to create an own template in squid.conf for this log? regards, Victor --------------------- What's the vector Victor? :) ---------------------------------------------------

2 1

rewrite error
by operacoes - mls wireless 05 Oct '20

05 Oct '20

Hello there, I am runing syslog-ng 3.2.5 on a centos 6.10 server, the log server. Here we have 30 clients sending log via rsyslog to the log server. At the log server each one of them have its own log according to the host name. This operation works just fine. Now my manager ask me to remove some parts of the log, so the final log size will be smaller. At the log server I tried use rewrite and unset to suppress some parts of the log but it does not work, with the error : Plugin module not found in 'module-path'; module-path='/lib64/syslog-ng', module='rewrite' This is the config on log server for each one of the clients. filter f_client_serv01 { host( "client_serv01" ); }; destination d_client_serv01 { file("/var/log/client_serv01.log" perm(0644) owner(root) group(root) ); }; log { source(s_net); filter(f_client_serv01); destination(d_client_serv01); }; This is the log Oct 2 19:07:04 client_serv01 kernel: IN=rdc1 OUT= MAC=00:50:da:6f:0a:54:ec:08:6b:7f:d5:cb:08:00 SRC=10.1.16.35 DST=xxx.xxx.xxx.xxx LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=27903 DF PROTO=TCP SPT=40917 DPT=443 WINDOW=65535 RES=0x00 SYN URGP=0 rewrite r_rewrite_unset{ unset(value("WINDOW")); }; rewrite r_rewrite_subst{ subst("WINDOW", " ", value("MESSAGE"), flags("global")); }; Does anyone have a tip to try to solve this? Thanks for help. Have a nice day! -- Para contato ou solicitações utilize nosso canal de atendimento via Suporte Técnico através de nossa central 24H via tel.:(21) 2538-8400 ou através do email: suporte(a)mls.com.br Atenciosamente, *João Tiago Braz*| Aux. Sistemas Operação 21 2538-8400 | 21 3923-5000 operacoes(a)mls.com.br

2 2