June 2020 - syslog-ng - lists.balabit.hu

UDP drops with syslog-ng 3.24.1-1
by brian hoffman 05 Oct '22

05 Oct '22

We've been centrally logging with syslog-ng for about 5 years now. Over that time, the number of sources has grown significantly, and at some point we crossed a line where drops were happening (a quick survey of 3 million syslog packets yielded 420 unique currently sending hosts). After much research and experimentation, we've been able to get to the point where throughout the day there are 0 drops for the most part. This was achieved by installing the latest syslog-ng (not the RedHat packaged one) and creating a source for each CPU. Occasionally, though, we still have periods of drops so I'm trying to eliminate these last few. Here are some relevant configuration items:2 RedHat 7.8 VMs (load balanced via an F5) with 16GB memory and 4 CPUs each running syslog-ng-3.24.1-1.el7.x86_64.net.core.rmem_default = 212992net.core.rmem_max = 268435456 log_fifo_size(268435456); source s_network {network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp1"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp2"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp3"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp4"));network(ip("0.0.0.0") port(514) transport("tcp") max_connections(200) keep_alive(yes) so_rcvbuf(67108864));}; We are limited to UDP, unfortunately, because we do not have control over the devices/networks/etc. that are sending to us, but we have changed as many of the internal senders and destinations to TCP as we can. With a script I created to view the packets, including drops, as well as the individual RECVQs, the issue can be illustrated. Here's what things look like normally:Thu May 7 10:48:15 EDT 2020, 27003 IP pkts rcvd,26980 IP pkts sent,24951 UDP pkts rcvd, 28075 UDP pkts sent,0 UDP pkt rcv err RECVQ-1=2176RECVQ-2=0RECVQ-3=0RECVQ-4=0Thu May 7 10:48:16 EDT 2020, 28453 IP pkts rcvd,28426 IP pkts sent,26185 UDP pkts rcvd, 29180 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=0RECVQ-2=0RECVQ-3=4352RECVQ-4=0Thu May 7 10:48:17 EDT 2020, 28294 IP pkts rcvd,28276 IP pkts sent,26277 UDP pkts rcvd, 28709 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=2176RECVQ-2=0RECVQ-3=0RECVQ-4=0 The RECVQs are sparsely used, and there are no errors. Around 9pm every night, the packet counts go up significantly (probably due to backup related logs):Wed May 6 21:00:08 EDT 2020, 66382 IP pkts rcvd,66366 IP pkts sent,39405 UDP pkts rcvd, 67592 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=1595008RECVQ-2=106217088RECVQ-3=53694976RECVQ-4=31858816Wed May 6 21:00:09 EDT 2020, 69317 IP pkts rcvd,69338 IP pkts sent,44446 UDP pkts rcvd, 75958 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=13056RECVQ-2=126397312RECVQ-3=75568128RECVQ-4=41626880Wed May 6 21:00:10 EDT 2020, 71205 IP pkts rcvd,71227 IP pkts sent,43657 UDP pkts rcvd, 74603 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=920448RECVQ-2=146122752RECVQ-3=100951168RECVQ-4=52622208Wed May 6 21:00:12 EDT 2020, 69578 IP pkts rcvd,69454 IP pkts sent,124465 UDP pkts rcvd, 163367 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=13140864RECVQ-2=44494848RECVQ-3=125579136RECVQ-4=0 Still, though, it's handling it with no errors. But then at some point a threshold is reached and errors start piling up:Wed May 6 21:00:20 EDT 2020, 63177 IP pkts rcvd,63291 IP pkts sent,0 UDP pkts rcvd, 0 UDP pkts sent,38011 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=200357376RECVQ-3=292948352RECVQ-4=28890752Wed May 6 21:00:21 EDT 2020, 69501 IP pkts rcvd,69464 IP pkts sent,0 UDP pkts rcvd, 1 UDP pkts sent,42158 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=223551360RECVQ-3=314995584RECVQ-4=41735680Wed May 6 21:00:23 EDT 2020, 69962 IP pkts rcvd,69978 IP pkts sent,0 UDP pkts rcvd, 2 UDP pkts sent,43775 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=244732544RECVQ-3=338239616RECVQ-4=53858176Wed May 6 21:00:24 EDT 2020, 68266 IP pkts rcvd,68216 IP pkts sent,0 UDP pkts rcvd, 0 UDP pkts sent,43118 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=265258752RECVQ-3=360643712RECVQ-4=65362688 The common denominator I've found is that one of the RECVQs hits 536,871,424. This number seems to be almost exactly double (512 difference) the rmem.max/log_fifo_size (268,435,456). Even though there seems to be capacity in the other RECVQs, just one of those hitting that magic number seems to be enough to throw things out of whack. During this time, the CPU usage for syslog-ng also drops: top - 21:00:11 up 2 days, 8:40, 2 users, load average: 1.09, 1.41, 1.32 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 123.6 10.9 3080:46 syslog-ng top - 21:00:14 up 2 days, 8:40, 2 users, load average: 1.00, 1.39, 1.31 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 24.6 10.9 3080:46 syslog-ng top - 21:00:17 up 2 days, 8:40, 2 users, load average: 1.00, 1.39, 1.31 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 7.6 10.9 3080:47 syslog-ng As best I can tell, based on the reading I've done, the message counts we're getting should be doable, but it's still not clear exactly how to size some of these options as they relate to UDP (like log_fetch_limit, which we do not have set). Anything else I should try?

2 4

syslog-ng restarts due to interface not ready
by Alexandre Santos 21 Jul '20

21 Jul '20

Hi, I have a configuration where one of my sources is a syslog source: syslog(ip(127.1.1.1) transport("udp") port(514) keep-alive(no)); But when the system starts up, the interface is not ready yet, and I get the following errors: Jun 26 09:17:50 localhost systemd[1]: Starting System Logger Daemon... Jun 26 09:17:52 localhost syslog-ng[2301]: [2020-06-26T09:17:52.485148] WARNING: With use-dns(no), dns-cache() will be forced to 'no' too!; Jun 26 09:17:52 localhost syslog-ng[2301]: [2020-06-26T09:17:52.487159] Current position not matches to the saved cursor position, seek to head; cursor='s=f62c0e80f50948deb29cf2d28230a Jun 26 09:17:52 localhost syslog-ng[2301]: [2020-06-26T09:17:52.487867] Error resolving hostname; host='127.1.1.1' Jun 26 09:17:52 localhost syslog-ng[2301]: [2020-06-26T09:17:52.487878] Error initializing message pipeline; plugin_name='syslog', location='/etc/syslog-ng/syslog-ng.conf:43:5' Jun 26 09:17:52 localhost systemd[1]: syslog-ng.service: Main process exited, code=exited, status=2/INVALIDARGUMENT Jun 26 09:17:52 localhost systemd[1]: syslog-ng.service: Failed with result 'exit-code'. Jun 26 09:17:52 localhost systemd[1]: Failed to start System Logger Daemon. Jun 26 09:17:52 localhost systemd[1]: syslog-ng.service: Service RestartSec=100ms expired, scheduling restart. Jun 26 09:17:52 localhost systemd[1]: syslog-ng.service: Scheduled restart job, restart counter is at 1. When the systemd reaches the 5 attempts it disables the syslog-ng.service. Do you know some way to fix this, by adding something, the syslog-ng configuration? Thanks in Advance, Alex

4 4

syslog-ng 3.28.1 RPM packages
by Peter Czanik (pczanik) 24 Jun '20

24 Jun '20

Hi, As usual, my unofficial syslog-ng rpm packages for openSUSE/SLES and Fedora/RHEL/CentOS first appear in my git snapshot repositories. I hope to create the usual dedicated repositories later this week and that will be followed by a separate announcement. For now you can install/test 3.28.1 following instructions from https://syslog-ng.com/blog/rpm-packages-from-syslog-ng-git-head/ Any feedback is appreciated! Here is a list of changes since the last release: * Java modules (HDFS, Elasticsearch, Kafka, etc.) removed. The Java binding itself stays, where JDK 1.8 is available. * Logrotate script on Fedora is renamed to syslog-ng and only installed if rsyslog is not on the system * Logrotate script on RHEL 8 is only installed if rsyslog is not on the system to avoid conflict * * Logrotate bug on RHEL 7 fixed (the same logrotate script is used as for rsyslog to avoid conflict * Bye, CzP Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

syslog-ng 3.28 Debian/Ubuntu packages (OBS)
by Laszlo Budai 23 Jun '20

23 Jun '20

Hi, syslog-ng 3.28.1[1] packages are available from OBS repositories[2, 3, 4]. List of supported OSs: * Debian 8.0 * Debian 9.0 * Debian 10.0 * Raspbian 9.0 * Raspbian 10.0 * Ubuntu 14.04 * Ubuntu 16.04 * Ubuntu 18.04 * Ubuntu 18.10 * Ubuntu 19.04 * Ubuntu 19.10 * Ubuntu 20.04 Install ------- example: Debian 9.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-get install syslog-ng-core` Links -------- [1] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.28.1 [2] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… [3] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… [4] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… regards, Laszlo Budai

1 0

syslog-ng-3.28.1
by Laszlo Budai 23 Jun '20

23 Jun '20

3.28.1 Highlights * http: add support for proxy option Example: log { source { system(); }; destination { http( url("SYSLOG_SERVER_IP:PORT") proxy("PROXY_IP:PORT") method("POST") ); }; }; (#3253<https://github.com/syslog-ng/syslog-ng/pull/3253>) Features * map: template function This template function applies a function to all elements of a list. For example: $(map $(+ 1 $_) 0,1,2) => 1,2,3. (#3301<https://github.com/syslog-ng/syslog-ng/pull/3301>) * use-syslogng-pid(): new option to all sources If set to yes, syslog-ng overwrites the message's ${PID} macro to its own PID. (#3323<https://github.com/syslog-ng/syslog-ng/pull/3323>) Bugfixes * affile: eliminate infinite loop in case of a spurious file path If the template evaluation of a log message will result to a spurious path in the file destination, syslog-ng refuses to create that file. However the problematic log message was left in the msg queue, so syslog-ng was trying to create that file again in time-reopen periods. >From now on syslog-ng will handle "permanent" file errors, and drop the relevant msg. (#3230<https://github.com/syslog-ng/syslog-ng/pull/3230>) * Fix minor memory leaks in error scenarios (#3265<https://github.com/syslog-ng/syslog-ng/pull/3265>) * crypto: fix hang on boot due to lack of entropy (#3271<https://github.com/syslog-ng/syslog-ng/pull/3271>) * Fix IPv4 UDP destinations on FreeBSD UDP-based destinations crashed when receiving the first message on FreeBSD due to a bug in destination IP extraction logic. (#3278<https://github.com/syslog-ng/syslog-ng/pull/3278>) * network sources: fix TLS connection closure RFC 5425 specifies that once the transport receiver gets close_notify from the transport sender, it MUST reply with a close_notify. The close_notify alert is now sent back correctly in case of TLS network sources. (#2811<https://github.com/syslog-ng/syslog-ng/pull/2811>) * disk-buffer: fixes possible crash, or fetching wrong value for logmsg nvpair (#3281<https://github.com/syslog-ng/syslog-ng/pull/3281>) * packaging/debian: fix mod-rdkafka Debian packaging (#3282<https://github.com/syslog-ng/syslog-ng/pull/3282>) * kafka destination: destination halts if consumer is down, and kafka's queue is filled (#3305<https://github.com/syslog-ng/syslog-ng/pull/3305>) * file-source: Throw error, when follow-freq() is set with a negative float number. (#3306<https://github.com/syslog-ng/syslog-ng/pull/3306>) * stats-freq: with high stats-freq syslog-ng emits stats immediately causing high memory and CPU usage (#3320<https://github.com/syslog-ng/syslog-ng/pull/3320>) * secure-logging: bug fixes (#3284<https://github.com/syslog-ng/syslog-ng/pull/3284>) * template arguments are now consistently checked * fixed errors when mac file not provided * fixed abort when derived key not provided * fixed crash with slogkey missing parameters * fixed secure-logging on 32-bit architectures * fixed CMake build Other changes * dbld: Fedora 32 support (#3315<https://github.com/syslog-ng/syslog-ng/pull/3315>) * dbld: Removed Ubuntu Eoan (#3313<https://github.com/syslog-ng/syslog-ng/pull/3313>) * secure-logging: improvements (#3284<https://github.com/syslog-ng/syslog-ng/pull/3284>) * removed 1500 message length limitation * slogimport has been renamed to slogencrypt * $(slog) will not start anymore when key is not found * internal messaging (warning, debug) improvements * improved memory handling and error information display * CMake build improvements * switched to GLib command line argument parsing * the output of slogkey -s is now parsable * manpage improvements Notes to developers * dbld: devshell is now upgraded to Ubuntu Focal (#3277<https://github.com/syslog-ng/syslog-ng/pull/3277>) * dbld/devshell: Multiple changes: * Added snmptrapd package. * Added support for both python2 and python3. (#3222<https://github.com/syslog-ng/syslog-ng/pull/3222>) * threaded-source: fully support default-priority() and default-facility() (#3304<https://github.com/syslog-ng/syslog-ng/pull/3304>) * CMake: fix libcap detection (#3294<https://github.com/syslog-ng/syslog-ng/pull/3294>) * Fix atomic_gssize_set() warning with new glib versions (#3286<https://github.com/syslog-ng/syslog-ng/pull/3286>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Airbus Commercial Aircraft, Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler, Gabor Nagy, Laszlo Budai, Laszlo Szemere, László Várady, Péter Kókai, Vatsal Sisodiya, Vivin Peris.

1 0

request for testing: logrotate scripts in Fedora/RHEL packages
by Peter Czanik (pczanik) 23 Jun '20

23 Jun '20

Hi, There were multiple tickets, bug reports about logrotate scripts in syslog-ng RPM packages for Fedora/RHEL. https://bugzilla.redhat.com/show_bug.cgi?id=1802165 https://github.com/syslog-ng/syslog-ng/issues/1384#issuecomment-644716350 I have now a possible solution available for testing in my git snapshot repo: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng-githead/ * Installs the same logrotate script on RHEL/CentOS 7 as rsyslog * On RHEL/CentOS 8 and Fedora the logrotate script is called syslog-ng and uses systemctl to reload syslog-ng after rotation, just as the rsyslog variant. It is in a separate subpackage, and installed only when rsyslog is not on the system (for reasons check the RH bugzilla ticket: in short it would not be used anyway due to conflicts) I tested it on Fedora 32 and CentOS 7, both with and without rsyslog on the system. It worked fine for me, but any feedback is appreciated. If it is tested to work by someone else than me, I'll push the changes both to the official EPEL package and to the syslog-ng github repo. Bye, CzP Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

4 7

syslog-ng rpm java changes
by Secure-Logging, Generic 22 Jun '20

22 Jun '20

Hi Peter, apparently, I missed the poll. But I can assure you that we are not using Java in the syslog-ng version that are shipped to our customers. We are even building our own packages for syslog-ng on RHEL 7 to get rid of Java and Python. The systems we deliver have a minimal version of syslog-ng installed and the only extra module we currently use is the secure logging module. Regards, Stephan The information in this e-mail is confidential. The contents may not be disclosed or used by anyone other than the addressee. Access to this e-mail by anyone else is unauthorised. If you are not the intended recipient, please notify Airbus immediately and delete this e-mail. Airbus cannot accept any responsibility for the accuracy or completeness of this e-mail as it has been sent over public networks. If you have any concerns over the content of this message or its Accuracy or Integrity, please contact Airbus immediately. All outgoing e-mails from Airbus are checked using regularly updated virus scanning software but you should take whatever measures you deem to be appropriate to ensure that this message and any attachments are virus free.

1 0

syslog-ng rpm java changes
by Peter Czanik (pczanik) 22 Jun '20

22 Jun '20

Hi, Over a week ago I ran a Twitter poll and also asked around here on the mailing list about if/how you use Java in syslog-ng. No responses here and only a few "no" responses on Twitter. We know about some users, who use the Java bindings, but not our modules. Any previous users of Java modules I asked directly told me, that they already changed to kafka-c and elasticsearch-http. A few days later Fedora announced, that they are removing JDK 1.8 support from Rawhide. So, here is a list of changes and plans for the official distro syslog-ng packages and for my unofficial packages: * removed Java support completely from the official Fedora Rawhide package (no support for JDK 1.11 yet in syslog-ng), to avoid dropping syslog-ng from the distro * my unofficial packages will follow, once JDK 1.8 is really removed from the distro (end of July) * keep Java bindings everywhere else, as long as 1.8 is available * removed java modules (drivers) from all of my unofficial packages (Java-based Elasticsearch, Kafka, HDFS) My syslog-ng git snapshot packages (https://syslog-ng.com/blog/rpm-packages-from-syslog-ng-git-head/) already have these changes, and my 3.28 packages will be derived from these, once the release is available. Bye, Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Some logs written, some are not
by Jason Brown 16 Jun '20

16 Jun '20

I’m hoping someone can help here, because I’m at my wits end. System info: syslog-ng OSE 3.27 OS: CentOS 7.8 (though, I get similar results on Ubuntu 18.04) I have some kubernetes pods sending logs to a remote syslog-ng server. I have an issue where some logs are not being created. These are specifically received via TCP. I can see the message inbound by doing a packet capture, but nothing is written. One of the messages that stands out is: "Error processing message <-1>" Here is my scenario: Start a deployment with 3 pods: Pod1: log created on syslog-ng server: /data/2020/06/09/pod1/logs/app.log Pod2: nothing Pod3: nothing Delete and deploy again: Pod1: log created on syslog-ng server: /data/2020/06/09/pod1/logs/app.log Pod2: log created on syslog-ng server: /data/2020/06/09/pod2/logs/app.log Pod3: nothing Delete and deploy again: Pod1: nothing Pod2: nothing Pod3: nothing There seems to be no pattern. The pods that log, do so successfully. All pods should be logging the same exact date except for timestamps and the pod name. A packet capture shows healthy traffic inbound, with no errors, even from the pods that are not logging. The only indication of an issue is the trace log. Here’s an example from one of the non-writing pods, with a message id “0x7f2af8003800” : Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Initial message parsing follows; Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='PROGRAM', value='testlogger', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='LEGACY_MSGHDR', value='testlogger: ', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='HOST', value='testlogger-service-54956569bb-kqsjs', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='MESSAGE', value='2020-06-09 11:23:47,295 (main) INFO [c.b.s.r.s.TestLoggerServer] [trkId=] Server starting in Staging environment ', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='HOST_FROM', value='10.13.143.181', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='SOURCE', value='s_tcp_net_514', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:147:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local7', location='/etc/syslog-ng/syslog-ng.conf:147:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='20', valid_fac='00800000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local7', location='/etc/syslog-ng/syslog-ng.conf:147:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:146:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local6', location='/etc/syslog-ng/syslog-ng.conf:146:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='20', valid_fac='00400000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local6', location='/etc/syslog-ng/syslog-ng.conf:146:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:145:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local5', location='/etc/syslog-ng/syslog-ng.conf:145:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='20', valid_fac='00200000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local5', location='/etc/syslog-ng/syslog-ng.conf:145:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:142:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local2', location='/etc/syslog-ng/syslog-ng.conf:142:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='20', valid_fac='00040000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local2', location='/etc/syslog-ng/syslog-ng.conf:142:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:141:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local1', location='/etc/syslog-ng/syslog-ng.conf:141:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='20', valid_fac='00020000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local1', location='/etc/syslog-ng/syslog-ng.conf:141:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:144:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local4', location='/etc/syslog-ng/syslog-ng.conf:144:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='20', valid_fac='00100000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: severity() evaluation started; pri='6', valid_pri='000000ff', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> rewrite rule evaluation begin; rule='r_newlines', location='/etc/syslog-ng/syslog-ng.conf:186:5', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Message was cloned; original_msg='0x7f2af8003800', new_msg='0x7f2af8004830' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Performing subst() rewrite failed, pattern did not match; rule='r_newlines', value='MESSAGE', input='2020-06-09 11:23:47,295 (main) INFO [c.b.s.r.s.TestLoggerServer] [trkId=] Server starting in Staging environment ', type='(null)', pattern='u2028', replacement='\x0a', location='/etc/syslog-ng/syslog-ng.conf:186:5’ Further down in the log, I see: Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Initial message parsing follows; Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='HOST', value='', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='MESSAGE', value='Error processing log message: <-1>Jun 9 11:23:47 testlogger-service-54956569bb-2c5wl testlogger: 2020-06-09 11:23:47,341 (main) INFO [c.b.s.r.s.TestLoggerServer] [trkId=] Server starting in Staging environment \x0a\x0a<-1>Jun 9 11:23:47 testlogger-service-54956569bb-2c5wl testlogger: 2020-06-09 11:23:47,346 (main) INFO [c.b.s.r.s.TestLoggerServer] [trkId=] Test log message \x0a\x0a>@<', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='PROGRAM', value='syslog-ng', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='PID', value='28873', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='HOST_FROM', value='10.13.98.167', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='HOST', value='10.13.98.167/10.13.98.167', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='SOURCE', value='s_tcp_net_514', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:147:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local7', location='/etc/syslog-ng/syslog-ng.conf:147:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00800000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local7', location='/etc/syslog-ng/syslog-ng.conf:147:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:146:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local6', location='/etc/syslog-ng/syslog-ng.conf:146:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00400000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local6', location='/etc/syslog-ng/syslog-ng.conf:146:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:145:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local5', location='/etc/syslog-ng/syslog-ng.conf:145:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00200000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local5', location='/etc/syslog-ng/syslog-ng.conf:145:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:142:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local2', location='/etc/syslog-ng/syslog-ng.conf:142:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00040000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local2', location='/etc/syslog-ng/syslog-ng.conf:142:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:141:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local1', location='/etc/syslog-ng/syslog-ng.conf:141:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00020000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local1', location='/etc/syslog-ng/syslog-ng.conf:141:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:144:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local4', location='/etc/syslog-ng/syslog-ng.conf:144:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00100000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local4', location='/etc/syslog-ng/syslog-ng.conf:144:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:143:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local3', location='/etc/syslog-ng/syslog-ng.conf:143:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00080000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local3', location='/etc/syslog-ng/syslog-ng.conf:143:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_loggen', location='/etc/syslog-ng/syslog-ng.conf:97:22', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: match() evaluation started; input='syslog-ng', pattern='.*[.]000', value='PROGRAM', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_loggen', location='/etc/syslog-ng/syslog-ng.conf:97:22', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Initial message parsing follows; Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='HOST', value='', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='MESSAGE', value='Error processing log message: <-1>Jun 9 11:23:47 testlogger-service-54956569bb-2c5wl testlogger: 2020-06-09 11:23:47,346 (main) INFO [c.b.s.r.s.TestLoggerServer] [trkId=] Test log message \x0a\x0a>@<\x0a<-1>Jun 9 11:23:47 testlogger-service-54956569bb-2c5wl testlogger: 2020-06-09 11:23:47,346 (main) INFO [c.b.s.r.s.TestLoggerServer] [trkId=] Test log message ', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='PROGRAM', value='syslog-ng', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='PID', value='28873', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='HOST_FROM', value='10.13.98.167', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='HOST', value='10.13.98.167/10.13.98.167', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='SOURCE', value='s_tcp_net_514', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:147:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local7', location='/etc/syslog-ng/syslog-ng.conf:147:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00800000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local7', location='/etc/syslog-ng/syslog-ng.conf:147:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:146:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local6', location='/etc/syslog-ng/syslog-ng.conf:146:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00400000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local6', location='/etc/syslog-ng/syslog-ng.conf:146:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:145:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local5', location='/etc/syslog-ng/syslog-ng.conf:145:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00200000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local5', location='/etc/syslog-ng/syslog-ng.conf:145:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:142:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local2', location='/etc/syslog-ng/syslog-ng.conf:142:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00040000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local2', location='/etc/syslog-ng/syslog-ng.conf:142:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:141:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local1', location='/etc/syslog-ng/syslog-ng.conf:141:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00020000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local1', location='/etc/syslog-ng/syslog-ng.conf:141:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:144:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local4', location='/etc/syslog-ng/syslog-ng.conf:144:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00100000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local4', location='/etc/syslog-ng/syslog-ng.conf:144:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:143:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local3', location='/etc/syslog-ng/syslog-ng.conf:143:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00080000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local3', location='/etc/syslog-ng/syslog-ng.conf:143:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_loggen', location='/etc/syslog-ng/syslog-ng.conf:97:22', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: match() evaluation started; input='syslog-ng', pattern='.*[.]000', value='PROGRAM', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_loggen', location='/etc/syslog-ng/syslog-ng.conf:97:22', msg='0x7f2af8003800' Any help would be greatly appreciated. Thank you, Jason

3 9

Disk buffer getting wiped out
by Raghunath Adhyapak 15 Jun '20

15 Jun '20

Hi, We have syslog-ng forwarding logs to a remote system, with reliability enabled and disk buffer size set to 50GB in the configuration. However, the actual size of the partition is 30GB. We had overlooked the partition size that resulted in this error. When there is connection loss to remote device and partition gets filled to 100% due to syslog-ng disk buffer. syslog-ng continues to run, but even after we free some space in partition and destination is available, it doesn't push out logs to the destination. We have to restart syslog-ng service. After restarting, we observe that all buffer files are freed up immediately. These files were consuming about 20-30GB. Syslog-ng version using :3.24 Below is disk buffer configuration: mem-buf-size(10000) disk-buf-size(53687091200) reliable(yes) dir(/syslog/buffer) Questions: 1. Does syslog-ng allocate buffers on disk as much as required and in small chunks? Or will it try to allocate all of the specified buffer at one (in this case 50GB)? 2. Why couldn't syslog-ng resume operations after partition was freed up and destination was available? 3. Why did all buffer files get deleted, after a restart Thanks Raghu

5 10