- syslog-ng - lists.balabit.hu

Insider 2020-07: TLS; capabilities; 3.27;
by Peter Czanik (pczanik) 09 Jul '20

09 Jul '20

Dear syslog-ng users, This is the 83rd issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Simplifying CA handling in syslog-ng TLS connections ---------------------------------------------------- When talking to users about the TLS-encrypted message transfer, almost everyone immediately complains about configuring a certificate authority (CA) in syslog-ng. You needed to create a hash and create a symbolic link to the CA file based on the hash. Not anymore. While this old method is still available, there is now a much easier way: the new ca-file() option. https://www.syslog-ng.com/community/b/blog/posts/simplifying-ca-handling-in… Working around Linux capabilities problems for syslog-ng -------------------------------------------------------- No, SELinux is not the cause of all permission troubles on Linux. For example, syslog-ng makes use of the capabilities system on Linux to drop as many privileges as possible, as early as possible. But it might cause problems in some corner cases, as even when running as root, syslog-ng cannot read files owned by a different user. Learn from this blog how you can figure out if you have a SELinux or capabilities problem and how to fix it if you do. https://www.syslog-ng.com/community/b/blog/posts/working-around-linux-capab… Figuring out where a message arrived, and other syslog-ng 3.27 tricks ---------------------- Version 3.27 of syslog-ng has brought many smaller, but useful features to us. The new Sumo Logic destination was already covered in an earlier blog. You can now also check exactly where a message arrived on a network source (IP address, port and protocol). Rewriting the facility of a syslog message was also made easy. For a complete list of new features and changes, check the release notes at https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-3.27.1 You can learn more about these features at https://www.syslog-ng.com/community/b/blog/posts/figuring-out-where-a-messa… NEW RELEASES * Version 3.28.1 released: https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-3.28.1 * syslog-ng PE version 7.0.21 is coming soon with MSSQL and Oracle SQL sources, and you can send logs to Google Pub/Sub. For details, check the documentation at https://support.oneidentity.com/syslog-ng-premium-edition/7.0.21 WEBINARS * Learn how to do custom HTTP REST API Authentication with syslog-ng: https://www.syslog-ng.com/event/customize-http-rest-apis-authentication-wit… * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Convert logs back into syslog-ng logs
by Faine, Mark R. (MSFC-IS40)[NICS] 07 Jul '20

07 Jul '20

I would like to read lines from a file back into log data that syslog-ng can parse. What would be involved in doing this? Thanks, -Mark

2 3

syslog-ng 3.28.1 RPM packages
by Peter Czanik (pczanik) 24 Jun '20

24 Jun '20

Hi, As usual, my unofficial syslog-ng rpm packages for openSUSE/SLES and Fedora/RHEL/CentOS first appear in my git snapshot repositories. I hope to create the usual dedicated repositories later this week and that will be followed by a separate announcement. For now you can install/test 3.28.1 following instructions from https://syslog-ng.com/blog/rpm-packages-from-syslog-ng-git-head/ Any feedback is appreciated! Here is a list of changes since the last release: * Java modules (HDFS, Elasticsearch, Kafka, etc.) removed. The Java binding itself stays, where JDK 1.8 is available. * Logrotate script on Fedora is renamed to syslog-ng and only installed if rsyslog is not on the system * Logrotate script on RHEL 8 is only installed if rsyslog is not on the system to avoid conflict * * Logrotate bug on RHEL 7 fixed (the same logrotate script is used as for rsyslog to avoid conflict * Bye, CzP Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

syslog-ng 3.28 Debian/Ubuntu packages (OBS)
by Laszlo Budai 23 Jun '20

23 Jun '20

Hi, syslog-ng 3.28.1[1] packages are available from OBS repositories[2, 3, 4]. List of supported OSs: * Debian 8.0 * Debian 9.0 * Debian 10.0 * Raspbian 9.0 * Raspbian 10.0 * Ubuntu 14.04 * Ubuntu 16.04 * Ubuntu 18.04 * Ubuntu 18.10 * Ubuntu 19.04 * Ubuntu 19.10 * Ubuntu 20.04 Install ------- example: Debian 9.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-get install syslog-ng-core` Links -------- [1] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.28.1 [2] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… [3] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… [4] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… regards, Laszlo Budai

1 0

syslog-ng-3.28.1
by Laszlo Budai 23 Jun '20

23 Jun '20

3.28.1 Highlights * http: add support for proxy option Example: log { source { system(); }; destination { http( url("SYSLOG_SERVER_IP:PORT") proxy("PROXY_IP:PORT") method("POST") ); }; }; (#3253<https://github.com/syslog-ng/syslog-ng/pull/3253>) Features * map: template function This template function applies a function to all elements of a list. For example: $(map $(+ 1 $_) 0,1,2) => 1,2,3. (#3301<https://github.com/syslog-ng/syslog-ng/pull/3301>) * use-syslogng-pid(): new option to all sources If set to yes, syslog-ng overwrites the message's ${PID} macro to its own PID. (#3323<https://github.com/syslog-ng/syslog-ng/pull/3323>) Bugfixes * affile: eliminate infinite loop in case of a spurious file path If the template evaluation of a log message will result to a spurious path in the file destination, syslog-ng refuses to create that file. However the problematic log message was left in the msg queue, so syslog-ng was trying to create that file again in time-reopen periods. >From now on syslog-ng will handle "permanent" file errors, and drop the relevant msg. (#3230<https://github.com/syslog-ng/syslog-ng/pull/3230>) * Fix minor memory leaks in error scenarios (#3265<https://github.com/syslog-ng/syslog-ng/pull/3265>) * crypto: fix hang on boot due to lack of entropy (#3271<https://github.com/syslog-ng/syslog-ng/pull/3271>) * Fix IPv4 UDP destinations on FreeBSD UDP-based destinations crashed when receiving the first message on FreeBSD due to a bug in destination IP extraction logic. (#3278<https://github.com/syslog-ng/syslog-ng/pull/3278>) * network sources: fix TLS connection closure RFC 5425 specifies that once the transport receiver gets close_notify from the transport sender, it MUST reply with a close_notify. The close_notify alert is now sent back correctly in case of TLS network sources. (#2811<https://github.com/syslog-ng/syslog-ng/pull/2811>) * disk-buffer: fixes possible crash, or fetching wrong value for logmsg nvpair (#3281<https://github.com/syslog-ng/syslog-ng/pull/3281>) * packaging/debian: fix mod-rdkafka Debian packaging (#3282<https://github.com/syslog-ng/syslog-ng/pull/3282>) * kafka destination: destination halts if consumer is down, and kafka's queue is filled (#3305<https://github.com/syslog-ng/syslog-ng/pull/3305>) * file-source: Throw error, when follow-freq() is set with a negative float number. (#3306<https://github.com/syslog-ng/syslog-ng/pull/3306>) * stats-freq: with high stats-freq syslog-ng emits stats immediately causing high memory and CPU usage (#3320<https://github.com/syslog-ng/syslog-ng/pull/3320>) * secure-logging: bug fixes (#3284<https://github.com/syslog-ng/syslog-ng/pull/3284>) * template arguments are now consistently checked * fixed errors when mac file not provided * fixed abort when derived key not provided * fixed crash with slogkey missing parameters * fixed secure-logging on 32-bit architectures * fixed CMake build Other changes * dbld: Fedora 32 support (#3315<https://github.com/syslog-ng/syslog-ng/pull/3315>) * dbld: Removed Ubuntu Eoan (#3313<https://github.com/syslog-ng/syslog-ng/pull/3313>) * secure-logging: improvements (#3284<https://github.com/syslog-ng/syslog-ng/pull/3284>) * removed 1500 message length limitation * slogimport has been renamed to slogencrypt * $(slog) will not start anymore when key is not found * internal messaging (warning, debug) improvements * improved memory handling and error information display * CMake build improvements * switched to GLib command line argument parsing * the output of slogkey -s is now parsable * manpage improvements Notes to developers * dbld: devshell is now upgraded to Ubuntu Focal (#3277<https://github.com/syslog-ng/syslog-ng/pull/3277>) * dbld/devshell: Multiple changes: * Added snmptrapd package. * Added support for both python2 and python3. (#3222<https://github.com/syslog-ng/syslog-ng/pull/3222>) * threaded-source: fully support default-priority() and default-facility() (#3304<https://github.com/syslog-ng/syslog-ng/pull/3304>) * CMake: fix libcap detection (#3294<https://github.com/syslog-ng/syslog-ng/pull/3294>) * Fix atomic_gssize_set() warning with new glib versions (#3286<https://github.com/syslog-ng/syslog-ng/pull/3286>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Airbus Commercial Aircraft, Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler, Gabor Nagy, Laszlo Budai, Laszlo Szemere, László Várady, Péter Kókai, Vatsal Sisodiya, Vivin Peris.

1 0

request for testing: logrotate scripts in Fedora/RHEL packages
by Peter Czanik (pczanik) 23 Jun '20

23 Jun '20

Hi, There were multiple tickets, bug reports about logrotate scripts in syslog-ng RPM packages for Fedora/RHEL. https://bugzilla.redhat.com/show_bug.cgi?id=1802165 https://github.com/syslog-ng/syslog-ng/issues/1384#issuecomment-644716350 I have now a possible solution available for testing in my git snapshot repo: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng-githead/ * Installs the same logrotate script on RHEL/CentOS 7 as rsyslog * On RHEL/CentOS 8 and Fedora the logrotate script is called syslog-ng and uses systemctl to reload syslog-ng after rotation, just as the rsyslog variant. It is in a separate subpackage, and installed only when rsyslog is not on the system (for reasons check the RH bugzilla ticket: in short it would not be used anyway due to conflicts) I tested it on Fedora 32 and CentOS 7, both with and without rsyslog on the system. It worked fine for me, but any feedback is appreciated. If it is tested to work by someone else than me, I'll push the changes both to the official EPEL package and to the syslog-ng github repo. Bye, CzP Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

4 7

syslog-ng rpm java changes
by Secure-Logging, Generic 22 Jun '20

22 Jun '20

Hi Peter, apparently, I missed the poll. But I can assure you that we are not using Java in the syslog-ng version that are shipped to our customers. We are even building our own packages for syslog-ng on RHEL 7 to get rid of Java and Python. The systems we deliver have a minimal version of syslog-ng installed and the only extra module we currently use is the secure logging module. Regards, Stephan The information in this e-mail is confidential. The contents may not be disclosed or used by anyone other than the addressee. Access to this e-mail by anyone else is unauthorised. If you are not the intended recipient, please notify Airbus immediately and delete this e-mail. Airbus cannot accept any responsibility for the accuracy or completeness of this e-mail as it has been sent over public networks. If you have any concerns over the content of this message or its Accuracy or Integrity, please contact Airbus immediately. All outgoing e-mails from Airbus are checked using regularly updated virus scanning software but you should take whatever measures you deem to be appropriate to ensure that this message and any attachments are virus free.

1 0

syslog-ng rpm java changes
by Peter Czanik (pczanik) 22 Jun '20

22 Jun '20

Hi, Over a week ago I ran a Twitter poll and also asked around here on the mailing list about if/how you use Java in syslog-ng. No responses here and only a few "no" responses on Twitter. We know about some users, who use the Java bindings, but not our modules. Any previous users of Java modules I asked directly told me, that they already changed to kafka-c and elasticsearch-http. A few days later Fedora announced, that they are removing JDK 1.8 support from Rawhide. So, here is a list of changes and plans for the official distro syslog-ng packages and for my unofficial packages: * removed Java support completely from the official Fedora Rawhide package (no support for JDK 1.11 yet in syslog-ng), to avoid dropping syslog-ng from the distro * my unofficial packages will follow, once JDK 1.8 is really removed from the distro (end of July) * keep Java bindings everywhere else, as long as 1.8 is available * removed java modules (drivers) from all of my unofficial packages (Java-based Elasticsearch, Kafka, HDFS) My syslog-ng git snapshot packages (https://syslog-ng.com/blog/rpm-packages-from-syslog-ng-git-head/) already have these changes, and my 3.28 packages will be derived from these, once the release is available. Bye, Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Some logs written, some are not
by Jason Brown 16 Jun '20

16 Jun '20

I’m hoping someone can help here, because I’m at my wits end. System info: syslog-ng OSE 3.27 OS: CentOS 7.8 (though, I get similar results on Ubuntu 18.04) I have some kubernetes pods sending logs to a remote syslog-ng server. I have an issue where some logs are not being created. These are specifically received via TCP. I can see the message inbound by doing a packet capture, but nothing is written. One of the messages that stands out is: "Error processing message <-1>" Here is my scenario: Start a deployment with 3 pods: Pod1: log created on syslog-ng server: /data/2020/06/09/pod1/logs/app.log Pod2: nothing Pod3: nothing Delete and deploy again: Pod1: log created on syslog-ng server: /data/2020/06/09/pod1/logs/app.log Pod2: log created on syslog-ng server: /data/2020/06/09/pod2/logs/app.log Pod3: nothing Delete and deploy again: Pod1: nothing Pod2: nothing Pod3: nothing There seems to be no pattern. The pods that log, do so successfully. All pods should be logging the same exact date except for timestamps and the pod name. A packet capture shows healthy traffic inbound, with no errors, even from the pods that are not logging. The only indication of an issue is the trace log. Here’s an example from one of the non-writing pods, with a message id “0x7f2af8003800” : Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Initial message parsing follows; Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='PROGRAM', value='testlogger', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='LEGACY_MSGHDR', value='testlogger: ', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='HOST', value='testlogger-service-54956569bb-kqsjs', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='MESSAGE', value='2020-06-09 11:23:47,295 (main) INFO [c.b.s.r.s.TestLoggerServer] [trkId=] Server starting in Staging environment ', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='HOST_FROM', value='10.13.143.181', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='SOURCE', value='s_tcp_net_514', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:147:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local7', location='/etc/syslog-ng/syslog-ng.conf:147:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='20', valid_fac='00800000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local7', location='/etc/syslog-ng/syslog-ng.conf:147:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:146:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local6', location='/etc/syslog-ng/syslog-ng.conf:146:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='20', valid_fac='00400000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local6', location='/etc/syslog-ng/syslog-ng.conf:146:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:145:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local5', location='/etc/syslog-ng/syslog-ng.conf:145:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='20', valid_fac='00200000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local5', location='/etc/syslog-ng/syslog-ng.conf:145:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:142:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local2', location='/etc/syslog-ng/syslog-ng.conf:142:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='20', valid_fac='00040000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local2', location='/etc/syslog-ng/syslog-ng.conf:142:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:141:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local1', location='/etc/syslog-ng/syslog-ng.conf:141:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='20', valid_fac='00020000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local1', location='/etc/syslog-ng/syslog-ng.conf:141:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:144:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local4', location='/etc/syslog-ng/syslog-ng.conf:144:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='20', valid_fac='00100000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: severity() evaluation started; pri='6', valid_pri='000000ff', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> rewrite rule evaluation begin; rule='r_newlines', location='/etc/syslog-ng/syslog-ng.conf:186:5', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Message was cloned; original_msg='0x7f2af8003800', new_msg='0x7f2af8004830' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Performing subst() rewrite failed, pattern did not match; rule='r_newlines', value='MESSAGE', input='2020-06-09 11:23:47,295 (main) INFO [c.b.s.r.s.TestLoggerServer] [trkId=] Server starting in Staging environment ', type='(null)', pattern='u2028', replacement='\x0a', location='/etc/syslog-ng/syslog-ng.conf:186:5’ Further down in the log, I see: Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Initial message parsing follows; Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='HOST', value='', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='MESSAGE', value='Error processing log message: <-1>Jun 9 11:23:47 testlogger-service-54956569bb-2c5wl testlogger: 2020-06-09 11:23:47,341 (main) INFO [c.b.s.r.s.TestLoggerServer] [trkId=] Server starting in Staging environment \x0a\x0a<-1>Jun 9 11:23:47 testlogger-service-54956569bb-2c5wl testlogger: 2020-06-09 11:23:47,346 (main) INFO [c.b.s.r.s.TestLoggerServer] [trkId=] Test log message \x0a\x0a>@<', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='PROGRAM', value='syslog-ng', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='PID', value='28873', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='HOST_FROM', value='10.13.98.167', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='HOST', value='10.13.98.167/10.13.98.167', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='SOURCE', value='s_tcp_net_514', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:147:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local7', location='/etc/syslog-ng/syslog-ng.conf:147:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00800000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local7', location='/etc/syslog-ng/syslog-ng.conf:147:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:146:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local6', location='/etc/syslog-ng/syslog-ng.conf:146:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00400000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local6', location='/etc/syslog-ng/syslog-ng.conf:146:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:145:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local5', location='/etc/syslog-ng/syslog-ng.conf:145:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00200000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local5', location='/etc/syslog-ng/syslog-ng.conf:145:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:142:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local2', location='/etc/syslog-ng/syslog-ng.conf:142:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00040000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local2', location='/etc/syslog-ng/syslog-ng.conf:142:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:141:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local1', location='/etc/syslog-ng/syslog-ng.conf:141:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00020000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local1', location='/etc/syslog-ng/syslog-ng.conf:141:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:144:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local4', location='/etc/syslog-ng/syslog-ng.conf:144:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00100000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local4', location='/etc/syslog-ng/syslog-ng.conf:144:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:143:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local3', location='/etc/syslog-ng/syslog-ng.conf:143:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00080000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local3', location='/etc/syslog-ng/syslog-ng.conf:143:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_loggen', location='/etc/syslog-ng/syslog-ng.conf:97:22', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: match() evaluation started; input='syslog-ng', pattern='.*[.]000', value='PROGRAM', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_loggen', location='/etc/syslog-ng/syslog-ng.conf:97:22', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Initial message parsing follows; Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='HOST', value='', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='MESSAGE', value='Error processing log message: <-1>Jun 9 11:23:47 testlogger-service-54956569bb-2c5wl testlogger: 2020-06-09 11:23:47,346 (main) INFO [c.b.s.r.s.TestLoggerServer] [trkId=] Test log message \x0a\x0a>@<\x0a<-1>Jun 9 11:23:47 testlogger-service-54956569bb-2c5wl testlogger: 2020-06-09 11:23:47,346 (main) INFO [c.b.s.r.s.TestLoggerServer] [trkId=] Test log message ', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='PROGRAM', value='syslog-ng', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='PID', value='28873', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='HOST_FROM', value='10.13.98.167', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='HOST', value='10.13.98.167/10.13.98.167', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='SOURCE', value='s_tcp_net_514', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:147:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local7', location='/etc/syslog-ng/syslog-ng.conf:147:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00800000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local7', location='/etc/syslog-ng/syslog-ng.conf:147:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:146:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local6', location='/etc/syslog-ng/syslog-ng.conf:146:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00400000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local6', location='/etc/syslog-ng/syslog-ng.conf:146:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:145:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local5', location='/etc/syslog-ng/syslog-ng.conf:145:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00200000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local5', location='/etc/syslog-ng/syslog-ng.conf:145:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:142:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local2', location='/etc/syslog-ng/syslog-ng.conf:142:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00040000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local2', location='/etc/syslog-ng/syslog-ng.conf:142:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:141:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local1', location='/etc/syslog-ng/syslog-ng.conf:141:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00020000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local1', location='/etc/syslog-ng/syslog-ng.conf:141:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:144:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local4', location='/etc/syslog-ng/syslog-ng.conf:144:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00100000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local4', location='/etc/syslog-ng/syslog-ng.conf:144:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:143:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local3', location='/etc/syslog-ng/syslog-ng.conf:143:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00080000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local3', location='/etc/syslog-ng/syslog-ng.conf:143:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_loggen', location='/etc/syslog-ng/syslog-ng.conf:97:22', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: match() evaluation started; input='syslog-ng', pattern='.*[.]000', value='PROGRAM', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_loggen', location='/etc/syslog-ng/syslog-ng.conf:97:22', msg='0x7f2af8003800' Any help would be greatly appreciated. Thank you, Jason

3 9

Disk buffer getting wiped out
by Raghunath Adhyapak 15 Jun '20

15 Jun '20

Hi, We have syslog-ng forwarding logs to a remote system, with reliability enabled and disk buffer size set to 50GB in the configuration. However, the actual size of the partition is 30GB. We had overlooked the partition size that resulted in this error. When there is connection loss to remote device and partition gets filled to 100% due to syslog-ng disk buffer. syslog-ng continues to run, but even after we free some space in partition and destination is available, it doesn't push out logs to the destination. We have to restart syslog-ng service. After restarting, we observe that all buffer files are freed up immediately. These files were consuming about 20-30GB. Syslog-ng version using :3.24 Below is disk buffer configuration: mem-buf-size(10000) disk-buf-size(53687091200) reliable(yes) dir(/syslog/buffer) Questions: 1. Does syslog-ng allocate buffers on disk as much as required and in small chunks? Or will it try to allocate all of the specified buffer at one (in this case 50GB)? 2. Why couldn't syslog-ng resume operations after partition was freed up and destination was available? 3. Why did all buffer files get deleted, after a restart Thanks Raghu

5 10