- syslog-ng - lists.balabit.hu

Kafka performance
by Ronald Fenner 07 Aug '20

07 Aug '20

I'm trying to figure out why the performance of syslog-ng sending to kafka has tanked considerably. Everything was running fine up until 7/30 and then all of sudden syslog-ng is having problems pushing messages to kafka at a quick pace. I've updated to 3.28.1 and it's still having problems. I was using disk buffering but I removed it an d doesn't seem to be the bottleneck. I feel it's something with syslog ng as I can use the kafka perf test to push messages at a very quick rate to the cluster. I was able to push 1 million 1k test messages in several seconds. Syslog-ng seems to be able to only push a couple of messages a second. Any thoughts on what could be tanking it's performance? It's running on Amazon Linux 2. I've rebuilt our base image with the latest version so we can switch between kafka-c and the java version however since the kafka-c topic can't be templated at the moment until a pull request is released I can't switch to it to see if it fairs better. Ronald Fenner Network Architect Game Circus LLC. rfenner(a)gamecircus.com

2 3

Sylog-ng and F5 Load balancer.
by Peter Griggs 07 Aug '20

07 Aug '20

Hello Does anyone have any experience of syslog-ng behind F5 load balancer and preserving source IP address? The F5 can put the X-Forwarding header or TCP Options value but I don't believe syslog-ng can understand either of these - Am I wrong? If I am then I am assuming I can have a filter that overwrites $HOST with the value of X-Forwarding or TCP Options. Any help appreciated. Thanks Peter.

3 7

EWMM and sudo app-parser
by Fabien Wernli 06 Aug '20

06 Aug '20

Hi, I'm investigating using the EWMM forwarding model. Consider the following setup: Linux hosts collect logs using `system()` send them over using `syslog-ng()` destination to a remote host that collects them using `default-network-drivers()` source. It seems to me that the sudo app parsing is fired up twice: 1. On the sender side because `system()` expands to something including the `sudo-parser()` SCL 2. On the receiver side because `default-network-drivers()` expands to something involving the `app-parser()` This happens also when using `syslog()` source on the sender side, which is why I noticed this behaviour. So my question is, is there something wrong with that model ?

2 4

Kafka-c topics dynamic
by Ronald Fenner 06 Aug '20

06 Aug '20

When I was using the java implementation I extracted the topic dynamically from the message to route the message to the proper topic. I see in Kafka c implementation that the topic can't use templates so my old method of doing this topic("${topic}.${environment}") No longer works. Is there a way to reproduce this or do I need to go back to the java version for now? Ronald Fenner Network Architect Game Circus LLC. rfenner(a)gamecircus.com

2 2

journald prefix
by Fabien Wernli 05 Aug '20

05 Aug '20

Hi, I'm in the process of simplifying my structured logging config. Most of my systems now have journald shipping local logs to syslog-ng via the `system()` auto-expanding source. While this is convenient, as it makes it possible to use the same config for all my systems (systemd and rest of world), it also makes it impossible to configure the underlying driver's defaults. The task at hand is shipping the journald additional fields, which would be easy to do using the following excerpt from the documentation: source s_journald { systemd-journal(prefix(".SDATA.journald.")); }; This is not possible of course using `system()`. I've got a couple of ideas on how to handle this, but I'd like to hear other (probably less insane than my own) ways first :-) Cheers

2 2

Error trying to use kafka-c
by Ronald Fenner 05 Aug '20

05 Aug '20

I'm trying to switch from the java version of kafka to the c version and have built syslog with the following config: ./configure --prefix=/opt/syslog-ng --enable-json --enable-python --with-python=2.7 --enable-kafka However when I try syntax check the config it's returning this error: Error parsing destination statement, destination plugin kafka-c not found in block destination kafka() at /opt/syslog-ng/share/syslog-ng/include/scl/kafka/kafka.conf:24:4:5-4:12: 1 2 #Start Block block destination kafka() at /opt/syslog-ng/share/syslog-ng/include/scl/kafka/kafka.conf:24 3 4-----> kafka-c(bootstrap_servers("kafka.service.consul:9092") topic("syslog-ng-production") ); 4-----> ^^^^^^^ 5 6 #End Block block destination kafka() at /opt/syslog-ng/share/syslog-ng/include/scl/kafka/kafka.conf:24 7 Included from /opt/syslog-ng/etc/syslog-ng.conf:33:5-36:6: 28 source s_rtl_raw_backup { 29 file("/var/gamecircus/log/rtl-raw-backup.log" flags(no-parse) log_msg_size(1000000)); 30 }; 31 32 destination d_kafka_unstructured { 33----> kafka ( 33----> ^^^^^^^^ 34----> bootstrap-servers("kafka.service.consul:9092") 35----> topic("syslog-ng-`DeployedEnvironment`") 36----> ); 37 }; 38 The top of my config looks like this @version: 3.28 @include "scl.conf" @include "conf.d/gc-parsers.conf" @define kafka-implementation kafka-c Everything looks correct just not sure why it's having a hard time finding it. Ronald Fenner Network Architect Game Circus LLC. rfenner(a)gamecircus.com

2 1

Deadlock with full disk buffer
by Ronald Fenner 03 Aug '20

03 Aug '20

I've run into a problem where we had enabled reliable disk buffering on our Kafka destination and it's disk buffer eventually filled the drive it's stored on party because the drive is sized to the buffer it's holding. When restarting syslog-ng it complains with "Daemon exited due to a deadlock/signal/failure, restarting; exitcode='6'" The version we are using is 3.23.1 I'm not sure why the disk buffer filled up the only thing I can think of is that syslog-ng was unable to send message to the Kafka destination, however our Kafka cluster is up and function fine and I can even consume messages from a topic that we are pushing messages to on one of the instances affected. A question once a reliable message is sent is it removed from the buffer, i assume that's the case. The other question is it possible the java Kafka plugin can't keep up with the amount of mesages we are sending though syslog-ng. We see 12.5 million requests per day to the servers with the disk buffers. Each request generates at least 2 messages, one access log message, one application log so that's roughly 25 million messages a day. I designed the disk buffers to hold roughly 2 days worth of data if for some reason the Kafka cluster went down which in this case doesn't seem to be the case since monitoring tools show it's online and none of our hourly consumers complained about being able to connect. I'll have to build a recovery server for the buffers since they are housed on attached drives the drive are preserved when the instance is terminated this was done to make sure we didn't lose any messages not sent when the autoscaling scaled out a instance. I'm currently manually terminating the affected instances to keep those buffer and restore out logging. Once I have it, if more info is needed I can see what happens with trying to start syslog-ng up when it's also not receiving a bunch of messages as well. Ronald Fenner Network Architect Game Circus LLC. rfenner(a)gamecircus.com

2 3

Error message in syslog-ng (LOGMSG_REFCACHE_VALUE_TO_REF(old_value) >= 1)
by Ahmed Soyef 29 Jul '20

29 Jul '20

Hi All, We're seeing some instances of this error message ERROR:lib/logmsg/logmsg.c:1521:log_msg_ref: assertion failed: (LOGMSG_REFCACHE_VALUE_TO_REF(old_value) >= 1) I've had a quick search about and can't figure out what it actually means. Can anyone shed some light and possibly some guidance on how to avoid it happening? Regards, Soyef ********************************************************************** This email and any files transmitted with it are private and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please return it to the address it came from telling them it is not for you and then delete it from your system. This email message has been swept for computer viruses. **********************************************************************

2 1

File destination is not being written, when it can not bind remote host address
by Alexandre Santos 29 Jul '20

29 Jul '20

Hi, I do not know if this a seen issue or not. I am using syslog-ng version 3.19 on a Debian 10. I have the configuration which is attached in this email: syslog-ng.conf, that has one local file destination and one remote host destination. 1. Start the syslog-ng The localip of syslog destination is not available and therefore I have these errors, as expected: syslog-ng[1482]: [2020-07-27T12:58:00.069422] Error binding socket; addr='AF_INET(10.0.2.6:0)', error='Cannot assign requested address (99)' syslog-ng[1482]: [2020-07-27T12:58:00.069439] Initiating connection failed, reconnecting; time_reopen='60' 2. After some time running syslog-ng stops sending the logs to the local file. Why? 3. If I reload the syslog-ng configuration, without the remote destination, the old logs are immediately flushed to the local file. Jul 27 *15:05:55* localhost syslog-ng[29680]: [*2020-07-27T15:05:55.378*019] Incoming log entry from journal; message='[*2020-07-27T13:44:06.890*952] Outgoing message; message=\'<30>1 2020-07-27T13:44:06.772+00:00 localhost syslog-ng 29680 - - [2020-07-27T13:44:06.772722] Processing the time zone file (32bit part); filename=\\'/usr/share/zoneinfo/UTC\\'\x0a\'' Jul 27 15:05:55 localhost syslog-ng[29680]: [2020-07-27T15:05:55.378032] json-parser(): no marker at the beginning of the message, skipping JSON parsing ; input='[2020-07-27T13:44:06.890952] Outgoing message; message=\'<30>1 2020-07-27T13:44:06.772+00:00 localhost syslog-ng 29680 - - [2020-07-27T13:44:06.772722] Processing the time zone file (32bit part); filename=\\'/usr/share/zoneinfo/UTC\\'\x0a\'', marker='@cim:' Is this a new issue? Is there any configuration to prevent this? In attachment also the journalctl of syslog-ng in debug mode (syslog-ng -Fvde). Thanks and regards, Alex

2 1

setting sequenceId in forwarded log messages read from journald reader
by Peter Vollmer 29 Jul '20

29 Jul '20

Hi, I am currently trying to find a way to set meta.sequenceId of log messages that have been read from the locally running systemd-journal to forward them to a remote syslog server that expects the logs to contain a sequenceId according to RFC 5424 section 7.3.1. I found that a sequence number could be taken from the __CURSOR field "i=..." of the journald log: # journalctl -o json-pretty -f ... "__CURSOR" : "s=02a7b30ba17b4a43846f265706bd3a70;i=f01;b=ba633698f20848e480bca4e72476e4d3;m=1a355c1d5;t=5ab670340c8ea;x=33389988ef680e7e", ... My problem is that the journal reader does not seem to parse the __CURSOR string when reading from journald logs. Is there a way to get this information into meta.sequenceId of the forwarded log without modifying the systemd-journal module in syslog-ng ? Thank you for any ideas and best regards Peter Vollmer

3 2

Certificate ip authentication
by Alexandre Damas 27 Jul '20

27 Jul '20

Hi, As my objective on the utilization of syslog-ng is the authentication and certification of received security and auditing events, I implemented an internal CA which generates a certificate per syslog-ng client. Once there was someone that misconfigured a client and got a certificate for a different client. The funny part of it is that no one observed any problem as the certificate, which was generated for a client having an ip (different from the one configured on the machine), was working and communications (using ALTP with TLS) went up for the exchange of messages. Has anyone experienced this? Does someone have a clue on how to prevent certificate reutilization on the client side? On Linux side I did not find any way of preventing the utilization of a certificate for one machine, that was issued for a different machine, having a different ip. If there is no cross checking on the server for the ip addresses on the certificate with the ip address on the received event, I don't have a way of non-repudiate a received event and the client can reuse the certificate for other machines, allowing the events to be received on the server. Kind Regards Alexandre Damas

2 2

Configuration validation Issue with snmptrap module
by Luca Da Col 23 Jul '20

23 Jul '20

Running syslog-ng 3.27.1 on Centos 7. Configuration validation returns an error with snmptrap command. Configuration example has been taken directly from the OSE documentation. Error error was reproducible with version available on epel repository (3.5), though documentation indicates the snmptrap is available from 3.10 Configuration file includes statements: @version:3.22 @include "scl.conf" Any assistance is appreciated, Thank you, Luca [root@adev syslog-ng]# syslog-ng --syntax-only Error parsing source statement, source plugin snmptrap not found in /etc/syslog-ng/syslog-ng.conf:92:5-92:13: 90 log { 91 source { 92----> snmptrap(filename("/var/log/snmptrapd.log")); 92----> ^^^^^^^^ 93 }; 94 destination { 95 file("/var/log/example.log"); 96 }; 97 }; [root@dev syslog-ng]# syslog-ng -V syslog-ng 3 (3.27.1) Config version: 3.22 Installer-Version: 3.27.1 Revision: Compile-Date: Jun 8 2020 07:54:17 Module-Directory: /usr/lib64/syslog-ng Module-Path: /usr/lib64/syslog-ng Include-Path: /usr/share/syslog-ng/include Available-Modules: affile,afprog,afsocket,afstomp,afuser,basicfuncs,confgen,cryptofuncs,csvparser,dbparser,linux-kmsg-format,syslogformat,system-source,add-contextual-data,appmodel,cef,disk-buffer,examples,graphite,hook-commands,json-plugin,kvformat,map-value-pairs,pseudofile,sdjournal,stardate,tags-parser,tfgetent,timestamp,xml Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Systemd: on [root@dev syslog-ng]#

2 1

syslog-ng multiple VRF
by Alexandre Santos 22 Jul '20

22 Jul '20

Hi All, I have a question regarding syslog-ng and VRF. I want to read from a syslog source, which interface is in the default VRF, and send the logs to a syslog/network destination interface which is in a MGMT VRF. Can syslog-ng support this? If yes, what are the aspects I should be careful about? Thanks and regards, Alex

2 2

syslog-ng restarts due to interface not ready
by Alexandre Santos 21 Jul '20

21 Jul '20

Hi, I have a configuration where one of my sources is a syslog source: syslog(ip(127.1.1.1) transport("udp") port(514) keep-alive(no)); But when the system starts up, the interface is not ready yet, and I get the following errors: Jun 26 09:17:50 localhost systemd[1]: Starting System Logger Daemon... Jun 26 09:17:52 localhost syslog-ng[2301]: [2020-06-26T09:17:52.485148] WARNING: With use-dns(no), dns-cache() will be forced to 'no' too!; Jun 26 09:17:52 localhost syslog-ng[2301]: [2020-06-26T09:17:52.487159] Current position not matches to the saved cursor position, seek to head; cursor='s=f62c0e80f50948deb29cf2d28230a Jun 26 09:17:52 localhost syslog-ng[2301]: [2020-06-26T09:17:52.487867] Error resolving hostname; host='127.1.1.1' Jun 26 09:17:52 localhost syslog-ng[2301]: [2020-06-26T09:17:52.487878] Error initializing message pipeline; plugin_name='syslog', location='/etc/syslog-ng/syslog-ng.conf:43:5' Jun 26 09:17:52 localhost systemd[1]: syslog-ng.service: Main process exited, code=exited, status=2/INVALIDARGUMENT Jun 26 09:17:52 localhost systemd[1]: syslog-ng.service: Failed with result 'exit-code'. Jun 26 09:17:52 localhost systemd[1]: Failed to start System Logger Daemon. Jun 26 09:17:52 localhost systemd[1]: syslog-ng.service: Service RestartSec=100ms expired, scheduling restart. Jun 26 09:17:52 localhost systemd[1]: syslog-ng.service: Scheduled restart job, restart counter is at 1. When the systemd reaches the 5 attempts it disables the syslog-ng.service. Do you know some way to fix this, by adding something, the syslog-ng configuration? Thanks in Advance, Alex

4 4

disk buffer files permissions
by Pal, Laszlo 13 Jul '20

13 Jul '20

Hi, We want to monitor disk-buffer files, so we can detect if there is are some issues with the destinations. syslog-ng creates these files with permissions like this otal 360 -rw-------. 1 root root 4096 Oct 10 2019 syslog-ng-00000.qf -rw-------. 1 root root 4096 Jul 12 03:49 syslog-ng-00001.qf So, our monitor script (for zabbix) cannot stat files because it is running under user zabbix. I'm not sure how to submit a feature request nowadays, but it would be great of perm() option added to disk-buffer() Any chance? Thx L:

3 4

Insider 2020-07: TLS; capabilities; 3.27;
by Peter Czanik (pczanik) 09 Jul '20

09 Jul '20

Dear syslog-ng users, This is the 83rd issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Simplifying CA handling in syslog-ng TLS connections ---------------------------------------------------- When talking to users about the TLS-encrypted message transfer, almost everyone immediately complains about configuring a certificate authority (CA) in syslog-ng. You needed to create a hash and create a symbolic link to the CA file based on the hash. Not anymore. While this old method is still available, there is now a much easier way: the new ca-file() option. https://www.syslog-ng.com/community/b/blog/posts/simplifying-ca-handling-in… Working around Linux capabilities problems for syslog-ng -------------------------------------------------------- No, SELinux is not the cause of all permission troubles on Linux. For example, syslog-ng makes use of the capabilities system on Linux to drop as many privileges as possible, as early as possible. But it might cause problems in some corner cases, as even when running as root, syslog-ng cannot read files owned by a different user. Learn from this blog how you can figure out if you have a SELinux or capabilities problem and how to fix it if you do. https://www.syslog-ng.com/community/b/blog/posts/working-around-linux-capab… Figuring out where a message arrived, and other syslog-ng 3.27 tricks ---------------------- Version 3.27 of syslog-ng has brought many smaller, but useful features to us. The new Sumo Logic destination was already covered in an earlier blog. You can now also check exactly where a message arrived on a network source (IP address, port and protocol). Rewriting the facility of a syslog message was also made easy. For a complete list of new features and changes, check the release notes at https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-3.27.1 You can learn more about these features at https://www.syslog-ng.com/community/b/blog/posts/figuring-out-where-a-messa… NEW RELEASES * Version 3.28.1 released: https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-3.28.1 * syslog-ng PE version 7.0.21 is coming soon with MSSQL and Oracle SQL sources, and you can send logs to Google Pub/Sub. For details, check the documentation at https://support.oneidentity.com/syslog-ng-premium-edition/7.0.21 WEBINARS * Learn how to do custom HTTP REST API Authentication with syslog-ng: https://www.syslog-ng.com/event/customize-http-rest-apis-authentication-wit… * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Convert logs back into syslog-ng logs
by Faine, Mark R. (MSFC-IS40)[NICS] 07 Jul '20

07 Jul '20

I would like to read lines from a file back into log data that syslog-ng can parse. What would be involved in doing this? Thanks, -Mark

2 3

syslog-ng 3.28.1 RPM packages
by Peter Czanik (pczanik) 24 Jun '20

24 Jun '20

Hi, As usual, my unofficial syslog-ng rpm packages for openSUSE/SLES and Fedora/RHEL/CentOS first appear in my git snapshot repositories. I hope to create the usual dedicated repositories later this week and that will be followed by a separate announcement. For now you can install/test 3.28.1 following instructions from https://syslog-ng.com/blog/rpm-packages-from-syslog-ng-git-head/ Any feedback is appreciated! Here is a list of changes since the last release: * Java modules (HDFS, Elasticsearch, Kafka, etc.) removed. The Java binding itself stays, where JDK 1.8 is available. * Logrotate script on Fedora is renamed to syslog-ng and only installed if rsyslog is not on the system * Logrotate script on RHEL 8 is only installed if rsyslog is not on the system to avoid conflict * * Logrotate bug on RHEL 7 fixed (the same logrotate script is used as for rsyslog to avoid conflict * Bye, CzP Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

syslog-ng 3.28 Debian/Ubuntu packages (OBS)
by Laszlo Budai 23 Jun '20

23 Jun '20

Hi, syslog-ng 3.28.1[1] packages are available from OBS repositories[2, 3, 4]. List of supported OSs: * Debian 8.0 * Debian 9.0 * Debian 10.0 * Raspbian 9.0 * Raspbian 10.0 * Ubuntu 14.04 * Ubuntu 16.04 * Ubuntu 18.04 * Ubuntu 18.10 * Ubuntu 19.04 * Ubuntu 19.10 * Ubuntu 20.04 Install ------- example: Debian 9.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-get install syslog-ng-core` Links -------- [1] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.28.1 [2] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… [3] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… [4] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… regards, Laszlo Budai

1 0

syslog-ng-3.28.1
by Laszlo Budai 23 Jun '20

23 Jun '20

3.28.1 Highlights * http: add support for proxy option Example: log { source { system(); }; destination { http( url("SYSLOG_SERVER_IP:PORT") proxy("PROXY_IP:PORT") method("POST") ); }; }; (#3253<https://github.com/syslog-ng/syslog-ng/pull/3253>) Features * map: template function This template function applies a function to all elements of a list. For example: $(map $(+ 1 $_) 0,1,2) => 1,2,3. (#3301<https://github.com/syslog-ng/syslog-ng/pull/3301>) * use-syslogng-pid(): new option to all sources If set to yes, syslog-ng overwrites the message's ${PID} macro to its own PID. (#3323<https://github.com/syslog-ng/syslog-ng/pull/3323>) Bugfixes * affile: eliminate infinite loop in case of a spurious file path If the template evaluation of a log message will result to a spurious path in the file destination, syslog-ng refuses to create that file. However the problematic log message was left in the msg queue, so syslog-ng was trying to create that file again in time-reopen periods. >From now on syslog-ng will handle "permanent" file errors, and drop the relevant msg. (#3230<https://github.com/syslog-ng/syslog-ng/pull/3230>) * Fix minor memory leaks in error scenarios (#3265<https://github.com/syslog-ng/syslog-ng/pull/3265>) * crypto: fix hang on boot due to lack of entropy (#3271<https://github.com/syslog-ng/syslog-ng/pull/3271>) * Fix IPv4 UDP destinations on FreeBSD UDP-based destinations crashed when receiving the first message on FreeBSD due to a bug in destination IP extraction logic. (#3278<https://github.com/syslog-ng/syslog-ng/pull/3278>) * network sources: fix TLS connection closure RFC 5425 specifies that once the transport receiver gets close_notify from the transport sender, it MUST reply with a close_notify. The close_notify alert is now sent back correctly in case of TLS network sources. (#2811<https://github.com/syslog-ng/syslog-ng/pull/2811>) * disk-buffer: fixes possible crash, or fetching wrong value for logmsg nvpair (#3281<https://github.com/syslog-ng/syslog-ng/pull/3281>) * packaging/debian: fix mod-rdkafka Debian packaging (#3282<https://github.com/syslog-ng/syslog-ng/pull/3282>) * kafka destination: destination halts if consumer is down, and kafka's queue is filled (#3305<https://github.com/syslog-ng/syslog-ng/pull/3305>) * file-source: Throw error, when follow-freq() is set with a negative float number. (#3306<https://github.com/syslog-ng/syslog-ng/pull/3306>) * stats-freq: with high stats-freq syslog-ng emits stats immediately causing high memory and CPU usage (#3320<https://github.com/syslog-ng/syslog-ng/pull/3320>) * secure-logging: bug fixes (#3284<https://github.com/syslog-ng/syslog-ng/pull/3284>) * template arguments are now consistently checked * fixed errors when mac file not provided * fixed abort when derived key not provided * fixed crash with slogkey missing parameters * fixed secure-logging on 32-bit architectures * fixed CMake build Other changes * dbld: Fedora 32 support (#3315<https://github.com/syslog-ng/syslog-ng/pull/3315>) * dbld: Removed Ubuntu Eoan (#3313<https://github.com/syslog-ng/syslog-ng/pull/3313>) * secure-logging: improvements (#3284<https://github.com/syslog-ng/syslog-ng/pull/3284>) * removed 1500 message length limitation * slogimport has been renamed to slogencrypt * $(slog) will not start anymore when key is not found * internal messaging (warning, debug) improvements * improved memory handling and error information display * CMake build improvements * switched to GLib command line argument parsing * the output of slogkey -s is now parsable * manpage improvements Notes to developers * dbld: devshell is now upgraded to Ubuntu Focal (#3277<https://github.com/syslog-ng/syslog-ng/pull/3277>) * dbld/devshell: Multiple changes: * Added snmptrapd package. * Added support for both python2 and python3. (#3222<https://github.com/syslog-ng/syslog-ng/pull/3222>) * threaded-source: fully support default-priority() and default-facility() (#3304<https://github.com/syslog-ng/syslog-ng/pull/3304>) * CMake: fix libcap detection (#3294<https://github.com/syslog-ng/syslog-ng/pull/3294>) * Fix atomic_gssize_set() warning with new glib versions (#3286<https://github.com/syslog-ng/syslog-ng/pull/3286>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Airbus Commercial Aircraft, Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler, Gabor Nagy, Laszlo Budai, Laszlo Szemere, László Várady, Péter Kókai, Vatsal Sisodiya, Vivin Peris.

1 0

request for testing: logrotate scripts in Fedora/RHEL packages
by Peter Czanik (pczanik) 23 Jun '20

23 Jun '20

Hi, There were multiple tickets, bug reports about logrotate scripts in syslog-ng RPM packages for Fedora/RHEL. https://bugzilla.redhat.com/show_bug.cgi?id=1802165 https://github.com/syslog-ng/syslog-ng/issues/1384#issuecomment-644716350 I have now a possible solution available for testing in my git snapshot repo: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng-githead/ * Installs the same logrotate script on RHEL/CentOS 7 as rsyslog * On RHEL/CentOS 8 and Fedora the logrotate script is called syslog-ng and uses systemctl to reload syslog-ng after rotation, just as the rsyslog variant. It is in a separate subpackage, and installed only when rsyslog is not on the system (for reasons check the RH bugzilla ticket: in short it would not be used anyway due to conflicts) I tested it on Fedora 32 and CentOS 7, both with and without rsyslog on the system. It worked fine for me, but any feedback is appreciated. If it is tested to work by someone else than me, I'll push the changes both to the official EPEL package and to the syslog-ng github repo. Bye, CzP Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

4 7

syslog-ng rpm java changes
by Secure-Logging, Generic 22 Jun '20

22 Jun '20

Hi Peter, apparently, I missed the poll. But I can assure you that we are not using Java in the syslog-ng version that are shipped to our customers. We are even building our own packages for syslog-ng on RHEL 7 to get rid of Java and Python. The systems we deliver have a minimal version of syslog-ng installed and the only extra module we currently use is the secure logging module. Regards, Stephan The information in this e-mail is confidential. The contents may not be disclosed or used by anyone other than the addressee. Access to this e-mail by anyone else is unauthorised. If you are not the intended recipient, please notify Airbus immediately and delete this e-mail. Airbus cannot accept any responsibility for the accuracy or completeness of this e-mail as it has been sent over public networks. If you have any concerns over the content of this message or its Accuracy or Integrity, please contact Airbus immediately. All outgoing e-mails from Airbus are checked using regularly updated virus scanning software but you should take whatever measures you deem to be appropriate to ensure that this message and any attachments are virus free.

1 0

syslog-ng rpm java changes
by Peter Czanik (pczanik) 22 Jun '20

22 Jun '20

Hi, Over a week ago I ran a Twitter poll and also asked around here on the mailing list about if/how you use Java in syslog-ng. No responses here and only a few "no" responses on Twitter. We know about some users, who use the Java bindings, but not our modules. Any previous users of Java modules I asked directly told me, that they already changed to kafka-c and elasticsearch-http. A few days later Fedora announced, that they are removing JDK 1.8 support from Rawhide. So, here is a list of changes and plans for the official distro syslog-ng packages and for my unofficial packages: * removed Java support completely from the official Fedora Rawhide package (no support for JDK 1.11 yet in syslog-ng), to avoid dropping syslog-ng from the distro * my unofficial packages will follow, once JDK 1.8 is really removed from the distro (end of July) * keep Java bindings everywhere else, as long as 1.8 is available * removed java modules (drivers) from all of my unofficial packages (Java-based Elasticsearch, Kafka, HDFS) My syslog-ng git snapshot packages (https://syslog-ng.com/blog/rpm-packages-from-syslog-ng-git-head/) already have these changes, and my 3.28 packages will be derived from these, once the release is available. Bye, Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Some logs written, some are not
by Jason Brown 16 Jun '20

16 Jun '20

I’m hoping someone can help here, because I’m at my wits end. System info: syslog-ng OSE 3.27 OS: CentOS 7.8 (though, I get similar results on Ubuntu 18.04) I have some kubernetes pods sending logs to a remote syslog-ng server. I have an issue where some logs are not being created. These are specifically received via TCP. I can see the message inbound by doing a packet capture, but nothing is written. One of the messages that stands out is: "Error processing message <-1>" Here is my scenario: Start a deployment with 3 pods: Pod1: log created on syslog-ng server: /data/2020/06/09/pod1/logs/app.log Pod2: nothing Pod3: nothing Delete and deploy again: Pod1: log created on syslog-ng server: /data/2020/06/09/pod1/logs/app.log Pod2: log created on syslog-ng server: /data/2020/06/09/pod2/logs/app.log Pod3: nothing Delete and deploy again: Pod1: nothing Pod2: nothing Pod3: nothing There seems to be no pattern. The pods that log, do so successfully. All pods should be logging the same exact date except for timestamps and the pod name. A packet capture shows healthy traffic inbound, with no errors, even from the pods that are not logging. The only indication of an issue is the trace log. Here’s an example from one of the non-writing pods, with a message id “0x7f2af8003800” : Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Initial message parsing follows; Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='PROGRAM', value='testlogger', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='LEGACY_MSGHDR', value='testlogger: ', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='HOST', value='testlogger-service-54956569bb-kqsjs', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='MESSAGE', value='2020-06-09 11:23:47,295 (main) INFO [c.b.s.r.s.TestLoggerServer] [trkId=] Server starting in Staging environment ', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='HOST_FROM', value='10.13.143.181', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='SOURCE', value='s_tcp_net_514', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:147:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local7', location='/etc/syslog-ng/syslog-ng.conf:147:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='20', valid_fac='00800000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local7', location='/etc/syslog-ng/syslog-ng.conf:147:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:146:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local6', location='/etc/syslog-ng/syslog-ng.conf:146:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='20', valid_fac='00400000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local6', location='/etc/syslog-ng/syslog-ng.conf:146:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:145:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local5', location='/etc/syslog-ng/syslog-ng.conf:145:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='20', valid_fac='00200000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local5', location='/etc/syslog-ng/syslog-ng.conf:145:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:142:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local2', location='/etc/syslog-ng/syslog-ng.conf:142:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='20', valid_fac='00040000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local2', location='/etc/syslog-ng/syslog-ng.conf:142:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:141:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local1', location='/etc/syslog-ng/syslog-ng.conf:141:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='20', valid_fac='00020000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local1', location='/etc/syslog-ng/syslog-ng.conf:141:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:144:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local4', location='/etc/syslog-ng/syslog-ng.conf:144:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='20', valid_fac='00100000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: severity() evaluation started; pri='6', valid_pri='000000ff', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> rewrite rule evaluation begin; rule='r_newlines', location='/etc/syslog-ng/syslog-ng.conf:186:5', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Message was cloned; original_msg='0x7f2af8003800', new_msg='0x7f2af8004830' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Performing subst() rewrite failed, pattern did not match; rule='r_newlines', value='MESSAGE', input='2020-06-09 11:23:47,295 (main) INFO [c.b.s.r.s.TestLoggerServer] [trkId=] Server starting in Staging environment ', type='(null)', pattern='u2028', replacement='\x0a', location='/etc/syslog-ng/syslog-ng.conf:186:5’ Further down in the log, I see: Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Initial message parsing follows; Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='HOST', value='', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='MESSAGE', value='Error processing log message: <-1>Jun 9 11:23:47 testlogger-service-54956569bb-2c5wl testlogger: 2020-06-09 11:23:47,341 (main) INFO [c.b.s.r.s.TestLoggerServer] [trkId=] Server starting in Staging environment \x0a\x0a<-1>Jun 9 11:23:47 testlogger-service-54956569bb-2c5wl testlogger: 2020-06-09 11:23:47,346 (main) INFO [c.b.s.r.s.TestLoggerServer] [trkId=] Test log message \x0a\x0a>@<', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='PROGRAM', value='syslog-ng', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='PID', value='28873', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='HOST_FROM', value='10.13.98.167', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='HOST', value='10.13.98.167/10.13.98.167', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='SOURCE', value='s_tcp_net_514', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:147:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local7', location='/etc/syslog-ng/syslog-ng.conf:147:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00800000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local7', location='/etc/syslog-ng/syslog-ng.conf:147:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:146:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local6', location='/etc/syslog-ng/syslog-ng.conf:146:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00400000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local6', location='/etc/syslog-ng/syslog-ng.conf:146:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:145:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local5', location='/etc/syslog-ng/syslog-ng.conf:145:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00200000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local5', location='/etc/syslog-ng/syslog-ng.conf:145:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:142:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local2', location='/etc/syslog-ng/syslog-ng.conf:142:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00040000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local2', location='/etc/syslog-ng/syslog-ng.conf:142:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:141:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local1', location='/etc/syslog-ng/syslog-ng.conf:141:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00020000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local1', location='/etc/syslog-ng/syslog-ng.conf:141:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:144:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local4', location='/etc/syslog-ng/syslog-ng.conf:144:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00100000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local4', location='/etc/syslog-ng/syslog-ng.conf:144:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:143:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local3', location='/etc/syslog-ng/syslog-ng.conf:143:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00080000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local3', location='/etc/syslog-ng/syslog-ng.conf:143:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_loggen', location='/etc/syslog-ng/syslog-ng.conf:97:22', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: match() evaluation started; input='syslog-ng', pattern='.*[.]000', value='PROGRAM', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_loggen', location='/etc/syslog-ng/syslog-ng.conf:97:22', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Initial message parsing follows; Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='HOST', value='', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='MESSAGE', value='Error processing log message: <-1>Jun 9 11:23:47 testlogger-service-54956569bb-2c5wl testlogger: 2020-06-09 11:23:47,346 (main) INFO [c.b.s.r.s.TestLoggerServer] [trkId=] Test log message \x0a\x0a>@<\x0a<-1>Jun 9 11:23:47 testlogger-service-54956569bb-2c5wl testlogger: 2020-06-09 11:23:47,346 (main) INFO [c.b.s.r.s.TestLoggerServer] [trkId=] Test log message ', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='PROGRAM', value='syslog-ng', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='PID', value='28873', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='HOST_FROM', value='10.13.98.167', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='HOST', value='10.13.98.167/10.13.98.167', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Setting value; name='SOURCE', value='s_tcp_net_514', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:147:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local7', location='/etc/syslog-ng/syslog-ng.conf:147:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00800000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local7', location='/etc/syslog-ng/syslog-ng.conf:147:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:146:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local6', location='/etc/syslog-ng/syslog-ng.conf:146:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00400000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local6', location='/etc/syslog-ng/syslog-ng.conf:146:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:145:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local5', location='/etc/syslog-ng/syslog-ng.conf:145:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00200000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local5', location='/etc/syslog-ng/syslog-ng.conf:145:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:142:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local2', location='/etc/syslog-ng/syslog-ng.conf:142:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00040000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local2', location='/etc/syslog-ng/syslog-ng.conf:142:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:141:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local1', location='/etc/syslog-ng/syslog-ng.conf:141:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00020000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local1', location='/etc/syslog-ng/syslog-ng.conf:141:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:144:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local4', location='/etc/syslog-ng/syslog-ng.conf:144:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00100000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local4', location='/etc/syslog-ng/syslog-ng.conf:144:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:143:32' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_local3', location='/etc/syslog-ng/syslog-ng.conf:143:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: facility() evaluation started; fac='5', valid_fac='00080000', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_local3', location='/etc/syslog-ng/syslog-ng.conf:143:32', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: >>>>>> filter rule evaluation begin; rule='f_loggen', location='/etc/syslog-ng/syslog-ng.conf:97:22', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: match() evaluation started; input='syslog-ng', pattern='.*[.]000', value='PROGRAM', msg='0x7f2af8003800' Jun 9 11:23:47 s_local@syslog syslog-ng[28873]: <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_loggen', location='/etc/syslog-ng/syslog-ng.conf:97:22', msg='0x7f2af8003800' Any help would be greatly appreciated. Thank you, Jason

3 9

Disk buffer getting wiped out
by Raghunath Adhyapak 15 Jun '20

15 Jun '20

Hi, We have syslog-ng forwarding logs to a remote system, with reliability enabled and disk buffer size set to 50GB in the configuration. However, the actual size of the partition is 30GB. We had overlooked the partition size that resulted in this error. When there is connection loss to remote device and partition gets filled to 100% due to syslog-ng disk buffer. syslog-ng continues to run, but even after we free some space in partition and destination is available, it doesn't push out logs to the destination. We have to restart syslog-ng service. After restarting, we observe that all buffer files are freed up immediately. These files were consuming about 20-30GB. Syslog-ng version using :3.24 Below is disk buffer configuration: mem-buf-size(10000) disk-buf-size(53687091200) reliable(yes) dir(/syslog/buffer) Questions: 1. Does syslog-ng allocate buffers on disk as much as required and in small chunks? Or will it try to allocate all of the specified buffer at one (in this case 50GB)? 2. Why couldn't syslog-ng resume operations after partition was freed up and destination was available? 3. Why did all buffer files get deleted, after a restart Thanks Raghu

5 10