- syslog-ng - lists.balabit.hu

Re: [syslog-ng] syslog-ng multiple VRF
by Terry Slattery 10 Aug '20

10 Aug '20

Alex, The simplest solution is to configure the network to share routes between the two VRFs. Linux now supports VRF, however, my quick read on the subject suggests that its configuration is complex. Applications like syslog-ng are rarely VRF-aware. It would have to bind to two different interfaces, one in each VRF. -tcs On 7/22/20 8:00 AM, syslog-ng-request(a)lists.balabit.hu wrote: > Date: Tue, 21 Jul 2020 15:18:46 +0100 > From: Alexandre Santos <alexandre.rosas.santos(a)gmail.com> > To: "Syslog-ng users' and developers' mailing list" > <syslog-ng(a)lists.balabit.hu> > Subject: [syslog-ng] syslog-ng multiple VRF > Message-ID: > <CAHxG_72hhkfCBu3j9nPxr_MVwKevJtFcpcRhVJLYPPYx4Mys1g(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi All, > > I have a question regarding syslog-ng and VRF. > I want to read from a syslog source, which interface is in the default VRF, > and send the logs to a syslog/network destination interface which is in a > MGMT VRF. > > Can syslog-ng support this? > If yes, what are the aspects I should be careful about? > > Thanks and regards, > Alex -- Terry Slattery CCIE #1026 NetCraftsmen www.netcraftsmen.com

5 7

Kafka performance
by Ronald Fenner 07 Aug '20

07 Aug '20

I'm trying to figure out why the performance of syslog-ng sending to kafka has tanked considerably. Everything was running fine up until 7/30 and then all of sudden syslog-ng is having problems pushing messages to kafka at a quick pace. I've updated to 3.28.1 and it's still having problems. I was using disk buffering but I removed it an d doesn't seem to be the bottleneck. I feel it's something with syslog ng as I can use the kafka perf test to push messages at a very quick rate to the cluster. I was able to push 1 million 1k test messages in several seconds. Syslog-ng seems to be able to only push a couple of messages a second. Any thoughts on what could be tanking it's performance? It's running on Amazon Linux 2. I've rebuilt our base image with the latest version so we can switch between kafka-c and the java version however since the kafka-c topic can't be templated at the moment until a pull request is released I can't switch to it to see if it fairs better. Ronald Fenner Network Architect Game Circus LLC. rfenner(a)gamecircus.com

2 3

Sylog-ng and F5 Load balancer.
by Peter Griggs 07 Aug '20

07 Aug '20

Hello Does anyone have any experience of syslog-ng behind F5 load balancer and preserving source IP address? The F5 can put the X-Forwarding header or TCP Options value but I don't believe syslog-ng can understand either of these - Am I wrong? If I am then I am assuming I can have a filter that overwrites $HOST with the value of X-Forwarding or TCP Options. Any help appreciated. Thanks Peter.

3 7

EWMM and sudo app-parser
by Fabien Wernli 06 Aug '20

06 Aug '20

Hi, I'm investigating using the EWMM forwarding model. Consider the following setup: Linux hosts collect logs using `system()` send them over using `syslog-ng()` destination to a remote host that collects them using `default-network-drivers()` source. It seems to me that the sudo app parsing is fired up twice: 1. On the sender side because `system()` expands to something including the `sudo-parser()` SCL 2. On the receiver side because `default-network-drivers()` expands to something involving the `app-parser()` This happens also when using `syslog()` source on the sender side, which is why I noticed this behaviour. So my question is, is there something wrong with that model ?

2 4

Kafka-c topics dynamic
by Ronald Fenner 06 Aug '20

06 Aug '20

When I was using the java implementation I extracted the topic dynamically from the message to route the message to the proper topic. I see in Kafka c implementation that the topic can't use templates so my old method of doing this topic("${topic}.${environment}") No longer works. Is there a way to reproduce this or do I need to go back to the java version for now? Ronald Fenner Network Architect Game Circus LLC. rfenner(a)gamecircus.com

2 2

journald prefix
by Fabien Wernli 05 Aug '20

05 Aug '20

Hi, I'm in the process of simplifying my structured logging config. Most of my systems now have journald shipping local logs to syslog-ng via the `system()` auto-expanding source. While this is convenient, as it makes it possible to use the same config for all my systems (systemd and rest of world), it also makes it impossible to configure the underlying driver's defaults. The task at hand is shipping the journald additional fields, which would be easy to do using the following excerpt from the documentation: source s_journald { systemd-journal(prefix(".SDATA.journald.")); }; This is not possible of course using `system()`. I've got a couple of ideas on how to handle this, but I'd like to hear other (probably less insane than my own) ways first :-) Cheers

2 2

Error trying to use kafka-c
by Ronald Fenner 05 Aug '20

05 Aug '20

I'm trying to switch from the java version of kafka to the c version and have built syslog with the following config: ./configure --prefix=/opt/syslog-ng --enable-json --enable-python --with-python=2.7 --enable-kafka However when I try syntax check the config it's returning this error: Error parsing destination statement, destination plugin kafka-c not found in block destination kafka() at /opt/syslog-ng/share/syslog-ng/include/scl/kafka/kafka.conf:24:4:5-4:12: 1 2 #Start Block block destination kafka() at /opt/syslog-ng/share/syslog-ng/include/scl/kafka/kafka.conf:24 3 4-----> kafka-c(bootstrap_servers("kafka.service.consul:9092") topic("syslog-ng-production") ); 4-----> ^^^^^^^ 5 6 #End Block block destination kafka() at /opt/syslog-ng/share/syslog-ng/include/scl/kafka/kafka.conf:24 7 Included from /opt/syslog-ng/etc/syslog-ng.conf:33:5-36:6: 28 source s_rtl_raw_backup { 29 file("/var/gamecircus/log/rtl-raw-backup.log" flags(no-parse) log_msg_size(1000000)); 30 }; 31 32 destination d_kafka_unstructured { 33----> kafka ( 33----> ^^^^^^^^ 34----> bootstrap-servers("kafka.service.consul:9092") 35----> topic("syslog-ng-`DeployedEnvironment`") 36----> ); 37 }; 38 The top of my config looks like this @version: 3.28 @include "scl.conf" @include "conf.d/gc-parsers.conf" @define kafka-implementation kafka-c Everything looks correct just not sure why it's having a hard time finding it. Ronald Fenner Network Architect Game Circus LLC. rfenner(a)gamecircus.com

2 1

Deadlock with full disk buffer
by Ronald Fenner 03 Aug '20

03 Aug '20

I've run into a problem where we had enabled reliable disk buffering on our Kafka destination and it's disk buffer eventually filled the drive it's stored on party because the drive is sized to the buffer it's holding. When restarting syslog-ng it complains with "Daemon exited due to a deadlock/signal/failure, restarting; exitcode='6'" The version we are using is 3.23.1 I'm not sure why the disk buffer filled up the only thing I can think of is that syslog-ng was unable to send message to the Kafka destination, however our Kafka cluster is up and function fine and I can even consume messages from a topic that we are pushing messages to on one of the instances affected. A question once a reliable message is sent is it removed from the buffer, i assume that's the case. The other question is it possible the java Kafka plugin can't keep up with the amount of mesages we are sending though syslog-ng. We see 12.5 million requests per day to the servers with the disk buffers. Each request generates at least 2 messages, one access log message, one application log so that's roughly 25 million messages a day. I designed the disk buffers to hold roughly 2 days worth of data if for some reason the Kafka cluster went down which in this case doesn't seem to be the case since monitoring tools show it's online and none of our hourly consumers complained about being able to connect. I'll have to build a recovery server for the buffers since they are housed on attached drives the drive are preserved when the instance is terminated this was done to make sure we didn't lose any messages not sent when the autoscaling scaled out a instance. I'm currently manually terminating the affected instances to keep those buffer and restore out logging. Once I have it, if more info is needed I can see what happens with trying to start syslog-ng up when it's also not receiving a bunch of messages as well. Ronald Fenner Network Architect Game Circus LLC. rfenner(a)gamecircus.com

2 3

Error message in syslog-ng (LOGMSG_REFCACHE_VALUE_TO_REF(old_value) >= 1)
by Ahmed Soyef 29 Jul '20

29 Jul '20

Hi All, We're seeing some instances of this error message ERROR:lib/logmsg/logmsg.c:1521:log_msg_ref: assertion failed: (LOGMSG_REFCACHE_VALUE_TO_REF(old_value) >= 1) I've had a quick search about and can't figure out what it actually means. Can anyone shed some light and possibly some guidance on how to avoid it happening? Regards, Soyef ********************************************************************** This email and any files transmitted with it are private and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please return it to the address it came from telling them it is not for you and then delete it from your system. This email message has been swept for computer viruses. **********************************************************************

2 1

File destination is not being written, when it can not bind remote host address
by Alexandre Santos 29 Jul '20

29 Jul '20

Hi, I do not know if this a seen issue or not. I am using syslog-ng version 3.19 on a Debian 10. I have the configuration which is attached in this email: syslog-ng.conf, that has one local file destination and one remote host destination. 1. Start the syslog-ng The localip of syslog destination is not available and therefore I have these errors, as expected: syslog-ng[1482]: [2020-07-27T12:58:00.069422] Error binding socket; addr='AF_INET(10.0.2.6:0)', error='Cannot assign requested address (99)' syslog-ng[1482]: [2020-07-27T12:58:00.069439] Initiating connection failed, reconnecting; time_reopen='60' 2. After some time running syslog-ng stops sending the logs to the local file. Why? 3. If I reload the syslog-ng configuration, without the remote destination, the old logs are immediately flushed to the local file. Jul 27 *15:05:55* localhost syslog-ng[29680]: [*2020-07-27T15:05:55.378*019] Incoming log entry from journal; message='[*2020-07-27T13:44:06.890*952] Outgoing message; message=\'<30>1 2020-07-27T13:44:06.772+00:00 localhost syslog-ng 29680 - - [2020-07-27T13:44:06.772722] Processing the time zone file (32bit part); filename=\\'/usr/share/zoneinfo/UTC\\'\x0a\'' Jul 27 15:05:55 localhost syslog-ng[29680]: [2020-07-27T15:05:55.378032] json-parser(): no marker at the beginning of the message, skipping JSON parsing ; input='[2020-07-27T13:44:06.890952] Outgoing message; message=\'<30>1 2020-07-27T13:44:06.772+00:00 localhost syslog-ng 29680 - - [2020-07-27T13:44:06.772722] Processing the time zone file (32bit part); filename=\\'/usr/share/zoneinfo/UTC\\'\x0a\'', marker='@cim:' Is this a new issue? Is there any configuration to prevent this? In attachment also the journalctl of syslog-ng in debug mode (syslog-ng -Fvde). Thanks and regards, Alex

2 1