- syslog-ng - lists.balabit.hu

Error trying to use kafka-c
by Ronald Fenner 05 Aug '20

05 Aug '20

I'm trying to switch from the java version of kafka to the c version and have built syslog with the following config: ./configure --prefix=/opt/syslog-ng --enable-json --enable-python --with-python=2.7 --enable-kafka However when I try syntax check the config it's returning this error: Error parsing destination statement, destination plugin kafka-c not found in block destination kafka() at /opt/syslog-ng/share/syslog-ng/include/scl/kafka/kafka.conf:24:4:5-4:12: 1 2 #Start Block block destination kafka() at /opt/syslog-ng/share/syslog-ng/include/scl/kafka/kafka.conf:24 3 4-----> kafka-c(bootstrap_servers("kafka.service.consul:9092") topic("syslog-ng-production") ); 4-----> ^^^^^^^ 5 6 #End Block block destination kafka() at /opt/syslog-ng/share/syslog-ng/include/scl/kafka/kafka.conf:24 7 Included from /opt/syslog-ng/etc/syslog-ng.conf:33:5-36:6: 28 source s_rtl_raw_backup { 29 file("/var/gamecircus/log/rtl-raw-backup.log" flags(no-parse) log_msg_size(1000000)); 30 }; 31 32 destination d_kafka_unstructured { 33----> kafka ( 33----> ^^^^^^^^ 34----> bootstrap-servers("kafka.service.consul:9092") 35----> topic("syslog-ng-`DeployedEnvironment`") 36----> ); 37 }; 38 The top of my config looks like this @version: 3.28 @include "scl.conf" @include "conf.d/gc-parsers.conf" @define kafka-implementation kafka-c Everything looks correct just not sure why it's having a hard time finding it. Ronald Fenner Network Architect Game Circus LLC. rfenner(a)gamecircus.com

2 1

Deadlock with full disk buffer
by Ronald Fenner 04 Aug '20

04 Aug '20

I've run into a problem where we had enabled reliable disk buffering on our Kafka destination and it's disk buffer eventually filled the drive it's stored on party because the drive is sized to the buffer it's holding. When restarting syslog-ng it complains with "Daemon exited due to a deadlock/signal/failure, restarting; exitcode='6'" The version we are using is 3.23.1 I'm not sure why the disk buffer filled up the only thing I can think of is that syslog-ng was unable to send message to the Kafka destination, however our Kafka cluster is up and function fine and I can even consume messages from a topic that we are pushing messages to on one of the instances affected. A question once a reliable message is sent is it removed from the buffer, i assume that's the case. The other question is it possible the java Kafka plugin can't keep up with the amount of mesages we are sending though syslog-ng. We see 12.5 million requests per day to the servers with the disk buffers. Each request generates at least 2 messages, one access log message, one application log so that's roughly 25 million messages a day. I designed the disk buffers to hold roughly 2 days worth of data if for some reason the Kafka cluster went down which in this case doesn't seem to be the case since monitoring tools show it's online and none of our hourly consumers complained about being able to connect. I'll have to build a recovery server for the buffers since they are housed on attached drives the drive are preserved when the instance is terminated this was done to make sure we didn't lose any messages not sent when the autoscaling scaled out a instance. I'm currently manually terminating the affected instances to keep those buffer and restore out logging. Once I have it, if more info is needed I can see what happens with trying to start syslog-ng up when it's also not receiving a bunch of messages as well. Ronald Fenner Network Architect Game Circus LLC. rfenner(a)gamecircus.com

2 3

Error message in syslog-ng (LOGMSG_REFCACHE_VALUE_TO_REF(old_value) >= 1)
by Ahmed Soyef 29 Jul '20

29 Jul '20

Hi All, We're seeing some instances of this error message ERROR:lib/logmsg/logmsg.c:1521:log_msg_ref: assertion failed: (LOGMSG_REFCACHE_VALUE_TO_REF(old_value) >= 1) I've had a quick search about and can't figure out what it actually means. Can anyone shed some light and possibly some guidance on how to avoid it happening? Regards, Soyef ********************************************************************** This email and any files transmitted with it are private and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please return it to the address it came from telling them it is not for you and then delete it from your system. This email message has been swept for computer viruses. **********************************************************************

2 1

File destination is not being written, when it can not bind remote host address
by Alexandre Santos 29 Jul '20

29 Jul '20

Hi, I do not know if this a seen issue or not. I am using syslog-ng version 3.19 on a Debian 10. I have the configuration which is attached in this email: syslog-ng.conf, that has one local file destination and one remote host destination. 1. Start the syslog-ng The localip of syslog destination is not available and therefore I have these errors, as expected: syslog-ng[1482]: [2020-07-27T12:58:00.069422] Error binding socket; addr='AF_INET(10.0.2.6:0)', error='Cannot assign requested address (99)' syslog-ng[1482]: [2020-07-27T12:58:00.069439] Initiating connection failed, reconnecting; time_reopen='60' 2. After some time running syslog-ng stops sending the logs to the local file. Why? 3. If I reload the syslog-ng configuration, without the remote destination, the old logs are immediately flushed to the local file. Jul 27 *15:05:55* localhost syslog-ng[29680]: [*2020-07-27T15:05:55.378*019] Incoming log entry from journal; message='[*2020-07-27T13:44:06.890*952] Outgoing message; message=\'<30>1 2020-07-27T13:44:06.772+00:00 localhost syslog-ng 29680 - - [2020-07-27T13:44:06.772722] Processing the time zone file (32bit part); filename=\\'/usr/share/zoneinfo/UTC\\'\x0a\'' Jul 27 15:05:55 localhost syslog-ng[29680]: [2020-07-27T15:05:55.378032] json-parser(): no marker at the beginning of the message, skipping JSON parsing ; input='[2020-07-27T13:44:06.890952] Outgoing message; message=\'<30>1 2020-07-27T13:44:06.772+00:00 localhost syslog-ng 29680 - - [2020-07-27T13:44:06.772722] Processing the time zone file (32bit part); filename=\\'/usr/share/zoneinfo/UTC\\'\x0a\'', marker='@cim:' Is this a new issue? Is there any configuration to prevent this? In attachment also the journalctl of syslog-ng in debug mode (syslog-ng -Fvde). Thanks and regards, Alex

2 1

setting sequenceId in forwarded log messages read from journald reader
by Peter Vollmer 29 Jul '20

29 Jul '20

Hi, I am currently trying to find a way to set meta.sequenceId of log messages that have been read from the locally running systemd-journal to forward them to a remote syslog server that expects the logs to contain a sequenceId according to RFC 5424 section 7.3.1. I found that a sequence number could be taken from the __CURSOR field "i=..." of the journald log: # journalctl -o json-pretty -f ... "__CURSOR" : "s=02a7b30ba17b4a43846f265706bd3a70;i=f01;b=ba633698f20848e480bca4e72476e4d3;m=1a355c1d5;t=5ab670340c8ea;x=33389988ef680e7e", ... My problem is that the journal reader does not seem to parse the __CURSOR string when reading from journald logs. Is there a way to get this information into meta.sequenceId of the forwarded log without modifying the systemd-journal module in syslog-ng ? Thank you for any ideas and best regards Peter Vollmer

3 2

Certificate ip authentication
by Alexandre Damas 27 Jul '20

27 Jul '20

Hi, As my objective on the utilization of syslog-ng is the authentication and certification of received security and auditing events, I implemented an internal CA which generates a certificate per syslog-ng client. Once there was someone that misconfigured a client and got a certificate for a different client. The funny part of it is that no one observed any problem as the certificate, which was generated for a client having an ip (different from the one configured on the machine), was working and communications (using ALTP with TLS) went up for the exchange of messages. Has anyone experienced this? Does someone have a clue on how to prevent certificate reutilization on the client side? On Linux side I did not find any way of preventing the utilization of a certificate for one machine, that was issued for a different machine, having a different ip. If there is no cross checking on the server for the ip addresses on the certificate with the ip address on the received event, I don't have a way of non-repudiate a received event and the client can reuse the certificate for other machines, allowing the events to be received on the server. Kind Regards Alexandre Damas

2 2

Configuration validation Issue with snmptrap module
by Luca Da Col 23 Jul '20

23 Jul '20

Running syslog-ng 3.27.1 on Centos 7. Configuration validation returns an error with snmptrap command. Configuration example has been taken directly from the OSE documentation. Error error was reproducible with version available on epel repository (3.5), though documentation indicates the snmptrap is available from 3.10 Configuration file includes statements: @version:3.22 @include "scl.conf" Any assistance is appreciated, Thank you, Luca [root@adev syslog-ng]# syslog-ng --syntax-only Error parsing source statement, source plugin snmptrap not found in /etc/syslog-ng/syslog-ng.conf:92:5-92:13: 90 log { 91 source { 92----> snmptrap(filename("/var/log/snmptrapd.log")); 92----> ^^^^^^^^ 93 }; 94 destination { 95 file("/var/log/example.log"); 96 }; 97 }; [root@dev syslog-ng]# syslog-ng -V syslog-ng 3 (3.27.1) Config version: 3.22 Installer-Version: 3.27.1 Revision: Compile-Date: Jun 8 2020 07:54:17 Module-Directory: /usr/lib64/syslog-ng Module-Path: /usr/lib64/syslog-ng Include-Path: /usr/share/syslog-ng/include Available-Modules: affile,afprog,afsocket,afstomp,afuser,basicfuncs,confgen,cryptofuncs,csvparser,dbparser,linux-kmsg-format,syslogformat,system-source,add-contextual-data,appmodel,cef,disk-buffer,examples,graphite,hook-commands,json-plugin,kvformat,map-value-pairs,pseudofile,sdjournal,stardate,tags-parser,tfgetent,timestamp,xml Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Systemd: on [root@dev syslog-ng]#

2 1

syslog-ng multiple VRF
by Alexandre Santos 22 Jul '20

22 Jul '20

Hi All, I have a question regarding syslog-ng and VRF. I want to read from a syslog source, which interface is in the default VRF, and send the logs to a syslog/network destination interface which is in a MGMT VRF. Can syslog-ng support this? If yes, what are the aspects I should be careful about? Thanks and regards, Alex

2 2

syslog-ng restarts due to interface not ready
by Alexandre Santos 21 Jul '20

21 Jul '20

Hi, I have a configuration where one of my sources is a syslog source: syslog(ip(127.1.1.1) transport("udp") port(514) keep-alive(no)); But when the system starts up, the interface is not ready yet, and I get the following errors: Jun 26 09:17:50 localhost systemd[1]: Starting System Logger Daemon... Jun 26 09:17:52 localhost syslog-ng[2301]: [2020-06-26T09:17:52.485148] WARNING: With use-dns(no), dns-cache() will be forced to 'no' too!; Jun 26 09:17:52 localhost syslog-ng[2301]: [2020-06-26T09:17:52.487159] Current position not matches to the saved cursor position, seek to head; cursor='s=f62c0e80f50948deb29cf2d28230a Jun 26 09:17:52 localhost syslog-ng[2301]: [2020-06-26T09:17:52.487867] Error resolving hostname; host='127.1.1.1' Jun 26 09:17:52 localhost syslog-ng[2301]: [2020-06-26T09:17:52.487878] Error initializing message pipeline; plugin_name='syslog', location='/etc/syslog-ng/syslog-ng.conf:43:5' Jun 26 09:17:52 localhost systemd[1]: syslog-ng.service: Main process exited, code=exited, status=2/INVALIDARGUMENT Jun 26 09:17:52 localhost systemd[1]: syslog-ng.service: Failed with result 'exit-code'. Jun 26 09:17:52 localhost systemd[1]: Failed to start System Logger Daemon. Jun 26 09:17:52 localhost systemd[1]: syslog-ng.service: Service RestartSec=100ms expired, scheduling restart. Jun 26 09:17:52 localhost systemd[1]: syslog-ng.service: Scheduled restart job, restart counter is at 1. When the systemd reaches the 5 attempts it disables the syslog-ng.service. Do you know some way to fix this, by adding something, the syslog-ng configuration? Thanks in Advance, Alex

4 4

disk buffer files permissions
by Pal, Laszlo 13 Jul '20

13 Jul '20

Hi, We want to monitor disk-buffer files, so we can detect if there is are some issues with the destinations. syslog-ng creates these files with permissions like this otal 360 -rw-------. 1 root root 4096 Oct 10 2019 syslog-ng-00000.qf -rw-------. 1 root root 4096 Jul 12 03:49 syslog-ng-00001.qf So, our monitor script (for zabbix) cannot stat files because it is running under user zabbix. I'm not sure how to submit a feature request nowadays, but it would be great of perm() option added to disk-buffer() Any chance? Thx L:

3 4