- syslog-ng - lists.balabit.hu

Re: [syslog-ng] multi-line logs and program/facility filters
by Michael Thénault 14 Aug '18

14 Aug '18

Ok, well that cannot be a solution for us because of different reasons: performance (embedded environment), probability to add bugs ... I guess we'll have to go reconsider keeping syslogd which doesn't have this issue. Regards, Michael

4 5

documentation for snmp destination driver
by Armando Mártires 14 Aug '18

14 Aug '18

Hello everyone, I'm trying to forward some log messages through SNMP with syslog-ng 3.16 buy I can't seem to find the documentation for the snmp() driver, can anyone help me with this? Thank you, Armando Mártires

2 1

found a release-critical bug, 3.17.2
by Laszlo Budai 10 Aug '18

10 Aug '18

Hi, we are in the middle of an investigation of a partial-write issue and found a bug that can lead to message loss in some circumstances, even when reliable diskbuffer is used. We marked this bug as release critical. Fortunately we fixed the issue a few minutes ago, so we are going to release 3.17.2 that contains the fix ( https://github.com/balabit/syslog-ng/pull/2224 ). We don't have an automated testcase for this, actually it is in progress(this is why we didn't catch the bug), but sharing the manual reproduction: * listen with netcat (and use pv tool to make partial write happen) nc -l 61002 | pv -L 40 * run syslog-ng with the following config: @version: 3.17 options { time_reopen(3); stats_level(1); }; source s_network { network(ip("127.0.0.1") port(5555)); }; destination d_network { network("127.0.0.1" disk_buffer(mem-buf-size(1000) disk-buf-size(1048576) reliable(yes) dir('/tmp/dq')) ip_protocol(4) port(6666) so-sndbuf(4096)); }; log { source(s_network); destination(d_network); flags(flow-control); }; * send logs loggen -r 100000 -I 200 127.0.0.1 10001 --inet * wait a few seconds then stop netcat * result: syslog-ng will drops all the incoming messages regards, Laszlo Budai

1 1

multi-line logs and program/facility filters
by Michael Thénault 10 Aug '18

10 Aug '18

Hello, I have an Issue with syslog-ng 3.16.1 and multi-line logs. I try to configure per-application filters using either the program name or a facility. The applications use the traditional syslog() from syslog.h. When an application logs multiple lines, only the first line is filtered. Indeed, the program name or facility is only applied to the first line. Example : $ logger -t testprog "line1 line2 line3" $ cat /var/log/messages 2018-08-10T16:26:14.000000+02:00 testprog: line1 2018-08-10T16:26:14.899505+02:00 line2 2018-08-10T16:26:14.899505+02:00 line3 The log source is unix-stream("/dev/log" ); What can I do to fix this ? Thanks in advance for your help. Thanks & Regards, Michael

2 1

unofficial packages for syslog-ng (Debian/Ubuntu)
by Laszlo Budai 10 Aug '18

10 Aug '18

Hi, syslog-ng 3.17.1[1] packages are available in OBS repo[2]. List of supported OSs: * Debian 8.0 * Debian 9.0 * Ubuntu 14.04 * Ubuntu 16.04 * Ubuntu 16.10 * Ubuntu 17.04 * Ubuntu 17.10 * Ubuntu 18.04 In case of Debian 9.0, armv7l architecture also supported, so you can install it to your Raspberry Pi. Install ------- example: Debian 9.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-get install syslog-ng-core` Links -------- [1] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.17.1 [2] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… regards, Laszlo Budai

5 5

OSE 3.17.1
by Laszlo Budai 09 Aug '18

09 Aug '18

3.17.1 Features * Client side failback mode (#2183<https://github.com/balabit/syslog-ng/pull/2183>) * New linux-audit() source as SCL (#2186<https://github.com/balabit/syslog-ng/pull/2186>) * Decorating generated configuration (#2218<https://github.com/balabit/syslog-ng/pull/2218>) * Introduce ewmm() source (#2199<https://github.com/balabit/syslog-ng/pull/2199>, #2209<https://github.com/balabit/syslog-ng/pull/2209>) * Add parsing of Cisco unified call manager (#2134<https://github.com/balabit/syslog-ng/pull/2134>) * Mandatory parameters for cfg-block (SCL) (#2088<https://github.com/balabit/syslog-ng/pull/2088>) Bugfixes * dqtool cat print backlog items (#2213<https://github.com/balabit/syslog-ng/pull/2213>) * Rewind backlog in case of stateless LogProtoClient (#2214<https://github.com/balabit/syslog-ng/pull/2214>) * Filter out incorrectly parsed sudo logs (#2208<https://github.com/balabit/syslog-ng/pull/2208>) * Minor fixes related to client-lib-dir, loggen and eventlog (#2204<https://github.com/balabit/syslog-ng/pull/2204>) * Minor stats-query fixes and refactor (#2193<https://github.com/balabit/syslog-ng/pull/2193>) * Reliable disk buffer non-empty backlog (#2192<https://github.com/balabit/syslog-ng/pull/2192>) * Fix pip package versions on older distro releases (dbld) (#2188<https://github.com/balabit/syslog-ng/pull/2188>) * Fix a groupset/groupunset and map-value-pairs() crash (#2184<https://github.com/balabit/syslog-ng/pull/2184>) * Make g_atomic_counter_set() atomic and update minimum glib version to 2.26 (#2182<https://github.com/balabit/syslog-ng/pull/2182>) * Aligning java related SCLs with mandatory parameters (#2160<https://github.com/balabit/syslog-ng/pull/2160>) * Loggen minor fixes (#2150<https://github.com/balabit/syslog-ng/pull/2150>) * grab-logging should be installed as a header (#2151<https://github.com/balabit/syslog-ng/pull/2151>) * Fix possible underflow of memory_usage (afsql, logqueue-fifo) (#2140<https://github.com/balabit/syslog-ng/pull/2140>) * Fix SELinux module version inconsistency (#2133<https://github.com/balabit/syslog-ng/pull/2133>) * Fix CMake unit test compilation (no-pie) (#2137<https://github.com/balabit/syslog-ng/pull/2137>) * Fix possible crash in syslog-parser() (#2128<https://github.com/balabit/syslog-ng/pull/2128>) * Disable ack for mark mode (#2129<https://github.com/balabit/syslog-ng/pull/2129>) * Fixing a Telegram destination bug with bot id (#2142<https://github.com/balabit/syslog-ng/pull/2142>) * All drivers should support inner destination or source plugins (#2143<https://github.com/balabit/syslog-ng/pull/2143>) * Fix default file and directory creation ownership (#2119<https://github.com/balabit/syslog-ng/pull/2119>) * Fix global "center;;received" counter when systemd-journal() is used (#2121<https://github.com/balabit/syslog-ng/pull/2121>) * Link everything to libsecret-storage (#2100<https://github.com/balabit/syslog-ng/pull/2100>) * Inform about the right dns-cache() configuration (warning message typo) (#2145<https://github.com/balabit/syslog-ng/pull/2145>) * Adjusting window size for internal mark mode (#2146<https://github.com/balabit/syslog-ng/pull/2146>) * Fix memory leaks in disk-buffer() (#2153<https://github.com/balabit/syslog-ng/pull/2153>) * Fix undefined behavior in log multiplexer (#2154<https://github.com/balabit/syslog-ng/pull/2154>) * Fix static linking mode (autotools) (#2155<https://github.com/balabit/syslog-ng/pull/2155>) * Fix internal mark mode infinite loop with old ivykis (#2157<https://github.com/balabit/syslog-ng/pull/2157>) * Fix missing normalize flags (#2162<https://github.com/balabit/syslog-ng/pull/2162>) * Keep JVM running on reload if once configured (#2164<https://github.com/balabit/syslog-ng/pull/2164>, #2211<https://github.com/balabit/syslog-ng/pull/2211>) * Fix a race condition (suspend) in LogSource (#2167<https://github.com/balabit/syslog-ng/pull/2167>) * Add @requires json-plugin to the cim() parser (#2181<https://github.com/balabit/syslog-ng/pull/2181>) * Added exclude_kmsg option to system source (#2166<https://github.com/balabit/syslog-ng/pull/2166>) * Fix padding template function (#2174<https://github.com/balabit/syslog-ng/pull/2174>) * Leak & invalid memory access (#2173<https://github.com/balabit/syslog-ng/pull/2173>) * FreeBSD 11.2 builderror SOCK_STREAM (#2170<https://github.com/balabit/syslog-ng/pull/2170>) * Add ref-counted TLSVerifier struct (use after free fix) (#2168<https://github.com/balabit/syslog-ng/pull/2168>) Other changes * Improve loggen's file message parser (#2205<https://github.com/balabit/syslog-ng/pull/2205>) * syslog-ng-debun improvements (#2201<https://github.com/balabit/syslog-ng/pull/2201>) * Goodbye "goto relex" (refactor) (#2198<https://github.com/balabit/syslog-ng/pull/2198>) * Refactor the callback registration mechanism of WildcardFileReader (#2185<https://github.com/balabit/syslog-ng/pull/2185>) * Extended Linux capabilities detection (pkg-config) (#2169<https://github.com/balabit/syslog-ng/pull/2169>) * Add atomic gssize (#2159<https://github.com/balabit/syslog-ng/pull/2159>) * Lower the message level of @requires to debug (#2147<https://github.com/balabit/syslog-ng/pull/2147>) * macOS warning elimination (#2139<https://github.com/balabit/syslog-ng/pull/2139>) * Remove a misleading rewrite-related debug message (#2132<https://github.com/balabit/syslog-ng/pull/2132>) * Minor updates to SELinux policy installer script (#2127<https://github.com/balabit/syslog-ng/pull/2127>) * More robust GLib detection (CMake) (#2125<https://github.com/balabit/syslog-ng/pull/2125>) * Logthreaded nonfunctional changes (#2123<https://github.com/balabit/syslog-ng/pull/2123>) * Confgen and pragma improvements (#2122<https://github.com/balabit/syslog-ng/pull/2122>) * Flush before stopping syslog-ng (functional tests) (#2216<https://github.com/balabit/syslog-ng/pull/2216>) * Port unit tests into criterion (test_filters_netmask6, test_findeom, csv_parser, patternDB) (#2217<https://github.com/balabit/syslog-ng/pull/2217>, #2175<https://github.com/balabit/syslog-ng/pull/2175>, #2118<https://github.com/balabit/syslog-ng/pull/2118>) * Libtest refactors (#2149<https://github.com/balabit/syslog-ng/pull/2149>) * Add missing files to the source tarball (#2114<https://github.com/balabit/syslog-ng/pull/2114>) * Better python binary detection (#2092<https://github.com/balabit/syslog-ng/pull/2092>) Notes to the developers * LogThreadedDestDriver batching (#2063<https://github.com/balabit/syslog-ng/pull/2063>) * Add sqlite3 and riemann to dbld devshell (#2210<https://github.com/balabit/syslog-ng/pull/2210>) * Make mock-transport inheritable (#2120<https://github.com/balabit/syslog-ng/pull/2120>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Balazs Scheidler, Bernie Harris, Bertrand Jacquin, Gabor Nagy, Gergely Nagy, German Service Network, Janos SZIGETVARI, Laszlo Budai, Laszlo Szemere, László Várady, Norbert Takacs, Peter Czanik, Peter Kokai, Szigetvari Janos, Terez Nemes, Viktor Juhasz. https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.17.1 [https://avatars1.githubusercontent.com/u/986981?s=400&v=4]<https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.17.1> balabit/syslog-ng<https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.17.1> syslog-ng is an enhanced log daemon, supporting a wide range of input and output methods: syslog, unstructured text, queueing, SQL & NoSQL. github.com regards, Laszlo Budai

1 0

syslog-ng-3.13.2 compilation issue
by R.Naveen Naidu 07 Aug '18

07 Aug '18

Hi, I am trying to cross compile "syslog-ng-3.13.2" with my Linux and getting below message. Do I need to include the PCRE libs to my syslog-ng Makefile where I am including the syslog-ng. Please suggest me how to avoid this error. *Logs:* checking for inotify_init... yes checking for TCP wrapper library... -lwrap checking whether to enable TCP wrapper support... yes checking for dlsym in -ldl... yes checking for LIBDBI... no checking for dbi_initialize_r in -ldbi... no checking whether to enable SQL support... no checking for GLIB... yes checking for g_mapped_file_unref... yes checking for g_list_copy_deep... no checking sanity checking Glib headers... yes checking for GEOIP... no checking for MAXMINDDB... checking for MMDB_open in -lmaxminddb... no checking for PCRE... no configure: error: Cannot find pcre version >= 6.1 it is a hard dependency from syslog-ng 3.6 onwards make[11]: *** [syslog-ng-3.13.2/config.status] Error 1 -- Thanks & Regards Naveen

3 2

Syslog-ng and RSA Archer
by Carter, Jed 31 Jul '18

31 Jul '18

Hi, We are trying to configure RSA Archer to send its logs to our syslog-ng server. When the engineers push data to syslog-ng we see it in the logs but the Archer engineers say they are the three-way handshake occur so Archer stops sending data. They seem to think it's our config but I have RSA Netwitness sending logs to syslog-ng without any issues. Has anyone seen this problem before or know of a solution that I can try? Thanks, Jed The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.

2 1

Forcing remote syslog request from 1 IP into 1 log file
by freebsd＠tango.lu 27 Jul '18

27 Jul '18

Hello, I have a syslog server setup which works quite well for most of the hosts: destination d_net_debug { file("/var/log/$HOST/debug"); }; destination d_net_error { file("/var/log/$HOST/error"); }; This way I dont have to define every host which logs there but they will be autocreated. I have a quite misbehaving Asus router device however which keep sending strings like: FTP WAN(11) WAN(8) WAN(3) as host therefore syslog-ng interpret these messages like it would be coming from different $HOSTs and keep creating directories for them. 2018-07-22T20:45:59+02:00 WAN Connection: Wan link down. 2018-07-24T16:12:20+02:00 WAN Connection: Wan link down. 2018-07-22T20:45:59+02:00 WAN Connection: Wan link down. 2018-07-24T16:12:20+02:00 WAN Connection: Wan link down. How do I force all the logs into one logfile for this one specific host? If possible I don't want to change my current rules just extend them. Thank you.

4 3

Asynchronous address resolution using getaddrinfo_a()
by Naveen Revanna 26 Jul '18

26 Jul '18

Hi Developers, When DNS server is unreachable, getaddrinfo() function will block (until it times out after few tens of seconds). If syslog-ng application is configured with a remote syslog server using its hostname, afsocket_dd_try_connect() will try to resolve this address in a loop (using a timer). Since getaddrinfo() is blocking, execution of this task will take few seconds thereby delaying the execution of other tasks. Eventually this will reach a state in which there will be lot of backlogs of tasks and syslog() appears to hang, delaying the execution of all shell commands (by upto a minute). Resolving the remote syslog server's address by having an entry in /etc/host could be a possible workaround (as indicated in a previous thread). However, this is not a desirable solution for our use case. Here is my thought on a possible fix that I am thinking to work on for which I am looking for feedback. 1. I am thinking of using getaddrinfo_a() ( http://man7.org/linux/man-pages/man3/getaddrinfo_a.3.html) in the afsocket_dd_try_connect() loop function. We can have this async call only in case it starts looping and not for the first attempt. The first attempt can continue to do the current sync way. 2. If I were to take the above approach, there are two ways in which we can know the status of address resolution. Which one do you folks think would be better: 1. SIGEV_SIGNAL: We can receive a signal when a look-up completes. We can take further action in the handler function. 2. SIGEV_THREAD: A notification function will be called. This results in creation of a new thread (pthread?). What I am not sure of is the impact of this thread creation on the existing thread infrastructure in syslog-ng through ivykis. Repro: It is fairly easy to reproduce this issue 1. Configure a remote syslog server using it's hostname. 2. Make the dns unreachable (Edit resolve.conf and put wrong IP(s) for 'nameserver' entries) 3. $ syslog-ng-ctl reload 4. Observe that any command executed on the shell takes unusually long time. I can elaborate on any of the items here if something is not clear. Appreciate any pointers. Thanks, Naveen

4 4