- syslog-ng - lists.balabit.hu

Insider 2021-03: Kafka; Windows; Bastille;
by Peter Czanik (pczanik) 11 Mar '21

11 Mar '21

Dear syslog-ng users, This is the 89th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Consuming logs from a Kafka topic --------------------------------- There is no official Kafka source in syslog-ng, but because this question comes up often enough, I created one. It is just a temporary workaround using the program() source, but it works. It involves Java and installing Kafka manually, but it was fast and reliabe in my tests: ingesting 50,000–100,000 messages a second on my laptop in a resource-constrained virtual machine. https://www.syslog-ng.com/community/b/blog/posts/consuming-logs-from-a-kafk… Collecting logs from Windows ---------------------------- Normally I cover free and open-source software in the syslog-ng blog. But recently quite a few members of the community reached out to me and asked about collecting logs from Windows. So, I prepared a quick overview of the topic. The good news is, that syslog-ng supports collecting logs from Windows in multiple ways. The not so good news is that Windows support is only available in the commercial version of syslog-ng. There are multiple ways for collecting log messages from Windows. You can either install syslog-ng agents on Windows hosts, or you can use the Windows Event Collector (WEC) component of syslog-ng PE. https://www.syslog-ng.com/community/b/blog/posts/collecting-logs-from-windo… Running syslog-ng in Bastille ----------------------------- Bastille is a container management system for FreeBSD, similar to Docker or Podman on Linux. The historical name of containers on FreeBSD is jail, and they appeared a lot earlier than containers on Linux. Managing jails was not always easy. When I started to use this technology in production in 2001, nothing was automated. Using Bastille, you can easily create, configure, or update jails at scale. It has a template system to install applications in containers and there is a template also for syslog-ng. From this blog, you can learn how to get started with Bastille and how to create and run a syslog-ng jail using the freshly released 0.8 version of Bastille. https://www.syslog-ng.com/community/b/blog/posts/running-syslog-ng-in-basti… Syslog-ng OSE 3.31.1 released ----------------------------- Version 3.31.1 of syslog-ng OSE was released with the Fortigate parser, many parsing-related enhancements, silent Telegram messages, and more. For a complete list of changes, check https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-3.31.1 WEBINARS * Scale log collection from Windows endpoints with WEC clustering in syslog-ng https://www.syslog-ng.com/event/scale-log-collection-from-windows-endpoints… * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Syslog-ng segfault at version 3.8.1
by Sam Kappen 08 Mar '21

08 Mar '21

Hi All, We have an issue from a customer who uses syslog-ng version 3.8.1 on the 3.10.108 kernel(Hardware: freescale-p2020ds-3.10-1.4). Customer observed a segmentation fault after syslog-ng being stuck for more than one hour. There are no reproducible steps available, the issue is seen as random. As per the backtrace segfault happened from "malloc_consolidate" which looks an odd behaviour.On checking the frame it looks mchunkptr "p" has the value 0x2 which caused this crash, but not been able to find out what is the root cause for this. Has there been any known issue with syslog-ng? Any idea if this issue is fixed in version 3.16.1. Any help or pointers would be appreciated. Attached the back trace. Back trace snippet: Program terminated with signal 11, Segmentation fault. #0 malloc_consolidate (av=av@entry=0xffc738c <main_arena>) at malloc.c:4070 4070 malloc.c: No such file or directory. (gdb) bt #0 malloc_consolidate (av=av@entry=0xffc738c <main_arena>) at malloc.c:4070 #1 0x0fec5cb4 in _int_malloc (av=av@entry=0xffc738c <main_arena>, bytes=bytes@entry=556) at malloc.c:3374 #2 0x0fec6f64 in __GI___libc_malloc (bytes=556) at malloc.c:2874 #3 0x0f90d58c in standard_malloc (n_bytes=<optimized out>) at gmem.c:85 #4 0x0f90d9f0 in g_malloc (n_bytes=n_bytes@entry=556) at gmem.c:159 #5 0x0f7be00c in log_msg_alloc (payload_size=256) at lib/logmsg/logmsg.c:1084 #6 log_msg_new_empty () at lib/logmsg/logmsg.c:1150 #7 0x0f7be088 in log_msg_new_local () at lib/logmsg/logmsg.c:1160 #8 0x0f7be194 in log_msg_new_mark () at lib/logmsg/logmsg.c:1260 #9 0x0f7713e8 in afinter_source_mark (s=0x10863690) at lib/afinter.c:142 #10 0x0f7c4c74 in iv_run_timers (st=st@entry=0x10284d30) at iv_timer.c:149 #11 0x0f7c72bc in iv_main () at iv_main_posix.c:107 #12 0x0f78b990 in main_loop_run () at lib/mainloop.c:497 #13 0x10001260 in main (argc=1, argv=0xbfbe7724) at syslog-ng/main.c:270 (gdb) Regards, Sam

1 0

syslog-ng-3.31.1
by Peter Kokai (pkokai) 08 Mar '21

08 Mar '21

3.31.1 ====== ## Highlights * fortigate-parser(): new parser to parse fortigate logs Example: ``` log { source { network(transport("udp") flags(no-parse)); }; parser { fortigate-parser(); }; destination { }; }; ``` An adapter to automatically recognize fortigate logs in app-parser() has also been added. ([#3536](https://github.com/syslog-ng/syslog-ng/pull/3536)) * `patterndb`: Added `OPTIONALSET` parser. It works the same as `SET`, but continues, even if none of the characters is found. ([#3540](https://github.com/syslog-ng/syslog-ng/pull/3540)) ## Features * `syslog-parser()`: add no-header flag to tell syslog-ng to parse only the PRI field of an incoming message, everything else is just put into $MSG. ([#3538](https://github.com/syslog-ng/syslog-ng/pull/3538)) * `set-pri()`: this new rewrite operation allows you to change the PRI value of a message based on the string directly parsed out of a syslog header. ([#3546](https://github.com/syslog-ng/syslog-ng/pull/3546)) * telegram: option to send silent message Example: ``` destination { telegram(bot-id(...) chat-id(...) disable_notification(true)); }; ``` ([#3558](https://github.com/syslog-ng/syslog-ng/pull/3558)) * `app-parser()`: added automatic classification & parsing for project Lumberjack/Mitre CEE formatted logs ([#3569](https://github.com/syslog-ng/syslog-ng/pull/3569)) * diskq: if the dir() path provided by the user does not exists, syslog-ng creates the path with the same permission as the running instance ([#3550](https://github.com/syslog-ng/syslog-ng/pull/3550)) ## Bugfixes * `network()`, `syslog()` destinations: fix reconnection timer when DNS lookups are slow After a long-lasting DNS query, syslog-ng did not wait the specified time (`time_reopen()`) before reconnecting to a destination. This has been fixed. ([#3526](https://github.com/syslog-ng/syslog-ng/pull/3526)) * cmake: minor fixes ([#3523](https://github.com/syslog-ng/syslog-ng/pull/3523)) * `stats-level()`: fix processing the changes in the stats-level() global option: changes in stats-level() were not reflected in syslog facility/severity related and message tag related counters after first configuration reload. These counters continued to operate according to the value of stats-level() at the first reload. ([#3561](https://github.com/syslog-ng/syslog-ng/pull/3561)) * `date-parser()`: fix hour-only timezone parsing If the timestamp contains a short timezone offset (e.g. hours only), the recent change in strptime() introduces an error, it interprets those numbers as minutes instead of hours. For example: Jan 16 2019 18:23:12 +05 ([#3555](https://github.com/syslog-ng/syslog-ng/pull/3555)) * `loggen`: fix undefined timeout while connecting to network sources (`glib < 2.32`) When compiling syslog-ng with old glib versions (< 2.32), `loggen` could fail due a timeout bug. This has been fixed. ([#3504](https://github.com/syslog-ng/syslog-ng/pull/3504)) * `grouping-by()`: fix deadlock when context expires In certain cases, the `grouping-by()` parser could get stuck when a message context expired, causing a deadlock in syslog-ng. This has been fixed. ([#3515](https://github.com/syslog-ng/syslog-ng/pull/3515)) * `date-parser`: Fixed a crash, which occured sometimes when `%z` was used. ([#3553](https://github.com/syslog-ng/syslog-ng/pull/3553)) * `date-parser`: `%z`. We no longer ignore daylight saving time when calculating the GMT offset. ([#3553](https://github.com/syslog-ng/syslog-ng/pull/3553)) * `kafka-c`: fix a double LogMessage acknowledgement bug, which can cause crash with segmentation fault or exit with sigabrt. The issue affects both flow-controlled and non-flow-controlled log paths and it's triggered in case previously published messages failed to be delivered to Kafka. ([#3583](https://github.com/syslog-ng/syslog-ng/pull/3583)) * `python destination`: Fixed a rare crash during reload. ([#3568](https://github.com/syslog-ng/syslog-ng/pull/3568)) * `date-parser()`: fix non-mandatory parsing of timezone name When %Z is used, the presence of the timezone qualifier is not mandatory, so don't fail that case. ([#3555](https://github.com/syslog-ng/syslog-ng/pull/3555)) * `wildcard-file()`: fix infrequent crash when file renamed/recreated The wildcard-file source crashed when a file being processed was replaced by a new one on the same path (renamed, deleted+recreated, rotated, etc.). ([#3513](https://github.com/syslog-ng/syslog-ng/pull/3513)) * Remove the no-parse flag in system() source from FreeBSD kernel messages, so the message header is no more part of the message. ([#3586](https://github.com/syslog-ng/syslog-ng/pull/3586)) * Fix abort on macOS Big Sur A basic subset of syslog-ng's functionality now works on the latest macOS version. ([#3522](https://github.com/syslog-ng/syslog-ng/pull/3522)) * `affile`: Fix improper initialization in affile and LogWriter to avoid memory leak when reloading ([#3574](https://github.com/syslog-ng/syslog-ng/pull/3574)) * `udp destination`: Fixed a bug, where the packet's checksum was not calculated, when `spoof-source(yes)` and `ip-protocol(6)` were set. ([#3528](https://github.com/syslog-ng/syslog-ng/pull/3528)) * `python`: fix LogMessage.keys() listing non-existenting keys and duplicates ([#3557](https://github.com/syslog-ng/syslog-ng/pull/3557)) ## Packaging * Simplify spec file by removing obsolete technologies: - remove RHEL 6 support - remove Python 2 support - keep Java support, but remove Java-based drivers (HDFS, etc.) ([#3587](https://github.com/syslog-ng/syslog-ng/pull/3587)) * `libnet`: Minimal libnet version is now 1.1.6. ([#3528](https://github.com/syslog-ng/syslog-ng/pull/3528)) * configure: added new --enable-manpages-install option along with the existing --enable-manpages. The new option would install pre-existing manpages even without the DocBook tools installed. ([#3493](https://github.com/syslog-ng/syslog-ng/pull/3493)) ## Notes to developers * `apphook`: the concept of hook run modes were introduced, adding support for two modes: AHM_RUN_ONCE (the original behavior) and AHM_RUN_REPEAT (the new behavior with the hook repeatedly called after registration). ([#3561](https://github.com/syslog-ng/syslog-ng/pull/3561)) ## Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: 0140454, Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler, egorbeliy, Gabor Nagy, Laszlo Budai, Laszlo Szemere, László Várady, Michael Ducharme, Norbert Takacs, Peter Czanik, Peter Kokai, Pratik raj, Ryan Faircloth, Zoltan Pallagi

1 0

Syslog-ng 3.4.5 support question
by mukesh kumar 05 Mar '21

05 Mar '21

Hi, We are using syslog-ng 3.4.5 version and we are using it to utilize patterndb. We are not looking for bug fix but using it for just all the different ways we can use the patterndb xml. Please provide me the info on such as the IF and ELSE statements, For loops in the xml file. Thanks, Mukesh Kumar

4 5

CentOS Stream
by Peter Czanik (pczanik) 03 Mar '21

03 Mar '21

Hi, CentOS is right now the most popular platform to run syslog-ng. On the other hand, they are shifting focus to CentOS Stream: https://blog.centos.org/2020/12/future-is-centos-stream/ Due to these changes many people are looking now for alternative platforms. Replacements for regular CentOS releases are under way: AlmaLinux is already in RC phase, Rocky Linux is arriving soon. I heard many people switching to openSUSE. And of course, there are also non-rpm alternatives, like Ubuntu or even FreeBSD. As I'm doing syslog-ng packaging for our most popular platforms (Fedora/CentOS/RHEL, openSUSE/SLES, FreeBSD), I'm curious what to expect on the packaging frontier. What will be our future platforms? I'm running a poll on Twitter: https://twitter.com/PCzanik/status/1366391907748831235 If your choice is "other", I'd love to know what is that other platform! Thanks for your feedback, Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

syslog-ng dumps core (twice yesterday) how do I investigate cause?
by David Lacerte 01 Mar '21

01 Mar '21

Running syslog-ng-3.5.4.1/ on CentOS 6.5 started seeing core dumps in var.log/kern approximately once per week. The "supervising syslog-ng" process seems to restart everything so the problem is not really being noticed (except we have gaps, pauses during which logging for some systems is not occurring) Here is /var/log/kern from yesterday showing two process core dumps: Feb 28 18:05:03 sl.its.uwo.ca kernel: syslog-ng[45160]: segfault at 0 ip 00007f0a767d0acc sp 00007f0a6fffca50 error 6 in libsyslog-ng-3.5.4.1.so[7f0a7679e000+8b000] Feb 28 23:04:02 sl.its.uwo.ca kernel: syslog-ng[5442]: segfault at 0 ip 00007f0a767d0acc sp 00007f0a6fffca50 error 6 in libsyslog-ng-3.5.4.1.so[7f0a7679e000+8b000] What is the approach I should use to investigate this (seemingly random) seg fault problem? Dave Lacerte UWO.ca

2 1

Console logging stops after some time.
by Alexandre Santos 24 Feb '21

24 Feb '21

Hi All, I am using a Debian10 with syslog-ng 3.19. I have a simple configuration formed by a log to a file and a log to the console "usertty(*)". Please check the attached file. *After some time the logging to the console stops, but the logging to the file keeps working.* I used strace to debug this. (strace -p <syslog-ng-pid> -yy -f -s 256 -e trace=write) *root@localhost:~# w root 12:50:41 up 3:16, 5 users, load average: 0.21, 0.35, 0.37USER TTY FROM LOGIN@ IDLE JCPU PCPU WHATroot ttyS0 - 09:36 45:18 0.02s 0.02s -bashroot pts/1 10.0.2.2 09:40 1.00s 0.26s 0.00s w rootroot pts/4 10.0.2.2 09:56 24:08 3.03s 2.84s strace -p 23429 -yy -f -s 256 -e trace=writeroot pts/5 10.0.2.2 10:26 22:01 0.01s 0.01s tail -f /var/log/apps.logroot pts/0 10.0.2.2 12:31 19:25 21.77s 0.00s sleep 60root@localhost:~# strace -p 23429 -yy -f -s 256 -e trace=write[pid 24379] write(26</dev/ttyS0<char 4:64>>, "2021 Feb 19 12:31:16 localhost 8389:Usm INFO {Orchestrator.cpp:1329} attr_id=10931 moid=8398\n", 107) = 107[pid 24379] write(26</dev/pts/1<char 136:1>>, "2021 Feb 19 12:31:16 localhost 8389:Usm INFO {Orchestrator.cpp:1329} attr_id=10931 moid=8398\n", 107) = 107[pid 24379] write(26</dev/pts/4<char 136:4>>, "2021 Feb 19 12:31:16 localhost 8389:Usm INFO {Orchestrator.cpp:1329} attr_id=10931 moid=8398\n", 1072021 Feb 19 12:31:16 localhost 8389:Usm INFO {Orchestrator.cpp:1329} attr_id=10931 moid=8398[pid 24379] write(26</dev/pts/4<char 136:4>>, "2021 Feb 19 12:31:16 localhost 8389:Usm INFO {Orchestrator.cpp:1329} attr_id=10931 moid=8398\n", 1072021 Feb 19 12:31:16 localhost 8389:Usm INFO {Orchestrator.cpp:1329} attr_id=10931 moid=8398[pid 24379] write(26</dev/pts/5<char 136:5>>, "2021 Feb 19 12:31:16 localhost 8389:Usm INFO {Orchestrator.cpp:1329} attr_id=10931 moid=8398\n", 107) = 107[pid 24379] write(26</dev/pts/0<char 136:0>>, "2021 Feb 19 12:31:16 localhost 8389:Usm INFO {Orchestrator.cpp:1329} attr_id=10931 moid=8398\n", 107) = 107[pid 24379] write(23<pipe:[1957563]>, "<190>1 2021-02-19T12:31:16.855+00:00 localhost usm 6802 - - 8389:Usm INFO {Orchestrator.cpp:1329} attr_id=10931 moid=8398\n", 136) = 136[pid 24379] write(23<pipe:[1957563]>, "<190>1 2021-02-19T12:31:16.885+00:00 localhost usm 6802 - - 8395:Usm INFO {Orchestrator.cpp:1329} attr_id=10931 moid=8398\n", 136) = 136[pid 23703] write(1</var/log/apps.log>, "<190>1 2021-02-19T12:31:16.855+00:00 localhost usm 6802 - - 8389:Usm INFO {Orchestrator.cpp:1329} attr_id=10931 moid=8398\n", 136) = 136[pid 23703] write(1</var/log/apps.log>, "<190>1 2021-02-19T12:31:16.885+00:00 localhost usm 6802 - - 8395:Usm INFO {Orchestrator.cpp:1329} attr_id=10931 moid=8398\n", 136) = 136* After some time I stop seeing writes to the console's, and only see writes to the apps.log file. *[pid 24379] write(23<pipe:[1957563]>, "<190>1 2021-02-19T12:31:21.334+00:00 localhost dbf 7884 - - 8159:Dbf INFO {transaction_context.cpp:20} dtor of TransCtxt, tid: 120\n", 145) = 145[pid 23703] write(1</var/log/apps.log>, "<190>1 2021-02-19T12:31:21.334+00:00 localhost dbf 7884 - - 8159:Dbf INFO {transaction_context.cpp:20} dtor of TransCtxt, tid: 120\n", 145) = 145* It seems that due to the high amount of messages the logging to the console is discarded. Can you help me to solve this, Thanks in advance, Alex

2 5

query on using throttle in syslog-ng.conf file
by SIMON BABY 20 Feb '21

20 Feb '21

Hello Team, I am new to this group and I have a query on adding the throttle attribute in syslog-ng.conf file. My target is little slow to process all the messages sent by the sender and sometimes the link connected to the target device is broken. I am thinking of slowing down the sender by adding the throttle attribute. I have the below queries: 1) What exactly the throttle confoguration does? 2) What does throttle(500) mean ? will it send 500 Bytes per second or 500 messages per second? What does the message here mean ? Can it be the entire message sent by the application ? Is there an upper limit and lower limit ? 3) Any side effect of my system if I am going to use throttle(). 4) Any other method in syslog-ng to delay the logs sending at the sender? 5) My destination configuration is below. Is it a valid configuration ? destination logFiler { file("/var/log/wq.log" template("${FULLDATE}${TZ} ${HOST} ${PROGRAM} [$LEVEL] ${MSG}\n") template_escape(yes) throttle(500));}; Thank you for your time. Regards Simon

2 9

tty10 on Debian under LXC
by John Kristoff 16 Feb '21

16 Feb '21

I ran into the exact problem succinctly and completely written up here: <https://www.claudiokuenzler.com/blog/988/syslog-ng-lxc-container-no-space-l…> It doesn't appear this was brought to the attention of the devs nor an issue even filed in github. I'd like to make sure I didn't miss something, so posting here for comment before I submit a github issue. In summary, LXC sets up a small number of tty devices by default. However, syslog-ng in Debian has this config statement: destination d_console_all { file(`tty10`); }; This results in syslog-ng writing the a new /dev/tty10 file instead of to a device, which will pretty quickly fill up an LXC /dev partition, causing syslog-ng to soon start failing when it sees the device associated with tty10 file handle full. I have modified my syslog-ng.conf directly to use tty2 as the blog post suggested, but it would be nice if there was a "cleaner" work-around in the future. John

2 2

Insider 2021-02: proxy protocol; sudo JSON; Kafka;
by Peter Czanik (pczanik) 11 Feb '21

11 Feb '21

Dear syslog-ng users, This is the 88th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Finding the real source IP: using the PROXY protocol ---------------------------------------------------- Until now collecting logs behind proxies or load balancers needed some compromises. You either trusted the host information included in the log messages or you could only see the proxy as the sender host. Starting with syslog-ng 3.30 there is a third option available: using the PROXY protocol. While not an official Internet standard, it is supported by a number of popular software, like HAProxy. Other software can be extended to use it, like F5 load balancers using iRules. This way crucial information about the original network connection is not lost, but it is forwarded to the server by the proxy. https://www.syslog-ng.com/community/b/blog/posts/finding-the-real-source-ip… Parsing sudo JSON logs: building a syslog-ng configuration ---------------------------------------------------------- The latest version of sudo, version 1.9.4 includes support for JSON formatted logging. Compared to traditional sudo logs, it has the advantage of containing more information in a structured way. While traditional sudo logs are also parsed automatically by syslog-ng, it is worth taking a look at the new JSON formatted logs. >From this blog, you can learn how the new logs look like and also a configuration working with these logs. Instead of just posting a complex configuration, I try to show you how my configuration was built. Creating a new configuration in smaller iterations makes the resulting configurations easier to debug. https://www.syslog-ng.com/community/b/blog/posts/parsing-sudo-json-logs-bui… Kafka destination improved with template support ------------------------------------------------ The C implementation of the Kafka destination in syslog-ng has been improved in version 3.30. Support for templates in topic names was added as a result of a Google Summer of Code (GSoC) project. The advantage of the new template support feature is that you no longer have to use a static topic name. For example, you can include the name of your host or the application sending the log in the topic name. From this blog you can learn about a minimal Kafka setup, configuring syslog-ng and testing syslog-ng with Kafka. https://www.syslog-ng.com/community/b/blog/posts/kafka-destination-improved… Syslog-ng PE 7.0.23 released ---------------------------- Version 7.0.23 of syslog-ng PE was released with clustering support for Windows Event Collector (WEC). You can learn more about it from the documentation or from this short video: https://www.youtube.com/watch?v=bYdIJaM24Z8 WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0