- syslog-ng - lists.balabit.hu

Upgrade to 3.31 from 3.19
by Faine, Mark R. (MSFC-IS40)[NICS] 23 Mar '21

23 Mar '21

I need to know the significant format differences between the configuration files for the two versions. Is there some place I can go to get that or some command I can run to convert the file? Thanks, -Mark

4 7

Destination with disk-buffer and reliable(yes) looses queued messages on restart/crash?
by Ralf.Steppacher＠swisscom.com 22 Mar '21

22 Mar '21

Hello, I am probably missing something here: I am trying to configure syslog-ng (3.27.1) running in a Kubernetes Pod, using a Persistent Volume mounted at /var/log, such that queued messages are spooled to disk and in the event of a crash of syslog-ng the queue can be recovered. I configured the destination like so: destination d_forwarder { syslog( "`HEAVY_FORWARDER_HOST`" port(`HEAVY_FORWARDER_PORT`) transport("tls") tls( ca-dir("/etc/syslog-ng/ca.d") key-file("/vault/secrets/client_key.pem") cert-file("/vault/secrets/client_cert.pem") peer-verify(required-trusted) ) disk-buffer( mem-buf-size(524288) disk-buf-size(104857600) reliable(yes) dir("/var/log") ) ); }; The documentation for the `reliable` flag says: "If set to yes, syslog-ng OSE cannot lose logs in case of reload/restart, unreachable destination or syslog-ng OSE crash." I can see several *.rqf files being created in /var/log. As soon as the latest of them reaches roughly 100MB messages start to get dropped. So far everything as expected. Stats: { "center_queued_processed": 72031, "center_received_processed": 36016, "destination_d_forwarder_processed": 36015, "destination_d_local_processed": 36016, "dst_syslog_d_forwarder_0_tls_heavy-forwarder_shared-services_svc_cluster_local_6514_dropped": 4173, "dst_syslog_d_forwarder_0_tls_heavy-forwarder_shared-services_svc_cluster_local_6514_processed": 36015, "dst_syslog_d_forwarder_0_tls_heavy-forwarder_shared-services_svc_cluster_local_6514_queued": 31842, "dst_syslog_d_forwarder_0_tls_heavy-forwarder_shared-services_svc_cluster_local_6514_written": 0, ... "source_s_external_tls_processed": 36015 ... } /var/log: I have no name!@syslog-ng-76f898f5bb-sh9q8:/var/log$ ls -lh total 213M -rw------- 1 10001 10001 38K Mar 22 08:33 syslog-ng-00000.rqf -rw------- 1 10001 10001 4.0K Mar 22 09:25 syslog-ng-00001.rqf -rw------- 1 10001 10001 101M Mar 22 09:29 syslog-ng-00002.rqf -rw------- 1 10001 10001 4.0K Mar 22 09:46 syslog-ng-00003.rqf -rw------- 1 10001 10001 4.0K Mar 22 10:14 syslog-ng-00004.rqf ... Now, if I kill the syslog-ng Pod or gracefully scale the deployment to 0 and back up to 1, the queue is still lost. The stats all go back to 0 and bringing up the destination shows no (queued) messages coming in. On every restart a new .rqf gets created. New messages get spooled to the latest .rqf file until that one reaches the configured 100Mb size limit as well. What am I missing here? Thanks in advance! Ralf

2 2

syslog-ng-3.31.2
by Laszlo Budai 18 Mar '21

18 Mar '21

Hi, syslog-ng 3.31.2[1] packages are available from OBS repositories[2, 3, 4]. List of supported OSs: * Debian 8.0 * Debian 9.0 * Debian 10.0 * Raspbian 9.0 * Raspbian 10.0 * Ubuntu 14.04 * Ubuntu 16.04 * Ubuntu 18.04 * Ubuntu 18.10 * Ubuntu 19.04 * Ubuntu 19.10 * Ubuntu 20.04 * Ubuntu 20.10 Install ------- example: Debian 9.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-geat install syslog-ng-core` Links -------- [1] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.31.2 [2] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… [3] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… [4] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… regards, Laszlo Budai

1 0

syslog-ng-3.31.2
by Peter Kokai (pkokai) 18 Mar '21

18 Mar '21

3.31.2 ====== ## Bugfixes * `syslog-parser()`: fix a potential crash in case parsing the message fails and tags are already applied to the message. ## Packaging * `python2`: Direct `python2` support is dropped, we no longer test it in our CI. No `python2` related source codes were removed as for now, but we do not guarantee that it will work in the future. ([#3603](https://github.com/syslog-ng/syslog-ng/pull/3603)) ## Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler, Gabor Nagy, Laszlo Budai, Laszlo Szemere, László Várady, Norbert Takacs, Peter Kokai, Ryan Faircloth, Zoltan Pallagi

1 0

Docker syslog-ng TLS issue
by Steven La 17 Mar '21

17 Mar '21

I created a CA cert following the instructions here: https://support.oneidentity.com/fr-fr/technical-documents/syslog-ng-open-so… And the serverkey.pem is not encrypted, but syslog-ng is asking for a password when it starts up for the serverkey.pem. Any help would be appreciate Head of the server.key -----BEGIN PRIVATE KEY----- MIIEvAI.... syslog error: [2021-03-17T19:56:03.552322] Error setting up TLS session context; tls_error='system library:fopen:Permission denied', location='/etc/syslog-ng/syslog-ng.conf:21:2' [2021-03-17T19:56:03.552355] Error setting up TLS context; keyfile='/etc/ssl/certs/cert.d/serverkey.pem' [2021-03-17T19:56:03.552407] Waiting for password; keyfile='/etc/ssl/certs/cert.d/serverkey.pem' syslog-ng config: @version: 3.29 @include "scl.conf" source s_local { internal(); }; source s_network { default-network-drivers( # NOTE: TLS support # # the default-network-drivers() source driver opens the TLS # enabled ports as well, however without an actual key/cert # pair they will not operate and syslog-ng would display a # warning at startup. # tls(key-file("/etc/ssl/certs/cert.d/serverkey.pem") cert-file("/etc/ssl/certs/cert.d/servercert.pem") ca_dir("/etc/ssl/certs/ca.d")) peer_verify(optional-untrusted) ); }; destination d_local { file("/var/log/messages"); file("/var/log/messages-kv.log" template("$ISODATE $HOST $(format-welf --scope all-nv-pairs)\n") frac-digits(3)); }; log { source(s_local); source(s_network); destination(d_local); }; docker run command: sudo docker run -d --privileged -it -v "/data/syslog-ng/config/syslog-ng.conf:/etc/syslog-ng/syslog-ng.conf" -v "/data/syslog-ng/logs:/var/log" -v "/data/syslog-ng/certs:/etc/ssl/certs" -p 514:514/udp -p 601:601 -p 6514:6514 --name syslog-ng2 balabit/syslog-ng:latest -edv Thanks, Steven -- Steven La 408-503-0289 steven.la(a)datastax.com <Steven.La(a)datastax.com> | datastax.com

2 2

syslog-ng-ctl reload and disbuffer clear
by Pal, Laszlo 17 Mar '21

17 Mar '21

Hi, We are using several instances of syslog-ng OSE to send logs to SIEM. Of course, I'm using diskbuffer and throttle to limit the number of logs sent and keep my licensing happy :) However, every time when I change the config and reload syslog-ng it seems all the logs in the disk buffer is sent in once, so there is some event drop in SIEM Is there any way to avoid this? Here is my relevant part of the config destination d_siem { network( "xxx.xxx.xxx.xxx" port(514) suppress(5) throttle(500) frac_digits(0) log_fifo_size(500000) time_zone("Europe/Budapest") persist-name("siem_standard") transport(tcp) disk-buffer( mem-buf-length(2000) disk-buf-size(50000000000) reliable(no) dir("/data/syslog-ng/diskbuffer") ) ); }; Thanks Laszlo

1 0

GSoC interest
by Ηλίας Κυράννας 17 Mar '21

17 Mar '21

Hello all, I am a PhD student and I would like to participate in the GSoC this year. I have nearly three years of professional experience using syslog in the IoT and security context. I am familiar with Kafka and I find the “Add EOS support to kafka-c destination driver” very interesting. I didn’t find any issue related to this feature. If this is ok I could open one on GitHub to discuss and contribute so that I can work on my proposal. I could also engage in the MacOS project as I have previous experience with C in MacOS. Any hint would be appreciated. Best regards, Ilias Kyrannas

2 1

Insider 2021-03: Kafka; Windows; Bastille;
by Peter Czanik (pczanik) 11 Mar '21

11 Mar '21

Dear syslog-ng users, This is the 89th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Consuming logs from a Kafka topic --------------------------------- There is no official Kafka source in syslog-ng, but because this question comes up often enough, I created one. It is just a temporary workaround using the program() source, but it works. It involves Java and installing Kafka manually, but it was fast and reliabe in my tests: ingesting 50,000–100,000 messages a second on my laptop in a resource-constrained virtual machine. https://www.syslog-ng.com/community/b/blog/posts/consuming-logs-from-a-kafk… Collecting logs from Windows ---------------------------- Normally I cover free and open-source software in the syslog-ng blog. But recently quite a few members of the community reached out to me and asked about collecting logs from Windows. So, I prepared a quick overview of the topic. The good news is, that syslog-ng supports collecting logs from Windows in multiple ways. The not so good news is that Windows support is only available in the commercial version of syslog-ng. There are multiple ways for collecting log messages from Windows. You can either install syslog-ng agents on Windows hosts, or you can use the Windows Event Collector (WEC) component of syslog-ng PE. https://www.syslog-ng.com/community/b/blog/posts/collecting-logs-from-windo… Running syslog-ng in Bastille ----------------------------- Bastille is a container management system for FreeBSD, similar to Docker or Podman on Linux. The historical name of containers on FreeBSD is jail, and they appeared a lot earlier than containers on Linux. Managing jails was not always easy. When I started to use this technology in production in 2001, nothing was automated. Using Bastille, you can easily create, configure, or update jails at scale. It has a template system to install applications in containers and there is a template also for syslog-ng. From this blog, you can learn how to get started with Bastille and how to create and run a syslog-ng jail using the freshly released 0.8 version of Bastille. https://www.syslog-ng.com/community/b/blog/posts/running-syslog-ng-in-basti… Syslog-ng OSE 3.31.1 released ----------------------------- Version 3.31.1 of syslog-ng OSE was released with the Fortigate parser, many parsing-related enhancements, silent Telegram messages, and more. For a complete list of changes, check https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-3.31.1 WEBINARS * Scale log collection from Windows endpoints with WEC clustering in syslog-ng https://www.syslog-ng.com/event/scale-log-collection-from-windows-endpoints… * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Syslog-ng segfault at version 3.8.1
by Sam Kappen 08 Mar '21

08 Mar '21

Hi All, We have an issue from a customer who uses syslog-ng version 3.8.1 on the 3.10.108 kernel(Hardware: freescale-p2020ds-3.10-1.4). Customer observed a segmentation fault after syslog-ng being stuck for more than one hour. There are no reproducible steps available, the issue is seen as random. As per the backtrace segfault happened from "malloc_consolidate" which looks an odd behaviour.On checking the frame it looks mchunkptr "p" has the value 0x2 which caused this crash, but not been able to find out what is the root cause for this. Has there been any known issue with syslog-ng? Any idea if this issue is fixed in version 3.16.1. Any help or pointers would be appreciated. Attached the back trace. Back trace snippet: Program terminated with signal 11, Segmentation fault. #0 malloc_consolidate (av=av@entry=0xffc738c <main_arena>) at malloc.c:4070 4070 malloc.c: No such file or directory. (gdb) bt #0 malloc_consolidate (av=av@entry=0xffc738c <main_arena>) at malloc.c:4070 #1 0x0fec5cb4 in _int_malloc (av=av@entry=0xffc738c <main_arena>, bytes=bytes@entry=556) at malloc.c:3374 #2 0x0fec6f64 in __GI___libc_malloc (bytes=556) at malloc.c:2874 #3 0x0f90d58c in standard_malloc (n_bytes=<optimized out>) at gmem.c:85 #4 0x0f90d9f0 in g_malloc (n_bytes=n_bytes@entry=556) at gmem.c:159 #5 0x0f7be00c in log_msg_alloc (payload_size=256) at lib/logmsg/logmsg.c:1084 #6 log_msg_new_empty () at lib/logmsg/logmsg.c:1150 #7 0x0f7be088 in log_msg_new_local () at lib/logmsg/logmsg.c:1160 #8 0x0f7be194 in log_msg_new_mark () at lib/logmsg/logmsg.c:1260 #9 0x0f7713e8 in afinter_source_mark (s=0x10863690) at lib/afinter.c:142 #10 0x0f7c4c74 in iv_run_timers (st=st@entry=0x10284d30) at iv_timer.c:149 #11 0x0f7c72bc in iv_main () at iv_main_posix.c:107 #12 0x0f78b990 in main_loop_run () at lib/mainloop.c:497 #13 0x10001260 in main (argc=1, argv=0xbfbe7724) at syslog-ng/main.c:270 (gdb) Regards, Sam

1 0

syslog-ng-3.31.1
by Peter Kokai (pkokai) 08 Mar '21

08 Mar '21

3.31.1 ====== ## Highlights * fortigate-parser(): new parser to parse fortigate logs Example: ``` log { source { network(transport("udp") flags(no-parse)); }; parser { fortigate-parser(); }; destination { }; }; ``` An adapter to automatically recognize fortigate logs in app-parser() has also been added. ([#3536](https://github.com/syslog-ng/syslog-ng/pull/3536)) * `patterndb`: Added `OPTIONALSET` parser. It works the same as `SET`, but continues, even if none of the characters is found. ([#3540](https://github.com/syslog-ng/syslog-ng/pull/3540)) ## Features * `syslog-parser()`: add no-header flag to tell syslog-ng to parse only the PRI field of an incoming message, everything else is just put into $MSG. ([#3538](https://github.com/syslog-ng/syslog-ng/pull/3538)) * `set-pri()`: this new rewrite operation allows you to change the PRI value of a message based on the string directly parsed out of a syslog header. ([#3546](https://github.com/syslog-ng/syslog-ng/pull/3546)) * telegram: option to send silent message Example: ``` destination { telegram(bot-id(...) chat-id(...) disable_notification(true)); }; ``` ([#3558](https://github.com/syslog-ng/syslog-ng/pull/3558)) * `app-parser()`: added automatic classification & parsing for project Lumberjack/Mitre CEE formatted logs ([#3569](https://github.com/syslog-ng/syslog-ng/pull/3569)) * diskq: if the dir() path provided by the user does not exists, syslog-ng creates the path with the same permission as the running instance ([#3550](https://github.com/syslog-ng/syslog-ng/pull/3550)) ## Bugfixes * `network()`, `syslog()` destinations: fix reconnection timer when DNS lookups are slow After a long-lasting DNS query, syslog-ng did not wait the specified time (`time_reopen()`) before reconnecting to a destination. This has been fixed. ([#3526](https://github.com/syslog-ng/syslog-ng/pull/3526)) * cmake: minor fixes ([#3523](https://github.com/syslog-ng/syslog-ng/pull/3523)) * `stats-level()`: fix processing the changes in the stats-level() global option: changes in stats-level() were not reflected in syslog facility/severity related and message tag related counters after first configuration reload. These counters continued to operate according to the value of stats-level() at the first reload. ([#3561](https://github.com/syslog-ng/syslog-ng/pull/3561)) * `date-parser()`: fix hour-only timezone parsing If the timestamp contains a short timezone offset (e.g. hours only), the recent change in strptime() introduces an error, it interprets those numbers as minutes instead of hours. For example: Jan 16 2019 18:23:12 +05 ([#3555](https://github.com/syslog-ng/syslog-ng/pull/3555)) * `loggen`: fix undefined timeout while connecting to network sources (`glib < 2.32`) When compiling syslog-ng with old glib versions (< 2.32), `loggen` could fail due a timeout bug. This has been fixed. ([#3504](https://github.com/syslog-ng/syslog-ng/pull/3504)) * `grouping-by()`: fix deadlock when context expires In certain cases, the `grouping-by()` parser could get stuck when a message context expired, causing a deadlock in syslog-ng. This has been fixed. ([#3515](https://github.com/syslog-ng/syslog-ng/pull/3515)) * `date-parser`: Fixed a crash, which occured sometimes when `%z` was used. ([#3553](https://github.com/syslog-ng/syslog-ng/pull/3553)) * `date-parser`: `%z`. We no longer ignore daylight saving time when calculating the GMT offset. ([#3553](https://github.com/syslog-ng/syslog-ng/pull/3553)) * `kafka-c`: fix a double LogMessage acknowledgement bug, which can cause crash with segmentation fault or exit with sigabrt. The issue affects both flow-controlled and non-flow-controlled log paths and it's triggered in case previously published messages failed to be delivered to Kafka. ([#3583](https://github.com/syslog-ng/syslog-ng/pull/3583)) * `python destination`: Fixed a rare crash during reload. ([#3568](https://github.com/syslog-ng/syslog-ng/pull/3568)) * `date-parser()`: fix non-mandatory parsing of timezone name When %Z is used, the presence of the timezone qualifier is not mandatory, so don't fail that case. ([#3555](https://github.com/syslog-ng/syslog-ng/pull/3555)) * `wildcard-file()`: fix infrequent crash when file renamed/recreated The wildcard-file source crashed when a file being processed was replaced by a new one on the same path (renamed, deleted+recreated, rotated, etc.). ([#3513](https://github.com/syslog-ng/syslog-ng/pull/3513)) * Remove the no-parse flag in system() source from FreeBSD kernel messages, so the message header is no more part of the message. ([#3586](https://github.com/syslog-ng/syslog-ng/pull/3586)) * Fix abort on macOS Big Sur A basic subset of syslog-ng's functionality now works on the latest macOS version. ([#3522](https://github.com/syslog-ng/syslog-ng/pull/3522)) * `affile`: Fix improper initialization in affile and LogWriter to avoid memory leak when reloading ([#3574](https://github.com/syslog-ng/syslog-ng/pull/3574)) * `udp destination`: Fixed a bug, where the packet's checksum was not calculated, when `spoof-source(yes)` and `ip-protocol(6)` were set. ([#3528](https://github.com/syslog-ng/syslog-ng/pull/3528)) * `python`: fix LogMessage.keys() listing non-existenting keys and duplicates ([#3557](https://github.com/syslog-ng/syslog-ng/pull/3557)) ## Packaging * Simplify spec file by removing obsolete technologies: - remove RHEL 6 support - remove Python 2 support - keep Java support, but remove Java-based drivers (HDFS, etc.) ([#3587](https://github.com/syslog-ng/syslog-ng/pull/3587)) * `libnet`: Minimal libnet version is now 1.1.6. ([#3528](https://github.com/syslog-ng/syslog-ng/pull/3528)) * configure: added new --enable-manpages-install option along with the existing --enable-manpages. The new option would install pre-existing manpages even without the DocBook tools installed. ([#3493](https://github.com/syslog-ng/syslog-ng/pull/3493)) ## Notes to developers * `apphook`: the concept of hook run modes were introduced, adding support for two modes: AHM_RUN_ONCE (the original behavior) and AHM_RUN_REPEAT (the new behavior with the hook repeatedly called after registration). ([#3561](https://github.com/syslog-ng/syslog-ng/pull/3561)) ## Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: 0140454, Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler, egorbeliy, Gabor Nagy, Laszlo Budai, Laszlo Szemere, László Várady, Michael Ducharme, Norbert Takacs, Peter Czanik, Peter Kokai, Pratik raj, Ryan Faircloth, Zoltan Pallagi

1 0

Syslog-ng 3.4.5 support question
by mukesh kumar 05 Mar '21

05 Mar '21

Hi, We are using syslog-ng 3.4.5 version and we are using it to utilize patterndb. We are not looking for bug fix but using it for just all the different ways we can use the patterndb xml. Please provide me the info on such as the IF and ELSE statements, For loops in the xml file. Thanks, Mukesh Kumar

4 5

CentOS Stream
by Peter Czanik (pczanik) 03 Mar '21

03 Mar '21

Hi, CentOS is right now the most popular platform to run syslog-ng. On the other hand, they are shifting focus to CentOS Stream: https://blog.centos.org/2020/12/future-is-centos-stream/ Due to these changes many people are looking now for alternative platforms. Replacements for regular CentOS releases are under way: AlmaLinux is already in RC phase, Rocky Linux is arriving soon. I heard many people switching to openSUSE. And of course, there are also non-rpm alternatives, like Ubuntu or even FreeBSD. As I'm doing syslog-ng packaging for our most popular platforms (Fedora/CentOS/RHEL, openSUSE/SLES, FreeBSD), I'm curious what to expect on the packaging frontier. What will be our future platforms? I'm running a poll on Twitter: https://twitter.com/PCzanik/status/1366391907748831235 If your choice is "other", I'd love to know what is that other platform! Thanks for your feedback, Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

syslog-ng dumps core (twice yesterday) how do I investigate cause?
by David Lacerte 01 Mar '21

01 Mar '21

Running syslog-ng-3.5.4.1/ on CentOS 6.5 started seeing core dumps in var.log/kern approximately once per week. The "supervising syslog-ng" process seems to restart everything so the problem is not really being noticed (except we have gaps, pauses during which logging for some systems is not occurring) Here is /var/log/kern from yesterday showing two process core dumps: Feb 28 18:05:03 sl.its.uwo.ca kernel: syslog-ng[45160]: segfault at 0 ip 00007f0a767d0acc sp 00007f0a6fffca50 error 6 in libsyslog-ng-3.5.4.1.so[7f0a7679e000+8b000] Feb 28 23:04:02 sl.its.uwo.ca kernel: syslog-ng[5442]: segfault at 0 ip 00007f0a767d0acc sp 00007f0a6fffca50 error 6 in libsyslog-ng-3.5.4.1.so[7f0a7679e000+8b000] What is the approach I should use to investigate this (seemingly random) seg fault problem? Dave Lacerte UWO.ca

2 1

Console logging stops after some time.
by Alexandre Santos 24 Feb '21

24 Feb '21

Hi All, I am using a Debian10 with syslog-ng 3.19. I have a simple configuration formed by a log to a file and a log to the console "usertty(*)". Please check the attached file. *After some time the logging to the console stops, but the logging to the file keeps working.* I used strace to debug this. (strace -p <syslog-ng-pid> -yy -f -s 256 -e trace=write) *root@localhost:~# w root 12:50:41 up 3:16, 5 users, load average: 0.21, 0.35, 0.37USER TTY FROM LOGIN@ IDLE JCPU PCPU WHATroot ttyS0 - 09:36 45:18 0.02s 0.02s -bashroot pts/1 10.0.2.2 09:40 1.00s 0.26s 0.00s w rootroot pts/4 10.0.2.2 09:56 24:08 3.03s 2.84s strace -p 23429 -yy -f -s 256 -e trace=writeroot pts/5 10.0.2.2 10:26 22:01 0.01s 0.01s tail -f /var/log/apps.logroot pts/0 10.0.2.2 12:31 19:25 21.77s 0.00s sleep 60root@localhost:~# strace -p 23429 -yy -f -s 256 -e trace=write[pid 24379] write(26</dev/ttyS0<char 4:64>>, "2021 Feb 19 12:31:16 localhost 8389:Usm INFO {Orchestrator.cpp:1329} attr_id=10931 moid=8398\n", 107) = 107[pid 24379] write(26</dev/pts/1<char 136:1>>, "2021 Feb 19 12:31:16 localhost 8389:Usm INFO {Orchestrator.cpp:1329} attr_id=10931 moid=8398\n", 107) = 107[pid 24379] write(26</dev/pts/4<char 136:4>>, "2021 Feb 19 12:31:16 localhost 8389:Usm INFO {Orchestrator.cpp:1329} attr_id=10931 moid=8398\n", 1072021 Feb 19 12:31:16 localhost 8389:Usm INFO {Orchestrator.cpp:1329} attr_id=10931 moid=8398[pid 24379] write(26</dev/pts/4<char 136:4>>, "2021 Feb 19 12:31:16 localhost 8389:Usm INFO {Orchestrator.cpp:1329} attr_id=10931 moid=8398\n", 1072021 Feb 19 12:31:16 localhost 8389:Usm INFO {Orchestrator.cpp:1329} attr_id=10931 moid=8398[pid 24379] write(26</dev/pts/5<char 136:5>>, "2021 Feb 19 12:31:16 localhost 8389:Usm INFO {Orchestrator.cpp:1329} attr_id=10931 moid=8398\n", 107) = 107[pid 24379] write(26</dev/pts/0<char 136:0>>, "2021 Feb 19 12:31:16 localhost 8389:Usm INFO {Orchestrator.cpp:1329} attr_id=10931 moid=8398\n", 107) = 107[pid 24379] write(23<pipe:[1957563]>, "<190>1 2021-02-19T12:31:16.855+00:00 localhost usm 6802 - - 8389:Usm INFO {Orchestrator.cpp:1329} attr_id=10931 moid=8398\n", 136) = 136[pid 24379] write(23<pipe:[1957563]>, "<190>1 2021-02-19T12:31:16.885+00:00 localhost usm 6802 - - 8395:Usm INFO {Orchestrator.cpp:1329} attr_id=10931 moid=8398\n", 136) = 136[pid 23703] write(1</var/log/apps.log>, "<190>1 2021-02-19T12:31:16.855+00:00 localhost usm 6802 - - 8389:Usm INFO {Orchestrator.cpp:1329} attr_id=10931 moid=8398\n", 136) = 136[pid 23703] write(1</var/log/apps.log>, "<190>1 2021-02-19T12:31:16.885+00:00 localhost usm 6802 - - 8395:Usm INFO {Orchestrator.cpp:1329} attr_id=10931 moid=8398\n", 136) = 136* After some time I stop seeing writes to the console's, and only see writes to the apps.log file. *[pid 24379] write(23<pipe:[1957563]>, "<190>1 2021-02-19T12:31:21.334+00:00 localhost dbf 7884 - - 8159:Dbf INFO {transaction_context.cpp:20} dtor of TransCtxt, tid: 120\n", 145) = 145[pid 23703] write(1</var/log/apps.log>, "<190>1 2021-02-19T12:31:21.334+00:00 localhost dbf 7884 - - 8159:Dbf INFO {transaction_context.cpp:20} dtor of TransCtxt, tid: 120\n", 145) = 145* It seems that due to the high amount of messages the logging to the console is discarded. Can you help me to solve this, Thanks in advance, Alex

2 5

query on using throttle in syslog-ng.conf file
by SIMON BABY 20 Feb '21

20 Feb '21

Hello Team, I am new to this group and I have a query on adding the throttle attribute in syslog-ng.conf file. My target is little slow to process all the messages sent by the sender and sometimes the link connected to the target device is broken. I am thinking of slowing down the sender by adding the throttle attribute. I have the below queries: 1) What exactly the throttle confoguration does? 2) What does throttle(500) mean ? will it send 500 Bytes per second or 500 messages per second? What does the message here mean ? Can it be the entire message sent by the application ? Is there an upper limit and lower limit ? 3) Any side effect of my system if I am going to use throttle(). 4) Any other method in syslog-ng to delay the logs sending at the sender? 5) My destination configuration is below. Is it a valid configuration ? destination logFiler { file("/var/log/wq.log" template("${FULLDATE}${TZ} ${HOST} ${PROGRAM} [$LEVEL] ${MSG}\n") template_escape(yes) throttle(500));}; Thank you for your time. Regards Simon

2 9

tty10 on Debian under LXC
by John Kristoff 16 Feb '21

16 Feb '21

I ran into the exact problem succinctly and completely written up here: <https://www.claudiokuenzler.com/blog/988/syslog-ng-lxc-container-no-space-l…> It doesn't appear this was brought to the attention of the devs nor an issue even filed in github. I'd like to make sure I didn't miss something, so posting here for comment before I submit a github issue. In summary, LXC sets up a small number of tty devices by default. However, syslog-ng in Debian has this config statement: destination d_console_all { file(`tty10`); }; This results in syslog-ng writing the a new /dev/tty10 file instead of to a device, which will pretty quickly fill up an LXC /dev partition, causing syslog-ng to soon start failing when it sees the device associated with tty10 file handle full. I have modified my syslog-ng.conf directly to use tty2 as the blog post suggested, but it would be nice if there was a "cleaner" work-around in the future. John

2 2

Insider 2021-02: proxy protocol; sudo JSON; Kafka;
by Peter Czanik (pczanik) 11 Feb '21

11 Feb '21

Dear syslog-ng users, This is the 88th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Finding the real source IP: using the PROXY protocol ---------------------------------------------------- Until now collecting logs behind proxies or load balancers needed some compromises. You either trusted the host information included in the log messages or you could only see the proxy as the sender host. Starting with syslog-ng 3.30 there is a third option available: using the PROXY protocol. While not an official Internet standard, it is supported by a number of popular software, like HAProxy. Other software can be extended to use it, like F5 load balancers using iRules. This way crucial information about the original network connection is not lost, but it is forwarded to the server by the proxy. https://www.syslog-ng.com/community/b/blog/posts/finding-the-real-source-ip… Parsing sudo JSON logs: building a syslog-ng configuration ---------------------------------------------------------- The latest version of sudo, version 1.9.4 includes support for JSON formatted logging. Compared to traditional sudo logs, it has the advantage of containing more information in a structured way. While traditional sudo logs are also parsed automatically by syslog-ng, it is worth taking a look at the new JSON formatted logs. >From this blog, you can learn how the new logs look like and also a configuration working with these logs. Instead of just posting a complex configuration, I try to show you how my configuration was built. Creating a new configuration in smaller iterations makes the resulting configurations easier to debug. https://www.syslog-ng.com/community/b/blog/posts/parsing-sudo-json-logs-bui… Kafka destination improved with template support ------------------------------------------------ The C implementation of the Kafka destination in syslog-ng has been improved in version 3.30. Support for templates in topic names was added as a result of a Google Summer of Code (GSoC) project. The advantage of the new template support feature is that you no longer have to use a static topic name. For example, you can include the name of your host or the application sending the log in the topic name. From this blog you can learn about a minimal Kafka setup, configuring syslog-ng and testing syslog-ng with Kafka. https://www.syslog-ng.com/community/b/blog/posts/kafka-destination-improved… Syslog-ng PE 7.0.23 released ---------------------------- Version 7.0.23 of syslog-ng PE was released with clustering support for Windows Event Collector (WEC). You can learn more about it from the documentation or from this short video: https://www.youtube.com/watch?v=bYdIJaM24Z8 WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

flags(final) - Question
by Matthias Gruber 09 Feb '21

09 Feb '21

Hi! After searching why a log-message doesnt got the right destination path, I propably got a bit crossed up with the flag(final) statement, perhaps someone can enlight me :-) Following scenario (no valid syntax, only for description) filter(f_a) { match("something");}; destination(d_a) { file("/destination_a"); }: destination(d_b) { file("/destination_b"); }: destination(d_c) { file("/destination_c"); }: log { source(s_remote_default_syslog_udp); if filter(f_a) { destination(d_a); flag(final); } else { destination(d_b); flag(final); }; destination(d_c); }; What I expect is if filter->f_a hits the message is written in destination->d_a (what he do), then hits the final, and ends the log-statement What I discovered he also write destination->d_c Is that the expected behaviour? And if so, how to accomplish, that e.g. if the if or the else part hit, no destination -> d_c is written. Yeah I know, why the destination->d_c :-), correct I stumbled over this, normally after such a if/elif/else-Block I dunno have this destination->d_c line in my syslog-ng environments. Only a bit courious about it and seeking for an explanation. cheers Matthias ------------------------------------------------------------------------------------ METZLER Informationstechnologie Matthias Gruber IT-Infrastruktur & -Betrieb B. Metzler seel. Sohn & Co. Kommanditgesellschaft auf Aktien Untermainanlage 1 60329 Frankfurt am Main Telefon (0 69) 21 04 - 43 30 Telefax (0 69) 21 04 - 40 40 MGruber(a)metzler.com www.metzler.com Persönlich haftende Gesellschafter: Harald Illy, Friedrich von Metzler, Emmerich Müller, Gerhard Wiesheu Vorsitzender des Aufsichtsrats: Dr. Christoph Schücking Sitz der Gesellschaft: Frankfurt am Main, Handelsregister-Nr. HRB 27 515 Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfänger sein, so bitten wir Sie höflich, dies unverzüglich dem Absender mitzuteilen und die Nachricht zu löschen. Es ist unzulässig, die Nachricht unbefugt weiterzuleiten oder zu kopieren. Da wir nicht die Echtheit oder Vollständigkeit der in dieser Nachricht enthaltenen Informationen garantieren oder zusichern können, sind die vorstehenden Ausführungen rechtlich nicht bindend. Eine Haftung hierfür wird ausgeschlossen. This message is confidential. If you are not the intended recipient, we kindly ask you to inform the sender and delete the information. Any unauthorised dissemination or copying hereof is prohibited. As we cannot guarantee or assure the genuineness or completeness of the information contained in this message, the statements set forth above are not legally binding. Accordingly we cannot accept any liability for their contents.

2 2

filtering based on text?
by Dan Egli 08 Feb '21

08 Feb '21

Hey folks, I've looked for a way to do this, and I guess my google-foo is weak today. I've got a situation where on one of my machines, an automated process FREQUENTLY calls sudo so it can gain the permissions it needs to do certain tasks. But that means my /var/log/messages is getting FILLED with sudo messages. I was hoping I could insert some kind of text filter that would allow me to shunt messages where one user (the automated process) calls sudo into another log file. It would basically need to be a nested filter, i.e.: If message_source = sudo then if user = X then log to auto_sudo.log If someone knows how this can be done, I'd appreciate it. Thanks! --- Dan

3 3

PostgreSQL destination - connection refused
by victorsugo 06 Feb '21

06 Feb '21

Hi, I'd like to store incoming log messages in a postgres db. Both db server and syslog-ng run in a docker container. I'm using docker-compose to manage containers. The syslog-ng is depending on the pg container, but... every time I start containers with docker-compose up -d, the syslog-ng starts, but can't connect to database. I found this in the /var/log/messages on the syslog-ng's container: err syslog-ng[1]: Error establishing SQL connection; type='pgsql', host='pgsrv.infra-net', port='', username='syslog', database='postgres', error='could not connect to server: Connection refused ... Is there any way to reconnect automatically or abort syslog-ng if it can't connect to the db? It is important if I don't want to lose some logs... (or do I need to forget the SQL backend if I want to operate a log server without loss of data??) Regards, Sent with [ProtonMail](https://protonmail.com) Secure Email.

1 1

problems with multiline log messages on ubuntu18
by Jerry Scharf 06 Feb '21

06 Feb '21

Hi, We have been struggling with problems with multiple log messages being broken apart. Our apps can send random things to the logger including entire stack traces. We have a classifier that sends unknown lines to their own log file, so this makes a complete mess. This was with the distro apt package. So I went to building 3.25.1 by hand. If I build it with nothing on the configure line, it processes the multiline log messages just fine but if I build it with the ubuntu18 debian rules, it breaks the same way. I narrowed it down to if --eneable-systemd is set it breaks. Here are some relevant things. DISTRIB_DESCRIPTION="Ubuntu 18.04.5 LTS" config cmd ../configure --prefix=/usr/local --sysconfdir=/etc/syslog-ng --enable-dynamic-linking --enable-json --enable-manpages --enable-python --enable-java=no --enable-mongodb=no --enable-legacy-mongodb-options=no SOURCE_REVISION=3.25.1-ir1 --enable-systemd=no attaching the conf and the file classifier (60...) thanks, jerry

2 1

Help with filtering at destination based on custom macro value match
by Akshay Joshi 30 Jan '21

30 Jan '21

My client is sending logs and it has the following config : *template t_global {template("<${PRI}>${LOGHOST} ${MSG_TAG}${MSGHDR}${MSG}\n"); };* *# Global logging remote destination:#-----------------------------------destination d_global_remote { tcp("proxy.dc.nuagedemo.net <http://proxy.dc.nuagedemo.net>" port(10514) template(t_global) tls(peer-verify(required-untrusted) ca-dir('/etc/default/bootstrap/keys') cert_file('/etc/default/bootstrap/keys/cert.pem') key-file('/etc/default/bootstrap/keys/key.pem') ) flags("threaded") );};source s_nuageDiag { file("/home/user/nuage/nuage_diagnostics_daemon.log" follow-freq(10) default-facility(local1) default-priority(info) tags("nuageDiag"));};rewrite w_nuageDiag { set("nuage-diag: ", value("MSG_TAG") condition(tags("nuageDiag"))); };* On the destination, I have this : *source s_network { tcp( port(10514) max-connections(1000) tls( peer-verify(required-untrusted) key-file("/opt/proxy/config/keys/proxy-Key.pem") cert_file("/opt/proxy/config/keys/proxyCert.pem") ca-dir("/opt/proxy/config/keys/proxy-CA.pem") ) );};* *filter nsg_diag { match("nuage-diag: " value("MSG_TAG"));};filter f_messages { (level(info..warn) and filter (nsg_diag)); };* *destination d_logs { file( "/var/log/syslog-ng/logs.txt" owner("root") group("root") perm(0777) );};log { source(s_sys); source(s_network); filter(f_messages); destination(d_logs); };* I can write logs locally without the filtering. But with filtering, it does not match "nuage-diag: " macro. This "MSG_TAG" does not seem to be a standard header but a custom one. I couldn't find many straightforward examples on forums etc.. as well. I am missing a trick or two config-wise for sure. Any pointers / help will be much appreciated. -- Regards, Akshay Joshi

3 2

syslog-ng won't talk to systemd?
by Dan Egli 26 Jan '21

26 Jan '21

I'm trying to setup syslog-ng on a systemd system. But even though I have several things writing to syslog, syslog-ng doesn't write ANYTHING to the /var/log/messages except kernel messages and it's own info. So I searched around, and found the systemd-journal() source. So I tried it. And I get an error that says syslog-ng can't initialize the logger plugin: # syslog-ng -F [2021-01-23T23:25:36.525097] The configuration must not contain more than one systemd-journal() source; [2021-01-23T23:25:36.525157] Error initializing journal_reader; [2021-01-23T23:25:36.525173] Error initializing message pipeline; plugin_name='systemd-journal', location='/etc/syslog-ng/syslog-ng.conf:26:5' Here's the config, basically copied from the syslog-ng.com page: @version: 3.26 # # Syslog-ng default configuration file for Gentoo Linux # https://bugs.gentoo.org/426814 @include "scl.conf" options { threaded(yes); chain_hostnames(no); stats_freq(43200); mark_freq(3600); }; source s_journald { systemd-journal(prefix(".SDATA.journald.")); }; source src { system(); internal(); file("/proc/kmsg"); }; destination messages { file("/var/log/messages"); }; destination console_all { file("/dev/tty12"); }; log { source(src); destination(messages); }; log { source(s_journald); destination(messages); }; log { source(src); destination(console_all); }; What am I doing wrong? I am POSITIVE things are sending to the syslog sub-system. I have ensured several programs have sent something. But without the systemd-journal line syslog-ng loads fine, recording /proc/kmsg and syslog-ng messages but nothing else. I am completely stumped here.

2 3

Polling additional machines
by Dan Egli 23 Jan '21

23 Jan '21

I have an interesting setup that was proposed to me, and while I _THINK_ syslog-ng can handle it I don't know for sure, let alone HOW. There are multiple service-specific machines in the network, behind the firewall. I'll just use the services they provide as their hostnames in this example. So firewall fw is a small linux machine running syslog-ng and fail2ban. Now fail2ban works off of the log files, but it also wants to apply changes to the firewall rules locally. So either www has its firewall rules while smtp has its rules and imap has a third set (ugly) or fail2ban can run on fw and modify the rules there. Trick is, in order to do that, all the logs have to be accessible in real time so that fail2ban can see them. My thought was to have syslog-ng on each sub machine somehow report to the syslog-ng on fw. Then fw's syslog-ng can write the files that fail2ban wants to read from. What would be the best way to go about this? I am still learning syslog-ng so if this is something simple, I'm sorry.

2 1

syslog-ng on CentOS 7 Aarch64
by Peter Czanik (pczanik) 20 Jan '21

20 Jan '21

Hi, I just learned that RHEL 7 on Aarch64 (64bit Arm) is end of life. Copr, where I build syslog-ng packages, removes RHEL/CentOS 7 Aarch64 packages in about half a year and it is not possible to build new packages there any more. CentOS 7 on the other hand keeps going on Aarch64 (using CentOS 8 kernel and CentOS 7 userland). There might be a workaround for the situation (using a different build service), but it's quite a lot extra effort. So, before investing any time and effort into keep building CentOS 7 Aarch64, I want to know, if there is a significant number of syslog-ng users, who keep using CentOS 7 on Aarch64. You can respond here or to my Twitter poll at: https://twitter.com/PCzanik/status/1351816930345291776 Thanks, Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Signed Syslog Messages RFC 5848
by Alexandre Santos 20 Jan '21

20 Jan '21

Hi, Are there any plans to add Signed Syslog Messages RFC 5848 to syslog-ng? Best Regards, Alex

3 4

Signed Syslog Messages RFC 5848
by Secure-Logging, Generic 19 Jan '21

19 Jan '21

Hi Alexandre, as a replacement for signed messages you might have a look at the secure logging module of syslog-ng. This module authentically encrypts each message, so you get an integrity guarantee which is stronger than a pure signature. You can then forward the message to a backend for verification and further processing. Regards, Stephan The information in this e-mail is confidential. The contents may not be disclosed or used by anyone other than the addressee. Access to this e-mail by anyone else is unauthorised. If you are not the intended recipient, please notify Airbus immediately and delete this e-mail. Airbus cannot accept any responsibility for the accuracy or completeness of this e-mail as it has been sent over public networks. If you have any concerns over the content of this message or its Accuracy or Integrity, please contact Airbus immediately. All outgoing e-mails from Airbus are checked using regularly updated virus scanning software but you should take whatever measures you deem to be appropriate to ensure that this message and any attachments are virus free.

1 0

Failover/Hi Availability
by Faine, Mark R. (MSFC-IS40)[NICS] 18 Jan '21

18 Jan '21

If I have two Syslog-ng servers receiving logs from the same source and writing to the exact same file system destination, is there a way to do this without running into issues with the two instances clobbering the files or file locking issues? Thanks, -Mark

2 4

Filtering Destination by Source
by Peter Griggs 12 Jan '21

12 Jan '21

Hello, We have a lot of network logs all being pointed to a central syslog however this is a mix of vendors (Cisco / Juniper / Checkpoint) etc. is there a way of splitting the destination file by vendor type / or source IP address? We ingest this data into Splunk so want to get the source typing right however I am unable to get the sources to point to various listeners and I would prefer. Thanks Peter.

3 3

Disk file truncation issue
by Raghunath Adhyapak 20 Dec '20

20 Dec '20

Hi, We have configured the destination with reliable(yes). We observe that many times the disk buffer file has zero messages, however the buffer file is not getting truncated. This syslog-ng service is running on 18.04 ubuntu VM. =============================================== Our destination configuration is as below: destination d_msg_syslog_01a_06 { tcp( "172.XX.XX.XX" port(514) # log-fifo-size should be atleast max-connections * log-fetch-limit # 500 * 10 (default value of log-fetch-limit) log-fifo-size(50000) # throttle to max 5k eps throttle(3000) disk-buffer( # number of bytes to store in memory mem-buf-size(10000) # number of bytes to store on disk disk-buf-size(75161927680) # 70GB reliable(yes) # directory location to persist messages dir("/datadisk/syslog-ng/disk-buffer/ne-collector-01a-06/") ) persist-name(d_msg_syslog_01a_06) ); }; =============================================== $ find /datadisk/syslog-ng/disk-buffer/ -type f -name "*.rqf" | xargs ls -lh -rw------- 1 root root 64G Dec 19 16:50 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-02/syslog-ng-00000.rqf -rw------- 1 root root 4.0K Dec 19 12:38 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-04/syslog-ng-00000.rqf -rw------- 1 root root 4.0K Dec 19 12:38 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-05/syslog-ng-00000.rqf -rw------- 1 root root 37G Dec 19 12:41 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-06/syslog-ng-00000.rqf -rw------- 1 root root 4.0K Dec 19 12:38 /datadisk/syslog-ng/disk-buffer/ne-collector-01a-07/syslog-ng-00000.rqf -rw------- 1 root root 4.0K Dec 19 16:50 /datadisk/syslog-ng/disk-buffer/ne-collector-01b-01/syslog-ng-00000.rqf Here are log snippets when we ran syslog-ng in debug mode. =============================================== [2020-12-19T16:44:30.986904] Module loaded and initialized successfully; module='syslogformat' [2020-12-19T16:44:30.988121] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-02//syslog-ng-00000.rqf', queue_length='39418538', size='-1990797846' [2020-12-19T16:44:30.988228] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000' [2020-12-19T16:44:30.988382] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-04//syslog-ng-00000.rqf', queue_length='0', size='0' [2020-12-19T16:44:30.988506] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000' [2020-12-19T16:44:30.988640] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-05//syslog-ng-00000.rqf', queue_length='0', size='0' [2020-12-19T16:44:30.988731] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000' [2020-12-19T16:44:30.988867] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-06//syslog-ng-00000.rqf', queue_length='0', size='0' [2020-12-19T16:44:30.988958] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000' [2020-12-19T16:44:30.989113] Reliable disk-buffer state loaded; filename='/datadisk/syslog-ng/disk-buffer/ne-collector-01a-07//syslog-ng-00000.rqf', queue_length='0', size='0' [2020-12-19T16:44:30.989231] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='2', new_log_iw_size='100', min_log_fifo_size='50000' =============================================== We are unable to figure out why the disk buffer file is not getting truncated. We have restarted syslog-ng multiple times. Please advise. Thanks Raghu

2 3

Re: [syslog-ng] elasticsearch and ILM
by Németh Balázs 20 Dec '20

20 Dec '20

Hi, Originally I responsed to a digest message so you might did not get it. You can make a copy of elasticsearch-http SCL like elasticsearch-http2 and adjust the http() block's body part to use create, instead of index. - body("$(format-json --scope none --omit-empty-values index._index=`index` index._type=`type` index._id=`custom_id`)\n`template`") + body("$(format-json --scope none --omit-empty-values create._index=`index` index._type=`type` index._id=`custom_id`)\n`template`") https://github.com/syslog-ng/syslog-ng/blob/master/scl/elasticsearch/elasti… Although this will make it work with data streams you will still have to take care of everything else just as with now called legacy indexes (templates, mapping). Data streams on Kibana are quite new (7.0.9 or 7.10 maybe). It is still in flux. ILM is mostly used for cases when you want to replace indexes according to storage tiers in multi node clusters (warm, hot, cold node) or want to have delete indexes older than a specified time. If you just need a separate index for example every month, then use the data macros in the elasticsearch-http destination when you specify the index name. / Warning I do not work for Elastic, these are my personal experiences. ILM highly depends on things like rollover alias, which is set by the index template - created by beats in advance - whenever a new index is created. And it is versioned. Elasticsearch - the vendor - puts a lot of effort into making beats + Elasticsearch nodes themselves to work together. However they always change something upon upgrades which requires great effort to accomodate in syslog-ng. If you use syslog-ng with Elasticsearch, I highly recommend to go full manually with everything, maybe add enrichments with Logstash. You should not mix things used by beats with syslog-ng (templates, mapping, ILM, pipelines, dashboards, etc). / If you need some maintained examples, have a look at my configs.:https://github.com/abalage/balagetech-openwrt-syslog-ng-elasticsea… Regards, Balázs > > ---------- Forwarded message ---------- > From: Russell Fulton <r.fulton(a)auckland.ac.nz> > To: "Syslog-ng users' and developers' mailing list" < > syslog-ng(a)lists.balabit.hu> > Cc: > Bcc: > Date: Sat, 19 Dec 2020 21:47:12 +0000 > Subject: Re: [syslog-ng] elasticsearch and ILM > Russell wishes to delete this post :) > > Russell did what he should have done before posting! He ran tcpdump and > found the error response from ES! > > Apologies for the noise. The original question is still open! > > On 20/12/2020, at 10:38 AM, Russell Fulton <r.fulton(a)auckland.ac.nz> > wrote: > > Caution - Forged Internal Domain! > This e-mail cannot be validated and may not have been sent by the sender > shown in the 'From' field. > If you were not expecting to receive this e-mail we recommend you contact > the sender to confirm that they sent it. > If you believe this email was legitimately sent, we suggest the sender > notify the Staff Service Centre that it has been received as a forged > (fake) e-mail. > Please contact the Staff Service Centre on extension 86000 if you require > further assistance. > > > > On 19/12/2020, at 2:40 PM, Russell Fulton <r.fulton(a)auckland.ac.nz> wrote: > > I have been using the elasticsearch_http driver to push stuff into ES for > well over a year. Now I am trying to use the Index Lifecycle Management > (ILM) to manage these indexes and finding it very confusing. There seem to > be holes in the ES docs and help on their forum is a bit erratic. > > The most confusing thing is the “rollover index alias” and I have yet to > find a coherent explanation of exactly what it does and how to set it. Most > of the documentation seems to assume you are using filebeat, logstash or > datastreams. > > So far as I can see you syslog-ng can not write to datastreams as you need > to use “create” rather than “index”. > > If anyone has a working set up that they are happy to share it would be > wonderful. > > Russell > > > This is getting weirder ! > > ES is now failing to create my new index at all in spite of the > destination definition being practically identical to one that works: > > destination d_auth_elastic { > elasticsearch_http( > index("auth_${YEAR}.${MONTH}.${DAY}") > type("_doc") > persist-name("auth") > template("$(format-json --scope nv-pairs --exclude HOST_FROM > > > --exclude HOST > > > --exclude srcip* > > > --exclude SOURCE > > > --exclude PROGRAM > > > --exclude 0 > > > --exclude 1 > > > --exclude PID > > > --exclude LEGACY_MSGHDR > > > --key ISODATE)\n" > ) > > url("http://secesprd01.its.auckland.ac.nz:9200/_bulk") > ); > }; > > destination d_authm_elastic { > elasticsearch_http( > index("auth-000001") > type("") > template("$(format-json --scope nv-pairs --exclude HOST_FROM > > > --exclude HOST > > > --exclude srcip* > > > --exclude SOURCE > > > --exclude PROGRAM > > > --exclude 0 > > > --exclude 1 > > > --exclude PID > > > --exclude LEGACY_MSGHDR )\n" > ) > > url("http://secesprd01.its.auckland.ac.nz:9200/_bulk") > ); > }; > > > > log { > source( s_loghost ); > parser( p_patterns ); > filter( f_classified ); > filter( f_ping999 ); > filter( f_forti_stats ); > rewrite( r_user1 ); > rewrite( r_user2 ); > rewrite( r_srcip ); > parser( p_srcip ); > rewrite( r_srcip_country ); > rewrite( r_srcip_city); > rewrite( r_src_loc); > rewrite( r_ISODATE ); > destination( d_auth_elastic ); > destination( d_authm_elastic); > flags( flow-control); > }; > > d_auth_elastic works fine but d_authm_elastic never gets created. No > errors on the ES end. > > Anyone have any clues as to what is going on? > > I am at the point of starting tcpdump and looking at what is going over > the network. > > > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > _______________________________________________ > syslog-ng maillist - syslog-ng(a)lists.balabit.hu > https://lists.balabit.hu/mailman/listinfo/syslog-ng >

1 0

elasticsearch and ILM
by Russell Fulton 19 Dec '20

19 Dec '20

I have been using the elasticsearch_http driver to push stuff into ES for well over a year. Now I am trying to use the Index Lifecycle Management (ILM) to manage these indexes and finding it very confusing. There seem to be holes in the ES docs and help on their forum is a bit erratic. The most confusing thing is the “rollover index alias” and I have yet to find a coherent explanation of exactly what it does and how to set it. Most of the documentation seems to assume you are using filebeat, logstash or datastreams. So far as I can see you syslog-ng can not write to datastreams as you need to use “create” rather than “index”. If anyone has a working set up that they are happy to share it would be wonderful. Russell

1 2

Can elasticsearch_http send data to ES datastreams ?
by Russell Fulton 19 Dec '20

19 Dec '20

Datastreams behave like indexes *except* when you use _bulk then instead of “index” you must use “create”. Datastreams are set up to handle timeseries log data and they make managing such data must easier. R

1 0

Re: [syslog-ng] syslog-ng Digest, Vol 188, Issue 15
by Németh Balázs 19 Dec '20

19 Dec '20

Hi, You can make a copy of elasticsearch-http SCL like elasticsearch-http2 and adjust the http() block's body part to use create, instead of index. - body("$(format-json --scope none --omit-empty-values index._index=`index` index._type=`type` index._id=`custom_id`)\n`template`") + body("$(format-json --scope none --omit-empty-values create._index=`index` index._type=`type` index._id=`custom_id`)\n`template`") https://github.com/syslog-ng/syslog-ng/blob/master/scl/elasticsearch/elasti… Although this will make it work with data streams you will still have to take care of everything else just as with now called legacy indexes (templates, mapping). Data streams on Kibana are quite new (7.0.9 or 7.10 maybe). It is still in flux. ILM is mostly used for cases when you want to replace indexes according to storage tiers in multi node clusters (warm, hot, cold node) or want to have delete indexes older than a specified time. If you just need a separate index for example every month, then use the data macros in the elasticsearch-http destination when you specify the index name. / Warning I do not work for Elastic, these are my personal experiences. ILM highly depends on things like rollover alias, which is set by the index template - created by beats in advance - whenever a new index is created. And it is versioned. Elasticsearch - the vendor - puts a lot of effort into making beats + Elasticsearch nodes themselves to work together. However they always change something upon upgrades which requires great effort to accomodate in syslog-ng. If you use syslog-ng with Elasticsearch, I highly recommend to go full manually with everything, maybe add enrichments with Logstash. You should not mix things used by beats with syslog-ng (templates, mapping, ILM, pipelines, dashboards, etc). / If you need some maintained examples, have a look at my configs.: https://github.com/abalage/balagetech-openwrt-syslog-ng-elasticsearch/blob/… Regards, Balázs <syslog-ng-request(a)lists.balabit.hu> ezt írta (időpont: 2020. dec. 19., Szo, 13:00): > Send syslog-ng mailing list submissions to > syslog-ng(a)lists.balabit.hu > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.balabit.hu/mailman/listinfo/syslog-ng > or, via email, send a message with subject or body 'help' to > syslog-ng-request(a)lists.balabit.hu > > You can reach the person managing the list at > syslog-ng-owner(a)lists.balabit.hu > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of syslog-ng digest..." > Today's Topics: > > 1. elasticsearch and ILM (Russell Fulton) > > > > ---------- Forwarded message ---------- > From: Russell Fulton <r.fulton(a)auckland.ac.nz> > To: "Syslog-ng users' and developers' mailing list" < > syslog-ng(a)lists.balabit.hu> > Cc: > Bcc: > Date: Sat, 19 Dec 2020 01:40:42 +0000 > Subject: [syslog-ng] elasticsearch and ILM > I have been using the elasticsearch_http driver to push stuff into ES for > well over a year. Now I am trying to use the Index Lifecycle Management > (ILM) to manage these indexes and finding it very confusing. There seem to > be holes in the ES docs and help on their forum is a bit erratic. > > The most confusing thing is the “rollover index alias” and I have yet to > find a coherent explanation of exactly what it does and how to set it. > Most of the documentation seems to assume you are using filebeat, logstash > or datastreams. > > So far as I can see you syslog-ng can not write to datastreams as you need > to use “create” rather than “index”. > > If anyone has a working set up that they are happy to share it would be > wonderful. > > Russell_______________________________________________ > syslog-ng maillist - syslog-ng(a)lists.balabit.hu > https://lists.balabit.hu/mailman/listinfo/syslog-ng >

1 0

What am I doing wrong?
by Dan Egli 16 Dec '20

16 Dec '20

Help me understand this, please? I have ISC dhcpd configured to log to syslog.local7 (since I don't see an option to force it into it's own log file). So I went into my syslog-ng file and created two filters, just like on the example page of syslog-ng.com: filter dhcpmsgs { facility(23) ); filter non_dhcp { NOT filter(dhcpmsgs) ) I quoted almost directly from the example page on syslog-ng.com, but I keep getting this error when I reload syslog-ng's config: Error parsing filter expression, filter plugin NOT not found OR you may not used double quotes in your filter expression in /etc/syslog-ng/syslog-ng.conf:25:18-25:21: What did I do wrong? Here's the lines I modified from the syslog-ng page: filter demo_filter { host("example") and match("deny" value("MESSAGE")) }; filter inverted_demo_filter { NOT filter(demo_filter) } You can see the page at: https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edi… -- Dan Egli From my Test Server

2 6

syslog-ng and DTLS
by Secure-Logging, Generic 12 Dec '20

12 Dec '20

Dear syslog-ng users and developers, I know that syslog-ng is able to communicate using TCP and also supports TLS connections for improved security. Does syslog-ng also support support secure connections based UDP using DTLS? I am trying to implement a secure connection on a system with a network stack that does not support TCP because of resource constraints. Can I use syslog-ng for this? As syslog-ng security is based on OpenSSL and OpenSSL supports DTLS, I hope that this will be possible but I rely on your feedback concerning this topic. Regards, Stephan The information in this e-mail is confidential. The contents may not be disclosed or used by anyone other than the addressee. Access to this e-mail by anyone else is unauthorised. If you are not the intended recipient, please notify Airbus immediately and delete this e-mail. Airbus cannot accept any responsibility for the accuracy or completeness of this e-mail as it has been sent over public networks. If you have any concerns over the content of this message or its Accuracy or Integrity, please contact Airbus immediately. All outgoing e-mails from Airbus are checked using regularly updated virus scanning software but you should take whatever measures you deem to be appropriate to ensure that this message and any attachments are virus free.

2 1

Insider 2020-12: web interfaces; Grafana Loki; Amazon Linux 2;
by Peter Czanik (pczanik) 10 Dec '20

10 Dec '20

Dear syslog-ng users, This is the 87th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Web interfaces for your syslog server – an overview --------------------------------------------------- This is the 2020 edition of my most read blog entry about syslog-ng web-based graphical user interfaces (web GUIs). Many things have changed in the past few years. In 2011, only a single logging as a service solution was available, while nowadays, I regularly run into others. Also, while some software disappeared, the number of logging-related GUIs is growing. This is why in this post, I will mostly focus on generic log management and open source instead of highly specialized software, like SIEMs. https://www.syslog-ng.com/community/b/blog/posts/web-interfaces-for-your-sy… Sending logs from syslog-ng to Grafana Loki ------------------------------------------- Loki is one of the latest applications that lets you aggregate and query log messages, and of course to visualize logs using Grafana. It does not index the contents of log messages, only the labels associated with logs. This way, processing and storing log messages requires less resources, making Loki more cost-effective. Promtail, the log collector component of Loki, can collect log messages using the new, RFC5424 syslog protocol. This is where syslog-ng can send its log messages. >From this blog, you can learn a minimal Loki & Promtail setup. We will send logs from syslog-ng, and as a first step, will check them with logcli, a command line utility for Loki. Once it works, we will also install Grafana in a container and query Loki from there. https://www.syslog-ng.com/community/b/blog/posts/sending-logs-from-syslog-n… Installing syslog-ng in Amazon Linux 2, including Graviton2 ----------------------------------------------------------- Amazon Linux 2 is just one of many Linux distributions and other operating systems running in AWS. As Amazon Linux 2 is a close relative of RHEL 7 / CentOS 7, the user experience is also very close to these operating systems. All you need to learn is an extra command for enabling EPEL (Extra Packages for Enterprise Linux). Best of all: the very same commands work both on x86 and Graviton2 (Arm/Aarch64) instances. https://www.syslog-ng.com/community/b/blog/posts/installing-syslog-ng-in-am… WEBINARS * Using syslog-ng with Google Pub/Sub: https://www.syslog-ng.com/event/using-syslogng-with-google-pubsub-8145754/ * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Poll: RHEL 6 / CentOS 6 packages
by Peter Czanik (pczanik) 09 Dec '20

09 Dec '20

Hi, Now, that CentOS 6 (RHEL 6) is end of life and removed from mirrors (https://twitter.com/Arrfab/status/1334064890261147650) it is time to think what to do with my unofficial CentOS 6 syslog-ng packages on Copr (https://copr.fedorainfracloud.org/coprs/czanik/) They definitely do not receive any updates. Personally I would delete the CentOS 6 syslog-ng repositories today to free up resources, but before doing so I am asking your opinion. The repositories will be deleted by Copr anyway in about half a year (180 days after a release becomes EoL). You can answer here or my poll on Twitter: https://twitter.com/PCzanik/status/1334071192748167168 Bye, CzP Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 1

Filter matching not working
by Saqib M 09 Dec '20

09 Dec '20

Greetings - I have been trying to create a very basic filter that looks up a string in the incoming log. However, it would not match any filter and would go to the default filter. I have tried both match() and message(), neither worked for me. Please let me know if you think I am missing something. Following are the chunks from the syslog-ng.conf source s_net { # All syslog traffic on port 514 - this is direct from network devices. udp(port (514)); network(transport("tcp") max-connections(20000) log_iw_size(100000000) ); # tags("fortigate", "cisco", "default") ); }; filter f_discreg { message("default send string") }; log { source(s_net); filter(f_dlptracker); destination(d_dlptracker); flags(final); }; Here is the log from the test I ran. [2020-12-02T22:00:36+0000] Incoming log entry; source='s_net#0', line='default send string' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match', filter_type='OR' [2020-12-02T22:00:46+0000] Filter rule evaluation result; filter_result='not-match', filter_rule='f_tanium' [2020-12-02T22:00:46+0000] Filter rule evaluation begins; filter_rule='f_palo_alto' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match', filter_type='OR' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match', filter_type='OR' [2020-12-02T22:00:46+0000] Filter rule evaluation result; filter_result='not-match', filter_rule='f_palo_alto' [2020-12-02T22:00:46+0000] Filter rule evaluation begins; filter_rule='f_dlptracker' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match' [2020-12-02T22:00:46+0000] Filter rule evaluation result; filter_result='not-match', filter_rule='f_dlptracker' [2020-12-02T22:00:46+0000] Outgoing message; destination='d_fallback#0', message='2020-12-02T22:00:46+00:00 172.17.236.3 default send string\x0a' Regards, Saqib M Cybersecurity Co-op Global Cybersecurity Technologies Email: saqib.m(a)cummins.com<mailto:saqib.m@cummins.com> Cummins Inc.

4 5

Syslog-ng Windows Agent
by Nik Ambrosch 08 Dec '20

08 Dec '20

Hi, I was wondering if there is an OSE syslog-ng windows agent or if that's PE only? The "officially supported binaries" link is broken (bottom of the page) - https://www.syslog-ng.com/products/open-source-log-management/3rd-party-bin… We have some windows servers that I'd like to integrate into our log management system and the built-in syslog support is... not great. Thanks.

2 1

Don't change permissions
by Faine, Mark R. (MSFC-IS40)[NICS] 08 Dec '20

08 Dec '20

I'm monitoring the ACL on log files with monit and comparing them to a known good ACL to make sure everything is correct and running a correction script in response to a failure, my problem is that my compare keeps failing because syslog-ng is changing the effective permissions. Is there a way I can disable this? Thanks, -Mark

2 1

Re: [syslog-ng] syslog-ng version 26.1 running in a docker image from docker hub (balabit/syslog-ng)
by Martin, Uwe 01 Dec '20

01 Dec '20

Hello Laci, thanks for your support and sorry for the delay. After some debugs and installing the old version 3.5.6 on the host again, which was also not working I found the problem in the secondary ip address we used on the host. After removing the secondary address syslog-ng is working again without a problem. At the moment we will use this old version, but I will try to use the docker version on a test system to check if it works again and we can use the new features of the 3.29 version. Thanks again and kind regards Kind regards Uwe Von: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu> Im Auftrag von Laszlo Szemere (lszemere) Gesendet: Freitag, 13. November 2020 12:04 An: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Betreff: Re: [syslog-ng] syslog-ng version 26.1 running in a docker image from docker hub (balabit/syslog-ng) CAUTION: This e-mail is from an external sender. Do not click links or open attachments unless you recognize the sender and know the content is safe. Never provide your password! Hello Uwe, first of all: thank you for sharing your config and exact Syslog-ng versions. It helps a lot. At first look I cannot see any problem with your config. However, there were a lot of changes between version 3.5.6 and 3.26.1 (You might want to take a look at "cisco-parser()") If you suspect this is a filter problem, may I suggest starting Syslog-ng in a debug mode? example: https://github.com/balabit/syslog-ng-docker/issues/58#issuecomment-680674916 In this case you should see messages like: >>>>>> filter rule evaluation begin; rule='foobar', location='/conf/syslog-ng.conf:15:16', msg='0x7efd38015c40' <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='foobar', location='/conf/syslog-ng.conf:15:16', msg='0x7efd38015c40' If this does not help finding the problematic filter, I might need some example logs to trace down any parsing issues. Best regards, Laci ________________________________ From: syslog-ng on behalf of Martin, Uwe Sent: Friday, November 13, 2020 11:15 To: syslog-ng(a)lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu> Subject: [syslog-ng] syslog-ng version 26.1 running in a docker image from docker hub (balabit/syslog-ng) CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Hello, We have a problem after an upgrade form syslog-ng version 3.5.6 to version 26.1 in a docker container. CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d8c8c3380a71 balabit/syslog-ng:3.26.1 "/usr/sbin/syslog-..." About an hour ago Up About an hour Now not all logs from the devices are seen and forwarding to another log gateway is also not working. With tcpdump I see the packets on the interface. I seems some filter will not work. I add our config. Anybody an idea how to troubleshoot or fix this problem? Kind regards Uwe Festo SE & Co. KG Uwe Martin Abteilung IM-IHN IT Communication Networks Gottlieb-Stoll-Straße 29 66386 St. Ingbert Deutschland Telefon +49(6894)591-6323 Telefax +49(711)34754-6323 http://www.festo.com Der Inhalt dieser E-Mail und moeglicher Anhaenge sind ausschliesslich fuer den bezeichneten Adressaten bestimmt. Jede Form der Kenntnisnahme, Veroeffentlichung, Vervielfaeltigung oder Weitergabe des Inhalts dieser E-Mail und moeglicher Anhaenge durch unberechtigte Dritte ist unzulaessig. Wir bitten Sie, sich mit dem Absender der E-Mail in Verbindung zu setzen, falls Sie nicht der Adressat dieser E-Mail sind sowie das Material von Ihrem Computer zu loeschen. This e-mail and any attachments are confidential and intended solely for the addressee. The perusal, publication, copying or dissemination of the contents of this e-mail by unauthorised third parties is prohibited. If you are not the intended recipient of this e-mail, please delete it and immediately notify the sender. Rechtsform: Kommanditgesellschaft, Sitz: Esslingen a.N., Registergericht Stuttgart HRA 211583, Umsatzsteuerident-Nummer: DE 145339206 Persoenlich haftende Gesellschafterin: Festo Management SE, Sitz: Frankfurt a.M., Registergericht Frankfurt a.M., HRB 115998 Vorstand: Dipl.-Ing. Gerhard Borho, Dipl.-Ing. Dr. h.c. Oliver D. Jung (Vorsitzender), Dr. Ansgar Kriwet, Dr. Frank Melzer, Dipl.-Ing. (FH) Frank Notz, Dr. Jaroslav Patka Aufsichtsratsvorsitzender: Dr. Friedrich Eichiner Bitte beachten Sie: die Festo AG & Co. KG firmiert seit 31.01.2020 unter Festo SE & Co. KG. Please note, on 2020-01-31 Festo AG & Co. KG changed its company name to Festo SE & Co. KG.

1 0

poll: web interface for syslog-ng
by Peter Czanik (pczanik) 27 Nov '20

27 Nov '20

Hi, Now, that the 2020 edition of the syslog-ng web gui blog is out (https://www.syslog-ng.com/community/b/blog/posts/web-interfaces-for-your-sy…) it's time for a poll: Which web interface do you use to search / analyze / alert on your logs collected by syslog-ng? Splunk? Elasticsearch? Other? Or still grep/awk? 🙂 You can answer here or take the poll on Twitter: https://twitter.com/PCzanik/status/1331607033817280513 Thanks, Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

4 3

Re: [syslog-ng] poll: web interface for syslog-ng
by Clayton Dukes 25 Nov '20

25 Nov '20

Well naturally, my vote is for LogZilla 😉 Peter, I should probably get you an updated description of the product, it has grown a lot! Here are a few useful links: LogZilla in 4 minutes https://www.logzilla.net/logzilla-in-4-minutes.html Automation/Orchestration https://www.logzilla.net/tcat.html ELK Revealed https://www.logzilla.net/the-real-cost-of-elk.html [photo] [signature_543644359] Clayton Dukes CEO, LogZilla Corporation [signature_672330100]<http://www.linkedin.com/in/lzcdukes/> [signature_753363620]<http://twitter.com/logzilla> [signature_14708953] (855) LogZilla<tel:(855)%20LogZilla> [signature_1738705128] 919-600-3198<tel:919-600-3198> [signature_579604246] cdukes(a)logzilla.net<mailto:cdukes@logzilla.net> [signature_1641853981] logzilla.net<http://logzilla.net/> [signature_1306807084] 4819 Emperor Blvd., Suite 400, Raleigh, NC, 27703 [signature_560762648]<https://calendly.com/teamneo/logzilla-neo-discovery> Schedule a 15 minute demo<https://calendly.com/teamneo/logzilla-neo-discovery> [signature_457621985]<https://youtu.be/yGQqkXegdoo> [signature_39202885]LZ in 4 Minutes<https://youtu.be/yGQqkXegdoo> LogZilla Corporation delivers a Centralized Log Management (CLM) platform, designed for IT Operations, Security, and Risk Management leaders who want to gain better incident investigation capabilities by capturing all network and security related logs into a single log collection platform. [Logo Description automatically generated] From: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu> on behalf of "Peter Czanik (pczanik)" <Peter.Czanik(a)oneidentity.com> Reply-To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Date: Wednesday, November 25, 2020 at 9:46 AM To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: [syslog-ng] poll: web interface for syslog-ng Hi, Now, that the 2020 edition of the syslog-ng web gui blog is out (https://www.syslog-ng.com/community/b/blog/posts/web-interfaces-for-your-sy…<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.syslo…>) it's time for a poll: Which web interface do you use to search / analyze / alert on your logs collected by syslog-ng? Splunk? Elasticsearch? Other? Or still grep/awk? 🙂 You can answer here or take the poll on Twitter: https://twitter.com/PCzanik/status/1331607033817280513<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.c…> Thanks, Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsyslog-ng…> https://twitter.com/PCzanik<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.c…> IMPORTANT NOTICE: This e-mail message is intended to be received only by persons entitled to receive the confidential information it may contain. E-mail messages to clients of Logzilla Corporation may contain information that is confidential and legally privileged. Please do not read, copy, forward, or store this message unless you are an intended recipient of it. If you have received this message in error, please forward it to the sender and delete it completely from your computer system.

1 0

syslog-ng 3.30 Debian/Ubuntu packages (OBS)
by Laszlo Budai 20 Nov '20

20 Nov '20

Hi, syslog-ng 3.30.1[1] packages are available from OBS repositories[2, 3, 4]. List of supported OSs: * Debian 8.0 * Debian 9.0 * Debian 10.0 * Raspbian 9.0 * Raspbian 10.0 * Ubuntu 14.04 * Ubuntu 16.04 * Ubuntu 18.04 * Ubuntu 18.10 * Ubuntu 19.04 * Ubuntu 19.10 * Ubuntu 20.04 * Ubuntu 20.10 (build is still in progress in OBS) Install ------- example: Debian 9.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-geat install syslog-ng-core` Links -------- [1] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.30.1 [2] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… [3] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… [4] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… regards, Laszlo Budai

1 0

issues with starting syslog-ng from systemd
by Russell Fulton 20 Nov '20

20 Nov '20

version 3.26.1 — ubuntu 14.04 I can start syslog-ng from the command line without issues (apart form a warning to stderr) but when I try and start it from systemd it starts up (it prints the warning message to the logs) and then exits silently. This version was compiled locally i.e. it was not installed from a package. The .service file was adapted from a sample in the contrib directory. I also had the same behaviour from and other .service file that I put together myself. It is presumably some to do with the environment. I did try starting it with -i (interactive) in the hope that it would spit something out on the console but instead it hung and I had to ^C it Any thoughts on how to figure out what is going on? ===================================== here are two different attempts (first commented) rful011@secmgrprd02:~$ cat /etc/systemd/system/syslog-ng.service #[Unit] #Description=syslog next generation #After=network.target auditd.service #[Service] #EnvironmentFile=/etc/default/syslog-ng #ExecStart=/usr/local/sbin/syslog-ng -f /usr/local/etc/syslog-ng.conf #ExecReload=/usr/local/sbin/syslog-ng-ctl reload #Restart=on-failure #RestartPreventExitStatus=255 #Type=notify #[Install] #WantedBy=multi-user.target #Alias=syslog-ng.service [Unit] Description=System Logger Daemon Documentation=man:syslog-ng(8) Conflicts=emergency.service emergency.target Wants=network.target network-online.target After=network.target network-online.target [Service] Type=notify EnvironmentFile=-/etc/default/syslog-ng EnvironmentFile=-/etc/sysconfig/syslog-ng ExecStart=/usr/local/sbin/syslog-ng -f /usr/local/etc/syslog-ng.conf ExecReload=/usr/local/sbin/syslog-ng-ctl reload StandardOutput=journal StandardError=journal Restart=on-failure [Install] WantedBy=multi-user.target

3 2

syslog-ng-3.30.1
by Laszlo Budai 19 Nov '20

19 Nov '20

3.30.1 Highlights * filter template function * support proxy-protocol<https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt> Features * kafka (C implementation): * Added template support to topic(). * Added fallback-topic() option, which will be used, if the templated topic() yields an invalid topic name. (#3372<https://github.com/syslog-ng/syslog-ng/pull/3372>) * transport: add proxy-protocol support http://www.haproxy.org/download/1.8/doc/proxy-protocol.txt Example config, click to expand! ([#3437<https://github.com/syslog-ng/syslog-ng/pull/3437>](https://github.com//pull/3437<https://github.com/syslog-ng/syslog-ng/pull/3437>)) * filter: new template function The new introduced filter template function will allow filtering lists based on a filter expression. For example this snippet removes odd numbers log { source { example-msg-generator(num(1) values(INPUT => "0,1,2,3")); }; destination { file("/dev/stdout" template("$(filter ('$(% $_ 2)' eq '0') $INPUT)\n)") ); }; }; (#3426<https://github.com/syslog-ng/syslog-ng/pull/3426>) * file, network, program destinations: : new truncate_size option introduced to truncate an output message to a specified max size. default value is -1 (disabled). network("127.0.0.1" truncate_size(100)); new stats counters: dst.network;d_local#0;udp,127.0.0.1:1111;a;truncated_count;1 dst.network;d_local#0;udp,127.0.0.1:1111;a;truncated_bytes;1 (#3474<https://github.com/syslog-ng/syslog-ng/pull/3474>) * network: add FreeBSD support for the so_reuseport(yes) the same as in linux SO_REUSEPORT (FreeBSD uses SO_REUSEPORT_LB flag). (#3438<https://github.com/syslog-ng/syslog-ng/pull/3438>) * date-parser: %z accepts local timezone std format as well (#3453<https://github.com/syslog-ng/syslog-ng/pull/3453>) * syslog-format: accepting longer sdata keys Triggered by #3197<https://github.com/syslog-ng/syslog-ng/issues/3197> At the end of the discussion in the mentioned issue, we decided to change the parser and accept longer than 32 character ID's. (#3244<https://github.com/syslog-ng/syslog-ng/pull/3244>) * systemd-journal: add namespace() option This option accepts a string which is identical to the --namespace option of journalctl. For systems defining this option with a systemd version older than v245 a warning is issued. (#3358<https://github.com/syslog-ng/syslog-ng/pull/3358>) Bugfixes * date-parse: %Z should parse the same timezones as %z not just local and gmt (#3453<https://github.com/syslog-ng/syslog-ng/pull/3453>) * python: printing the exception instead of None (if compiled with clang) (#3405<https://github.com/syslog-ng/syslog-ng/pull/3405>) * network/udp: message was lost (not sent) if it was too large, and a time reopen amount of time needed to expire to send the next message lowering the thoughtput. now it is truncated at 65507. (#3474<https://github.com/syslog-ng/syslog-ng/pull/3474>) * tlscontext: support IPv6 X509v3 Subject Alternative Name Fixes #3465<https://github.com/syslog-ng/syslog-ng/issues/3465> (#3466<https://github.com/syslog-ng/syslog-ng/pull/3466>) * map: pass $_ to if correctly. Prior this patchset, if did not receive $_ correctly. After this change, these configurations will work: log { source { example-msg-generator(num(1) values(INPUT => "0,1,2,3")); }; destination { file("/dev/stdout" template("$(map $(if ('$(% $_ 2)' eq '0') 'even' 'odd') $INPUT)'\n)") ); }; }; (#3426<https://github.com/syslog-ng/syslog-ng/pull/3426>) * systemd-journal: add namespace to the persist name (#3407<https://github.com/syslog-ng/syslog-ng/pull/3407>) * syslog-ng: fixed numerous spelling mistakes in messages generated by syslog-ng (#3398<https://github.com/syslog-ng/syslog-ng/pull/3398>) * network: fix TLS certificate hostname verification when using failover() servers For TLS certificate hostname verification, the certificate's hostname needs to be compared to the configured hostname of the primary and each failover server. syslog-ng used always the primary server's name incorrectly. (#3447<https://github.com/syslog-ng/syslog-ng/pull/3447>) * afsocket: syslog-ng fails to bind() after config revert When having a program source or destination and a network destination in the config, if we reload with an invalid config, syslog-ng crashes, as it cannot init the old network source, because its address is in use. (#3416<https://github.com/syslog-ng/syslog-ng/pull/3416>) * syslog-ng-ctl: when syslog-ng gets stuck on executing a heavy stats-ctl command, should be able to do a graceful shutdown when it is requested. (#3349<https://github.com/syslog-ng/syslog-ng/pull/3349>) * json-parser: fix parsing 64 bit numbers (currently 32 bit was a limit) (#3403<https://github.com/syslog-ng/syslog-ng/pull/3403>) * usertty(): on each tty open error an error mesage and a 10 minutes long disabling of the usertty() destination has been added. Until now, the usertty() destination were only disabled for blocking write() calls. (#3473<https://github.com/syslog-ng/syslog-ng/pull/3473>) Notes to developers * Proxy protocol support added to loggen. Four new options added to loggen to suppport the proxy protocol: * --proxied : Generate PROXY protocol v1 header * --proxy-src-ip : Set the source IP for the PROXY protocol v1 header. If not specified a random IP address generated (192.168.1.X). * --proxy-dst-ip : Set the destination IP for the PROXY protocol v1 header. If not specified a random IP address generated (192.168.1.X). * --proxy-src-port : Set the source port for the PROXY protocol v1 header. If not specified a random port generated in the range 5000-10000. * --proxy-dst-port : Set the destination port for the PROXY protocol v1 header. If not specified the port number 514 will be used. (#3462<https://github.com/syslog-ng/syslog-ng/pull/3462>) * bison: Minimum required version is now 3.4.2. You still only need bison, if you are building from git source or changing the grammar in the released source tarball. (#2526<https://github.com/syslog-ng/syslog-ng/pull/2526>) * Template evaluation related function signatures changed. A new structure LogTemplateEvalOptions is introduced to group parameters together. (#3426<https://github.com/syslog-ng/syslog-ng/pull/3426>) Other changes * json-parser: change every per message logs that was higher than debug to debug (#3401<https://github.com/syslog-ng/syslog-ng/pull/3401>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler, Boris Korzun, Gabor Nagy, Laszlo Budai, Laszlo Szemere, László Várady, Norbert Takacs, Peter Kokai, Viktor Juhasz, Vivin Peris, Zoltan Pallagi, bjoe2k4

1 0

syslog-ng version 26.1 running in a docker image from docker hub (balabit/syslog-ng)
by Martin, Uwe 13 Nov '20

13 Nov '20

Hello, We have a problem after an upgrade form syslog-ng version 3.5.6 to version 26.1 in a docker container. CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d8c8c3380a71 balabit/syslog-ng:3.26.1 "/usr/sbin/syslog-..." About an hour ago Up About an hour Now not all logs from the devices are seen and forwarding to another log gateway is also not working. With tcpdump I see the packets on the interface. I seems some filter will not work. I add our config. Anybody an idea how to troubleshoot or fix this problem? Kind regards Uwe

2 1

Insider 2020-10: LaaS; PAN-OS; Security Onion; New Relic;
by Peter Czanik (pczanik) 12 Nov '20

12 Nov '20

Dear syslog-ng users, This is the 86th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS How to use syslog-ng with LaaS and why? --------------------------------------- The first Logging as a Service (LaaS) I learned about many years ago was created by Loggly. Of course there are many more LaaS providers now. While most services also provide their own clients for sending log messages, many of them also document sending log messages to them using syslog-ng. >From this blog, you can learn about the advantages of using syslog-ng with LaaS, or with any cloud based SIEM solution or a Managed Security Service Provider (MSSP), and also how easy it is to create a configuration block for a new provider. While creating a configuration block takes some time, the simplification it provides can make your life easier on the long term. https://www.syslog-ng.com/community/b/blog/posts/how-to-use-syslog-ng-with-… Parsing PAN-OS logs using syslog-ng ----------------------------------- Version 3.29 of syslog-ng was released recently including a user-contributed feature: the panos-parser(). It is parsing log messages from PAN-OS (Palo Alto Networks Operating System). Unlike some other networking devices, the message headers of PAN-OS syslog messages are standards-compliant. However, if you want to act on your messages (filtering, alerting), you still need to parse the message part. The panos-parser() helps you create name-value pairs from the message part of the logs. >From this blog you can learn why it is useful to parse PAN-OS log messages and how to use the panos-parser(). https://www.syslog-ng.com/community/b/blog/posts/parsing-pan-os-logs-using-… Syslog-ng and Security Onion ---------------------------- One of the most interesting projects utilizing syslog-ng is Security Onion, a free and open source Linux distribution for threat hunting, enterprise security monitoring, and log management. It is utilizing syslog-ng for log collection and log transfer and uses the Elastic stack to store and search log messages. Even if you do not use its advanced security features, you can still use it for centralized log collection and as a nice web interface for your logs. But it is also worth getting acquainted with its security monitoring features, as it can show you useful insights about your network. Best of all, Security Onion is completely free and open source, with commercial support available for it. https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-and-security-oni… How to Use Syslog-ng with New Relic ----------------------------------- Learn how to use the http() destination to send logs to New Relic: https://blog.newrelic.com/product-news/how-to-use-syslog-ng-with-new-relic/ WEBINARS * Improve SIEM performance and cut SIEM costs with log management solutions: https://www.syslog-ng.com/event/improve-siem-performance-and-cut-siem-costs… * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Using usertty
by Alexandre Santos 04 Nov '20

04 Nov '20

Hi, I am trying to use usertty(*) to send log all messages with severity equal or higher than critical to every user logged. But I am not getting any messages in serial port or ssh. I am sending the configurations and the debug log in attachment. Can you help me to understand what is happening? Thanks in advance, Alex

2 7