- syslog-ng - lists.balabit.hu

The syslog-ng Insider 2024-11: testing; Quickwit; MacPorts
by Peter Czanik (pczanik) 14 Nov '24

14 Nov '24

Dear syslog-ng users, This is the 125th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS A call for syslog-ng testing ---------------------------- After the last syslog-ng release, we started a campaign to close open issues on GitHub. We'd like to continue this effort and call for collaboration from our users and contributors to make OSE even more stable. While unit tests are great (and we do many tests in-house), nothing can replace using syslog-ng in real-world situations. This blog collects some resources about how you can start testing the latest syslog-ng release from GitHub. https://www.syslog-ng.com/community/b/blog/posts/a-call-for-syslog-ng-testi… Working with Quickwit --------------------- We are always looking for new ways to store log messages. Quickwit is a new contender, designed for log storage, and among others, it also provides an Elasticsearch-compatible API. From this blog, you can learn about Quickwit, and how to forward log messages from syslog-ng to it using the Elasticsearch-compatible API. https://www.syslog-ng.com/community/b/blog/posts/first-steps-with-quickwit-… You can also use the OpenTelemetry protocol to send logs to Quickwit with syslog-ng. https://www.syslog-ng.com/community/b/blog/posts/sending-logs-to-quickwit-u… Huge improvements for syslog-ng in MacPorts ------------------------------------------- Recently I wrote about a campaign that we started to resolve issues on GitHub. Some of the fixes are coming from our enthusiastic community. Thanks to this, there is a new syslog-ng-devel port in MacPorts, where you can enable almost all syslog-ng features even for older MacOS versions and PowerPC hardware. Some of the freshly enabled modules include support for Kafka, GeoIP or OpenTelemetry. From this blog entry, you can learn how to install a legacy or an up-to-date syslog-ng version from MacPorts. https://www.syslog-ng.com/community/b/blog/posts/huge-improvements-for-sysl… WEBINARS * New webcast: “Log management superpowers: a PAM Essentials case study”. You can watch it at https://www.syslog-ng.com/webcast-ondemand/log-management-superpowers-a-pam… * You can learn about upcoming webinars and browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

can kv-parser options be from macros
by Evan Rempel 02 Nov '24

02 Nov '24

Does anyone know if I can use the enriching data to set macros that can then be used as kv-parser options. These values might change on a message by message basis. For example: parser p_kv { kv-parser ( prefix("$kv_prefix") value-separator("$kv_vseparator") pair-separator("$kv_pseparator") ); }; -- Evan

2 2

certificate errors result in excessive logging.
by Evan Rempel 21 Oct '24

21 Oct '24

I am using tls configuration with a locally signed certificate. This mans that I have to configure a custom root CA on to all client systems for them to be able to establish the tls connection to my syslog server. When the clients are unable to verify the server certificate, the server logs three messages for every connection attempt syslog-ng[452597]: SSL error while reading stream; tls_error='error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca', location='/etc/syslog-ng/syslog-ng.server.conf:71:17' syslog-ng[452597]: Error reading RFC6587 style framed data; fd='21', error='Connection reset by peer (104)' syslog-ng[452597]: Syslog connection closed; fd='21', client='AF_INET(1.2.3.4:1234)', local='AF_INET(1.2.3.4:1234)' If there are 100's of clients, and they try to reconnect at a fast rate (every 5 seconds) this can result in a large volume of messages. Is there any way to configure the logging rate of these types of errors or get rid of it altogether. Anyone have any comments on this? -- Evan

2 3

Re: [syslog-ng] syslog-ng Digest, Vol 234, Issue 4
by brian hoffman 15 Oct '24

15 Oct '24

Yes, you were spot on. I realized not long after I sent this that the .0 was missing on the end.Thanks!-Brian Date: Tue, 15 Oct 2024 00:32:37 +0000 From: Evan Rempel <erempel(a)uvic.ca> To: "syslog-ng(a)lists.balabit.hu" <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] netmask filter issue Message-ID: <YT4P288MB03355F0F965988C4C22978C0B9452(a)YT4P288MB0335.CANP288.PROD.OUTLOOK.COM> Content-Type: text/plain; charset="iso-8859-1" First off, to write a CIDR address correctly you need to use something of the form w.x.y.z/M Your specification is missing the ".z" prtion of the ip address. I think there is also some confusion regarding how netmasks work. The ip address 192.168.128.0 (I assume you want a .0 at the end) written in binary is 11000000.10101000.10000000.00000000 The /17 subnet mask written in binary is 11111111.11111111.10000000.00000000 If we perform a binary AND of these two addresses we get 11000000.10101000.10000000.000000 written in decimal is 192.168.128.0 This means that the FIRST address of this range is 192.168.128.0 If we perform the binary NOT of the subnet mask we get 00000000.00000000.01111111.11111111 When we take the binary OR of this with the IP address (w.x.y.z) we get 11000000.10101000.11111111.1111111 in decimal is 192.168.255.255 which is the last address if the range. The result is that 192.168.1.0/17 means the range 192.168.128.0 -- 192.168.255.255 I hope that helps. -- Evan

1 0

netmask filter issue
by brian hoffman 15 Oct '24

15 Oct '24

I set up a filter to send certain messages matching an IP range to a destination (a simplified version of my actual config).filter f_siem { netmask(192.168.128/17) or}; For some reason this is picking up the address 192.168.63.41, even though it's not in the range. 192.168.63.41 (10101100.00010111.0) does not match the network portion of 192.168.128.0/17 (10101100.00010111.1) but for some reason it's still getting picked up. I know for sure that it is this particular line because when I comment it out the messages are no longer forwarded for that device. Is there something different about the way this netmask directive works in syslog-ng that would cause it to get matched by this filter?-Brian

2 1

The syslog-ng Insider 2024-10: 4.8.0 release; version number; Debian Stable
by Peter Czanik (pczanik) 10 Oct '24

10 Oct '24

Dear syslog-ng users, This is the 124th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Version 4.8.0 of syslog-ng improves FreeBSD and MacOS support ------------------------------------------------------------- Recently One Identity released version 4.8.0 of its open-source log management application. Learn about some of the new features and bug fixes: why upgrade to the latest syslog-ng version, not only on FreeBSD :-) https://www.syslog-ng.com/community/b/blog/posts/version-4-8-0-of-syslog-ng… Why it is useful to set the version number in the syslog-ng configuration --------------------------------- The syslog-ng configuration starts with a version number declaration. Up until recently, if it was missing, syslog-ng did not start. With syslog-ng 4.8, this is changing. From this blog, you can learn why version information is useful, what workaround you can use if you do not want to edit your syslog-ng configuration on each update, and what changed in version 4.8. https://www.syslog-ng.com/community/b/blog/posts/why-it-is-useful-to-set-th… We are switching syslog-ng containers from Debian Testing to Stable ----------------------------------------------------- For many years, the official syslog-ng container and development containers were based on Debian Testing. We are switching to Debian Stable now. Learn about the history and the reasons for the change now. https://www.syslog-ng.com/community/b/blog/posts/we-are-switching-syslog-ng… WEBINARS * New webcast: “Log management superpowers: a PAM Essentials case study”. You can watch it at https://www.syslog-ng.com/webcast-ondemand/log-management-superpowers-a-pam… * You can learn about upcoming webinars and browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Syslog-ng needs some karma on Fedora
by Peter Czanik (pczanik) 10 Oct '24

10 Oct '24

Hi, Version 4.8.1 of syslog-ng was released last week. It is a bugfix release, and it contains fixes for problems also reported by members of the Fedora community. The Fedora 41 release is near, so package updates now need some additional testing, and “karma” in Bodhi. You can find information on how to install syslog-ng 4.8.1 from a testing repo on Fedora 41 beta at https://bodhi.fedoraproject.org/updates/FEDORA-2024-4e812b8a23 . This is also the place where you can provide feedback and karma. Thanks for your help! Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

syslog-ng 4.8.1 packages
by Peter Czanik (pczanik) 04 Oct '24

04 Oct '24

Hi, Version 4.8.1 of syslog-ng was released yesterday. Here is the current status of packaging: * As usual, Debian and Ubuntu packages were made available as part of the release process: https://www.syslog-ng.com/community/b/blog/posts/installing-the-latest-sysl… * openSUSE / SLES and Fedora / RHEL unofficial repositories are maintained by me, and now updated to 4.8.1: https://www.syslog-ng.com/community/b/blog/posts/installing-latest-syslog-n… you use the syslog-ng48 repo, then you do not have to change anything, just start an update. * my syslog-ng-stable repos for RPM distros will be updated next week, as I always wait a few days. * updates for FreeBSD ports are now submitted, but might take some time. If you urgently want to test 4.8.1, then you can use ports files from my GitHub repo: https://www.syslog-ng.com/community/b/blog/posts/installing-a-syslog-ng-4-d… Not maintained by us, but as far as I can see, Homebrew and Alpine Linux both already have syslog-ng 4.8.1. Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

fatal error is syslog-ng 4.8 from copr when reading pipe sources.
by Evan Rempel 04 Oct '24

04 Oct '24

I upgraded syslog-ng today from 4.6 to 4.8 and my disk drive almost immediately filled due to the local error from syslog-ng 2024-09-20T11:59:38.760-07:00 hostname syslog.err syslog-ng[600690]: Error invoking seek on file; filename='/var/log/syslog.pipes/mailstats2', error='Illegal seek (29)' This logged repeatedly until the disk filled up. My source definition for that pipe is source mailstats2 { pipe("/var/log/syslog.pipes/mailstats2" log_iw_size(900000) log_fetch_limit(500) flags(no-parse) ); }; Any insight would be useful. -- Evan

2 2

Version 4.8.1 of syslog-ng is now available
by Peter Czanik (pczanik) 03 Oct '24

03 Oct '24

Hi, I'm happy to share that version 4.8.1 of syslog-ng is now available. The focus of this release was bug fixing, but there are a few minor new features also available, like MacPorts support and ElasticSearch datastream support. Please help to spread the word, and share / retweet the release announcement on social media: * https://www.linkedin.com/posts/peterczanik_release-syslog-ng-481-syslog-ngs… * https://fosstodon.org/@PCzanik/113243820863895946 * https://x.com/PCzanik/status/1841838695659274333 For a complete list of changes you can read the release notes: Highlights * elasticsearch-datastream() destinations can be used to feed Elasticsearch data streams<https://www.elastic.co/guide/en/elasticsearch/reference/current/data-stream…>. Example config: elasticsearch-datastream( url("https://elastic-endpoint:9200/my-data-stream/_bulk") user("elastic") password("ba3DI8u5qX61We7EP748V8RZ") ); * (#5069<https://github.com/syslog-ng/syslog-ng/pull/5069>) * building: thanks to Sergey Fedorov and Marius Schamschula, macOS builds now support gcc again. They also updated the MacPort version<https://github.com/macports/macports-ports/blob/30c55ba04d8c0693c18cdf84014…> of syslog-ng (develop). Great work, and thank you so much for your contribution! (#5108<https://github.com/syslog-ng/syslog-ng/pull/5108>) Features * tls(): expose the key fingerprint of the peer in ${.tls.x509_fp} if trusted-keys() is used to retain the actual peer identity in the received messages. (#5068<https://github.com/syslog-ng/syslog-ng/pull/5068>) * syslog-parser: Added the no-piggyback-errors and the piggyback-errors flags to control whether the message retains the original message or not on parse error(s). By default the old behaviour/piggyback-errors flag is active. The following new tags can be added by the syslog-parser to the message when the parsing failed: * no-piggyback-errors: On failure, the original message will be left as it was before parsing, the value of $MSGFORMAT will be set to syslog-error, and a tag will be placed on the message corresponding to the parser's failure. * piggyback-errors: On failure, the old behaviour is used (clearing the entire message then syslog-ng will generate a new message in place of the old one describing the parser's error). * syslog.rfc5424_missing_hostname * syslog.rfc5424_missing_app_name * syslog.rfc5424_missing_procid * syslog.rfc5424_missing_msgid * syslog.rfc5424_missing_sdata * syslog.rfc5424_invalid_sdata * syslog.rfc5424_missing_message (#5063<https://github.com/syslog-ng/syslog-ng/pull/5063>) Bugfixes * syslog-ng-ctl: fix escaping of stats prometheus Metric labels (for example, the ones produced by metrics-probe()) may contain control characters, invalid UTF-8 or \ characters. In those specific rare cases, the escaping of the stats prometheus output was incorrect. (#5046<https://github.com/syslog-ng/syslog-ng/pull/5046>) * wildcard-file(): fix crashes can occure if the same wildcard file is used in multiple sources Because of some persistent name construction and validation bugs the following config crashed syslog-ng (if there were more than one log file is in the /path folder) @version: current @include "scl.conf" source s_files1 { file("/path/*.log" persist-name("p1") ); }; source s_files2 { file("/path/*.log" persist-name("p2") ); }; destination s_stdout { stdout(); }; log { source(s_files1); destination(s_stdout); }; log { source(s_files2); destination(s_stdout); }; * NOTE: * The issue occurred regardless of the presence of the persist-name() option. * It affected not only the simplified example of the legacy wildcard file() but also the new wildcard-file() source. (#5091<https://github.com/syslog-ng/syslog-ng/pull/5091>) * syslog-ng-ctl: fix crash of syslog-ng service in g_hash_table lookup function after syslog-ng-ctl reload (#5087<https://github.com/syslog-ng/syslog-ng/pull/5087>) * file(), stdout(): fix log sources getting stuck Due to an acknowledgment bug in the file() and stdout() destinations, sources routed to those destinations may have gotten stuck as they were flow-controlled incorrectly. This issue occured only in extremely rare cases with regular files, but it occured frequently with /dev/stderr and other slow pseudo-devices. (#5134<https://github.com/syslog-ng/syslog-ng/pull/5134>) * directory-monitor: fixed a main thread assertion crash that might have occurred during syslog-ng stop or restart (#5086<https://github.com/syslog-ng/syslog-ng/pull/5086>) * Config @version: fixed compat-mode inconsistencies when @version was not specified at the top of the configuration file or was not specified at all (#5145<https://github.com/syslog-ng/syslog-ng/pull/5145>) * grpc: Fix potential memoryleak when the grpc module is loaded but not used. (#5062<https://github.com/syslog-ng/syslog-ng/pull/5062>) * s3(): Eliminated indefinite memory usage increase for each reload. The increased memory usage is caused by the botocore library, which caches the session information. We only need the Session object, if role() is set. The increased memory usage still happens with that set, currently we only fixed the unset case. (#5149<https://github.com/syslog-ng/syslog-ng/pull/5149>) * opentelemetry() sources: fix crash when workers() is set to > 1 (#5138<https://github.com/syslog-ng/syslog-ng/pull/5138>) * opentelemetry() sources: fix source hang-up on flow-controlled paths (#5148<https://github.com/syslog-ng/syslog-ng/pull/5148>) * metrics-probe(): fix disappearing metrics from stats prometheus output metrics-probe() metrics became orphaned and disappeared from the syslog-ng-ctl stats prometheus output whenever an ivykis worker stopped (after 10 seconds of inactivity). (#5075<https://github.com/syslog-ng/syslog-ng/pull/5075>) * affile: Fix an invalid lseek call mainly on the pipe() source, but also possible if using affile on pipe like files (pipe, socket and FIFO). (#5058<https://github.com/syslog-ng/syslog-ng/pull/5058>) Other changes * format-json: spaces around = in $(format-json) template function could cause a crash<https://github.com/syslog-ng/syslog-ng/issues/5065>. The fix of the issue also introduced an enhancement, from now on spaces are allowed around the = operator, so the following $(format-json) template function calls are all valid: $(format-json foo =alma) $(format-json foo= alma) $(format-json foo = alma) $(format-json foo=\" alma \") $(format-json foo= \" alma \") $(format-json foo1= alma foo2 =korte foo3 = szilva foo4 = \" meggy \" foo5=\"\") * Please note the usage of the escaped strings like \" meggy \", and the (escaped and) quoted form that used for an empty value \"\", the latter is a breaking change as earlier an expression like key= led to a json key-value pair with an empty value {"key":""} that will not work anymore. (#5080<https://github.com/syslog-ng/syslog-ng/pull/5080>) * building: fixed multiple potentional FreeBSD build errors (#5099<https://github.com/syslog-ng/syslog-ng/pull/5099>) * docker: Changed the container image's base to debian:bookworm. (#5056<https://github.com/syslog-ng/syslog-ng/pull/5056>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Attila Szakacs, Balazs Scheidler, Hofi, Kovács Gergő Ferenc, László Várady, Mate Ory, Peter Czanik (CzP), Sergey Fedorov, Marius Schamschula, Szilard Parrag, Tamas Pal, shifter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0