- syslog-ng - lists.balabit.hu

netmask filter issue
by brian hoffman 15 Oct '24

15 Oct '24

I set up a filter to send certain messages matching an IP range to a destination (a simplified version of my actual config).filter f_siem { netmask(192.168.128/17) or}; For some reason this is picking up the address 192.168.63.41, even though it's not in the range. 192.168.63.41 (10101100.00010111.0) does not match the network portion of 192.168.128.0/17 (10101100.00010111.1) but for some reason it's still getting picked up. I know for sure that it is this particular line because when I comment it out the messages are no longer forwarded for that device. Is there something different about the way this netmask directive works in syslog-ng that would cause it to get matched by this filter?-Brian

2 1

The syslog-ng Insider 2024-10: 4.8.0 release; version number; Debian Stable
by Peter Czanik (pczanik) 10 Oct '24

10 Oct '24

Dear syslog-ng users, This is the 124th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Version 4.8.0 of syslog-ng improves FreeBSD and MacOS support ------------------------------------------------------------- Recently One Identity released version 4.8.0 of its open-source log management application. Learn about some of the new features and bug fixes: why upgrade to the latest syslog-ng version, not only on FreeBSD :-) https://www.syslog-ng.com/community/b/blog/posts/version-4-8-0-of-syslog-ng… Why it is useful to set the version number in the syslog-ng configuration --------------------------------- The syslog-ng configuration starts with a version number declaration. Up until recently, if it was missing, syslog-ng did not start. With syslog-ng 4.8, this is changing. From this blog, you can learn why version information is useful, what workaround you can use if you do not want to edit your syslog-ng configuration on each update, and what changed in version 4.8. https://www.syslog-ng.com/community/b/blog/posts/why-it-is-useful-to-set-th… We are switching syslog-ng containers from Debian Testing to Stable ----------------------------------------------------- For many years, the official syslog-ng container and development containers were based on Debian Testing. We are switching to Debian Stable now. Learn about the history and the reasons for the change now. https://www.syslog-ng.com/community/b/blog/posts/we-are-switching-syslog-ng… WEBINARS * New webcast: “Log management superpowers: a PAM Essentials case study”. You can watch it at https://www.syslog-ng.com/webcast-ondemand/log-management-superpowers-a-pam… * You can learn about upcoming webinars and browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Syslog-ng needs some karma on Fedora
by Peter Czanik (pczanik) 10 Oct '24

10 Oct '24

Hi, Version 4.8.1 of syslog-ng was released last week. It is a bugfix release, and it contains fixes for problems also reported by members of the Fedora community. The Fedora 41 release is near, so package updates now need some additional testing, and “karma” in Bodhi. You can find information on how to install syslog-ng 4.8.1 from a testing repo on Fedora 41 beta at https://bodhi.fedoraproject.org/updates/FEDORA-2024-4e812b8a23 . This is also the place where you can provide feedback and karma. Thanks for your help! Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

syslog-ng 4.8.1 packages
by Peter Czanik (pczanik) 04 Oct '24

04 Oct '24

Hi, Version 4.8.1 of syslog-ng was released yesterday. Here is the current status of packaging: * As usual, Debian and Ubuntu packages were made available as part of the release process: https://www.syslog-ng.com/community/b/blog/posts/installing-the-latest-sysl… * openSUSE / SLES and Fedora / RHEL unofficial repositories are maintained by me, and now updated to 4.8.1: https://www.syslog-ng.com/community/b/blog/posts/installing-latest-syslog-n… you use the syslog-ng48 repo, then you do not have to change anything, just start an update. * my syslog-ng-stable repos for RPM distros will be updated next week, as I always wait a few days. * updates for FreeBSD ports are now submitted, but might take some time. If you urgently want to test 4.8.1, then you can use ports files from my GitHub repo: https://www.syslog-ng.com/community/b/blog/posts/installing-a-syslog-ng-4-d… Not maintained by us, but as far as I can see, Homebrew and Alpine Linux both already have syslog-ng 4.8.1. Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

fatal error is syslog-ng 4.8 from copr when reading pipe sources.
by Evan Rempel 04 Oct '24

04 Oct '24

I upgraded syslog-ng today from 4.6 to 4.8 and my disk drive almost immediately filled due to the local error from syslog-ng 2024-09-20T11:59:38.760-07:00 hostname syslog.err syslog-ng[600690]: Error invoking seek on file; filename='/var/log/syslog.pipes/mailstats2', error='Illegal seek (29)' This logged repeatedly until the disk filled up. My source definition for that pipe is source mailstats2 { pipe("/var/log/syslog.pipes/mailstats2" log_iw_size(900000) log_fetch_limit(500) flags(no-parse) ); }; Any insight would be useful. -- Evan

2 2

Version 4.8.1 of syslog-ng is now available
by Peter Czanik (pczanik) 03 Oct '24

03 Oct '24

Hi, I'm happy to share that version 4.8.1 of syslog-ng is now available. The focus of this release was bug fixing, but there are a few minor new features also available, like MacPorts support and ElasticSearch datastream support. Please help to spread the word, and share / retweet the release announcement on social media: * https://www.linkedin.com/posts/peterczanik_release-syslog-ng-481-syslog-ngs… * https://fosstodon.org/@PCzanik/113243820863895946 * https://x.com/PCzanik/status/1841838695659274333 For a complete list of changes you can read the release notes: Highlights * elasticsearch-datastream() destinations can be used to feed Elasticsearch data streams<https://www.elastic.co/guide/en/elasticsearch/reference/current/data-stream…>. Example config: elasticsearch-datastream( url("https://elastic-endpoint:9200/my-data-stream/_bulk") user("elastic") password("ba3DI8u5qX61We7EP748V8RZ") ); * (#5069<https://github.com/syslog-ng/syslog-ng/pull/5069>) * building: thanks to Sergey Fedorov and Marius Schamschula, macOS builds now support gcc again. They also updated the MacPort version<https://github.com/macports/macports-ports/blob/30c55ba04d8c0693c18cdf84014…> of syslog-ng (develop). Great work, and thank you so much for your contribution! (#5108<https://github.com/syslog-ng/syslog-ng/pull/5108>) Features * tls(): expose the key fingerprint of the peer in ${.tls.x509_fp} if trusted-keys() is used to retain the actual peer identity in the received messages. (#5068<https://github.com/syslog-ng/syslog-ng/pull/5068>) * syslog-parser: Added the no-piggyback-errors and the piggyback-errors flags to control whether the message retains the original message or not on parse error(s). By default the old behaviour/piggyback-errors flag is active. The following new tags can be added by the syslog-parser to the message when the parsing failed: * no-piggyback-errors: On failure, the original message will be left as it was before parsing, the value of $MSGFORMAT will be set to syslog-error, and a tag will be placed on the message corresponding to the parser's failure. * piggyback-errors: On failure, the old behaviour is used (clearing the entire message then syslog-ng will generate a new message in place of the old one describing the parser's error). * syslog.rfc5424_missing_hostname * syslog.rfc5424_missing_app_name * syslog.rfc5424_missing_procid * syslog.rfc5424_missing_msgid * syslog.rfc5424_missing_sdata * syslog.rfc5424_invalid_sdata * syslog.rfc5424_missing_message (#5063<https://github.com/syslog-ng/syslog-ng/pull/5063>) Bugfixes * syslog-ng-ctl: fix escaping of stats prometheus Metric labels (for example, the ones produced by metrics-probe()) may contain control characters, invalid UTF-8 or \ characters. In those specific rare cases, the escaping of the stats prometheus output was incorrect. (#5046<https://github.com/syslog-ng/syslog-ng/pull/5046>) * wildcard-file(): fix crashes can occure if the same wildcard file is used in multiple sources Because of some persistent name construction and validation bugs the following config crashed syslog-ng (if there were more than one log file is in the /path folder) @version: current @include "scl.conf" source s_files1 { file("/path/*.log" persist-name("p1") ); }; source s_files2 { file("/path/*.log" persist-name("p2") ); }; destination s_stdout { stdout(); }; log { source(s_files1); destination(s_stdout); }; log { source(s_files2); destination(s_stdout); }; * NOTE: * The issue occurred regardless of the presence of the persist-name() option. * It affected not only the simplified example of the legacy wildcard file() but also the new wildcard-file() source. (#5091<https://github.com/syslog-ng/syslog-ng/pull/5091>) * syslog-ng-ctl: fix crash of syslog-ng service in g_hash_table lookup function after syslog-ng-ctl reload (#5087<https://github.com/syslog-ng/syslog-ng/pull/5087>) * file(), stdout(): fix log sources getting stuck Due to an acknowledgment bug in the file() and stdout() destinations, sources routed to those destinations may have gotten stuck as they were flow-controlled incorrectly. This issue occured only in extremely rare cases with regular files, but it occured frequently with /dev/stderr and other slow pseudo-devices. (#5134<https://github.com/syslog-ng/syslog-ng/pull/5134>) * directory-monitor: fixed a main thread assertion crash that might have occurred during syslog-ng stop or restart (#5086<https://github.com/syslog-ng/syslog-ng/pull/5086>) * Config @version: fixed compat-mode inconsistencies when @version was not specified at the top of the configuration file or was not specified at all (#5145<https://github.com/syslog-ng/syslog-ng/pull/5145>) * grpc: Fix potential memoryleak when the grpc module is loaded but not used. (#5062<https://github.com/syslog-ng/syslog-ng/pull/5062>) * s3(): Eliminated indefinite memory usage increase for each reload. The increased memory usage is caused by the botocore library, which caches the session information. We only need the Session object, if role() is set. The increased memory usage still happens with that set, currently we only fixed the unset case. (#5149<https://github.com/syslog-ng/syslog-ng/pull/5149>) * opentelemetry() sources: fix crash when workers() is set to > 1 (#5138<https://github.com/syslog-ng/syslog-ng/pull/5138>) * opentelemetry() sources: fix source hang-up on flow-controlled paths (#5148<https://github.com/syslog-ng/syslog-ng/pull/5148>) * metrics-probe(): fix disappearing metrics from stats prometheus output metrics-probe() metrics became orphaned and disappeared from the syslog-ng-ctl stats prometheus output whenever an ivykis worker stopped (after 10 seconds of inactivity). (#5075<https://github.com/syslog-ng/syslog-ng/pull/5075>) * affile: Fix an invalid lseek call mainly on the pipe() source, but also possible if using affile on pipe like files (pipe, socket and FIFO). (#5058<https://github.com/syslog-ng/syslog-ng/pull/5058>) Other changes * format-json: spaces around = in $(format-json) template function could cause a crash<https://github.com/syslog-ng/syslog-ng/issues/5065>. The fix of the issue also introduced an enhancement, from now on spaces are allowed around the = operator, so the following $(format-json) template function calls are all valid: $(format-json foo =alma) $(format-json foo= alma) $(format-json foo = alma) $(format-json foo=\" alma \") $(format-json foo= \" alma \") $(format-json foo1= alma foo2 =korte foo3 = szilva foo4 = \" meggy \" foo5=\"\") * Please note the usage of the escaped strings like \" meggy \", and the (escaped and) quoted form that used for an empty value \"\", the latter is a breaking change as earlier an expression like key= led to a json key-value pair with an empty value {"key":""} that will not work anymore. (#5080<https://github.com/syslog-ng/syslog-ng/pull/5080>) * building: fixed multiple potentional FreeBSD build errors (#5099<https://github.com/syslog-ng/syslog-ng/pull/5099>) * docker: Changed the container image's base to debian:bookworm. (#5056<https://github.com/syslog-ng/syslog-ng/pull/5056>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Attila Szakacs, Balazs Scheidler, Hofi, Kovács Gergő Ferenc, László Várady, Mate Ory, Peter Czanik (CzP), Sergey Fedorov, Marius Schamschula, Szilard Parrag, Tamas Pal, shifter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Suppress UTF-8 BOM in incoming network messages
by Jeremy Utley 25 Sep '24

25 Sep '24

I'm trying to use Syslog-NG to receive logs from our anti-virus web console (ESET Protect). I've got the logs coming in successfully, but they seem to be adding a UTF-8 BOM as part of the message: [2024-09-23T19:03:36.709950] Incoming log entry; input='<15>1 2024-09-23T19:03:36.455Z b35f2351-8f78-4414-b964-415939a7b0fb ERAServer 43 - - <feff>{"event_type" : "Audit_Event","ipv4" : "172.17.0.11","ipv6" : "","hostname" : "b35f2351-8f78-4414-b964-415939a7b0fb","source_uuid" : "b35f2351-8f78-4414-b964-415939a7b0fb","os_name" : "","occured" : "23-Sep-2024 19:03:36","group_name" : "","group_description" : "","severity" : "Information","domain" : "Mapped account","action" : "Logout","target" : "35396c68-5017-4ea9-b268-d209f8040de9","detail" : "Logging out mapped account \'Jeremy Utley\'.","user" : "Jeremy Utley","result" : "Success"}', msg='0x7f6020022150', rcptid='0' Because I only care about the JSON data being provided, I'm setting the "flags(no-parse)" option in my network source, and a rewrite filter to remove the initial data as follows: source s_network { network( port(6514) transport("tls") flags(no-parse) tls( key-file("/etc/syslog-ng/certs/syslog.key") cert-file("/etc/syslog-ng/certs/syslog.crt") peer-verify(optional-untrusted) ) ); }; rewrite r_trimjson { subst("^(.*)- - ", "", value("MESSAGE")); }; destination d_json { file("/var/log/json_data.log" template("${MESSAGE}\n")); }; But I can't seem to eliminate the BOM from the output, and that is messing with JSON parsing, since that is added to the beginning of every line (but the first) in my json_data.log file. I tried adding "\xfe\xff" to the rewrite in an attempt to eliminate it, but that didn't work - the rewrite does not even match at that point. Anyone have any suggestions on how I can deal with this? Thanks! Jeremy Utley

2 1

tls problems when chaning the server certificate syslog-ng-4.6.0
by Evan Rempel 13 Sep '24

13 Sep '24

I have a syslog server running syslog-ng-4.6.0 (from the copr repo). I am not seeing any TLS issues in the logs using the existing ca.d certificate and the current server tls certificate. The current tls certificate will expire soon, and the CA used to sign the server certificate can no longer be used. I have created a new server certificate, signed with a new Root CA. This new Root CA has been successfully added to the ca.d folder and is running without error with the current server certificate. When I replace the server certificate with the new one and restart syslog-ng, I start getting a lot of errors in the logs. SSL error while reading stream; tls_error='error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca' The new server certificate validates $openssl verify tls/server.crt tls/server.crt: OK The CA used to sign the certificate is in the ca.d folder with the correct hash. I have to assume that the error is actually revering to the server certificate, but it could be referring to a client certificate. The error goes away when I switch the server certificate back. The only things that changes for this error is using the new server certificate. How do I track this down? Are there any other suggestions on what might have gone wrong? -- Evan

2 2

rewrite in syslog-ng
by Wilson, Jonathan L 12 Sep '24

12 Sep '24

Running OSE version 3.38.1 and having difficulty with a rewrite rule. The logs that I'm trying to modify look like: 2024-09-12T06:39:31-05:00 hostname kernel: [*09/12/2024 11:39:31.9055] bwar: [7649:I:CN_ML] ... What I am trying to do is remove the extra timestamp in square brackets (the first field in square brackets above.) My rewrite rule looks like: rewrite r_bracketed_ts { subst( '^[.+]\s', '', type(pcre), value("MESSAGE")); }; It is invoked from this log statement: log { source(s_BSD_UDP_514); filter(f_something); rewrite(r_bracketed_ts); destination(d_something); flags(final,flow-control); }; The problem is that the rewrite appears to do nothing; log entries come out unmodified. Am I missing something? Thank you - Jon Wilson

2 1

The syslog-ng Insider 2024-09: documentation; TRANSPORT macro; rolling RPMs
by Peter Czanik (pczanik) 11 Sep '24

11 Sep '24

Dear syslog-ng users, This is the 123rd issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS You can also contribute to the syslog-ng OSE documentation ---------------------------------------------------------- The up-to-date syslog-ng Administration Guide received a new look and easier navigation, as well as better search experience. Best of all, you can now also easily contribute to the syslog-ng documentation. https://www.syslog-ng.com/community/b/blog/posts/you-can-also-contribute-to… The $TRANSPORT macro of syslog-ng --------------------------------- Do you want to know how your log messages arrived to syslog-ng? The new $TRANSPORT macro provides you with part of the answer. It shows you the protocol variant for network sources, or the kind of local source used. https://www.syslog-ng.com/community/b/blog/posts/the-transport-macro-of-sys… Rolling RPM platforms added to the syslog-ng package build system ----------------------------------------------------- Syslog-ng uses a container-based system to generate the official source tgz and to build ready-to-use packages for various Linux distributions. Recently, we added support for some rolling RPM Linux distributions, allowing us to spot breaking changes before a stable release of the OS would become available. https://www.syslog-ng.com/community/b/blog/posts/rolling-rpm-platforms-adde… WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0