From Peter.Czanik at oneidentity.com Wed Sep 11 10:55:36 2024 From: Peter.Czanik at oneidentity.com (Peter Czanik (pczanik)) Date: Wed, 11 Sep 2024 10:55:36 +0000 Subject: [syslog-ng] The syslog-ng Insider 2024-09: documentation; TRANSPORT macro; rolling RPMs Message-ID: Dear syslog-ng users, This is the 123rd issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS You can also contribute to the syslog-ng OSE documentation ---------------------------------------------------------- The up-to-date syslog-ng Administration Guide received a new look and easier navigation, as well as better search experience. Best of all, you can now also easily contribute to the syslog-ng documentation. https://www.syslog-ng.com/community/b/blog/posts/you-can-also-contribute-to-the-syslog-ng-ose-documentation The $TRANSPORT macro of syslog-ng --------------------------------- Do you want to know how your log messages arrived to syslog-ng? The new $TRANSPORT macro provides you with part of the answer. It shows you the protocol variant for network sources, or the kind of local source used. https://www.syslog-ng.com/community/b/blog/posts/the-transport-macro-of-syslog-ng Rolling RPM platforms added to the syslog-ng package build system ----------------------------------------------------- Syslog-ng uses a container-based system to generate the official source tgz and to build ready-to-use packages for various Linux distributions. Recently, we added support for some rolling RPM Linux distributions, allowing us to spot breaking changes before a stable release of the OS would become available. https://www.syslog-ng.com/community/b/blog/posts/rolling-rpm-platforms-added-to-the-syslog-ng-package-build-system WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik From jonathan.wilson at vumc.org Thu Sep 12 11:49:18 2024 From: jonathan.wilson at vumc.org (Wilson, Jonathan L) Date: Thu, 12 Sep 2024 11:49:18 +0000 Subject: [syslog-ng] rewrite in syslog-ng Message-ID: Running OSE version 3.38.1 and having difficulty with a rewrite rule. The logs that I'm trying to modify look like: 2024-09-12T06:39:31-05:00 hostname kernel: [*09/12/2024 11:39:31.9055] bwar: [7649:I:CN_ML] ... What I am trying to do is remove the extra timestamp in square brackets (the first field in square brackets above.) My rewrite rule looks like: rewrite r_bracketed_ts { subst( '^[.+]\s', '', type(pcre), value("MESSAGE")); }; It is invoked from this log statement: log { source(s_BSD_UDP_514); filter(f_something); rewrite(r_bracketed_ts); destination(d_something); flags(final,flow-control); }; The problem is that the rewrite appears to do nothing; log entries come out unmodified. Am I missing something? Thank you - Jon Wilson -------------- next part -------------- An HTML attachment was scrubbed... URL: From erempel at uvic.ca Thu Sep 12 12:58:13 2024 From: erempel at uvic.ca (Evan Rempel) Date: Thu, 12 Sep 2024 12:58:13 +0000 Subject: [syslog-ng] rewrite in syslog-ng In-Reply-To: References: Message-ID: The square brackets are special characters in a pcre expression and need to be escaped. The other tricky thing is that pcre expressions are greedy. By that I mean that this will match the first [ and then the LAST ] so in your example the following would be removed [*09/12/2024 11:39:31.9055] bwar: [7649:I:CN_ML] Or more if there is another ] in the message. The expression you are looking for is subst( '^\[\*\d{2}/\d{2}/\d{4} \d{2}:\d{2}:\d{2}\.\d+\]\s', '', type(pcre), value("MESSAGE")); -- Evan ________________________________ From: syslog-ng on behalf of Wilson, Jonathan L Sent: September 12, 2024 4:49 AM To: syslog-ng at lists.balabit.hu Subject: [syslog-ng] rewrite in syslog-ng You don't often get email from jonathan.wilson at vumc.org. Learn why this is important Running OSE version 3.38.1 and having difficulty with a rewrite rule. The logs that I?m trying to modify look like: 2024-09-12T06:39:31-05:00 hostname kernel: [*09/12/2024 11:39:31.9055] bwar: [7649:I:CN_ML] ? What I am trying to do is remove the extra timestamp in square brackets (the first field in square brackets above.) My rewrite rule looks like: rewrite r_bracketed_ts { subst( '^[.+]\s', '', type(pcre), value("MESSAGE")); }; It is invoked from this log statement: log { source(s_BSD_UDP_514); filter(f_something); rewrite(r_bracketed_ts); destination(d_something); flags(final,flow-control); }; The problem is that the rewrite appears to do nothing; log entries come out unmodified. Am I missing something? Thank you ? Jon Wilson -------------- next part -------------- An HTML attachment was scrubbed... URL: From erempel at uvic.ca Thu Sep 12 16:10:30 2024 From: erempel at uvic.ca (Evan Rempel) Date: Thu, 12 Sep 2024 16:10:30 +0000 Subject: [syslog-ng] tls problems when chaning the server certificate syslog-ng-4.6.0 Message-ID: I have a syslog server running syslog-ng-4.6.0 (from the copr repo). I am not seeing any TLS issues in the logs using the existing ca.d certificate and the current server tls certificate. The current tls certificate will expire soon, and the CA used to sign the server certificate can no longer be used. I have created a new server certificate, signed with a new Root CA. This new Root CA has been successfully added to the ca.d folder and is running without error with the current server certificate. When I replace the server certificate with the new one and restart syslog-ng, I start getting a lot of errors in the logs. SSL error while reading stream; tls_error='error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca' The new server certificate validates $openssl verify tls/server.crt tls/server.crt: OK The CA used to sign the certificate is in the ca.d folder with the correct hash. I have to assume that the error is actually revering to the server certificate, but it could be referring to a client certificate. The error goes away when I switch the server certificate back. The only things that changes for this error is using the new server certificate. How do I track this down? Are there any other suggestions on what might have gone wrong? -- Evan -------------- next part -------------- An HTML attachment was scrubbed... URL: From sandor.geller at ericsson.com Fri Sep 13 06:08:07 2024 From: sandor.geller at ericsson.com (Sandor Geller) Date: Fri, 13 Sep 2024 06:08:07 +0000 Subject: [syslog-ng] tls problems when chaning the server certificate syslog-ng-4.6.0 In-Reply-To: References: Message-ID: Hello, A packet capture would confirm but this TLS error has to come from the wire so the clients are rejecting the server's certificate. Have you distributed the new CAcert to the connecting clients, and is it properly set up, when using a directory then the hash symlink is present there as well? You could simulate your setup with openssl's s_client and s_server. Regards, Sandor On 2024. 09. 12. 18:10, Evan Rempel wrote: I have a syslog server running syslog-ng-4.6.0 (from the copr repo). I am not seeing any TLS issues in the logs using the existing ca.d certificate and the current server tls certificate. The current tls certificate will expire soon, and the CA used to sign the server certificate can no longer be used. I have created a new server certificate, signed with a new Root CA. This new Root CA has been successfully added to the ca.d folder and is running without error with the current server certificate. When I replace the server certificate with the new one and restart syslog-ng, I start getting a lot of errors in the logs. SSL error while reading stream; tls_error='error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca' The new server certificate validates $openssl verify tls/server.crt tls/server.crt: OK The CA used to sign the certificate is in the ca.d folder with the correct hash. I have to assume that the error is actually revering to the server certificate, but it could be referring to a client certificate. The error goes away when I switch the server certificate back. The only things that changes for this error is using the new server certificate. How do I track this down? Are there any other suggestions on what might have gone wrong? -- Evan ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq -------------- next part -------------- An HTML attachment was scrubbed... URL: From erempel at uvic.ca Fri Sep 13 15:54:52 2024 From: erempel at uvic.ca (Evan Rempel) Date: Fri, 13 Sep 2024 15:54:52 +0000 Subject: [syslog-ng] tls problems when chaning the server certificate syslog-ng-4.6.0 In-Reply-To: References: Message-ID: Thanks. That was exactly the issue. The clients were rejecting the server certificate. I was confused as how the client rejection gets logged into the server's logs. I guess there is a lot more information exchanged in a SSL handshake. Signing the new server certificate with the older CA which all of the clients already have resolved the issue. I will have to work on rolling out the new CA to all of the clients prior to resigning the server certificate. -- Evan ________________________________ From: syslog-ng on behalf of Sandor Geller Sent: September 12, 2024 11:08 PM To: syslog-ng at lists.balabit.hu Subject: Re: [syslog-ng] tls problems when chaning the server certificate syslog-ng-4.6.0 Hello, A packet capture would confirm but this TLS error has to come from the wire so the clients are rejecting the server's certificate. Have you distributed the new CAcert to the connecting clients, and is it properly set up, when using a directory then the hash symlink is present there as well? You could simulate your setup with openssl's s_client and s_server. Regards, Sandor On 2024. 09. 12. 18:10, Evan Rempel wrote: I have a syslog server running syslog-ng-4.6.0 (from the copr repo). I am not seeing any TLS issues in the logs using the existing ca.d certificate and the current server tls certificate. The current tls certificate will expire soon, and the CA used to sign the server certificate can no longer be used. I have created a new server certificate, signed with a new Root CA. This new Root CA has been successfully added to the ca.d folder and is running without error with the current server certificate. When I replace the server certificate with the new one and restart syslog-ng, I start getting a lot of errors in the logs. SSL error while reading stream; tls_error='error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca' The new server certificate validates $openssl verify tls/server.crt tls/server.crt: OK The CA used to sign the certificate is in the ca.d folder with the correct hash. I have to assume that the error is actually revering to the server certificate, but it could be referring to a client certificate. The error goes away when I switch the server certificate back. The only things that changes for this error is using the new server certificate. How do I track this down? Are there any other suggestions on what might have gone wrong? -- Evan -------------- next part -------------- An HTML attachment was scrubbed... URL: From erempel at uvic.ca Fri Sep 20 19:08:40 2024 From: erempel at uvic.ca (Evan Rempel) Date: Fri, 20 Sep 2024 19:08:40 +0000 Subject: [syslog-ng] fatal error is syslog-ng 4.8 from copr when reading pipe sources. Message-ID: I upgraded syslog-ng today from 4.6 to 4.8 and my disk drive almost immediately filled due to the local error from syslog-ng 2024-09-20T11:59:38.760-07:00 hostname syslog.err syslog-ng[600690]: Error invoking seek on file; filename='/var/log/syslog.pipes/mailstats2', error='Illegal seek (29)' This logged repeatedly until the disk filled up. My source definition for that pipe is source mailstats2 { pipe("/var/log/syslog.pipes/mailstats2" log_iw_size(900000) log_fetch_limit(500) flags(no-parse) ); }; Any insight would be useful. -- Evan -------------- next part -------------- An HTML attachment was scrubbed... URL: From erempel at uvic.ca Mon Sep 23 13:58:49 2024 From: erempel at uvic.ca (Evan Rempel) Date: Mon, 23 Sep 2024 13:58:49 +0000 Subject: [syslog-ng] new copr package required: fatal error is syslog-ng 4.8 from copr when reading pipe sources. In-Reply-To: References: Message-ID: I should have looked at reported issues. This issue was reported previously at https://github.com/syslog-ng/syslog-ng/issues/5057 And the PR has already been created. I will wait for an updated package from copr. -- Evan ________________________________ From: syslog-ng on behalf of Evan Rempel Sent: September 20, 2024 12:08 PM To: syslog-ng at lists.balabit.hu Subject: [syslog-ng] fatal error is syslog-ng 4.8 from copr when reading pipe sources. I upgraded syslog-ng today from 4.6 to 4.8 and my disk drive almost immediately filled due to the local error from syslog-ng 2024-09-20T11:59:38.760-07:00 hostname syslog.err syslog-ng[600690]: Error invoking seek on file; filename='/var/log/syslog.pipes/mailstats2', error='Illegal seek (29)' This logged repeatedly until the disk filled up. My source definition for that pipe is source mailstats2 { pipe("/var/log/syslog.pipes/mailstats2" log_iw_size(900000) log_fetch_limit(500) flags(no-parse) ); }; Any insight would be useful. -- Evan -------------- next part -------------- An HTML attachment was scrubbed... URL: From jutley at seedbox.com Mon Sep 23 19:17:32 2024 From: jutley at seedbox.com (Jeremy Utley) Date: Mon, 23 Sep 2024 14:17:32 -0500 Subject: [syslog-ng] Suppress UTF-8 BOM in incoming network messages Message-ID: I'm trying to use Syslog-NG to receive logs from our anti-virus web console (ESET Protect). I've got the logs coming in successfully, but they seem to be adding a UTF-8 BOM as part of the message: [2024-09-23T19:03:36.709950] Incoming log entry; input='<15>1 2024-09-23T19:03:36.455Z b35f2351-8f78-4414-b964-415939a7b0fb ERAServer 43 - - {"event_type" : "Audit_Event","ipv4" : "172.17.0.11","ipv6" : "","hostname" : "b35f2351-8f78-4414-b964-415939a7b0fb","source_uuid" : "b35f2351-8f78-4414-b964-415939a7b0fb","os_name" : "","occured" : "23-Sep-2024 19:03:36","group_name" : "","group_description" : "","severity" : "Information","domain" : "Mapped account","action" : "Logout","target" : "35396c68-5017-4ea9-b268-d209f8040de9","detail" : "Logging out mapped account \'Jeremy Utley\'.","user" : "Jeremy Utley","result" : "Success"}', msg='0x7f6020022150', rcptid='0' Because I only care about the JSON data being provided, I'm setting the "flags(no-parse)" option in my network source, and a rewrite filter to remove the initial data as follows: source s_network { network( port(6514) transport("tls") flags(no-parse) tls( key-file("/etc/syslog-ng/certs/syslog.key") cert-file("/etc/syslog-ng/certs/syslog.crt") peer-verify(optional-untrusted) ) ); }; rewrite r_trimjson { subst("^(.*)- - ", "", value("MESSAGE")); }; destination d_json { file("/var/log/json_data.log" template("${MESSAGE}\n")); }; But I can't seem to eliminate the BOM from the output, and that is messing with JSON parsing, since that is added to the beginning of every line (but the first) in my json_data.log file. I tried adding "\xfe\xff" to the rewrite in an attempt to eliminate it, but that didn't work - the rewrite does not even match at that point. Anyone have any suggestions on how I can deal with this? Thanks! Jeremy Utley -------------- next part -------------- An HTML attachment was scrubbed... URL: From bazsi77 at gmail.com Wed Sep 25 13:38:09 2024 From: bazsi77 at gmail.com (Balazs Scheidler) Date: Wed, 25 Sep 2024 15:38:09 +0200 Subject: [syslog-ng] Suppress UTF-8 BOM in incoming network messages In-Reply-To: References: Message-ID: Hi, You could add flags(syslog-protocol) to your network() source driver, so it recognizes the new 5424 style input. With that the BOM will be removed from MSG and you won't need the rewrite either. Balazs On Mon, Sep 23, 2024, 21:17 Jeremy Utley wrote: > I'm trying to use Syslog-NG to receive logs from our anti-virus web > console (ESET Protect). I've got the logs coming in successfully, but they > seem to be adding a UTF-8 BOM as part of the message: > > [2024-09-23T19:03:36.709950] Incoming log entry; input='<15>1 > 2024-09-23T19:03:36.455Z b35f2351-8f78-4414-b964-415939a7b0fb ERAServer 43 > - - {"event_type" : "Audit_Event","ipv4" : "172.17.0.11","ipv6" : > "","hostname" : "b35f2351-8f78-4414-b964-415939a7b0fb","source_uuid" : > "b35f2351-8f78-4414-b964-415939a7b0fb","os_name" : "","occured" : > "23-Sep-2024 19:03:36","group_name" : "","group_description" : > "","severity" : "Information","domain" : "Mapped account","action" : > "Logout","target" : "35396c68-5017-4ea9-b268-d209f8040de9","detail" : > "Logging out mapped account \'Jeremy Utley\'.","user" : "Jeremy > Utley","result" : "Success"}', msg='0x7f6020022150', rcptid='0' > > Because I only care about the JSON data being provided, I'm setting the > "flags(no-parse)" option in my network source, and a rewrite filter to > remove the initial data as follows: > > source s_network { > network( > port(6514) > transport("tls") > flags(no-parse) > tls( > key-file("/etc/syslog-ng/certs/syslog.key") > cert-file("/etc/syslog-ng/certs/syslog.crt") > peer-verify(optional-untrusted) > ) > ); > }; > > rewrite r_trimjson { > subst("^(.*)- - ", "", value("MESSAGE")); > }; > > destination d_json { > file("/var/log/json_data.log" template("${MESSAGE}\n")); > }; > > But I can't seem to eliminate the BOM from the output, and that is messing > with JSON parsing, since that is added to the beginning of every line (but > the first) in my json_data.log file. I tried adding "\xfe\xff" to the > rewrite in an attempt to eliminate it, but that didn't work - the rewrite > does not even match at that point. > > Anyone have any suggestions on how I can deal with this? > > Thanks! > > Jeremy Utley > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > -------------- next part -------------- An HTML attachment was scrubbed... URL: