- syslog-ng - lists.balabit.hu

Creation of manual pages
by Generic Secure-Logging 03 Jan '25

03 Jan '25

syslog-ng features manual pages for the Unix manual system in the doc/ subdirectory. However, these pages are not created when configuring, building and installing syslog-ng with CMake. The autotools config system creates a Makefile with a target manpages when calling configure with the option --enable-manpages. However, the autotools build is broken. The call of make manpages results in the following error ./configure: line 7001: syntax error near unexpected token `newline' ./configure: line 7001: `yes:' make: *** [Makefile:13726: config.status] Error 2 Does anyone know how to create the manual pages in the current version? The information in this e-mail is confidential. The contents may not be disclosed or used by anyone other than the addressee. Access to this e-mail by anyone else is unauthorised. If you are not the intended recipient, please notify Airbus immediately and delete this e-mail. Airbus cannot accept any responsibility for the accuracy or completeness of this e-mail as it has been sent over public networks. If you have any concerns over the content of this message or its Accuracy or Integrity, please contact Airbus immediately. All outgoing e-mails from Airbus are checked using regularly updated virus scanning software but you should take whatever measures you deem to be appropriate to ensure that this message and any attachments are virus free.

1 0

Syslog server chaining issue
by Maurya, Shivani 12 Dec '24

12 Dec '24

Hi All, I am using 2 syslog servers on version 3.31. The devices are sending syslog message to 1st syslog server. The 1st syslog server is forwarding the same message to 2nd syslog server. Device --> Syslog Server 1 --> Syslog Server 2 The configurations of 1st syslog server - Options - keep-hostname(yes); use-dns(yes); use-fqdn(yes); Destination - destination d_sec { udp("IP_of_second_syslog" port(514) template("${ISODATE} ${HOST} ${PRIORITY} ${MSG}\n") template-escape(no)); }; The configuration of 2nd syslog server - destination d_syslogFile { file("/var/log/syslog.log" template("${R_ISODATE} ${HOST} ${PRIORITY} ${FACILITY} ${PROGRAM} ${MSG}\n") template-escape(no)); }; Problem - When the syslog message is getting logged at 2nd Syslog server, the ${PRIORITY} of the message is always "notice". And the original severity/priotity of the message is getting captured in the ${PROGRAM} macro. How to capture the priority of the forwarded message on 2nd Syslog server in the ${PRIORITY} macro instead of ${PROGRAM} macro? Regards, Shivani Maurya

3 3

The syslog-ng Insider 2024-12: FreeBSD audit; 4.8.1; conferences
by Peter Czanik (pczanik) 05 Dec '24

05 Dec '24

Dear syslog-ng users, This is the 126th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS FreeBSD audit source for syslog-ng ---------------------------------- Two weeks ago, I was at EuroBSDcon and received a feature request for syslog-ng. The user wanted to collect FreeBSD audit logs together with other logs using syslog-ng. Writing a native driver in C is time consuming. However, creating an integration based on the program() source of syslog-ng is not that difficult. This blog shows you the current state of the FreeBSD audit source, how it works, and its limitations. It is also a request for feedback. https://www.syslog-ng.com/community/b/blog/posts/freebsd-audit-source-for-s… Version 4.8.1 of syslog-ng is now available ------------------------------------------- Version 4.8.1 was released recently. As you could guess from the version number change, it is primarily a bug fix release, but some minor features also slipped in. From this blog, you can learn what changed in syslog-ng 4.8.1 and where you can get its latest stable version. https://www.syslog-ng.com/community/b/blog/posts/version-4-8-1-of-syslog-ng… Where should I present syslog-ng and sudo? ------------------------------------------ Recently I was asked the same question both at my workplace and at EuroBSDCon, the conference where I was presenting: where do you talk next? I had no definite answer. Of course, I am looking forward to the FOSDEM CfP, but I am also looking for new conferences to present syslog-ng and sudo. Do you have any recommendations? https://www.syslog-ng.com/community/b/blog/posts/where-should-i-present-sys… WEBINARS * New webinar: “Performance tuning for syslog-ng deployments”. You can register for it at https://www.syslog-ng.com/event/performance-tuning-for-syslog-ng-deployment… * You can learn about upcoming webinars and browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

The syslog-ng Insider 2024-11: testing; Quickwit; MacPorts
by Peter Czanik (pczanik) 14 Nov '24

14 Nov '24

Dear syslog-ng users, This is the 125th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS A call for syslog-ng testing ---------------------------- After the last syslog-ng release, we started a campaign to close open issues on GitHub. We'd like to continue this effort and call for collaboration from our users and contributors to make OSE even more stable. While unit tests are great (and we do many tests in-house), nothing can replace using syslog-ng in real-world situations. This blog collects some resources about how you can start testing the latest syslog-ng release from GitHub. https://www.syslog-ng.com/community/b/blog/posts/a-call-for-syslog-ng-testi… Working with Quickwit --------------------- We are always looking for new ways to store log messages. Quickwit is a new contender, designed for log storage, and among others, it also provides an Elasticsearch-compatible API. From this blog, you can learn about Quickwit, and how to forward log messages from syslog-ng to it using the Elasticsearch-compatible API. https://www.syslog-ng.com/community/b/blog/posts/first-steps-with-quickwit-… You can also use the OpenTelemetry protocol to send logs to Quickwit with syslog-ng. https://www.syslog-ng.com/community/b/blog/posts/sending-logs-to-quickwit-u… Huge improvements for syslog-ng in MacPorts ------------------------------------------- Recently I wrote about a campaign that we started to resolve issues on GitHub. Some of the fixes are coming from our enthusiastic community. Thanks to this, there is a new syslog-ng-devel port in MacPorts, where you can enable almost all syslog-ng features even for older MacOS versions and PowerPC hardware. Some of the freshly enabled modules include support for Kafka, GeoIP or OpenTelemetry. From this blog entry, you can learn how to install a legacy or an up-to-date syslog-ng version from MacPorts. https://www.syslog-ng.com/community/b/blog/posts/huge-improvements-for-sysl… WEBINARS * New webcast: “Log management superpowers: a PAM Essentials case study”. You can watch it at https://www.syslog-ng.com/webcast-ondemand/log-management-superpowers-a-pam… * You can learn about upcoming webinars and browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

can kv-parser options be from macros
by Evan Rempel 02 Nov '24

02 Nov '24

Does anyone know if I can use the enriching data to set macros that can then be used as kv-parser options. These values might change on a message by message basis. For example: parser p_kv { kv-parser ( prefix("$kv_prefix") value-separator("$kv_vseparator") pair-separator("$kv_pseparator") ); }; -- Evan

2 2

certificate errors result in excessive logging.
by Evan Rempel 21 Oct '24

21 Oct '24

I am using tls configuration with a locally signed certificate. This mans that I have to configure a custom root CA on to all client systems for them to be able to establish the tls connection to my syslog server. When the clients are unable to verify the server certificate, the server logs three messages for every connection attempt syslog-ng[452597]: SSL error while reading stream; tls_error='error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca', location='/etc/syslog-ng/syslog-ng.server.conf:71:17' syslog-ng[452597]: Error reading RFC6587 style framed data; fd='21', error='Connection reset by peer (104)' syslog-ng[452597]: Syslog connection closed; fd='21', client='AF_INET(1.2.3.4:1234)', local='AF_INET(1.2.3.4:1234)' If there are 100's of clients, and they try to reconnect at a fast rate (every 5 seconds) this can result in a large volume of messages. Is there any way to configure the logging rate of these types of errors or get rid of it altogether. Anyone have any comments on this? -- Evan

2 3

Re: [syslog-ng] syslog-ng Digest, Vol 234, Issue 4
by brian hoffman 15 Oct '24

15 Oct '24

Yes, you were spot on. I realized not long after I sent this that the .0 was missing on the end.Thanks!-Brian Date: Tue, 15 Oct 2024 00:32:37 +0000 From: Evan Rempel <erempel(a)uvic.ca> To: "syslog-ng(a)lists.balabit.hu" <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] netmask filter issue Message-ID: <YT4P288MB03355F0F965988C4C22978C0B9452(a)YT4P288MB0335.CANP288.PROD.OUTLOOK.COM> Content-Type: text/plain; charset="iso-8859-1" First off, to write a CIDR address correctly you need to use something of the form w.x.y.z/M Your specification is missing the ".z" prtion of the ip address. I think there is also some confusion regarding how netmasks work. The ip address 192.168.128.0 (I assume you want a .0 at the end) written in binary is 11000000.10101000.10000000.00000000 The /17 subnet mask written in binary is 11111111.11111111.10000000.00000000 If we perform a binary AND of these two addresses we get 11000000.10101000.10000000.000000 written in decimal is 192.168.128.0 This means that the FIRST address of this range is 192.168.128.0 If we perform the binary NOT of the subnet mask we get 00000000.00000000.01111111.11111111 When we take the binary OR of this with the IP address (w.x.y.z) we get 11000000.10101000.11111111.1111111 in decimal is 192.168.255.255 which is the last address if the range. The result is that 192.168.1.0/17 means the range 192.168.128.0 -- 192.168.255.255 I hope that helps. -- Evan

1 0

netmask filter issue
by brian hoffman 15 Oct '24

15 Oct '24

I set up a filter to send certain messages matching an IP range to a destination (a simplified version of my actual config).filter f_siem { netmask(192.168.128/17) or}; For some reason this is picking up the address 192.168.63.41, even though it's not in the range. 192.168.63.41 (10101100.00010111.0) does not match the network portion of 192.168.128.0/17 (10101100.00010111.1) but for some reason it's still getting picked up. I know for sure that it is this particular line because when I comment it out the messages are no longer forwarded for that device. Is there something different about the way this netmask directive works in syslog-ng that would cause it to get matched by this filter?-Brian

2 1

The syslog-ng Insider 2024-10: 4.8.0 release; version number; Debian Stable
by Peter Czanik (pczanik) 10 Oct '24

10 Oct '24

Dear syslog-ng users, This is the 124th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Version 4.8.0 of syslog-ng improves FreeBSD and MacOS support ------------------------------------------------------------- Recently One Identity released version 4.8.0 of its open-source log management application. Learn about some of the new features and bug fixes: why upgrade to the latest syslog-ng version, not only on FreeBSD :-) https://www.syslog-ng.com/community/b/blog/posts/version-4-8-0-of-syslog-ng… Why it is useful to set the version number in the syslog-ng configuration --------------------------------- The syslog-ng configuration starts with a version number declaration. Up until recently, if it was missing, syslog-ng did not start. With syslog-ng 4.8, this is changing. From this blog, you can learn why version information is useful, what workaround you can use if you do not want to edit your syslog-ng configuration on each update, and what changed in version 4.8. https://www.syslog-ng.com/community/b/blog/posts/why-it-is-useful-to-set-th… We are switching syslog-ng containers from Debian Testing to Stable ----------------------------------------------------- For many years, the official syslog-ng container and development containers were based on Debian Testing. We are switching to Debian Stable now. Learn about the history and the reasons for the change now. https://www.syslog-ng.com/community/b/blog/posts/we-are-switching-syslog-ng… WEBINARS * New webcast: “Log management superpowers: a PAM Essentials case study”. You can watch it at https://www.syslog-ng.com/webcast-ondemand/log-management-superpowers-a-pam… * You can learn about upcoming webinars and browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Syslog-ng needs some karma on Fedora
by Peter Czanik (pczanik) 10 Oct '24

10 Oct '24

Hi, Version 4.8.1 of syslog-ng was released last week. It is a bugfix release, and it contains fixes for problems also reported by members of the Fedora community. The Fedora 41 release is near, so package updates now need some additional testing, and “karma” in Bodhi. You can find information on how to install syslog-ng 4.8.1 from a testing repo on Fedora 41 beta at https://bodhi.fedoraproject.org/updates/FEDORA-2024-4e812b8a23 . This is also the place where you can provide feedback and karma. Thanks for your help! Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

syslog-ng 4.8.1 packages
by Peter Czanik (pczanik) 04 Oct '24

04 Oct '24

Hi, Version 4.8.1 of syslog-ng was released yesterday. Here is the current status of packaging: * As usual, Debian and Ubuntu packages were made available as part of the release process: https://www.syslog-ng.com/community/b/blog/posts/installing-the-latest-sysl… * openSUSE / SLES and Fedora / RHEL unofficial repositories are maintained by me, and now updated to 4.8.1: https://www.syslog-ng.com/community/b/blog/posts/installing-latest-syslog-n… you use the syslog-ng48 repo, then you do not have to change anything, just start an update. * my syslog-ng-stable repos for RPM distros will be updated next week, as I always wait a few days. * updates for FreeBSD ports are now submitted, but might take some time. If you urgently want to test 4.8.1, then you can use ports files from my GitHub repo: https://www.syslog-ng.com/community/b/blog/posts/installing-a-syslog-ng-4-d… Not maintained by us, but as far as I can see, Homebrew and Alpine Linux both already have syslog-ng 4.8.1. Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

fatal error is syslog-ng 4.8 from copr when reading pipe sources.
by Evan Rempel 04 Oct '24

04 Oct '24

I upgraded syslog-ng today from 4.6 to 4.8 and my disk drive almost immediately filled due to the local error from syslog-ng 2024-09-20T11:59:38.760-07:00 hostname syslog.err syslog-ng[600690]: Error invoking seek on file; filename='/var/log/syslog.pipes/mailstats2', error='Illegal seek (29)' This logged repeatedly until the disk filled up. My source definition for that pipe is source mailstats2 { pipe("/var/log/syslog.pipes/mailstats2" log_iw_size(900000) log_fetch_limit(500) flags(no-parse) ); }; Any insight would be useful. -- Evan

2 2

Version 4.8.1 of syslog-ng is now available
by Peter Czanik (pczanik) 03 Oct '24

03 Oct '24

Hi, I'm happy to share that version 4.8.1 of syslog-ng is now available. The focus of this release was bug fixing, but there are a few minor new features also available, like MacPorts support and ElasticSearch datastream support. Please help to spread the word, and share / retweet the release announcement on social media: * https://www.linkedin.com/posts/peterczanik_release-syslog-ng-481-syslog-ngs… * https://fosstodon.org/@PCzanik/113243820863895946 * https://x.com/PCzanik/status/1841838695659274333 For a complete list of changes you can read the release notes: Highlights * elasticsearch-datastream() destinations can be used to feed Elasticsearch data streams<https://www.elastic.co/guide/en/elasticsearch/reference/current/data-stream…>. Example config: elasticsearch-datastream( url("https://elastic-endpoint:9200/my-data-stream/_bulk") user("elastic") password("ba3DI8u5qX61We7EP748V8RZ") ); * (#5069<https://github.com/syslog-ng/syslog-ng/pull/5069>) * building: thanks to Sergey Fedorov and Marius Schamschula, macOS builds now support gcc again. They also updated the MacPort version<https://github.com/macports/macports-ports/blob/30c55ba04d8c0693c18cdf84014…> of syslog-ng (develop). Great work, and thank you so much for your contribution! (#5108<https://github.com/syslog-ng/syslog-ng/pull/5108>) Features * tls(): expose the key fingerprint of the peer in ${.tls.x509_fp} if trusted-keys() is used to retain the actual peer identity in the received messages. (#5068<https://github.com/syslog-ng/syslog-ng/pull/5068>) * syslog-parser: Added the no-piggyback-errors and the piggyback-errors flags to control whether the message retains the original message or not on parse error(s). By default the old behaviour/piggyback-errors flag is active. The following new tags can be added by the syslog-parser to the message when the parsing failed: * no-piggyback-errors: On failure, the original message will be left as it was before parsing, the value of $MSGFORMAT will be set to syslog-error, and a tag will be placed on the message corresponding to the parser's failure. * piggyback-errors: On failure, the old behaviour is used (clearing the entire message then syslog-ng will generate a new message in place of the old one describing the parser's error). * syslog.rfc5424_missing_hostname * syslog.rfc5424_missing_app_name * syslog.rfc5424_missing_procid * syslog.rfc5424_missing_msgid * syslog.rfc5424_missing_sdata * syslog.rfc5424_invalid_sdata * syslog.rfc5424_missing_message (#5063<https://github.com/syslog-ng/syslog-ng/pull/5063>) Bugfixes * syslog-ng-ctl: fix escaping of stats prometheus Metric labels (for example, the ones produced by metrics-probe()) may contain control characters, invalid UTF-8 or \ characters. In those specific rare cases, the escaping of the stats prometheus output was incorrect. (#5046<https://github.com/syslog-ng/syslog-ng/pull/5046>) * wildcard-file(): fix crashes can occure if the same wildcard file is used in multiple sources Because of some persistent name construction and validation bugs the following config crashed syslog-ng (if there were more than one log file is in the /path folder) @version: current @include "scl.conf" source s_files1 { file("/path/*.log" persist-name("p1") ); }; source s_files2 { file("/path/*.log" persist-name("p2") ); }; destination s_stdout { stdout(); }; log { source(s_files1); destination(s_stdout); }; log { source(s_files2); destination(s_stdout); }; * NOTE: * The issue occurred regardless of the presence of the persist-name() option. * It affected not only the simplified example of the legacy wildcard file() but also the new wildcard-file() source. (#5091<https://github.com/syslog-ng/syslog-ng/pull/5091>) * syslog-ng-ctl: fix crash of syslog-ng service in g_hash_table lookup function after syslog-ng-ctl reload (#5087<https://github.com/syslog-ng/syslog-ng/pull/5087>) * file(), stdout(): fix log sources getting stuck Due to an acknowledgment bug in the file() and stdout() destinations, sources routed to those destinations may have gotten stuck as they were flow-controlled incorrectly. This issue occured only in extremely rare cases with regular files, but it occured frequently with /dev/stderr and other slow pseudo-devices. (#5134<https://github.com/syslog-ng/syslog-ng/pull/5134>) * directory-monitor: fixed a main thread assertion crash that might have occurred during syslog-ng stop or restart (#5086<https://github.com/syslog-ng/syslog-ng/pull/5086>) * Config @version: fixed compat-mode inconsistencies when @version was not specified at the top of the configuration file or was not specified at all (#5145<https://github.com/syslog-ng/syslog-ng/pull/5145>) * grpc: Fix potential memoryleak when the grpc module is loaded but not used. (#5062<https://github.com/syslog-ng/syslog-ng/pull/5062>) * s3(): Eliminated indefinite memory usage increase for each reload. The increased memory usage is caused by the botocore library, which caches the session information. We only need the Session object, if role() is set. The increased memory usage still happens with that set, currently we only fixed the unset case. (#5149<https://github.com/syslog-ng/syslog-ng/pull/5149>) * opentelemetry() sources: fix crash when workers() is set to > 1 (#5138<https://github.com/syslog-ng/syslog-ng/pull/5138>) * opentelemetry() sources: fix source hang-up on flow-controlled paths (#5148<https://github.com/syslog-ng/syslog-ng/pull/5148>) * metrics-probe(): fix disappearing metrics from stats prometheus output metrics-probe() metrics became orphaned and disappeared from the syslog-ng-ctl stats prometheus output whenever an ivykis worker stopped (after 10 seconds of inactivity). (#5075<https://github.com/syslog-ng/syslog-ng/pull/5075>) * affile: Fix an invalid lseek call mainly on the pipe() source, but also possible if using affile on pipe like files (pipe, socket and FIFO). (#5058<https://github.com/syslog-ng/syslog-ng/pull/5058>) Other changes * format-json: spaces around = in $(format-json) template function could cause a crash<https://github.com/syslog-ng/syslog-ng/issues/5065>. The fix of the issue also introduced an enhancement, from now on spaces are allowed around the = operator, so the following $(format-json) template function calls are all valid: $(format-json foo =alma) $(format-json foo= alma) $(format-json foo = alma) $(format-json foo=\" alma \") $(format-json foo= \" alma \") $(format-json foo1= alma foo2 =korte foo3 = szilva foo4 = \" meggy \" foo5=\"\") * Please note the usage of the escaped strings like \" meggy \", and the (escaped and) quoted form that used for an empty value \"\", the latter is a breaking change as earlier an expression like key= led to a json key-value pair with an empty value {"key":""} that will not work anymore. (#5080<https://github.com/syslog-ng/syslog-ng/pull/5080>) * building: fixed multiple potentional FreeBSD build errors (#5099<https://github.com/syslog-ng/syslog-ng/pull/5099>) * docker: Changed the container image's base to debian:bookworm. (#5056<https://github.com/syslog-ng/syslog-ng/pull/5056>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Attila Szakacs, Balazs Scheidler, Hofi, Kovács Gergő Ferenc, László Várady, Mate Ory, Peter Czanik (CzP), Sergey Fedorov, Marius Schamschula, Szilard Parrag, Tamas Pal, shifter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Suppress UTF-8 BOM in incoming network messages
by Jeremy Utley 25 Sep '24

25 Sep '24

I'm trying to use Syslog-NG to receive logs from our anti-virus web console (ESET Protect). I've got the logs coming in successfully, but they seem to be adding a UTF-8 BOM as part of the message: [2024-09-23T19:03:36.709950] Incoming log entry; input='<15>1 2024-09-23T19:03:36.455Z b35f2351-8f78-4414-b964-415939a7b0fb ERAServer 43 - - <feff>{"event_type" : "Audit_Event","ipv4" : "172.17.0.11","ipv6" : "","hostname" : "b35f2351-8f78-4414-b964-415939a7b0fb","source_uuid" : "b35f2351-8f78-4414-b964-415939a7b0fb","os_name" : "","occured" : "23-Sep-2024 19:03:36","group_name" : "","group_description" : "","severity" : "Information","domain" : "Mapped account","action" : "Logout","target" : "35396c68-5017-4ea9-b268-d209f8040de9","detail" : "Logging out mapped account \'Jeremy Utley\'.","user" : "Jeremy Utley","result" : "Success"}', msg='0x7f6020022150', rcptid='0' Because I only care about the JSON data being provided, I'm setting the "flags(no-parse)" option in my network source, and a rewrite filter to remove the initial data as follows: source s_network { network( port(6514) transport("tls") flags(no-parse) tls( key-file("/etc/syslog-ng/certs/syslog.key") cert-file("/etc/syslog-ng/certs/syslog.crt") peer-verify(optional-untrusted) ) ); }; rewrite r_trimjson { subst("^(.*)- - ", "", value("MESSAGE")); }; destination d_json { file("/var/log/json_data.log" template("${MESSAGE}\n")); }; But I can't seem to eliminate the BOM from the output, and that is messing with JSON parsing, since that is added to the beginning of every line (but the first) in my json_data.log file. I tried adding "\xfe\xff" to the rewrite in an attempt to eliminate it, but that didn't work - the rewrite does not even match at that point. Anyone have any suggestions on how I can deal with this? Thanks! Jeremy Utley

2 1

tls problems when chaning the server certificate syslog-ng-4.6.0
by Evan Rempel 13 Sep '24

13 Sep '24

I have a syslog server running syslog-ng-4.6.0 (from the copr repo). I am not seeing any TLS issues in the logs using the existing ca.d certificate and the current server tls certificate. The current tls certificate will expire soon, and the CA used to sign the server certificate can no longer be used. I have created a new server certificate, signed with a new Root CA. This new Root CA has been successfully added to the ca.d folder and is running without error with the current server certificate. When I replace the server certificate with the new one and restart syslog-ng, I start getting a lot of errors in the logs. SSL error while reading stream; tls_error='error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca' The new server certificate validates $openssl verify tls/server.crt tls/server.crt: OK The CA used to sign the certificate is in the ca.d folder with the correct hash. I have to assume that the error is actually revering to the server certificate, but it could be referring to a client certificate. The error goes away when I switch the server certificate back. The only things that changes for this error is using the new server certificate. How do I track this down? Are there any other suggestions on what might have gone wrong? -- Evan

2 2

rewrite in syslog-ng
by Wilson, Jonathan L 12 Sep '24

12 Sep '24

Running OSE version 3.38.1 and having difficulty with a rewrite rule. The logs that I'm trying to modify look like: 2024-09-12T06:39:31-05:00 hostname kernel: [*09/12/2024 11:39:31.9055] bwar: [7649:I:CN_ML] ... What I am trying to do is remove the extra timestamp in square brackets (the first field in square brackets above.) My rewrite rule looks like: rewrite r_bracketed_ts { subst( '^[.+]\s', '', type(pcre), value("MESSAGE")); }; It is invoked from this log statement: log { source(s_BSD_UDP_514); filter(f_something); rewrite(r_bracketed_ts); destination(d_something); flags(final,flow-control); }; The problem is that the rewrite appears to do nothing; log entries come out unmodified. Am I missing something? Thank you - Jon Wilson

2 1

The syslog-ng Insider 2024-09: documentation; TRANSPORT macro; rolling RPMs
by Peter Czanik (pczanik) 11 Sep '24

11 Sep '24

Dear syslog-ng users, This is the 123rd issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS You can also contribute to the syslog-ng OSE documentation ---------------------------------------------------------- The up-to-date syslog-ng Administration Guide received a new look and easier navigation, as well as better search experience. Best of all, you can now also easily contribute to the syslog-ng documentation. https://www.syslog-ng.com/community/b/blog/posts/you-can-also-contribute-to… The $TRANSPORT macro of syslog-ng --------------------------------- Do you want to know how your log messages arrived to syslog-ng? The new $TRANSPORT macro provides you with part of the answer. It shows you the protocol variant for network sources, or the kind of local source used. https://www.syslog-ng.com/community/b/blog/posts/the-transport-macro-of-sys… Rolling RPM platforms added to the syslog-ng package build system ----------------------------------------------------- Syslog-ng uses a container-based system to generate the official source tgz and to build ready-to-use packages for various Linux distributions. Recently, we added support for some rolling RPM Linux distributions, allowing us to spot breaking changes before a stable release of the OS would become available. https://www.syslog-ng.com/community/b/blog/posts/rolling-rpm-platforms-adde… WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Send GELF log messages to Syslog-NG server
by Roberto Carna 21 Aug '24

21 Aug '24

Hi people, a Graylog server is sending GELF log messages to my Syslog-NG server listening on the UDP/514 port. But when I was reading the message, I noticed it's in binary format. Maybe syslog-ng service converts GELF to binary, because the log is not in syslog format. Is it possible to make Syslog-NG listen on the UDP/12201 port for incoming GELF messages, in addition to UDP/514 syslog port ? Or what can I do to accept GELF messages in the Syslog-NG server and then read them in ASCII format ? Thanks a lot!

3 2

The syslog-ng Insider 2024-08: 4.8.0 release; Prometheus; Amazon Linux
by Peter Czanik (pczanik) 15 Aug '24

15 Aug '24

Dear syslog-ng users, This is the 122nd issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Version 4.8.0 of syslog-ng improves FreeBSD and MacOS support ------------------------------------------------------------- Recently One Identity released version 4.8.0 of its open-source log management application. Learn about some of the new features and bug fixes: why upgrade to the latest syslog-ng version, not only on FreeBSD :-) https://www.syslog-ng.com/community/b/blog/posts/version-4-8-0-of-syslog-ng… syslog-ng Prometheus exporter ----------------------------- Prometheus is an open-source monitoring system that collects metrics from your hosts and applications, allowing you to visualize and alert on them. The syslog-ng Prometheus exporter allows you to export syslog-ng statistics, so that Prometheus can collect it. While an implementation in Go has been available for years on GitHub (for more information, see this blog entry), that solution uses the old syslog-ng statistics interface. And while that Go-based implementation still works, syslog-ng 4.1 introduced a new interface that provides not just more information than the previous statistics interface, but does so in a Prometheus-friendly format. The information available through the new interface has been growing ever since. The syslog-ng Prometheus exporter is implemented in Python. It also uses the new statistics interface, making new fields automatically available when added. https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-prometheus-expor… Experimental syslog-ng packages for Amazon Linux 2023 ----------------------------------------------------- Last year, I received many requests about syslog-ng for Amazon Linux 2023, but I could not find an easy way to create syslog-ng packages. Recently, however, I found that Fedora Copr supports building packages for Amazon Linux 2023. So, with a little bit of experimentation, I got a cut down version of syslog-ng compiled. https://www.syslog-ng.com/community/b/blog/posts/experimental-syslog-ng-pac… WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Version 4.8.0 of syslog-ng is now available
by Peter Czanik (pczanik) 24 Jul '24

24 Jul '24

We have new documentation You can find our new up-to-date documentation in the new Administration Guide at syslog-ng.github.io<https://syslog-ng.github.io/admin-guide/README>. Highlights Default config version in configuration files cfg: allow usage of current in config @version by default if it is not presented This change allows syslog-ng to start even if the @version information is not present in the configuration file and treats the version as the latest in that case. NOTE: syslog-ng will still raise a warning if @version is not present. Please use @version: current to confirm the intention of always using the latest version and silence the warning. (#5030<https://github.com/syslog-ng/syslog-ng/pull/5030>) BSD directory monitoring with kqueue directory-monitor: Added a kqueue based directory monitor implementation. wildcard-file() sources are using a directory monitor<https://syslog-ng.github.io/admin-guide/060_Sources/020_File/001_File_follo…> as well to aid detection of changes in the folders of the followed files. The new kqueue-based directory monitor uses far fewer resources than the poll based version on BSD-based systems. (#5022<https://github.com/syslog-ng/syslog-ng/pull/5022>) See more at the new syslog-ng documentation<https://syslog-ng.github.io/admin-guide/060_Sources/020_File/001_File_follo…>. Wildcard file source fine-tuning wildcard-file(): Added a dedicated monitor_freq option to control the poll frequency of the change detection in the directories separately when the poll method is selected via the monitor-method() option. The monitor-method() option controls only the change detection method in the directories, not the following of the file changes, and if poll is the selected method the frequency must not necessarily be the same, e.g. if the (earlier) commonly used follow-freq() is set to 0 for switching to the poll_fd_events method for file content change detection, that also might be meant a directory change poll with zero delays (if monitor-method() was set to poll as well), and that could cause a heavy CPU load unnecessarily. (#4998<https://github.com/syslog-ng/syslog-ng/pull/4998>) See more at the new syslog-ng documentation<https://syslog-ng.github.io/admin-guide/060_Sources/020_File/001_File_follo…>. Features * s3(): Introduced server side encryption related options server-side-encryption() and kms-key() can be used to configure encryption. Currently only server-side-encryption("aws:kms") is supported. The kms-key() should be: To be able to use the aws:kms encryption the AWS Role or User has to have the following permissions on the given key: Check this<https://repost.aws/knowledge-center/s3-large-file-encryption-kms-key> page on why the kms:Decrypt is mandatory. Example config: destination d_s3 { s3( bucket("log-archive-bucket") object-key("logs/syslog") server-side-encryption("aws:kms") kms-key("alias/log-archive") ); }; See the S3<https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingKMSEncryption.ht…> documentation for more details. (#4993<https://github.com/syslog-ng/syslog-ng/pull/4993>) * an ID of a key * an alias of a key, but in that case you have to add the alias/prefix * an ARN of a key * kms:Decrypt * kms:Encrypt * kms:GenerateDataKey * filter: Added numerical severity settings. The level filter option now accepts numerical values similar to facility. Example config: filter f_severity { level(4) }; This is equivalent to filter f_severity { level("warning") }; For more information, consult the documentation<https://syslog-ng.github.io/admin-guide/080_Log/030_Filters/005_Filter_func…>. (#5016<https://github.com/syslog-ng/syslog-ng/pull/5016>) * opentelemetry(), loki(), bigquery(): Added headers() option Enables adding headers to RPC calls. Example config: opentelemetry( ... headers( "my_header" = "my_value" ) ); (#5012<https://github.com/syslog-ng/syslog-ng/pull/5012>) * Added new proxy options to the syslog() and network() source drivers The transport(proxied-tcp), transport(proxied-tls), and transport(proxied-tls-passthrough) options are now available when configuring syslog() and network() sources. (#4544<https://github.com/syslog-ng/syslog-ng/pull/4544>) Bugfixes * disk-buffer(): fix crash when pipeline initialization fails log_queue_disk_free_method: assertion failed: (!qdisk_started(self->qdisk)) (#4994<https://github.com/syslog-ng/syslog-ng/pull/4994>) * rate-limit(): Fixed a crash which occured on a config parse failure. (#5033<https://github.com/syslog-ng/syslog-ng/pull/5033>) * Fixed potential null pointer deref issues (#5035<https://github.com/syslog-ng/syslog-ng/pull/5035>) * wildcard-file(): fix a crash and detection of file delete/move when using ivykis poll events Two issues were fixed * Fixed a crash in log pipe queue during file deletion and EOF detection (#4989<https://github.com/syslog-ng/syslog-ng/issues/4989>) The crash was caused by a concurrency issue in the EOF and file deletion detection when using a wildcard-file() source. If a file is written after being deleted (e.g. with an application keeping the file open), or if these events happen concurrently, the file state change poller mechanism might schedule another read cycle even though the file has already been marked as fully read and deleted. To prevent this re-scheduling between these two checks, the following changes have been made: Instead of maintaining an internal EOF state in the WildcardFileReader, when a file deletion notification is received, the poller will be signaled to stop after reaching the next EOF. Only after both conditions are set the reader instance will be deleted. * Fixed the file deletion and removal detection when the file-reader uses poll_fd_events to follow file changes, which were mishandled. For example, files that were moved or deleted (such as those rolled by a log-rotator) were read to the end but never read again if they were not touched anymore, therefore switching to the new file never happened. (#4998<https://github.com/syslog-ng/syslog-ng/pull/4998>) * syslog-ng-ctl query: fix showing Prometheus metrics as unnamed values none.value=726685 (#4995<https://github.com/syslog-ng/syslog-ng/pull/4995>) * macros: Fixed a bug which always set certain macros to string type The affected macros are $PROGRAM, $HOST and $MESSAGE. (#5024<https://github.com/syslog-ng/syslog-ng/pull/5024>) * syslog-ng-ctl query: show timestamps and fix g_pattern_spec_match_string assert (#4995<https://github.com/syslog-ng/syslog-ng/pull/4995>) * csv-parser(): fix escape-backslash-with-sequences dialect on ARM csv-parser() produced invalid output on platforms where char is an unsigned type. (#4947<https://github.com/syslog-ng/syslog-ng/pull/4947>) Other changes * bigquery(), loki(), opentelemetry(), cloud-auth(): C++ modules can be compiled with clang Compiling and using these C++ modules are now easier on FreeBSD and macOS. (#4933<https://github.com/syslog-ng/syslog-ng/pull/4933>) * syslog-ng-ctl: do not show orphan metrics for stats prometheus As the stats prometheus command is intended to be used to forward metrics to Prometheus or any other time-series database, displaying orphaned metrics should be avoided in order not to insert new data points when a given metric is no longer alive. In case you are interested in the last known value of orphaned counters, use the stats or query subcommands. (#4921<https://github.com/syslog-ng/syslog-ng/pull/4921>) * s3(): new metric syslogng_output_event_bytes_total (#4958<https://github.com/syslog-ng/syslog-ng/pull/4958>) * multiline-options: Allow multi_line_timeout to be set to a non-integer value. Since multi_line_timeout is suggested to be set as a multiple of follow-freq, and follow-freq can be much smaller than one second, it makes sense to allow this value to be a non-integer as well. (#5002<https://github.com/syslog-ng/syslog-ng/pull/5002>) * packages/dbld: add support for Ubuntu 24.04 (Noble Numbat) (#4925<https://github.com/syslog-ng/syslog-ng/pull/4925>) * packages/dbld: add support for AlmaLinux 9 (#5009<https://github.com/syslog-ng/syslog-ng/pull/5009>) * packages/dbld: added support for Fedora Rawhide and CentOS Stream 9 as testing platforms (#5009<https://github.com/syslog-ng/syslog-ng/pull/5009>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Alex Becker, Andras Mitzki, Arpad Kunszt, Attila Szakacs, Balazs Scheidler, Bálint Horváth, Dmitry Levin, Hofi, Ilya Kheifets, joohoonmaeng, ktzsolt, László Várady, Mate Ory, Natanael Copa, Peter Czanik, qsunchiu, Robert Fekete, shifter, Szilárd Parrag, Tamas Pal, Wolfram Joost Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

The syslog-ng Insider 2024-07: logging trends; multiple platforms; sudo; retired platforms
by Peter Czanik (pczanik) 11 Jul '24

11 Jul '24

Dear syslog-ng users, This is the 121st issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Trends in open source logging ----------------------------- Recently I was asked by Chas Clawson of Sumo Logic what I think about open source log management and telemetry. Needless to say, I also talked about syslog_ng, and in the end I showed a couple of simple syslog-ng configurations sending logs to Sumo Logic. https://www.sumologic.com/blog/securing-open-source-infrastructure-log-all/ Using syslog-ng on multiple platforms ------------------------------------- Your favorite Linux distribution is X. You test everything there. However, your colleagues use distro Y, and another team distro Z. Nightmares start here: the same commands install a different set of syslog-ng features, configuration defaults and use different object names in the default configuration. I ran into these problems while working with Gábor Samu on his HPC logging blog. >From this blog you can learn about some of the main differences in packaging and configuration of syslog-ng in various Linux distributions and FreeBSD, and how to recognize these when configuring syslog-ng on a different platform. https://www.syslog-ng.com/community/b/blog/posts/using-syslog-ng-on-multipl… Working with sudo’s json_compact logs in syslog-ng -------------------------------------------------- Version 1.9.16 of sudo will feature a new option for logging: json_compact. Why is this important? This new format can easily be read and parsed by a log management software, like syslog-ng. Note that in this blog I am showing you a sudo feature which has not yet been released officially. You have to compile sudo yourself. By all means, if you have any other application writing JSON-formatted log messages, you can apply most of what you read here with slight modifications. https://www.syslog-ng.com/community/b/blog/posts/working-with-sudo-s-json_5… syslog-ng OSE 4.7.1 retires some old platforms ---------------------------------------------- Version 4.7.1 of syslog-ng OSE was released recently. It brings many smaller enhancements to metrics, OpenTelemetry and other features, while it also retires support for some older platforms. https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-4-7-1-retires-so… WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Help parsing logs
by John Norton 28 Jun '24

28 Jun '24

Hi All, I have a panorama system that sends logs from several devices (dvc_name). What I want to do is have it write to files based on the dvc_name. The dvc_name is in the following message as "ip-10-37-12-142". These are not static as they are spun up as needed and the dvc_name may change. Can you help me figure out how to create a parser or filter to take the following message, where it will read in the dvc_name "ip-10-37-12-142" and write logs that include that dvc_name to a separate file (and other dvc_names it sees in messages to their own file?: *Jun 27 19:08:15 TERL-PANORAMA-01.times.com <http://terl-panorama-01.times.com/> 1,2024/06/27 15:08:14,005655000463255,SYSTEM,dhcp,2817,2024/06/27 15:08:09,,if-renew-trigger,,0,0,general,informational,"DHCP RENEW: interface eth0, ip 10.37.12.142 netmask 255.255.255.224 dhcp server: 10.37.12.129",7383625404611793457,0x8000000000000000,0,0,0,0,,ip-10-37-12-142,0,0,2024-06-27T15:08:09.224-04:00* I've tried a few different things but can't figure out how to do it dynamically. This rewrite worked, but it requires a static entry which won't scale and all potential dvc_names cannot be known: rewrite r_panorama { set("ip-10-37-12-142", value("HOST") condition(message(".*ip-10-37-12-142"))); }; I've also tried creating a filter, but it doesn't actually store what the match finds in a macro that I can replace something like $HOST with. filter f_panorama { match("ip-\d+-\d+-\d+-\d+" value("MESSAGE")); }; I was also thinking that I could do something with the csv parser, but it didn't work like I was expecting, but am thinking a parser might work if I could get it right (I think partially it doesn't work because it is a syslog message and the first part of the message interferes with the csv parsing, not sure): parser p_panorama { csv-parser( columns("future_use1","receive_time","serial_number","log_type","log_subtype","version","generated_time","vsys","event_id","object","future_use3","future_use4","module","severity","description","sequence_number","action_flags","devicegroup_level1","devicegroup_level2","devicegroup_level3","devicegroup_level4","vsys_name","dvc_name") delimiters(",") quote-pairs('""[]') ); }; destination d_panorama {file("`mypath`/panorama/$dvc_name/$dvc_name.log"); }; log { source(s_panother); parser(p_panorama); destination(d_panorama); }; Is this something someone could help me with? Thank you, John

2 1

The syslog-ng Insider 2024-05: documentation; grouping-by(); PAM Essentials; health
by Peter Czanik (pczanik) 09 May '24

09 May '24

Dear syslog-ng users, This is the 120th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS The official syslog-ng OSE documentation got a new look ------------------------------------------------------- The syslog-ng Administration Guide received a new look and easier navigation. Not only that, but it is also up-to-date now. Besides, there are now contributor guides available both for the documentation and for syslog-ng developers. The admin guide is available at: https://syslog-ng.github.io/admin-guide/README You can reach all syslog-ng OSE-related documentation at: https://syslog-ng.github.io/ If you find any issues, pull requests and problem reports are welcome. The contributor guide describes how you can fix / extend the documentation. You can report issues at: https://github.com/syslog-ng/syslog-ng.github.io/issues Aggregating messages in syslog-ng using grouping-by() ----------------------------------------------------- Sometimes you have many log messages from an app, but none of them have the exact content you need. This is where the grouping-by() parser of syslog-ng can help. It allows you to aggregate information from multiple log messages into a single message. In this blog, I will show you how to parse sshd logs using the patterndb parser of syslog-ng, and then create an aggregate message from the opening and closing log message using grouping-by. https://www.syslog-ng.com/community/b/blog/posts/aggregating-messages-in-sy… Alerting on One Identity Cloud PAM Essentials logs using syslog-ng -------------------------------------------------- One Identity Cloud PAM Essentials is the latest security product by One Identity. It provides asset management as well as secure and monitored remote access for One Identity Cloud users to hosts on their local network. I had a chance to test PAM Essentials while still in development. While there, I also integrated it with syslog-ng. >From my previous blog, you could learn what PAM Essentials is, and how you can collect its logs using syslog-ng. This blog will show you how to work with the collected log messages and create alerts when somebody connects to a host on your local network using PAM Essentials. https://www.syslog-ng.com/community/b/blog/posts/alerting-on-one-identity-c… The syslog-ng health check -------------------------- Version 4.2 of syslog-ng introduced a healthcheck option to syslog-ng-ctl. It prints three syslog-ng-related metrics on screen – if it can reach syslog-ng, that is. You can use it from scripts to monitor the health of syslog-ng. https://www.syslog-ng.com/community/b/blog/posts/the-syslog-ng-health-check WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

we need to know if Syslog supports any of these virtualization solution - Harvester, CaaS or HyperV. If not, what other virtualization solution is supported by Syslog.
by Mayekar, PrachiX 24 Apr '24

24 Apr '24

We need to know which virtualization solution is supported by syslog - Harvester CaaS or HyperV ? Thanks & Regards, Prachi Mayekar ITI-Network Services A Contingent Worker at Intel For assistance, please visit us at https://it.intel.com<https://it.intel.com/>

3 2

syslog-ng 4.7.1
by László Várady 19 Apr '24

19 Apr '24

Dear syslog-ng users, We are pleased to announce the 4.7.1 version of syslog-ng, which has been released and is now available on GitHub: https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-4.7.1 Packages <https://github.com/syslog-ng/syslog-ng#installation-from-binaries> are also available for various platforms. The AxoSyslog project provides cloud-ready container images <https://axoflow.com/cloud-ready-syslog-ng-images/>, Helm charts <https://axoflow.com/axosyslog-log-collection-for-kubernetes/>. 4.7.1 *This is the combination of the news entries of 4.7.0 and 4.7.1.* *4.7.1 hotfixed two crashes related to configuration reload.* Read Axoflow's blog post <https://axoflow.com/axosyslog-release-4-7/> for more details. You can read more about the new features in the AxoSyslog documentation <https://axoflow.com/docs/axosyslog-core/>. HighlightsCollecting Jellyfin logs The new jellyfin() source, reads Jellyfin logs from its log file output. Example minimal config: source s_jellyfin { jellyfin( base-dir("/path/to/my/jellyfin/root/log/dir") filename-pattern("log_*.log") ); }; For more details about Jellyfin logging, see: - https://jellyfin.org/docs/general/administration/configuration/#main-config… - https://jellyfin.org/docs/general/administration/configuration/#log-directo… As the jellyfin() source is based on a wildcard-file() source, all of the wildcard-file() source options are applicable, too. (#4802 <https://github.com/syslog-ng/syslog-ng/pull/4802>) Collecting *arr logs Use the newly added *arr() sources to read various *arr logs: - lidarr() - prowlarr() - radarr() - readarr() - sonarr() - whisparr() Example minimal config: source s_radarr { radarr( dir("/path/to/my/radarr/log/dir") ); }; The logging module is stored in the <prefix><module> name-value pair, for example: .radarr.module => ImportListSyncService. The prefix can be modified with the prefix() option. (#4803 <https://github.com/syslog-ng/syslog-ng/pull/4803>) Features - opentelemetry(), syslog-ng-otlp() source: Added concurrent-requests() option. This option configures the maximal number of in-flight gRPC requests per worker. Setting this value to the range of 10s or 100s is recommended when there are a high number of clients sending simultaneously. Ideally, workers() * concurrent-requests() should be greater or equal to the number of clients, but this can increase the memory usage. (#4827 <https://github.com/syslog-ng/syslog-ng/pull/4827>) - loki(): Support multi-tenancy with the new tenant-id() option (#4812 <https://github.com/syslog-ng/syslog-ng/pull/4812>) - s3(): Added support for authentication from environment. The access-key() and secret-key() options are now optional, which makes it possible to use authentication methods originated from the environment, e.g. AWS_... environment variables or credentials files from the ~/.aws/ directory. For more info, see: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.h… (#4881 <https://github.com/syslog-ng/syslog-ng/pull/4881>) - gRPC based drivers: Added channel-args() option. Affected drivers are: - bigquery() destination - loki() destination - opentelemetry() source and destination - syslog-ng-otlp() source and destination The channel-args() option accepts name-value pairs and sets channel arguments defined in https://grpc.github.io/grpc/core/group__grpc__arg__keys.html Example config: opentelemetry( channel-args( "grpc.loadreporting" => 1 "grpc.minimal_stack" => 0 ) ); (#4827 <https://github.com/syslog-ng/syslog-ng/pull/4827>) - ${TRANSPORT} macro: Added support for locally created logs. New values are: - "local+unix-stream" - "local+unix-dgram" - "local+file" - "local+pipe" - "local+program" - "local+devkmsg" - "local+journal" - "local+afstreams" - "local+openbsd" (#4777 <https://github.com/syslog-ng/syslog-ng/pull/4777>) - tags: Added new built-in tags that help identifying parse errors. New tags are: - "message.utf8_sanitized" - "message.parse_error" - "syslog.missing_pri" - "syslog.missing_timestamp" - "syslog.invalid_hostname" - "syslog.unexpected_framing" - "syslog.rfc3164_missing_header" - "syslog.rfc5424_unquoted_sdata_value" (#4804 <https://github.com/syslog-ng/syslog-ng/pull/4804>) - mqtt() source: Added ${MQTT_TOPIC} name-value pair. It is useful for the cases where topic() contains wildcards. Example config: log { source { mqtt(topic("#")); }; destination { stdout(template("${MQTT_TOPIC} - ${MESSAGE}\n")); }; }; (#4824 <https://github.com/syslog-ng/syslog-ng/pull/4824>) - template(): Added a new template function: $(tags-head) This template function accepts multiple tag names, and returns the first one that is set. Example config: # resolves to "bar" if "bar" tag is set, but "foo" is not template("$(tags-head foo bar baz)") (#4804 <https://github.com/syslog-ng/syslog-ng/pull/4804>) - s3(): Use default AWS URL if url() is not set. (#4813 <https://github.com/syslog-ng/syslog-ng/pull/4813>) - opentelemetry(), syslog-ng-otlp() source: Added log-fetch-limit() option. This option can be used to fine tune the performance. To minimize locking while moving messages between source and destination side queues, syslog-ng can move messages in batches. The log-fetch-limit() option sets the maximal size of the batch moved by a worker. By default it is equal to log-iw-size() / workers(). (#4827 <https://github.com/syslog-ng/syslog-ng/pull/4827>) - dqtool: add option for truncating (compacting) abandoned disk-buffers (#4875 <https://github.com/syslog-ng/syslog-ng/pull/4875>) Bugfixes - opentelemetry(): fix crash when an invalid configuration needs to be reverted (#4910 <https://github.com/syslog-ng/syslog-ng/pull/4910>) - gRPC drivers: fixed a crash when gRPC drivers were used and syslog-ng was reloaded (#4909 <https://github.com/syslog-ng/syslog-ng/pull/4909>) - opentelemetry(), syslog-ng-otlp() source: Fixed a crash. It occurred with multiple workers() during high load. (#4827 <https://github.com/syslog-ng/syslog-ng/pull/4827>) - rename(): Fixed a bug, which always converted the renamed NV pair to string type. (#4847 <https://github.com/syslog-ng/syslog-ng/pull/4847>) - With IPv6 disabled, there were linking errors (#4880 <https://github.com/syslog-ng/syslog-ng/pull/4880>) Metrics - http(): Added a new counter for HTTP requests. It is activated on stats(level(1));. Example metrics: syslogng_output_http_requests_total{url="http://localhost:8888/bar",response_code="200",driver="http",id="#anon-destination0#0"} 16 syslogng_output_http_requests_total{url="http://localhost:8888/bar",response_code="401",driver="http",id="#anon-destination0#0"} 2 syslogng_output_http_requests_total{url="http://localhost:8888/bar",response_code="502",driver="http",id="#anon-destination0#0"} 1 syslogng_output_http_requests_total{url="http://localhost:8888/foo",response_code="200",driver="http",id="#anon-destination0#0"} 24 (#4805 <https://github.com/syslog-ng/syslog-ng/pull/4805>) - gRPC based destination drivers: Added gRPC request related metrics. Affected drivers: - opentelemetry() - syslog-ng-otlp() - bigquery() - loki() Example metrics: syslogng_output_grpc_requests_total{driver="syslog-ng-otlp",url="localhost:12345",response_code="ok"} 49 syslogng_output_grpc_requests_total{driver="syslog-ng-otlp",url="localhost:12345",response_code="unavailable"} 11 (#4811 <https://github.com/syslog-ng/syslog-ng/pull/4811>) - New metric to monitor destination reachability syslogng_output_unreachable is a bool-like metric, which shows whether a destination is reachable or not. sum() can be used to count all unreachable outputs, hence the negated name. It is currently available for the network(), syslog(), unix-*() destinations, and threaded destinations (http(), opentelemetry(), redis() , mongodb(), python(), etc.). (#4876 <https://github.com/syslog-ng/syslog-ng/pull/4876>) - destinations: Added "syslogng_output_event_retries_total" counter. This counter is available for the following destination drivers: - amqp() - bigquery() - http() and all http based drivers - java() - kafka() - loki() - mongodb() - mqtt() - opentelemetry() - python() and all python based drivers - redis() - riemann() - smtp() - snmp() - sql() - stomp() - syslog-ng-otlp() Example metrics: syslogng_output_event_retries_total{driver="http",url="http://localhost:8888/${path}",id="#anon-destination0#0"} 5 (#4807 <https://github.com/syslog-ng/syslog-ng/pull/4807>) - syslogng_memory_queue_capacity Shows the capacity (maximum possible size) of each queue. Note that this metric publishes log-fifo-size(), which only limits non-flow-controlled messages. Messages coming from flow-controlled paths are not limited by log-fifo-size(), their corresponding source log-iw-size() is the upper limit. (#4831 <https://github.com/syslog-ng/syslog-ng/pull/4831>) Other changes - opentelemetry(), syslog-ng-otlp() source: Changed the backpressure behavior. syslog-ng no longer returns UNAVAILABLE to the gRPC request, when it cannot forward the received message because of backpressure. Instead, syslog-ng will block until the destination can accept more messages. (#4827 <https://github.com/syslog-ng/syslog-ng/pull/4827>) - opentelemetry(), syslog-ng-otlp() source: log-iw-size() is now split between workers. (#4827 <https://github.com/syslog-ng/syslog-ng/pull/4827>) - APT packages: Dropped Debian Buster support. Old packages are still available, but new syslog-ng versions will not be available on Debian Buster (#4840 <https://github.com/syslog-ng/syslog-ng/pull/4840>) - dbld: AlmaLinux 8 support (#4902 <https://github.com/syslog-ng/syslog-ng/pull/4902>) syslog-ng Discord For a bit more interactive discussion, join our Discord server: [image: Axoflow Discord Server] <https://discord.gg/E65kP9aZGm> Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Arpad Kunszt, Attila Szakacs, Balazs Scheidler, Bálint Horváth, Hofi, Kovács, Gergő Ferenc, László Várady, Peter Marko, shifter

1 0

we need to know if Syslog supports any of these virtualization solution - Harvester, CaaS or HyperV. If not, what other virtualization solution is supported by Syslog.
by Mayekar, PrachiX 16 Apr '24

16 Apr '24

Hi All, We are migrating the VMs from VMware to new solution (Harvester, CaaS, HyperV). Hence, we need to know if Syslog supports any of these virtualization solution - Harvester, CaaS or HyperV. If not, what other virtualization solution is supported by Syslog. Thanks & Regards, Prachi Mayekar ITI-Network Services A Contingent Worker at Intel For assistance, please visit us at https://it.intel.com<https://it.intel.com/>

3 2

The syslog-ng Insider 2024-04: PAM Essentials; XML Eventlog; multi-line logs
by Peter Czanik (pczanik) 11 Apr '24

11 Apr '24

Dear syslog-ng users, This is the 119th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Collecting One Identity Cloud PAM Essentials logs using syslog-ng ----------------------------------------------- One Identity Cloud PAM Essentials is the latest security product by One Identity. It provides asset management as well as secure and monitored remote access for One Identity Cloud users to hosts on their local network. I had a chance to test PAM Essentials while still in development. While there, I also integrated it with syslog-ng. >From this blog, you can learn what PAM Essentials is, and how you can collect its logs using syslog-ng. My next blog will show you how to work with the collected log messages and create alerts when somebody connects to a host on your local network using PAM Essentials. https://www.syslog-ng.com/community/b/blog/posts/collecting-one-identity-cl… Dedicated Windows XML eventlog parser in syslog-ng -------------------------------------------------- Version 4.6 of syslog-ng introduced windows-eventlog-xml-parser(), a dedicated parser for XML-formatted event logs from Windows. It makes the EventData portion of log messages more useful, as it combines two arrays into a list of name-value pairs. https://www.syslog-ng.com/community/b/blog/posts/dedicated-windows-xml-even… Working with multi-line logs in syslog-ng ----------------------------------------- Most log messages fit on a single line. However, Windows and some developer tools and services, like Tomcat, write multi-line log messages. These can come in various formats. For example, new log messages start with a date in a specific format. You use the multi-line-prefix() of the syslog-ng file() source to send multi-line messages as single messages instead of line by line. I must admit that I have never seen multi-line logs in production. I am not a developer, do not run Tomcat or Windows. However, recently I tested a software on Windows, which produced multi-line log messages. https://www.syslog-ng.com/community/b/blog/posts/working-with-multi-line-lo… WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

How does syslog-ng detect the availability of the remote end
by Grant Guo 03 Apr '24

03 Apr '24

Hi, Recently I am using syslog-ng’s docker container as syslog generator as the following: docker run -d \ --cpus=2 \ --name $name \ -v ${PWD}/syslog.log:/data/syslog.log \ --entrypoint="" \ --network=host \ balabit/syslog-ng:latest \ /usr/bin/loggen --dgram --inet --dont-parse --loop-reading --size 8192 --rate $rate --interval $duration --read-file /data/syslog.log $ip 514 I found something interesting, if my syslog server stopped, the client syslog-ng containers stopped too. So I would like to know if syslog-ng detected the availability of the remote end? If so, how? Besides, I didn’t see ICMP packets between the client and server using tcpdump. Thanks Grant *** Please note that this message and any attachments may contain confidential and proprietary material and information and are intended only for the use of the intended recipient(s). If you are not the intended recipient, you are hereby notified that any review, use, disclosure, dissemination, distribution or copying of this message and any attachments is strictly prohibited. If you have received this email in error, please immediately notify the sender and destroy this e-mail and any attachments and all copies, whether electronic or printed. Please also note that any views, opinions, conclusions or commitments expressed in this message are those of the individual sender and do not necessarily reflect the views of Fortinet, Inc., its affiliates, and emails are not binding on Fortinet and only a writing manually signed by Fortinet's General Counsel can be a binding commitment of Fortinet to Fortinet's customers or partners. Thank you. ***

1 0

The syslog-ng Insider 2024-03: MacOS; OpenTelemetry;
by Peter Czanik (pczanik) 14 Mar '24

14 Mar '24

Dear syslog-ng users, This is the 118th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Native MacOS source in syslog-ng -------------------------------- You know that support for MacOS is important when every third visitor at the syslog-ng booth of Red Hat Summit asks if syslog-ng works on MacOS. With the upcoming syslog-ng version 4.6.0, syslog-ng not only compiles on MacOS, but it also collects local log messages natively. >From this blog you can learn how to compile syslog-ng yourself, options of the MacOS source, and also a bit of history. https://www.syslog-ng.com/community/b/blog/posts/native-macos-source-in-sys… Using OpenTelemetry between syslog-ng instances ----------------------------------------------- Do you have to forward large amounts of logs between two syslog-ng instances? OTLP (OpenTelemetry protocol) support in syslog-ng can solve this problem. Just like the ewmm() destination, syslog-ng-otlp() forwards most name-value pairs, however, unlike a tcp() connection, it scales well with multiple CPU cores. Support for OpenTelemetry was added to syslog-ng a couple of releases ago. OpenTelemetry is an observability framework, mainly used in Linux / Cloud / Kubernetes environments. However, I already had users asking to make this feature available on FreeBSD. (It already worked once, but now it fails to compile again.) Version 4.6.0 added many new OTLP-related enhancements. Batching and multiple workers make OTLP connections significantly faster, while compression can save you bandwidth at the expense of some more CPU usage. This changes the syslog-ng-otlp() destination from an interesting experiment into something really useful. It enables you to send a lot more log messages between two syslog-ng instances than with a tcp() connection, while using less bandwidth. https://www.syslog-ng.com/community/b/blog/posts/using-opentelemetry-betwee… Collecting even more logs on MacOS using syslog-ng -------------------------------------------------- Version 4.6.0 of syslog-ng features not just a new, native log collector for MacOS, but also darwin-oslog-stream(), which can also collect non-persistent log events. Beware that it can collect many megabytes of logs even in just a few minutes! However, if you need to debug MacOS, these logs might come handy. https://www.syslog-ng.com/community/b/blog/posts/collecting-even-more-logs-… WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

syslog-ng service keeps restarting on systemd
by Darren Fuller 29 Feb '24

29 Feb '24

Hello, I have installed syslog-ng OSE from source on Amazon Linux 2023 distribution (as there is no EPEL available on that distribution nor are there any precompiled binaries for it) When i start up syslog-ng in the local terminal with syslog-ng -Fedv , it runs fine forever.. But when i start it with systemd : systemctl start syslog-ng, the command times out (but does start in the background) and then it the service restarts every two minutes or so. I am stumped. any thoughts? where have i gone wrong? appreciate any assistance you can offer. thanks, Darren More information: ------------ i compiled with the following and saw no errors: > ./configure --disable-smtp --disable-http --disable-python --disable-json > make > make install ------------ I created the systemd service file as so: > cat /etc/systemd/system/syslog-ng.service [Unit] Description=System Logger Daemon Documentation=man:syslog-ng(8) After=network.target [Service] Type=notify PIDFile=/run/syslogd.pid ExecStart=/usr/local/sbin/syslog-ng -F -p /run/syslogd.pid ExecReload=/bin/kill -HUP $MAINPID StandardOutput=null Restart=on-failure [Install] WantedBy=multi-user.target ------------ My config is here: ------------ > cat /usr/local/etc/syslog-ng.conf @include “scl.conf” source s_local { system(); internal(); }; destination d_local { file(“/var/log/messages”); file(“/var/log/messages-kv.log” template(“$ISODATE $HOST $(format-welf --scope all-nv-pairs)\n”) frac-digits(3)); }; log { source(s_local); destination(d_local); }; @include /etc/syslog-ng.d/*.conf > cat /etc/syslog.d/syslognet.conf options { long_hostnames(off); sync(0); keep_hostname(yes); owner(“splunk”); group(“splunk”); perm(0640); create_dirs(yes); dir_perm(0750); }; source s_tcp_net { tcp(ip(0.0.0.0) port(41514)); }; source s_tls_net { network ( ip(0.0.0.0) port(42514) transport(“tls”) tls( key-file(“/path/to/syslog-ng.key”) cert-file(“/path/to/syslog-ng.crt”) peer-verify(no) ) ); }; destination d_net { file(“/data/syslog/app/${SOURCEIP}/${YEAR}${MONTH}${DAY}-app-data.log”); }; log { source(s_tcp_net); source(s_tls_net); destination(d_net); }; ------------ Here is what i see when i start it with systemd: > systemctl start syslog-ng Job for syslog-ng.service failed because a timeout was exceeded. See “systemctl status syslog-ng.service” and “journalctl -xeu syslog-ng.service” for details. > systemctl status syslog-ng.service syslog-ng.service - System Logger Daemon Loaded: loaded (/etc/systemd/system/syslog-ng.service; enabled; preset: enabled) Active: activating (start) since Wed 2024-02-28 15:25:01 UTC; 35s ago Docs: man:syslog-ng(8) Main PID: 488557 (syslog-ng) Tasks: 4 (limit: 2322) Memory: 2.5M CPU: 39ms CGroup: /system.slice/syslog-ng.service └─488557 /usr/local/sbin/syslog-ng -F -p /run/syslogd.pid Feb 28 15:25:01 ip-172-31-7-110.ca-central-1.compute.internal systemd[1]: Starting syslog-ng.service - System Logger Daemon... ------------ And here is what's in /var/log/messages showing the restart: > cat /var/log/messages | grep syslog-ng Feb 28 14:38:24 ip-172-31-7-110 syslog-ng[447117]: syslog-ng shutting down; version=‘4.6.0’ Feb 28 14:38:25 ip-172-31-7-110 syslog-ng[447177]: syslog-ng starting up; version=‘4.6.0’ Feb 28 14:38:26 ip-172-31-7-110 syslog-ng[447177]: Syslog connection accepted; fd=‘26’, client=‘AF_INET(1.2.3.4:27717)’, local=‘AF_INET(0.0.0.0:41514)’ Feb 28 14:38:26 ip-172-31-7-110 syslog-ng[447177]: Syslog connection accepted; fd=‘27’, client=‘AF_INET(1.2.3.4:2300)’, local=‘AF_INET(0.0.0.0:41514)’ Feb 28 14:38:26 ip-172-31-7-110 syslog-ng[447177]: Syslog connection accepted; fd=‘28’, client=‘AF_INET(1.2.3.4:31585)’, local=‘AF_INET(0.0.0.0:41514)’ Feb 28 14:38:26 ip-172-31-7-110 syslog-ng[447177]: Syslog connection accepted; fd=‘29’, client=‘AF_INET(1.2.3.4:17761)’, local=‘AF_INET(0.0.0.0:41514)’ Feb 28 14:39:55 ip-172-31-7-110 syslog-ng[447177]: syslog-ng shutting down; version=‘4.6.0’ Feb 28 14:39:55 ip-172-31-7-110 syslog-ng[447234]: syslog-ng starting up; version=‘4.6.0’ Feb 28 14:39:56 ip-172-31-7-110 syslog-ng[447234]: Syslog connection accepted; fd=‘25’, client=‘AF_INET(1.2.3.4:7213)’, local=‘AF_INET(0.0.0.0:41514)’ Feb 28 14:39:56 ip-172-31-7-110 syslog-ng[447234]: Syslog connection accepted; fd=‘26’, client=‘AF_INET(1.2.3.4:9268)’, local=‘AF_INET(0.0.0.0:41514)’ Feb 28 14:39:56 ip-172-31-7-110 syslog-ng[447234]: Syslog connection accepted; fd=‘27’, client=‘AF_INET(1.2.3.4:13128)’, local=‘AF_INET(0.0.0.0:41514)’ Feb 28 14:39:56 ip-172-31-7-110 syslog-ng[447234]: Syslog connection accepted; fd=‘28’, client=‘AF_INET(1.2.3.4:3928)’, local=‘AF_INET(0.0.0.0:41514)’ Feb 28 14:41:25 ip-172-31-7-110 syslog-ng[447234]: syslog-ng shutting down; version=‘4.6.0’ Feb 28 14:41:26 ip-172-31-7-110 syslog-ng[447385]: syslog-ng starting up; version=‘4.6.0’ Feb 28 14:41:27 ip-172-31-7-110 syslog-ng[447385]: Syslog connection accepted; fd=‘25’, client=‘AF_INET(1.2.3.4:1382)’, local=‘AF_INET(0.0.0.0:41514)’ Feb 28 14:41:27 ip-172-31-7-110 syslog-ng[447385]: Syslog connection accepted; fd=‘26’, client=‘AF_INET(1.2.3.4:5430)’, local=‘AF_INET(0.0.0.0:41514)’ Feb 28 14:41:27 ip-172-31-7-110 syslog-ng[447385]: Syslog connection accepted; fd=‘27’, client=‘AF_INET(1.2.3.4:6373)’, local=‘AF_INET(0.0.0.0:41514)’ Feb 28 14:41:27 ip-172-31-7-110 syslog-ng[447385]: Syslog connection accepted; fd=‘28’, client=‘AF_INET(1.2.3.4:26287)’, local=‘AF_INET(0.0.0.0:41514)’ Feb 28 14:42:56 ip-172-31-7-110 syslog-ng[447385]: syslog-ng shutting down; version=‘4.6.0’ Feb 28 14:42:56 ip-172-31-7-110 syslog-ng[447443]: syslog-ng starting up; version=‘4.6.0’ Feb 28 14:42:57 ip-172-31-7-110 syslog-ng[447443]: Syslog connection accepted; fd=‘25’, client=‘AF_INET(1.2.3.4:15165)’, local=‘AF_INET(0.0.0.0:41514)’ Feb 28 14:42:57 ip-172-31-7-110 syslog-ng[447443]: Syslog connection accepted; fd=‘26’, client=‘AF_INET(1.2.3.4:6967)’, local=‘AF_INET(0.0.0.0:41514)’ Feb 28 14:42:57 ip-172-31-7-110 syslog-ng[447443]: Syslog connection accepted; fd=‘27’, client=‘AF_INET(1.2.3.4:27444)’, local=‘AF_INET(0.0.0.0:41514)’ Feb 28 14:42:57 ip-172-31-7-110 syslog-ng[447443]: Syslog connection accepted; fd=‘28’, client=‘AF_INET(1.2.3.4:20758)’, local=‘AF_INET(0.0.0.0:41514)’

3 2

running RHEL 9
by Linn, William 19 Feb '24

19 Feb '24

Has anyone seen any issues with syslog-ng running on RHEL 9? Any pointers, caveats? Thanks William ---------------------------------------------------------------------- This message, and any attachment(s), is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/electronic-disclaimer. If you are not the intended recipient, please delete this message. For more information about how Bank of America protects your privacy, including specific rights that may apply, please visit the following pages: https://business.bofa.com/en-us/content/global-privacy-notices.html (which includes global privacy notices) and https://www.bankofamerica.com/security-center/privacy-overview/ (which includes US State specific privacy notices such as the http://www.bankofamerica.com/ccpa-notice)

3 3

The syslog-ng Insider 2024-02: OpenObserve; configuration check; build services;
by Peter Czanik (pczanik) 15 Feb '24

15 Feb '24

Dear syslog-ng users, This is the 117th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Version 4.5.0 of syslog-ng is now available with OpenObserve JSON API support ------------------------------------------------ Recently, syslog-ng 4.5.0 was released with many new features. These include sending logs to OpenObserve using its JSON API, support for Google Pub/Sub, a new macro describing message transport mechanisms like RFC 3164 + TCP, an SSL option to ignore validity periods, and many more. You can find a full list of new features and bug fixes in the release notes at: https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-4.5.0 In this blog, you can find some pointers on how to install the very latest syslog-ng version and learn how you can configure syslog-ng to use the OpenObserve JSON API. https://www.syslog-ng.com/community/b/blog/posts/version-4-5-0-of-syslog-ng… Syslog-ng PE can now send logs to Google BigQuery ------------------------------------------------- Version 7.0.34 of syslog-ng PE can now send logs to Google BigQuery destinations. You can learn more about how it works from the syslog-ng PE documentation: https://support.oneidentity.com/technical-documents/syslog-ng-premium-editi… Syslog-ng can now do a full configuration check ----------------------------------------------- One of the most frequent syslog-ng feature requests is now resolved. Welcome the --check-startup option, allowing you to check the syntax and also spot spelling mistakes! https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-can-now-do-a-ful… How build services make life easier for upstream developers ----------------------------------------------------------- Many Linux distributions provide build services under various names: openSUSE Build Service (OBS), Fedora Copr, and so on. These resources are indispensable for upstream developers, and also for their users. I will demonstrate this through some examples from the syslog-ng project. Note: this blog is loosely based on a talk idea I had for the FOSDEM Distributions Devroom. There is no deep technical information about syslog-ng in this blog. This is more like a history of syslog-ng packaging, and how the fantastic tools by openSUSE and Fedora made it a lot easier and made me an active part of these communities. https://www.syslog-ng.com/community/b/blog/posts/how-build-services-make-li… WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

The syslog-ng Insider 2024-01: HTTP; Cloudflare; systemd-journal; Humio / Logscale;
by Peter Czanik (pczanik) 18 Jan '24

18 Jan '24

Dear syslog-ng users, This is the 116th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Why use a http()-based destination in syslog-ng? ------------------------------------------------ Logging is not just syslog anymore. Still, many syslog-ng users stick to using one of the syslog protocols for log transport and flat files for log storage. While most SIEMs and log analytics tools can receive syslog messages or read them using their own agents, in most cases, you can use the http() destination of syslog-ng as well to send logs to them. You gain extreme performance and an architecture that is easier to maintain. Some of the drivers in syslog-ng built on top of the http() destination include elasticsearch-http() destination for sending logs to Elasticsearch / OpenSearch. Many other services utilizing the Elasticsearch Bulk API exist, and there are also others using Sumo Logic, Splunk, and so on. Of course, there are some destinations where performance is not a major concern (or can even be a drawback, in fact). For example,various instant messaging services used for alerting (like Telegram or Slack) belong in this category. You can also read the API docs and write a new destination based on http() yourself. https://www.syslog-ng.com/community/b/blog/posts/why-use-a-http--based-dest… An overview of Cloudflare's logging pipeline -------------------------------------------- One of the roles of Cloudflare's Observability Platform team is managing the operation, improvement, and maintenance of our internal logging pipelines. These pipelines are used to ship debugging logs from every service across Cloudflare’s infrastructure into a centralised location, allowing our engineers to operate and debug their services in near real time. In this post, we’re going to go over what that looks like, how we achieve high availability, and how we meet our Service Level Objectives (SLOs) while shipping close to a million log lines per second. https://blog.cloudflare.com/an-overview-of-cloudflares-logging-pipeline/ Working with multiple systemd-journal namespaces in syslog-ng ------------------------------------------------------------- Initial support for systemd-journal namespaces is available in syslog-ng 3.29. However, only version 4.4.0 allows you to work with multiple namespaces in your syslog-ng configuration. So, what changed in the latest version of syslog-ng? Previously, you could only configure a single systemd-journal() source in syslog-ng. By default, it collected logs from all namespaces, but you could configure it to collect log messages from a single one exclusively. This means that logs from other namespaces could not be collected by syslog-ng. Version 4.4.0 allows you to use multiple systemd-journal() source drivers in the configuration, as long as each source uses a unique namespace. https://www.syslog-ng.com/community/b/blog/posts/working-with-multiple-syst… Logging to Humio / Logscale simplified in syslog-ng --------------------------------------------------- Logging into Humio (which was recently re-branded to Falcon LogScale) was available for years, using their Elasticsearch compatible API. However, according to Humio developers, it is slightly slower than other APIs for log ingestion. Axoflow contributed a Logscale destination to syslog-ng, which uses Logscale’s native API. I did not measure if there is really a performance difference, however it is definitely easier to configure it. WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Syslog-ng Not Working properly
by Sumanta Banerjee 15 Jan '24

15 Jan '24

Hi Team, I am trying to configure syslog-ng in one our linux instance to get NGIPS/FMC data via udp connection on its default port (514). I have configured syslog-ng.conf under /etc/syslog-ng and then we have set SE Linux as Permissive. I am using RHEL 8.7 and syslog version 4.0. Apparently all looked good to me however while checking in the destination path that is mentioned I don't see any directory or logfile from for the said udp connection got created. Below is our observation and steps that we executed, can any of you please help me telling where I went wrong or if I am missing something, there is another testing in pipeline that is stalled for this - 1. Define source, destination and log_file in syslog-ng.conf (file attached). 2. Run the below SELinux command - # ausearch -c 'syslog-ng' --raw | audit2allow -M my-syslogng # semodule -X 300 -i my-syslogng.pp 1. Restart syslog-ng service - # systemctl restart syslog-ng.service (no error message received) 1. Check if the service is running - [cid:image002.png@01D99309.5147A9A0] 1. Check if syslog-ng is listening to udp port 514 - [cid:image003.png@01D99309.5147A9A0] 1. Checked and we have incoming data stream from source using the below command - tcpdump -i any -c10 -nn -A port 514 1. I have went through the syslog-ng troubleshooting steps mentioned in the link - https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edi… [cid:image004.png@01D9930B.D0B70940] syslog-ng -Fdev command output is also attached. 1. While running the following command got the below output - # watch '/usr/sbin/syslog-ng-ctl stats | grep "^center"' [cid:image005.png@01D99311.15E497D0] 1. # journaltctl command output (first 500 lines) attached 1. Current SE Linux status : [cid:image006.png@01D9931C.B8765370] 1. Our syslog-ng is logging to /var/log/messages and we are getting this message in /var/log/messages - [cid:image007.png@01D9931E.592725A0] Thanks & Regards, Sumanta Banerjee Splunk Admin | CISO | Aviva Group Tel: +91-8420892593 24x7x365: +44 1603 208 582 sumanta.banerjee(a)aviva.com<mailto:sumanta.banerjee@aviva.com> GlobalCyberSecurityEngineeringTeam(a)aviva.com<mailto:GlobalCyberSecurityEngineeringTeam@aviva.com> www.aviva.com<http://www.aviva.com> Wipro Technologies - SJP2, Bangalore, India [cid:image001.gif@01D99303.20AF3AC0] Aviva: Internal Aviva plc, registered Office: St. Helen's, 1 Undershaft, London EC3P 3DQ. Registered in England No. 02468686. www.aviva.com This message and any attachments may be confidential or legally privileged. If you are not the intended recipient, please telephone or e-mail the sender and delete this message and any attachments from your system. Also, if you are not the intended recipient you must not copy this message or attachments or disclose the contents to any other person. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Aviva.

2 4

Critical Severity Security Advisory: Apache Struts Vulnerability CVE-2023-50164
by Mayekar, PrachiX 16 Dec '23

16 Dec '23

Hi Team, Can you please confirm whether Syslog is affected by this vulnerability: Critical Severity Security Advisory: Apache Struts Vulnerability CVE-2023-50164 Thanks & Regards, Prachi Mayekar ITI-Network Services A Contingent Worker at Intel For assistance, please visit us at https://it.intel.com<https://it.intel.com/>

3 2

Insider 2023-12: compressed HTTP; packages; OpenObserve; duplicates;
by Peter Czanik (pczanik) 14 Dec '23

14 Dec '23

Dear syslog-ng users, This is the 115th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Compressing HTTP traffic in syslog-ng ------------------------------------- Network traffic is expensive in the cloud, and even a single syslog-ng instance can easily saturate the full bandwidth of a network connection. Compressing HTTP traffic was introduced in syslog-ng Version 4.4.0 and depending on your use case, you can cut down on your expenses on your networking or send more logs using the same budget or bandwidth. Development of this feature was done using a locally installed OpenResty web server, and later tested using Sumologic. However, according to the docs it should also work with Splunk, Elasticsearch, and many other services accessible using the http() destination. https://www.syslog-ng.com/community/b/blog/posts/compressing-http-traffic-i… Why is a feature not available in the syslog-ng package? -------------------------------------------------------- You can read about many interesting syslog-ng features in my blogs. However, it can happen that when you want to try them at home, you fail because the feature is missing. How can you solve such problems? In this blog, I discuss some of the possible solutions from installing sub-packages through using unofficial repositories, to upgrading your OS. This blog focuses on RPM packages for openSUSE / SLES, Fedora / RHEL, and FreeBSD, because these are the packages I know – I am their maintainer. However, these problems and their solutions also apply to Debian / Ubuntu, and other Linux distributions. https://www.syslog-ng.com/community/b/blog/posts/why-is-a-feature-not-avail… Sending logs to OpenObserve using syslog-ng ------------------------------------------- A question was asked if syslog-ng can send logs to OpenObserve. It has an Elasticsearch compatible API for log ingestion, but syslog-ng is not mentioned in the documentation. My plan was to document how to modify the syslog-ng elasticsearch-http() destination, based on API documentation. However, as it turned out, OpenObserve has a ready to use syslog-ng configuration example in the web UI. https://www.syslog-ng.com/community/b/blog/posts/sending-logs-to-openobserv… Removing duplicate messages with syslog-ng in a redundant logging environment ----------------------------- Creating highly available servers is difficult. Sending logs to two (or more) servers and hoping that at least one of them can collect logs any time is a lot easier. Since network traffic and storage are cheap, redundancy is usually not a problem. However, once you also want to analyze your log messages using a SIEM or other software, you do not want duplicate log messages. In this blog, I show you how you can make sure that each log message produced on your network reaches your SIEM system only once. Sending logs to a SIEM is expensive: both resource usage and licensing costs are much higher than at the log management level. https://www.syslog-ng.com/community/b/blog/posts/removing-duplicate-message… WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

syslog-ng 4.5.0
by Attila Szakács 30 Nov '23

30 Nov '23

Dear syslog-ng users, We are pleased to announce the 4.5.0 version of syslog-ng, which has been released and is now available on GitHub: https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-4.5.0 Packages <https://github.com/syslog-ng/syslog-ng#installation-from-binaries> are also available for various platforms. The AxoSyslog project provides cloud-ready container images <https://axoflow.com/cloud-ready-syslog-ng-images/>, Helm charts <https://axoflow.com/axosyslog-log-collection-for-kubernetes/>. 4.5.0 Read Axoflow's blog post <https://axoflow.com/axosyslog-release-4-5/> for more details. You can read more about the new features in the AxoSyslog documentation <https://axoflow.com/docs/axosyslog-core/>. HighlightsSending log messages to OpenObserve The openobserve-log() destination feeds OpenObserve via the JSON API <https://openobserve.ai/docs/api/ingestion/logs/json/>. Example config: openobserve-log( url("http://openobserve-endpoint") port(5080) stream("default") user("root(a)example.com") password("V2tsn88GhdNTKxaS") ); (#4698 <https://github.com/syslog-ng/syslog-ng/pull/4698>) Sending messages to Google Pub/Sub The google-pubsub() destination feeds Google Pub/Sub via the HTTP REST API <https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.topics/publ…> . Example config: google-pubsub( project("syslog-ng-project") topic("syslog-ng-topic") auth( service-account( key("/path/to/service-account-key.json") ) ) ); See the Google Pub/Sub documentation <https://cloud.google.com/pubsub/docs/building-pubsub-messaging-system> to learn more about configuring a service account. (#4651 <https://github.com/syslog-ng/syslog-ng/pull/4651>) Parsing PostgreSQL logs The postgresql-csvlog-parser(): add a new parser to process CSV log formatted by PostgreSQL ( https://www.postgresql.org/docs/current/runtime-config-logging.html) The CSV format is extracted into a set of name-value pairs. (#4586 <https://github.com/syslog-ng/syslog-ng/pull/4586>) Features - http(): Added support for using templates in the url() option. In syslog-ng a template can only be resolved on a single message, as the same template might have different resolutions on different messages. A http batch consists of multiple messages, so it is not trivial to decide which message should be used for the resolution. When batching is enabled and multiple workers are configured it is important to only batch messages which generate identical URLs. In this scenario one must set the worker-partition-key() option with a template that contains all the templates used in the url() option, otherwise messages will be mixed. For security reasons, all the templated contents in the url() option are getting URL encoded automatically. Also the following parts of the url cannot be templated: - scheme - host - port - user - password (#4663 <https://github.com/syslog-ng/syslog-ng/pull/4663>) - $TRANSPORT: this is a new name-value pair that syslog-ng populates automatically. It indicates the "transport" mechanism used to retrieve/receive the message. It is up to the source driver to determine the value. Currently the following values were implemented: BSD syslog drivers: tcp(), udp() & network() - rfc3164+tls - rfc3164+tcp - rfc3164+udp - rfc3164+proxied-tls - rfc3164+<custom logproto like altp> UNIX domain drivers: unix-dgram(), unix-stream() - unix-stream - unix-dgram RFC5424 style syslog: syslog(): - rfc5426: syslog over udp - rfc5425: syslog over tls - rfc6587: syslog over tcp - rfc5424+<custom logproto like altp>: syslog over a logproto plugin Other drivers: - otlp: otel() driver - mqtt: mqtt() driver - hypr-api: hypr-audit-source() driver $IP_PROTO: indicate the IP protocol version used to retrieve/receive the message. Contains either "4" to indicate IPv4 and "6" to indicate IPv6. (#4673 <https://github.com/syslog-ng/syslog-ng/pull/4673>) - network() and syslog() drivers: Added ignore-validity-period as a new flag to ssl-options(). By specifying ignore-validity-period, you can ignore the validity periods of certificates during the certificate validation process. (#4642 <https://github.com/syslog-ng/syslog-ng/pull/4642>) - tls() in udp()/tcp()/network() and syslog() drivers: add support for a new http() compatible ssl-version() option. This makes the TLS related options for http() and other syslog-like drivers more similar. This requires OpenSSL 1.1.0. (#4682 <https://github.com/syslog-ng/syslog-ng/pull/4682>) - cloud-auth(): Added a new plugin for drivers, which implements different cloud related authentications. Currently the only supported authentication is GCP's Service Account <https://cloud.google.com/iam/docs/service-account-overview> for the http() destination. Example config: http( cloud-auth( gcp( service-account( key("/path/to/service-account-key.json") audience("https://pubsub.googleapis.com/google.pubsub.v1.Publisher") ) ) ) ); (#4651 <https://github.com/syslog-ng/syslog-ng/pull/4651>) - csv-parser(): allow parsing the extracted values into matches ($1, $2, $3 ...) by omitting the columns() parameter, which normally specifies the column names. (#4678 <https://github.com/syslog-ng/syslog-ng/pull/4678>) - --check-startup: a new command line option for syslog-ng along with the existing --syntax-only. This new option will do a complete configuration initialization and then exit with exit code indicating the result. Since this also initializes things like network listeners, it will probably *not* work when there is another syslog-ng instance running in the background. The recommended use of this option is a dedicated config check container, as explained in #4592 <https://github.com/syslog-ng/syslog-ng/issues/4592>. (#4646 <https://github.com/syslog-ng/syslog-ng/pull/4646>) Bugfixes - s3: Fixed an ImportError. ImportError: cannot import name 'SharedBool' from 'syslogng.modules.s3.s3_object' (#4700 <https://github.com/syslog-ng/syslog-ng/pull/4700>) - loki(): fixed mixing non-related label values (#4713 <https://github.com/syslog-ng/syslog-ng/pull/4713>) - type hinting: Parsing and casting fractions are now done locale independently. (#4702 <https://github.com/syslog-ng/syslog-ng/pull/4702>) - metrics-probe(): Fixed a crash. This crash occurred when a metrics-probe() instance was used in multiple source threads, like a network() source with multiple connections. (#4685 <https://github.com/syslog-ng/syslog-ng/pull/4685>) - flags() argument to various drivers: fix a potential crash in case a flag with at least 32 characters is used. No such flag is defined by syslog-ng, so the only way to trigger the crash is to use an invalid configuration file. (#4689 <https://github.com/syslog-ng/syslog-ng/pull/4689>) - Fix $PROTO value for transport(tls) connections, previously it was set to "0" while in reality these are tcp connections (e.g. "6"). Fix how syslog-ng sets $HOST for V4-mapped addresses in case of IPv6 source drivers (e.g. udp6()/tcp6() or when using ip-protocol(6) for tcp()/udp() ). Previously V4-mapped addresses would be represented as "::ffff:<ipv4 address>". This is not wrong per-se, but would potentially cause the same host to be represented in multiple ways. With the fix, syslog-ng would just use "<ipv4 address>" in these cases. (#4673 <https://github.com/syslog-ng/syslog-ng/pull/4673>) - db-parser(): support nested match characters in @QSTRING@ pattern parser (#4717 <https://github.com/syslog-ng/syslog-ng/pull/4717>) Other changes - LogSource and LogFetcher: additional documentation was added to these Python classes to cover explicit source-side batching functionalities (e.g. the auto_close_batch attribute and the close_batch() method). (#4673 <https://github.com/syslog-ng/syslog-ng/pull/4673>) - rate-limit(): Renamed the template() option to key(), which better communicates the intention. (#4679 <https://github.com/syslog-ng/syslog-ng/pull/4679>) - templates: The template-escape() option now only escapes the top-level template function. Before syslog-ng 4.5.0 if you had embedded template functions, the template-escape(yes) setting escaped the output of each template function, so the parent template function received an already escaped string. This was never the intention of the template-escape() option. Although this is a breaking change, we do not except anyone having a config that is affected. If you have such a config, make sure to follow-up this change. If you need help with it, feel free to open an issue or discussion on GitHub, or contact us on the Axoflow Discord server. (#4666 <https://github.com/syslog-ng/syslog-ng/pull/4666>) - loki(): The timestamp() option now supports quoted strings. The valid values are the following, with or without quotes, case insensitive: - "current" - "received" - "msg" (#4688 <https://github.com/syslog-ng/syslog-ng/pull/4688>) syslog-ng Discord For a bit more interactive discussion, join our Discord server: [image: Axoflow Discord Server] <https://discord.gg/E65kP9aZGm> Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Attila Szakacs, Balazs Scheidler, Cedric Arickx, Fabrice Fontaine, Hofi, László Várady, Romain Tartière, Szilard Parrag, yashmathne

1 0

syslog-ng vs pcre2 without jit vs disable-jit config feature
by "Tóth Attila" 27 Nov '23

27 Nov '23

I'm using syslog-ng-4.4.0 on a Gentoo system, that also employs PaX hardening. Due to the necessity to elevate restrictions on pcre2 with jit enabled, I keep it disabled for this particular installation. Syslog-ng emits error messages during startup complaining about pcre2 and jit. I had studied the manual and found the disable-jit feature. https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edi… Maybe I'm using a wrong syntax, but syslog-ng doesn't seem to respect the option. Commenting out the jit feature in the source code works, but it would be much more comfortable to find the proper way to disable jit. Are there any other who managed to use disable-jit in action? Are there any tips or tricks aboutv what to pay attention on? Thx: Dw. -- dr Tóth Attila, Radiológus, 06-20-825-8057 Attila Toth MD, Radiologist, +36-20-825-8057

2 6

New install
by Klimek, Bryan J. 16 Nov '23

16 Nov '23

We've been using Syslog-ng for a long time. Our current deployment is on Red Hat 7 servers that we built 6 years ago. It's time to start a refresh of our Syslog-ng environment. Our plan is to use Red Hat 9 servers. Can you give me a jumping off point on where to obtain installation instructions and how to obtain the installation media. Thank you. Bryan Klimek IT Systems Architect I EIS Application Hosting Services E-mail: klimek.bryan(a)mayo.edu<mailto:klimek.bryan@mayo.edu>

2 1

Insider 2023-11: Splunk; configuration; journald;
by Peter Czanik (pczanik) 14 Nov '23

14 Nov '23

Dear syslog-ng users, This is the 114th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Sending logs to Splunk using syslog-ng -------------------------------------- There are many ways you can collect log messages using syslog-ng and forward them to Splunk. In this blog I collect the history of Splunk support in syslog-ng, and the advantages and disadvantages of various solutions, both open source and commercial. https://www.syslog-ng.com/community/b/blog/posts/sending-logs-to-splunk-usi… Developing a syslog-ng configuration ------------------------------------ This year I started publishing a syslog-ng tutorial series both on my blog and on YouTube: https://peter.czanik.hu/posts/syslog-ng-tutorial-toc/ And while the series was praised as the best possible introduction to syslog-ng, viewers also mentioned that one interesting element is missing from it: namely, it does not tell users how to develop a syslog-ng configuration. So, in this blog, learn how to develop a syslog-ng configuration from the ground up! I will explain not just the end result, but also the process and the steps to take to develop a configuration. It starts with a single source and destination, then concludes with a conditional log path and sending parsed and enriched logs to Elasticsearch (or a compatible document store). https://www.syslog-ng.com/community/b/blog/posts/developing-a-syslog-ng-con… Systemd-journald vs. syslog-ng ------------------------------ Even if most people ask me to compare systemd-journald vs. syslog-ng, I would say that they complement each other. Systemd-journald excels at collecting local log messages, including those of various system services. The focus of syslog-ng is on central log collection and forwarding the logs to a wide variety of destinations after processing and filtering. Combining the two gives you the most flexibility. https://www.syslog-ng.com/community/b/blog/posts/systemd-journald-vs-syslog… WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Custom script with map-value-pairs(pair(<script>))
by Faisal Chishti 18 Oct '23

18 Oct '23

Hi, I am trying to capture some custom information using a simple shell script. I am trying below but getting an error. Below is my parser, it works fine without the line that has custom_script_output. How do I get this to work? parser p_cr_syslog { # Convert the LEVEL field to uppercase. map-value-pairs(pair("jcnr.header.logLevel", "$(uppercase $LEVEL)")); map-value-pairs(pair("jc.header.custom_script_output", "$(script("/path/to/script.sh"))")); syslog-parser(flags(syslog-protocol) template("${MESSAGE}")); map-value-pairs( pair("jc.header.nodeName", "$HOST") pair("jc.header.eventDateTime", "$R_ISODATE") pair("jc.header.notificationType", "$MSGID") pair("jc.body", "$MSG") pair("jc.header.program", "$PROGRAM") ); # Log the output of the script. action { file { path "/var/log/syslog"; message "$jc.header.custom_script_output"; } } } Thanks in advance. -- Regards, Faisal

2 1

Re: [syslog-ng] Vulnerability making News - HTTP/2 Rapid Reset DDoS CVE-2023-44487
by Mayekar, PrachiX 16 Oct '23

16 Oct '23

Hi Team, Are syslog products vulnerable to this vulnerability ? Need to know if Syslog is affected: CVE-2023-44487 is a vulnerability in the HTTP/2 protocol that was recently used to launch DDoS attacks. The vulnerability allows for denial of service (DoS) because request cancellation can reset many streams quickly. https://www.bleepingcomputer.com/news/security/new-http-2-rapid-reset-zero-… Thanks & Regards, Prachi Mayekar ITI-Network Services A Contingent Worker at Intel For assistance, please visit us at https://it.intel.com<https://it.intel.com/>

2 2

Insider 2023-10: contribute; parallelize; compatibility;
by Peter Czanik (pczanik) 12 Oct '23

12 Oct '23

Dear syslog-ng users, This is the 113th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Why contribute to syslog-ng upstream? ------------------------------------- One of the returning questions I received recently: why contribute to the syslog-ng upstream? I guess it is a question many open-source projects receive regularly. There are many generic answers. Here I would like to focus more on syslog-ng, focusing on various parts of it. Of course, the generic answers also apply. Syslog-ng is an open-source project, free to use, modify, and extend. By contributing, you can give something back and improve syslog-ng for everyone. You do not have to be a developer in order to contribute: bug reports, configuration examples, PatternDB rules are all very useful contributions. https://www.syslog-ng.com/community/b/blog/posts/why-contribute-to-syslog-n… Accelerating single TCP connections in syslog-ng: parallelize() --------------------------------------------------------------- One of the highlights of the syslog-ng 4.3.0 release is parallelize(). Normally, syslog-ng processes incoming messages from a TCP connection in a single thread. While this works fine with many connections, it is a bottleneck when using a single or very few high-traffic connections. Using parallelize() allows syslog-ng to process log messages from a single high-traffic TCP connection in multiple threads, thus increasing processing performance on multi-core machines. https://www.syslog-ng.com/community/b/blog/posts/accelerating-single-tcp-co… Backward compatibility in syslog-ng by using the version number in syslog-ng.conf --------------------------------------------------------- Many users are annoyed by the version number included in the syslog-ng configuration. However, it ensures backward compatibility in syslog-ng. It is especially useful when updating to syslog-ng 4 from version 3, but also when updating within the same major version. https://www.syslog-ng.com/community/b/blog/posts/backward-compatibility-in-… WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Syslog-ng.com seems to be down
by Balazs Scheidler 18 Sep '23

18 Sep '23

Hi, As it seems syslog-ng.com has been down for a while (2 days at least). If your are looking for the syslog-ng documentation, a good alternative is the AxoSyslog documentation at the Axoflow website. https://axoflow.com/docs/axosyslog-core/

2 1

Insider 2023-09: MongoDB; UDP; sngbench; contribute;
by Peter Czanik (pczanik) 14 Sep '23

14 Sep '23

Dear syslog-ng users, This is the 112th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS The syslog-ng MongoDB destination receives bulk operations support ---------------------------------- The MongoDB destination of syslog-ng will receive another performance update. Starting with the upcoming version 4.3, it will support bulk operations. Depending on the configuration settings, this may result in a more than 300% performance increase. https://www.syslog-ng.com/community/b/blog/posts/the-syslog-ng-mongodb-dest… Syslog-ng 4.2: extra UDP performance ------------------------------------ No matter how awkward you feel when you hear about UDP syslog in the age of encrypted TCP connections, UDP syslog is here to stay in some special cases. The scalability issues of UDP log collection were first addressed in syslog-ng Open Source Edition (OSE) (the so-reuseport() parameter), and later a more advanced solution arrived to syslog-ng Premium Edition (PE) (the udp-balancer() source). The good news is that a new, open-source implementation is now available as part of syslog-ng 4.2.0. https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-4-2-extra-udp-pe… What I learned about syslog-ng performance using sngbench --------------------------------------------------------- A few weeks ago, I posted about sngbench, a shell script to measure syslog-ng performance. The performance of syslog-ng is influenced by many factors, including the hardware and OS it runs on, and syslog-ng itself. This blog summarizes some of my findings using the script. https://www.syslog-ng.com/community/b/blog/posts/what-i-learned-about-syslo… WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

syslog-ng stops logging some messages
by Lars Wagner 14 Sep '23

14 Sep '23

Hi everyone We’re having a very odd problem with syslog-ng. After some time syslog-ng just stops logging further messages. A restart is required that syslog-ng starts to log again. I already put some information in a GitHub issue: https://github.com/syslog-ng/syslog-ng/issues/4485 As there was nearly no response there I try it that way. Thanks anyone in advance. Regards, Lars

1 0

syslog-ng 4.2.0 changed the interpretation of regexp
by Evan Rempel 11 Sep '23

11 Sep '23

Prior to syslog-ng 4.2.0 I was able to use a filter specified as match("ABC\|DEF" value("MESSAGE")) to match the exact string ABC|DEF in the MESSAGE After upgrading to 4.2.0 this will match either ABC or DEF in the MESSAGE To match an exact string of ABC|DEF I now need to escape both the backslash and the pipe match("ABC\\\|DEF" value("MESSAGE")) I can't find anything in the release notes. Of course I can easily solve this with single quotes, but more complex examples would not be so straight forward. Can someone confirm that this was intentional, or if this is a bug. -- Evan

3 4

How to stop syslog-ng from generating HOSTID on start up
by Dineth Jayasekera 06 Sep '23

06 Sep '23

Hi everyone, When starting up, I've noticed that syslog-ng hangs on our embedded system when generating the HOSTID. This is because the HOSTID generation uses RAND_bytes (in_create_host_id <https://github.com/syslog-ng/syslog-ng/blob/66123c955b29298ba6e68cdc77b9a0f…>)- which needs entropy (which we don't have enough of, and don't want to generate). The logging is only used locally and there is no need for crypto functionality. Is there a way to avoid generating a cryptographically secure HOSTID on start up? Thank you!

2 2

How to enable Syslog-ng authentication with MongoDB
by Ibrahim Al Mahfooz 20 Aug '23

20 Aug '23

Hi, I would like to harden the security by enabling authentication between syslog-ng and MongoDB. I'm using syslog-ng v3.38 and MongoDB v6.0.8 At the destination I tried the below two methods but none of themt worked, am I doing something wrong? Or the authentication is not supported? *---> First method:* destination d_mongodb_01 { mongodb( uri("mongodb://SampleUser:SamplePass@172.16.17.16:27017/syslog-ng_01") collection("logs") value-pairs( pair("datetime" "${S_ISODATE}") pair("privateip" "${privateip}") pair("portstatus" "${PORTSTATUS}") pair("host" "${HOST}")) ); }; *---> Second method:* destination d_mongodb_01 { mongodb( uri("mongodb://172.16.17.16:27017/syslog-ng_01") username("SampleUser") password("SamplePass") collection("logs") value-pairs( pair("datetime" "${S_ISODATE}") pair("privateip" "${privateip}") pair("portstatus" "${PORTSTATUS}") pair("host" "${HOST}")) ); }; Appreciate your help Best regards

2 2

discussion: Should ptbtool test validate types
by Evan Rempel 19 Aug '23

19 Aug '23

Now that syslog-ng has type hinting should the pdbtool test also validate the types of any values specified for a pattern. <rule id="pid1" class="myclass" provider="me"> <patterns> <pattern>some pattern @NUMBER:mynum@ with number</pattern> </patterns> <tags> <tag>rate</tag> </tags> <values> <value name="more" type="integer">100A</value> </values> <examples> <example> <test_message program="some pattern 123 with number</test_message> <test_values> <test_value name="mynum" type="integer">123</test_value> </test_values> </example> </examples> </rule> Currently the test_values are verified for both the value, and the type that the pattern parses, however, none of the values that are used to enrich this message are verified for type. This results in a patterndb that passes validation, yet will fail at the time the pattern matches a log message. comments? -- Evan

2 1