- syslog-ng - lists.balabit.hu

Syslog-ng Not Working properly
by Sumanta Banerjee 15 Jan '24

15 Jan '24

Hi Team, I am trying to configure syslog-ng in one our linux instance to get NGIPS/FMC data via udp connection on its default port (514). I have configured syslog-ng.conf under /etc/syslog-ng and then we have set SE Linux as Permissive. I am using RHEL 8.7 and syslog version 4.0. Apparently all looked good to me however while checking in the destination path that is mentioned I don't see any directory or logfile from for the said udp connection got created. Below is our observation and steps that we executed, can any of you please help me telling where I went wrong or if I am missing something, there is another testing in pipeline that is stalled for this - 1. Define source, destination and log_file in syslog-ng.conf (file attached). 2. Run the below SELinux command - # ausearch -c 'syslog-ng' --raw | audit2allow -M my-syslogng # semodule -X 300 -i my-syslogng.pp 1. Restart syslog-ng service - # systemctl restart syslog-ng.service (no error message received) 1. Check if the service is running - [cid:image002.png@01D99309.5147A9A0] 1. Check if syslog-ng is listening to udp port 514 - [cid:image003.png@01D99309.5147A9A0] 1. Checked and we have incoming data stream from source using the below command - tcpdump -i any -c10 -nn -A port 514 1. I have went through the syslog-ng troubleshooting steps mentioned in the link - https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edi… [cid:image004.png@01D9930B.D0B70940] syslog-ng -Fdev command output is also attached. 1. While running the following command got the below output - # watch '/usr/sbin/syslog-ng-ctl stats | grep "^center"' [cid:image005.png@01D99311.15E497D0] 1. # journaltctl command output (first 500 lines) attached 1. Current SE Linux status : [cid:image006.png@01D9931C.B8765370] 1. Our syslog-ng is logging to /var/log/messages and we are getting this message in /var/log/messages - [cid:image007.png@01D9931E.592725A0] Thanks & Regards, Sumanta Banerjee Splunk Admin | CISO | Aviva Group Tel: +91-8420892593 24x7x365: +44 1603 208 582 sumanta.banerjee(a)aviva.com<mailto:sumanta.banerjee@aviva.com> GlobalCyberSecurityEngineeringTeam(a)aviva.com<mailto:GlobalCyberSecurityEngineeringTeam@aviva.com> www.aviva.com<http://www.aviva.com> Wipro Technologies - SJP2, Bangalore, India [cid:image001.gif@01D99303.20AF3AC0] Aviva: Internal Aviva plc, registered Office: St. Helen's, 1 Undershaft, London EC3P 3DQ. Registered in England No. 02468686. www.aviva.com This message and any attachments may be confidential or legally privileged. If you are not the intended recipient, please telephone or e-mail the sender and delete this message and any attachments from your system. Also, if you are not the intended recipient you must not copy this message or attachments or disclose the contents to any other person. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Aviva.

2 4

Critical Severity Security Advisory: Apache Struts Vulnerability CVE-2023-50164
by Mayekar, PrachiX 16 Dec '23

16 Dec '23

Hi Team, Can you please confirm whether Syslog is affected by this vulnerability: Critical Severity Security Advisory: Apache Struts Vulnerability CVE-2023-50164 Thanks & Regards, Prachi Mayekar ITI-Network Services A Contingent Worker at Intel For assistance, please visit us at https://it.intel.com<https://it.intel.com/>

3 2

Insider 2023-12: compressed HTTP; packages; OpenObserve; duplicates;
by Peter Czanik (pczanik) 14 Dec '23

14 Dec '23

Dear syslog-ng users, This is the 115th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Compressing HTTP traffic in syslog-ng ------------------------------------- Network traffic is expensive in the cloud, and even a single syslog-ng instance can easily saturate the full bandwidth of a network connection. Compressing HTTP traffic was introduced in syslog-ng Version 4.4.0 and depending on your use case, you can cut down on your expenses on your networking or send more logs using the same budget or bandwidth. Development of this feature was done using a locally installed OpenResty web server, and later tested using Sumologic. However, according to the docs it should also work with Splunk, Elasticsearch, and many other services accessible using the http() destination. https://www.syslog-ng.com/community/b/blog/posts/compressing-http-traffic-i… Why is a feature not available in the syslog-ng package? -------------------------------------------------------- You can read about many interesting syslog-ng features in my blogs. However, it can happen that when you want to try them at home, you fail because the feature is missing. How can you solve such problems? In this blog, I discuss some of the possible solutions from installing sub-packages through using unofficial repositories, to upgrading your OS. This blog focuses on RPM packages for openSUSE / SLES, Fedora / RHEL, and FreeBSD, because these are the packages I know – I am their maintainer. However, these problems and their solutions also apply to Debian / Ubuntu, and other Linux distributions. https://www.syslog-ng.com/community/b/blog/posts/why-is-a-feature-not-avail… Sending logs to OpenObserve using syslog-ng ------------------------------------------- A question was asked if syslog-ng can send logs to OpenObserve. It has an Elasticsearch compatible API for log ingestion, but syslog-ng is not mentioned in the documentation. My plan was to document how to modify the syslog-ng elasticsearch-http() destination, based on API documentation. However, as it turned out, OpenObserve has a ready to use syslog-ng configuration example in the web UI. https://www.syslog-ng.com/community/b/blog/posts/sending-logs-to-openobserv… Removing duplicate messages with syslog-ng in a redundant logging environment ----------------------------- Creating highly available servers is difficult. Sending logs to two (or more) servers and hoping that at least one of them can collect logs any time is a lot easier. Since network traffic and storage are cheap, redundancy is usually not a problem. However, once you also want to analyze your log messages using a SIEM or other software, you do not want duplicate log messages. In this blog, I show you how you can make sure that each log message produced on your network reaches your SIEM system only once. Sending logs to a SIEM is expensive: both resource usage and licensing costs are much higher than at the log management level. https://www.syslog-ng.com/community/b/blog/posts/removing-duplicate-message… WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

syslog-ng 4.5.0
by Attila Szakács 30 Nov '23

30 Nov '23

Dear syslog-ng users, We are pleased to announce the 4.5.0 version of syslog-ng, which has been released and is now available on GitHub: https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-4.5.0 Packages <https://github.com/syslog-ng/syslog-ng#installation-from-binaries> are also available for various platforms. The AxoSyslog project provides cloud-ready container images <https://axoflow.com/cloud-ready-syslog-ng-images/>, Helm charts <https://axoflow.com/axosyslog-log-collection-for-kubernetes/>. 4.5.0 Read Axoflow's blog post <https://axoflow.com/axosyslog-release-4-5/> for more details. You can read more about the new features in the AxoSyslog documentation <https://axoflow.com/docs/axosyslog-core/>. HighlightsSending log messages to OpenObserve The openobserve-log() destination feeds OpenObserve via the JSON API <https://openobserve.ai/docs/api/ingestion/logs/json/>. Example config: openobserve-log( url("http://openobserve-endpoint") port(5080) stream("default") user("root(a)example.com") password("V2tsn88GhdNTKxaS") ); (#4698 <https://github.com/syslog-ng/syslog-ng/pull/4698>) Sending messages to Google Pub/Sub The google-pubsub() destination feeds Google Pub/Sub via the HTTP REST API <https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.topics/publ…> . Example config: google-pubsub( project("syslog-ng-project") topic("syslog-ng-topic") auth( service-account( key("/path/to/service-account-key.json") ) ) ); See the Google Pub/Sub documentation <https://cloud.google.com/pubsub/docs/building-pubsub-messaging-system> to learn more about configuring a service account. (#4651 <https://github.com/syslog-ng/syslog-ng/pull/4651>) Parsing PostgreSQL logs The postgresql-csvlog-parser(): add a new parser to process CSV log formatted by PostgreSQL ( https://www.postgresql.org/docs/current/runtime-config-logging.html) The CSV format is extracted into a set of name-value pairs. (#4586 <https://github.com/syslog-ng/syslog-ng/pull/4586>) Features - http(): Added support for using templates in the url() option. In syslog-ng a template can only be resolved on a single message, as the same template might have different resolutions on different messages. A http batch consists of multiple messages, so it is not trivial to decide which message should be used for the resolution. When batching is enabled and multiple workers are configured it is important to only batch messages which generate identical URLs. In this scenario one must set the worker-partition-key() option with a template that contains all the templates used in the url() option, otherwise messages will be mixed. For security reasons, all the templated contents in the url() option are getting URL encoded automatically. Also the following parts of the url cannot be templated: - scheme - host - port - user - password (#4663 <https://github.com/syslog-ng/syslog-ng/pull/4663>) - $TRANSPORT: this is a new name-value pair that syslog-ng populates automatically. It indicates the "transport" mechanism used to retrieve/receive the message. It is up to the source driver to determine the value. Currently the following values were implemented: BSD syslog drivers: tcp(), udp() & network() - rfc3164+tls - rfc3164+tcp - rfc3164+udp - rfc3164+proxied-tls - rfc3164+<custom logproto like altp> UNIX domain drivers: unix-dgram(), unix-stream() - unix-stream - unix-dgram RFC5424 style syslog: syslog(): - rfc5426: syslog over udp - rfc5425: syslog over tls - rfc6587: syslog over tcp - rfc5424+<custom logproto like altp>: syslog over a logproto plugin Other drivers: - otlp: otel() driver - mqtt: mqtt() driver - hypr-api: hypr-audit-source() driver $IP_PROTO: indicate the IP protocol version used to retrieve/receive the message. Contains either "4" to indicate IPv4 and "6" to indicate IPv6. (#4673 <https://github.com/syslog-ng/syslog-ng/pull/4673>) - network() and syslog() drivers: Added ignore-validity-period as a new flag to ssl-options(). By specifying ignore-validity-period, you can ignore the validity periods of certificates during the certificate validation process. (#4642 <https://github.com/syslog-ng/syslog-ng/pull/4642>) - tls() in udp()/tcp()/network() and syslog() drivers: add support for a new http() compatible ssl-version() option. This makes the TLS related options for http() and other syslog-like drivers more similar. This requires OpenSSL 1.1.0. (#4682 <https://github.com/syslog-ng/syslog-ng/pull/4682>) - cloud-auth(): Added a new plugin for drivers, which implements different cloud related authentications. Currently the only supported authentication is GCP's Service Account <https://cloud.google.com/iam/docs/service-account-overview> for the http() destination. Example config: http( cloud-auth( gcp( service-account( key("/path/to/service-account-key.json") audience("https://pubsub.googleapis.com/google.pubsub.v1.Publisher") ) ) ) ); (#4651 <https://github.com/syslog-ng/syslog-ng/pull/4651>) - csv-parser(): allow parsing the extracted values into matches ($1, $2, $3 ...) by omitting the columns() parameter, which normally specifies the column names. (#4678 <https://github.com/syslog-ng/syslog-ng/pull/4678>) - --check-startup: a new command line option for syslog-ng along with the existing --syntax-only. This new option will do a complete configuration initialization and then exit with exit code indicating the result. Since this also initializes things like network listeners, it will probably *not* work when there is another syslog-ng instance running in the background. The recommended use of this option is a dedicated config check container, as explained in #4592 <https://github.com/syslog-ng/syslog-ng/issues/4592>. (#4646 <https://github.com/syslog-ng/syslog-ng/pull/4646>) Bugfixes - s3: Fixed an ImportError. ImportError: cannot import name 'SharedBool' from 'syslogng.modules.s3.s3_object' (#4700 <https://github.com/syslog-ng/syslog-ng/pull/4700>) - loki(): fixed mixing non-related label values (#4713 <https://github.com/syslog-ng/syslog-ng/pull/4713>) - type hinting: Parsing and casting fractions are now done locale independently. (#4702 <https://github.com/syslog-ng/syslog-ng/pull/4702>) - metrics-probe(): Fixed a crash. This crash occurred when a metrics-probe() instance was used in multiple source threads, like a network() source with multiple connections. (#4685 <https://github.com/syslog-ng/syslog-ng/pull/4685>) - flags() argument to various drivers: fix a potential crash in case a flag with at least 32 characters is used. No such flag is defined by syslog-ng, so the only way to trigger the crash is to use an invalid configuration file. (#4689 <https://github.com/syslog-ng/syslog-ng/pull/4689>) - Fix $PROTO value for transport(tls) connections, previously it was set to "0" while in reality these are tcp connections (e.g. "6"). Fix how syslog-ng sets $HOST for V4-mapped addresses in case of IPv6 source drivers (e.g. udp6()/tcp6() or when using ip-protocol(6) for tcp()/udp() ). Previously V4-mapped addresses would be represented as "::ffff:<ipv4 address>". This is not wrong per-se, but would potentially cause the same host to be represented in multiple ways. With the fix, syslog-ng would just use "<ipv4 address>" in these cases. (#4673 <https://github.com/syslog-ng/syslog-ng/pull/4673>) - db-parser(): support nested match characters in @QSTRING@ pattern parser (#4717 <https://github.com/syslog-ng/syslog-ng/pull/4717>) Other changes - LogSource and LogFetcher: additional documentation was added to these Python classes to cover explicit source-side batching functionalities (e.g. the auto_close_batch attribute and the close_batch() method). (#4673 <https://github.com/syslog-ng/syslog-ng/pull/4673>) - rate-limit(): Renamed the template() option to key(), which better communicates the intention. (#4679 <https://github.com/syslog-ng/syslog-ng/pull/4679>) - templates: The template-escape() option now only escapes the top-level template function. Before syslog-ng 4.5.0 if you had embedded template functions, the template-escape(yes) setting escaped the output of each template function, so the parent template function received an already escaped string. This was never the intention of the template-escape() option. Although this is a breaking change, we do not except anyone having a config that is affected. If you have such a config, make sure to follow-up this change. If you need help with it, feel free to open an issue or discussion on GitHub, or contact us on the Axoflow Discord server. (#4666 <https://github.com/syslog-ng/syslog-ng/pull/4666>) - loki(): The timestamp() option now supports quoted strings. The valid values are the following, with or without quotes, case insensitive: - "current" - "received" - "msg" (#4688 <https://github.com/syslog-ng/syslog-ng/pull/4688>) syslog-ng Discord For a bit more interactive discussion, join our Discord server: [image: Axoflow Discord Server] <https://discord.gg/E65kP9aZGm> Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Attila Szakacs, Balazs Scheidler, Cedric Arickx, Fabrice Fontaine, Hofi, László Várady, Romain Tartière, Szilard Parrag, yashmathne

1 0

syslog-ng vs pcre2 without jit vs disable-jit config feature
by "Tóth Attila" 27 Nov '23

27 Nov '23

I'm using syslog-ng-4.4.0 on a Gentoo system, that also employs PaX hardening. Due to the necessity to elevate restrictions on pcre2 with jit enabled, I keep it disabled for this particular installation. Syslog-ng emits error messages during startup complaining about pcre2 and jit. I had studied the manual and found the disable-jit feature. https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edi… Maybe I'm using a wrong syntax, but syslog-ng doesn't seem to respect the option. Commenting out the jit feature in the source code works, but it would be much more comfortable to find the proper way to disable jit. Are there any other who managed to use disable-jit in action? Are there any tips or tricks aboutv what to pay attention on? Thx: Dw. -- dr Tóth Attila, Radiológus, 06-20-825-8057 Attila Toth MD, Radiologist, +36-20-825-8057

2 6

New install
by Klimek, Bryan J. 16 Nov '23

16 Nov '23

We've been using Syslog-ng for a long time. Our current deployment is on Red Hat 7 servers that we built 6 years ago. It's time to start a refresh of our Syslog-ng environment. Our plan is to use Red Hat 9 servers. Can you give me a jumping off point on where to obtain installation instructions and how to obtain the installation media. Thank you. Bryan Klimek IT Systems Architect I EIS Application Hosting Services E-mail: klimek.bryan(a)mayo.edu<mailto:klimek.bryan@mayo.edu>

2 1

Insider 2023-11: Splunk; configuration; journald;
by Peter Czanik (pczanik) 14 Nov '23

14 Nov '23

Dear syslog-ng users, This is the 114th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Sending logs to Splunk using syslog-ng -------------------------------------- There are many ways you can collect log messages using syslog-ng and forward them to Splunk. In this blog I collect the history of Splunk support in syslog-ng, and the advantages and disadvantages of various solutions, both open source and commercial. https://www.syslog-ng.com/community/b/blog/posts/sending-logs-to-splunk-usi… Developing a syslog-ng configuration ------------------------------------ This year I started publishing a syslog-ng tutorial series both on my blog and on YouTube: https://peter.czanik.hu/posts/syslog-ng-tutorial-toc/ And while the series was praised as the best possible introduction to syslog-ng, viewers also mentioned that one interesting element is missing from it: namely, it does not tell users how to develop a syslog-ng configuration. So, in this blog, learn how to develop a syslog-ng configuration from the ground up! I will explain not just the end result, but also the process and the steps to take to develop a configuration. It starts with a single source and destination, then concludes with a conditional log path and sending parsed and enriched logs to Elasticsearch (or a compatible document store). https://www.syslog-ng.com/community/b/blog/posts/developing-a-syslog-ng-con… Systemd-journald vs. syslog-ng ------------------------------ Even if most people ask me to compare systemd-journald vs. syslog-ng, I would say that they complement each other. Systemd-journald excels at collecting local log messages, including those of various system services. The focus of syslog-ng is on central log collection and forwarding the logs to a wide variety of destinations after processing and filtering. Combining the two gives you the most flexibility. https://www.syslog-ng.com/community/b/blog/posts/systemd-journald-vs-syslog… WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Custom script with map-value-pairs(pair(<script>))
by Faisal Chishti 18 Oct '23

18 Oct '23

Hi, I am trying to capture some custom information using a simple shell script. I am trying below but getting an error. Below is my parser, it works fine without the line that has custom_script_output. How do I get this to work? parser p_cr_syslog { # Convert the LEVEL field to uppercase. map-value-pairs(pair("jcnr.header.logLevel", "$(uppercase $LEVEL)")); map-value-pairs(pair("jc.header.custom_script_output", "$(script("/path/to/script.sh"))")); syslog-parser(flags(syslog-protocol) template("${MESSAGE}")); map-value-pairs( pair("jc.header.nodeName", "$HOST") pair("jc.header.eventDateTime", "$R_ISODATE") pair("jc.header.notificationType", "$MSGID") pair("jc.body", "$MSG") pair("jc.header.program", "$PROGRAM") ); # Log the output of the script. action { file { path "/var/log/syslog"; message "$jc.header.custom_script_output"; } } } Thanks in advance. -- Regards, Faisal

2 1

Re: [syslog-ng] Vulnerability making News - HTTP/2 Rapid Reset DDoS CVE-2023-44487
by Mayekar, PrachiX 16 Oct '23

16 Oct '23

Hi Team, Are syslog products vulnerable to this vulnerability ? Need to know if Syslog is affected: CVE-2023-44487 is a vulnerability in the HTTP/2 protocol that was recently used to launch DDoS attacks. The vulnerability allows for denial of service (DoS) because request cancellation can reset many streams quickly. https://www.bleepingcomputer.com/news/security/new-http-2-rapid-reset-zero-… Thanks & Regards, Prachi Mayekar ITI-Network Services A Contingent Worker at Intel For assistance, please visit us at https://it.intel.com<https://it.intel.com/>

2 2

Insider 2023-10: contribute; parallelize; compatibility;
by Peter Czanik (pczanik) 12 Oct '23

12 Oct '23

Dear syslog-ng users, This is the 113th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Why contribute to syslog-ng upstream? ------------------------------------- One of the returning questions I received recently: why contribute to the syslog-ng upstream? I guess it is a question many open-source projects receive regularly. There are many generic answers. Here I would like to focus more on syslog-ng, focusing on various parts of it. Of course, the generic answers also apply. Syslog-ng is an open-source project, free to use, modify, and extend. By contributing, you can give something back and improve syslog-ng for everyone. You do not have to be a developer in order to contribute: bug reports, configuration examples, PatternDB rules are all very useful contributions. https://www.syslog-ng.com/community/b/blog/posts/why-contribute-to-syslog-n… Accelerating single TCP connections in syslog-ng: parallelize() --------------------------------------------------------------- One of the highlights of the syslog-ng 4.3.0 release is parallelize(). Normally, syslog-ng processes incoming messages from a TCP connection in a single thread. While this works fine with many connections, it is a bottleneck when using a single or very few high-traffic connections. Using parallelize() allows syslog-ng to process log messages from a single high-traffic TCP connection in multiple threads, thus increasing processing performance on multi-core machines. https://www.syslog-ng.com/community/b/blog/posts/accelerating-single-tcp-co… Backward compatibility in syslog-ng by using the version number in syslog-ng.conf --------------------------------------------------------- Many users are annoyed by the version number included in the syslog-ng configuration. However, it ensures backward compatibility in syslog-ng. It is especially useful when updating to syslog-ng 4 from version 3, but also when updating within the same major version. https://www.syslog-ng.com/community/b/blog/posts/backward-compatibility-in-… WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0