- syslog-ng - lists.balabit.hu

Send GELF log messages to Syslog-NG server
by Roberto Carna 21 Aug '24

21 Aug '24

Hi people, a Graylog server is sending GELF log messages to my Syslog-NG server listening on the UDP/514 port. But when I was reading the message, I noticed it's in binary format. Maybe syslog-ng service converts GELF to binary, because the log is not in syslog format. Is it possible to make Syslog-NG listen on the UDP/12201 port for incoming GELF messages, in addition to UDP/514 syslog port ? Or what can I do to accept GELF messages in the Syslog-NG server and then read them in ASCII format ? Thanks a lot!

3 2

The syslog-ng Insider 2024-08: 4.8.0 release; Prometheus; Amazon Linux
by Peter Czanik (pczanik) 15 Aug '24

15 Aug '24

Dear syslog-ng users, This is the 122nd issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Version 4.8.0 of syslog-ng improves FreeBSD and MacOS support ------------------------------------------------------------- Recently One Identity released version 4.8.0 of its open-source log management application. Learn about some of the new features and bug fixes: why upgrade to the latest syslog-ng version, not only on FreeBSD :-) https://www.syslog-ng.com/community/b/blog/posts/version-4-8-0-of-syslog-ng… syslog-ng Prometheus exporter ----------------------------- Prometheus is an open-source monitoring system that collects metrics from your hosts and applications, allowing you to visualize and alert on them. The syslog-ng Prometheus exporter allows you to export syslog-ng statistics, so that Prometheus can collect it. While an implementation in Go has been available for years on GitHub (for more information, see this blog entry), that solution uses the old syslog-ng statistics interface. And while that Go-based implementation still works, syslog-ng 4.1 introduced a new interface that provides not just more information than the previous statistics interface, but does so in a Prometheus-friendly format. The information available through the new interface has been growing ever since. The syslog-ng Prometheus exporter is implemented in Python. It also uses the new statistics interface, making new fields automatically available when added. https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-prometheus-expor… Experimental syslog-ng packages for Amazon Linux 2023 ----------------------------------------------------- Last year, I received many requests about syslog-ng for Amazon Linux 2023, but I could not find an easy way to create syslog-ng packages. Recently, however, I found that Fedora Copr supports building packages for Amazon Linux 2023. So, with a little bit of experimentation, I got a cut down version of syslog-ng compiled. https://www.syslog-ng.com/community/b/blog/posts/experimental-syslog-ng-pac… WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Version 4.8.0 of syslog-ng is now available
by Peter Czanik (pczanik) 24 Jul '24

24 Jul '24

We have new documentation You can find our new up-to-date documentation in the new Administration Guide at syslog-ng.github.io<https://syslog-ng.github.io/admin-guide/README>. Highlights Default config version in configuration files cfg: allow usage of current in config @version by default if it is not presented This change allows syslog-ng to start even if the @version information is not present in the configuration file and treats the version as the latest in that case. NOTE: syslog-ng will still raise a warning if @version is not present. Please use @version: current to confirm the intention of always using the latest version and silence the warning. (#5030<https://github.com/syslog-ng/syslog-ng/pull/5030>) BSD directory monitoring with kqueue directory-monitor: Added a kqueue based directory monitor implementation. wildcard-file() sources are using a directory monitor<https://syslog-ng.github.io/admin-guide/060_Sources/020_File/001_File_follo…> as well to aid detection of changes in the folders of the followed files. The new kqueue-based directory monitor uses far fewer resources than the poll based version on BSD-based systems. (#5022<https://github.com/syslog-ng/syslog-ng/pull/5022>) See more at the new syslog-ng documentation<https://syslog-ng.github.io/admin-guide/060_Sources/020_File/001_File_follo…>. Wildcard file source fine-tuning wildcard-file(): Added a dedicated monitor_freq option to control the poll frequency of the change detection in the directories separately when the poll method is selected via the monitor-method() option. The monitor-method() option controls only the change detection method in the directories, not the following of the file changes, and if poll is the selected method the frequency must not necessarily be the same, e.g. if the (earlier) commonly used follow-freq() is set to 0 for switching to the poll_fd_events method for file content change detection, that also might be meant a directory change poll with zero delays (if monitor-method() was set to poll as well), and that could cause a heavy CPU load unnecessarily. (#4998<https://github.com/syslog-ng/syslog-ng/pull/4998>) See more at the new syslog-ng documentation<https://syslog-ng.github.io/admin-guide/060_Sources/020_File/001_File_follo…>. Features * s3(): Introduced server side encryption related options server-side-encryption() and kms-key() can be used to configure encryption. Currently only server-side-encryption("aws:kms") is supported. The kms-key() should be: To be able to use the aws:kms encryption the AWS Role or User has to have the following permissions on the given key: Check this<https://repost.aws/knowledge-center/s3-large-file-encryption-kms-key> page on why the kms:Decrypt is mandatory. Example config: destination d_s3 { s3( bucket("log-archive-bucket") object-key("logs/syslog") server-side-encryption("aws:kms") kms-key("alias/log-archive") ); }; See the S3<https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingKMSEncryption.ht…> documentation for more details. (#4993<https://github.com/syslog-ng/syslog-ng/pull/4993>) * an ID of a key * an alias of a key, but in that case you have to add the alias/prefix * an ARN of a key * kms:Decrypt * kms:Encrypt * kms:GenerateDataKey * filter: Added numerical severity settings. The level filter option now accepts numerical values similar to facility. Example config: filter f_severity { level(4) }; This is equivalent to filter f_severity { level("warning") }; For more information, consult the documentation<https://syslog-ng.github.io/admin-guide/080_Log/030_Filters/005_Filter_func…>. (#5016<https://github.com/syslog-ng/syslog-ng/pull/5016>) * opentelemetry(), loki(), bigquery(): Added headers() option Enables adding headers to RPC calls. Example config: opentelemetry( ... headers( "my_header" = "my_value" ) ); (#5012<https://github.com/syslog-ng/syslog-ng/pull/5012>) * Added new proxy options to the syslog() and network() source drivers The transport(proxied-tcp), transport(proxied-tls), and transport(proxied-tls-passthrough) options are now available when configuring syslog() and network() sources. (#4544<https://github.com/syslog-ng/syslog-ng/pull/4544>) Bugfixes * disk-buffer(): fix crash when pipeline initialization fails log_queue_disk_free_method: assertion failed: (!qdisk_started(self->qdisk)) (#4994<https://github.com/syslog-ng/syslog-ng/pull/4994>) * rate-limit(): Fixed a crash which occured on a config parse failure. (#5033<https://github.com/syslog-ng/syslog-ng/pull/5033>) * Fixed potential null pointer deref issues (#5035<https://github.com/syslog-ng/syslog-ng/pull/5035>) * wildcard-file(): fix a crash and detection of file delete/move when using ivykis poll events Two issues were fixed * Fixed a crash in log pipe queue during file deletion and EOF detection (#4989<https://github.com/syslog-ng/syslog-ng/issues/4989>) The crash was caused by a concurrency issue in the EOF and file deletion detection when using a wildcard-file() source. If a file is written after being deleted (e.g. with an application keeping the file open), or if these events happen concurrently, the file state change poller mechanism might schedule another read cycle even though the file has already been marked as fully read and deleted. To prevent this re-scheduling between these two checks, the following changes have been made: Instead of maintaining an internal EOF state in the WildcardFileReader, when a file deletion notification is received, the poller will be signaled to stop after reaching the next EOF. Only after both conditions are set the reader instance will be deleted. * Fixed the file deletion and removal detection when the file-reader uses poll_fd_events to follow file changes, which were mishandled. For example, files that were moved or deleted (such as those rolled by a log-rotator) were read to the end but never read again if they were not touched anymore, therefore switching to the new file never happened. (#4998<https://github.com/syslog-ng/syslog-ng/pull/4998>) * syslog-ng-ctl query: fix showing Prometheus metrics as unnamed values none.value=726685 (#4995<https://github.com/syslog-ng/syslog-ng/pull/4995>) * macros: Fixed a bug which always set certain macros to string type The affected macros are $PROGRAM, $HOST and $MESSAGE. (#5024<https://github.com/syslog-ng/syslog-ng/pull/5024>) * syslog-ng-ctl query: show timestamps and fix g_pattern_spec_match_string assert (#4995<https://github.com/syslog-ng/syslog-ng/pull/4995>) * csv-parser(): fix escape-backslash-with-sequences dialect on ARM csv-parser() produced invalid output on platforms where char is an unsigned type. (#4947<https://github.com/syslog-ng/syslog-ng/pull/4947>) Other changes * bigquery(), loki(), opentelemetry(), cloud-auth(): C++ modules can be compiled with clang Compiling and using these C++ modules are now easier on FreeBSD and macOS. (#4933<https://github.com/syslog-ng/syslog-ng/pull/4933>) * syslog-ng-ctl: do not show orphan metrics for stats prometheus As the stats prometheus command is intended to be used to forward metrics to Prometheus or any other time-series database, displaying orphaned metrics should be avoided in order not to insert new data points when a given metric is no longer alive. In case you are interested in the last known value of orphaned counters, use the stats or query subcommands. (#4921<https://github.com/syslog-ng/syslog-ng/pull/4921>) * s3(): new metric syslogng_output_event_bytes_total (#4958<https://github.com/syslog-ng/syslog-ng/pull/4958>) * multiline-options: Allow multi_line_timeout to be set to a non-integer value. Since multi_line_timeout is suggested to be set as a multiple of follow-freq, and follow-freq can be much smaller than one second, it makes sense to allow this value to be a non-integer as well. (#5002<https://github.com/syslog-ng/syslog-ng/pull/5002>) * packages/dbld: add support for Ubuntu 24.04 (Noble Numbat) (#4925<https://github.com/syslog-ng/syslog-ng/pull/4925>) * packages/dbld: add support for AlmaLinux 9 (#5009<https://github.com/syslog-ng/syslog-ng/pull/5009>) * packages/dbld: added support for Fedora Rawhide and CentOS Stream 9 as testing platforms (#5009<https://github.com/syslog-ng/syslog-ng/pull/5009>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Alex Becker, Andras Mitzki, Arpad Kunszt, Attila Szakacs, Balazs Scheidler, Bálint Horváth, Dmitry Levin, Hofi, Ilya Kheifets, joohoonmaeng, ktzsolt, László Várady, Mate Ory, Natanael Copa, Peter Czanik, qsunchiu, Robert Fekete, shifter, Szilárd Parrag, Tamas Pal, Wolfram Joost Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

The syslog-ng Insider 2024-07: logging trends; multiple platforms; sudo; retired platforms
by Peter Czanik (pczanik) 11 Jul '24

11 Jul '24

Dear syslog-ng users, This is the 121st issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Trends in open source logging ----------------------------- Recently I was asked by Chas Clawson of Sumo Logic what I think about open source log management and telemetry. Needless to say, I also talked about syslog_ng, and in the end I showed a couple of simple syslog-ng configurations sending logs to Sumo Logic. https://www.sumologic.com/blog/securing-open-source-infrastructure-log-all/ Using syslog-ng on multiple platforms ------------------------------------- Your favorite Linux distribution is X. You test everything there. However, your colleagues use distro Y, and another team distro Z. Nightmares start here: the same commands install a different set of syslog-ng features, configuration defaults and use different object names in the default configuration. I ran into these problems while working with Gábor Samu on his HPC logging blog. >From this blog you can learn about some of the main differences in packaging and configuration of syslog-ng in various Linux distributions and FreeBSD, and how to recognize these when configuring syslog-ng on a different platform. https://www.syslog-ng.com/community/b/blog/posts/using-syslog-ng-on-multipl… Working with sudo’s json_compact logs in syslog-ng -------------------------------------------------- Version 1.9.16 of sudo will feature a new option for logging: json_compact. Why is this important? This new format can easily be read and parsed by a log management software, like syslog-ng. Note that in this blog I am showing you a sudo feature which has not yet been released officially. You have to compile sudo yourself. By all means, if you have any other application writing JSON-formatted log messages, you can apply most of what you read here with slight modifications. https://www.syslog-ng.com/community/b/blog/posts/working-with-sudo-s-json_5… syslog-ng OSE 4.7.1 retires some old platforms ---------------------------------------------- Version 4.7.1 of syslog-ng OSE was released recently. It brings many smaller enhancements to metrics, OpenTelemetry and other features, while it also retires support for some older platforms. https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-4-7-1-retires-so… WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Help parsing logs
by John Norton 28 Jun '24

28 Jun '24

Hi All, I have a panorama system that sends logs from several devices (dvc_name). What I want to do is have it write to files based on the dvc_name. The dvc_name is in the following message as "ip-10-37-12-142". These are not static as they are spun up as needed and the dvc_name may change. Can you help me figure out how to create a parser or filter to take the following message, where it will read in the dvc_name "ip-10-37-12-142" and write logs that include that dvc_name to a separate file (and other dvc_names it sees in messages to their own file?: *Jun 27 19:08:15 TERL-PANORAMA-01.times.com <http://terl-panorama-01.times.com/> 1,2024/06/27 15:08:14,005655000463255,SYSTEM,dhcp,2817,2024/06/27 15:08:09,,if-renew-trigger,,0,0,general,informational,"DHCP RENEW: interface eth0, ip 10.37.12.142 netmask 255.255.255.224 dhcp server: 10.37.12.129",7383625404611793457,0x8000000000000000,0,0,0,0,,ip-10-37-12-142,0,0,2024-06-27T15:08:09.224-04:00* I've tried a few different things but can't figure out how to do it dynamically. This rewrite worked, but it requires a static entry which won't scale and all potential dvc_names cannot be known: rewrite r_panorama { set("ip-10-37-12-142", value("HOST") condition(message(".*ip-10-37-12-142"))); }; I've also tried creating a filter, but it doesn't actually store what the match finds in a macro that I can replace something like $HOST with. filter f_panorama { match("ip-\d+-\d+-\d+-\d+" value("MESSAGE")); }; I was also thinking that I could do something with the csv parser, but it didn't work like I was expecting, but am thinking a parser might work if I could get it right (I think partially it doesn't work because it is a syslog message and the first part of the message interferes with the csv parsing, not sure): parser p_panorama { csv-parser( columns("future_use1","receive_time","serial_number","log_type","log_subtype","version","generated_time","vsys","event_id","object","future_use3","future_use4","module","severity","description","sequence_number","action_flags","devicegroup_level1","devicegroup_level2","devicegroup_level3","devicegroup_level4","vsys_name","dvc_name") delimiters(",") quote-pairs('""[]') ); }; destination d_panorama {file("`mypath`/panorama/$dvc_name/$dvc_name.log"); }; log { source(s_panother); parser(p_panorama); destination(d_panorama); }; Is this something someone could help me with? Thank you, John

2 1

The syslog-ng Insider 2024-05: documentation; grouping-by(); PAM Essentials; health
by Peter Czanik (pczanik) 09 May '24

09 May '24

Dear syslog-ng users, This is the 120th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS The official syslog-ng OSE documentation got a new look ------------------------------------------------------- The syslog-ng Administration Guide received a new look and easier navigation. Not only that, but it is also up-to-date now. Besides, there are now contributor guides available both for the documentation and for syslog-ng developers. The admin guide is available at: https://syslog-ng.github.io/admin-guide/README You can reach all syslog-ng OSE-related documentation at: https://syslog-ng.github.io/ If you find any issues, pull requests and problem reports are welcome. The contributor guide describes how you can fix / extend the documentation. You can report issues at: https://github.com/syslog-ng/syslog-ng.github.io/issues Aggregating messages in syslog-ng using grouping-by() ----------------------------------------------------- Sometimes you have many log messages from an app, but none of them have the exact content you need. This is where the grouping-by() parser of syslog-ng can help. It allows you to aggregate information from multiple log messages into a single message. In this blog, I will show you how to parse sshd logs using the patterndb parser of syslog-ng, and then create an aggregate message from the opening and closing log message using grouping-by. https://www.syslog-ng.com/community/b/blog/posts/aggregating-messages-in-sy… Alerting on One Identity Cloud PAM Essentials logs using syslog-ng -------------------------------------------------- One Identity Cloud PAM Essentials is the latest security product by One Identity. It provides asset management as well as secure and monitored remote access for One Identity Cloud users to hosts on their local network. I had a chance to test PAM Essentials while still in development. While there, I also integrated it with syslog-ng. >From my previous blog, you could learn what PAM Essentials is, and how you can collect its logs using syslog-ng. This blog will show you how to work with the collected log messages and create alerts when somebody connects to a host on your local network using PAM Essentials. https://www.syslog-ng.com/community/b/blog/posts/alerting-on-one-identity-c… The syslog-ng health check -------------------------- Version 4.2 of syslog-ng introduced a healthcheck option to syslog-ng-ctl. It prints three syslog-ng-related metrics on screen – if it can reach syslog-ng, that is. You can use it from scripts to monitor the health of syslog-ng. https://www.syslog-ng.com/community/b/blog/posts/the-syslog-ng-health-check WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

we need to know if Syslog supports any of these virtualization solution - Harvester, CaaS or HyperV. If not, what other virtualization solution is supported by Syslog.
by Mayekar, PrachiX 24 Apr '24

24 Apr '24

We need to know which virtualization solution is supported by syslog - Harvester CaaS or HyperV ? Thanks & Regards, Prachi Mayekar ITI-Network Services A Contingent Worker at Intel For assistance, please visit us at https://it.intel.com<https://it.intel.com/>

3 2

syslog-ng 4.7.1
by László Várady 19 Apr '24

19 Apr '24

Dear syslog-ng users, We are pleased to announce the 4.7.1 version of syslog-ng, which has been released and is now available on GitHub: https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-4.7.1 Packages <https://github.com/syslog-ng/syslog-ng#installation-from-binaries> are also available for various platforms. The AxoSyslog project provides cloud-ready container images <https://axoflow.com/cloud-ready-syslog-ng-images/>, Helm charts <https://axoflow.com/axosyslog-log-collection-for-kubernetes/>. 4.7.1 *This is the combination of the news entries of 4.7.0 and 4.7.1.* *4.7.1 hotfixed two crashes related to configuration reload.* Read Axoflow's blog post <https://axoflow.com/axosyslog-release-4-7/> for more details. You can read more about the new features in the AxoSyslog documentation <https://axoflow.com/docs/axosyslog-core/>. HighlightsCollecting Jellyfin logs The new jellyfin() source, reads Jellyfin logs from its log file output. Example minimal config: source s_jellyfin { jellyfin( base-dir("/path/to/my/jellyfin/root/log/dir") filename-pattern("log_*.log") ); }; For more details about Jellyfin logging, see: - https://jellyfin.org/docs/general/administration/configuration/#main-config… - https://jellyfin.org/docs/general/administration/configuration/#log-directo… As the jellyfin() source is based on a wildcard-file() source, all of the wildcard-file() source options are applicable, too. (#4802 <https://github.com/syslog-ng/syslog-ng/pull/4802>) Collecting *arr logs Use the newly added *arr() sources to read various *arr logs: - lidarr() - prowlarr() - radarr() - readarr() - sonarr() - whisparr() Example minimal config: source s_radarr { radarr( dir("/path/to/my/radarr/log/dir") ); }; The logging module is stored in the <prefix><module> name-value pair, for example: .radarr.module => ImportListSyncService. The prefix can be modified with the prefix() option. (#4803 <https://github.com/syslog-ng/syslog-ng/pull/4803>) Features - opentelemetry(), syslog-ng-otlp() source: Added concurrent-requests() option. This option configures the maximal number of in-flight gRPC requests per worker. Setting this value to the range of 10s or 100s is recommended when there are a high number of clients sending simultaneously. Ideally, workers() * concurrent-requests() should be greater or equal to the number of clients, but this can increase the memory usage. (#4827 <https://github.com/syslog-ng/syslog-ng/pull/4827>) - loki(): Support multi-tenancy with the new tenant-id() option (#4812 <https://github.com/syslog-ng/syslog-ng/pull/4812>) - s3(): Added support for authentication from environment. The access-key() and secret-key() options are now optional, which makes it possible to use authentication methods originated from the environment, e.g. AWS_... environment variables or credentials files from the ~/.aws/ directory. For more info, see: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.h… (#4881 <https://github.com/syslog-ng/syslog-ng/pull/4881>) - gRPC based drivers: Added channel-args() option. Affected drivers are: - bigquery() destination - loki() destination - opentelemetry() source and destination - syslog-ng-otlp() source and destination The channel-args() option accepts name-value pairs and sets channel arguments defined in https://grpc.github.io/grpc/core/group__grpc__arg__keys.html Example config: opentelemetry( channel-args( "grpc.loadreporting" => 1 "grpc.minimal_stack" => 0 ) ); (#4827 <https://github.com/syslog-ng/syslog-ng/pull/4827>) - ${TRANSPORT} macro: Added support for locally created logs. New values are: - "local+unix-stream" - "local+unix-dgram" - "local+file" - "local+pipe" - "local+program" - "local+devkmsg" - "local+journal" - "local+afstreams" - "local+openbsd" (#4777 <https://github.com/syslog-ng/syslog-ng/pull/4777>) - tags: Added new built-in tags that help identifying parse errors. New tags are: - "message.utf8_sanitized" - "message.parse_error" - "syslog.missing_pri" - "syslog.missing_timestamp" - "syslog.invalid_hostname" - "syslog.unexpected_framing" - "syslog.rfc3164_missing_header" - "syslog.rfc5424_unquoted_sdata_value" (#4804 <https://github.com/syslog-ng/syslog-ng/pull/4804>) - mqtt() source: Added ${MQTT_TOPIC} name-value pair. It is useful for the cases where topic() contains wildcards. Example config: log { source { mqtt(topic("#")); }; destination { stdout(template("${MQTT_TOPIC} - ${MESSAGE}\n")); }; }; (#4824 <https://github.com/syslog-ng/syslog-ng/pull/4824>) - template(): Added a new template function: $(tags-head) This template function accepts multiple tag names, and returns the first one that is set. Example config: # resolves to "bar" if "bar" tag is set, but "foo" is not template("$(tags-head foo bar baz)") (#4804 <https://github.com/syslog-ng/syslog-ng/pull/4804>) - s3(): Use default AWS URL if url() is not set. (#4813 <https://github.com/syslog-ng/syslog-ng/pull/4813>) - opentelemetry(), syslog-ng-otlp() source: Added log-fetch-limit() option. This option can be used to fine tune the performance. To minimize locking while moving messages between source and destination side queues, syslog-ng can move messages in batches. The log-fetch-limit() option sets the maximal size of the batch moved by a worker. By default it is equal to log-iw-size() / workers(). (#4827 <https://github.com/syslog-ng/syslog-ng/pull/4827>) - dqtool: add option for truncating (compacting) abandoned disk-buffers (#4875 <https://github.com/syslog-ng/syslog-ng/pull/4875>) Bugfixes - opentelemetry(): fix crash when an invalid configuration needs to be reverted (#4910 <https://github.com/syslog-ng/syslog-ng/pull/4910>) - gRPC drivers: fixed a crash when gRPC drivers were used and syslog-ng was reloaded (#4909 <https://github.com/syslog-ng/syslog-ng/pull/4909>) - opentelemetry(), syslog-ng-otlp() source: Fixed a crash. It occurred with multiple workers() during high load. (#4827 <https://github.com/syslog-ng/syslog-ng/pull/4827>) - rename(): Fixed a bug, which always converted the renamed NV pair to string type. (#4847 <https://github.com/syslog-ng/syslog-ng/pull/4847>) - With IPv6 disabled, there were linking errors (#4880 <https://github.com/syslog-ng/syslog-ng/pull/4880>) Metrics - http(): Added a new counter for HTTP requests. It is activated on stats(level(1));. Example metrics: syslogng_output_http_requests_total{url="http://localhost:8888/bar",response_code="200",driver="http",id="#anon-destination0#0"} 16 syslogng_output_http_requests_total{url="http://localhost:8888/bar",response_code="401",driver="http",id="#anon-destination0#0"} 2 syslogng_output_http_requests_total{url="http://localhost:8888/bar",response_code="502",driver="http",id="#anon-destination0#0"} 1 syslogng_output_http_requests_total{url="http://localhost:8888/foo",response_code="200",driver="http",id="#anon-destination0#0"} 24 (#4805 <https://github.com/syslog-ng/syslog-ng/pull/4805>) - gRPC based destination drivers: Added gRPC request related metrics. Affected drivers: - opentelemetry() - syslog-ng-otlp() - bigquery() - loki() Example metrics: syslogng_output_grpc_requests_total{driver="syslog-ng-otlp",url="localhost:12345",response_code="ok"} 49 syslogng_output_grpc_requests_total{driver="syslog-ng-otlp",url="localhost:12345",response_code="unavailable"} 11 (#4811 <https://github.com/syslog-ng/syslog-ng/pull/4811>) - New metric to monitor destination reachability syslogng_output_unreachable is a bool-like metric, which shows whether a destination is reachable or not. sum() can be used to count all unreachable outputs, hence the negated name. It is currently available for the network(), syslog(), unix-*() destinations, and threaded destinations (http(), opentelemetry(), redis() , mongodb(), python(), etc.). (#4876 <https://github.com/syslog-ng/syslog-ng/pull/4876>) - destinations: Added "syslogng_output_event_retries_total" counter. This counter is available for the following destination drivers: - amqp() - bigquery() - http() and all http based drivers - java() - kafka() - loki() - mongodb() - mqtt() - opentelemetry() - python() and all python based drivers - redis() - riemann() - smtp() - snmp() - sql() - stomp() - syslog-ng-otlp() Example metrics: syslogng_output_event_retries_total{driver="http",url="http://localhost:8888/${path}",id="#anon-destination0#0"} 5 (#4807 <https://github.com/syslog-ng/syslog-ng/pull/4807>) - syslogng_memory_queue_capacity Shows the capacity (maximum possible size) of each queue. Note that this metric publishes log-fifo-size(), which only limits non-flow-controlled messages. Messages coming from flow-controlled paths are not limited by log-fifo-size(), their corresponding source log-iw-size() is the upper limit. (#4831 <https://github.com/syslog-ng/syslog-ng/pull/4831>) Other changes - opentelemetry(), syslog-ng-otlp() source: Changed the backpressure behavior. syslog-ng no longer returns UNAVAILABLE to the gRPC request, when it cannot forward the received message because of backpressure. Instead, syslog-ng will block until the destination can accept more messages. (#4827 <https://github.com/syslog-ng/syslog-ng/pull/4827>) - opentelemetry(), syslog-ng-otlp() source: log-iw-size() is now split between workers. (#4827 <https://github.com/syslog-ng/syslog-ng/pull/4827>) - APT packages: Dropped Debian Buster support. Old packages are still available, but new syslog-ng versions will not be available on Debian Buster (#4840 <https://github.com/syslog-ng/syslog-ng/pull/4840>) - dbld: AlmaLinux 8 support (#4902 <https://github.com/syslog-ng/syslog-ng/pull/4902>) syslog-ng Discord For a bit more interactive discussion, join our Discord server: [image: Axoflow Discord Server] <https://discord.gg/E65kP9aZGm> Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Arpad Kunszt, Attila Szakacs, Balazs Scheidler, Bálint Horváth, Hofi, Kovács, Gergő Ferenc, László Várady, Peter Marko, shifter

1 0

we need to know if Syslog supports any of these virtualization solution - Harvester, CaaS or HyperV. If not, what other virtualization solution is supported by Syslog.
by Mayekar, PrachiX 16 Apr '24

16 Apr '24

Hi All, We are migrating the VMs from VMware to new solution (Harvester, CaaS, HyperV). Hence, we need to know if Syslog supports any of these virtualization solution - Harvester, CaaS or HyperV. If not, what other virtualization solution is supported by Syslog. Thanks & Regards, Prachi Mayekar ITI-Network Services A Contingent Worker at Intel For assistance, please visit us at https://it.intel.com<https://it.intel.com/>

3 2

The syslog-ng Insider 2024-04: PAM Essentials; XML Eventlog; multi-line logs
by Peter Czanik (pczanik) 11 Apr '24

11 Apr '24

Dear syslog-ng users, This is the 119th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Collecting One Identity Cloud PAM Essentials logs using syslog-ng ----------------------------------------------- One Identity Cloud PAM Essentials is the latest security product by One Identity. It provides asset management as well as secure and monitored remote access for One Identity Cloud users to hosts on their local network. I had a chance to test PAM Essentials while still in development. While there, I also integrated it with syslog-ng. >From this blog, you can learn what PAM Essentials is, and how you can collect its logs using syslog-ng. My next blog will show you how to work with the collected log messages and create alerts when somebody connects to a host on your local network using PAM Essentials. https://www.syslog-ng.com/community/b/blog/posts/collecting-one-identity-cl… Dedicated Windows XML eventlog parser in syslog-ng -------------------------------------------------- Version 4.6 of syslog-ng introduced windows-eventlog-xml-parser(), a dedicated parser for XML-formatted event logs from Windows. It makes the EventData portion of log messages more useful, as it combines two arrays into a list of name-value pairs. https://www.syslog-ng.com/community/b/blog/posts/dedicated-windows-xml-even… Working with multi-line logs in syslog-ng ----------------------------------------- Most log messages fit on a single line. However, Windows and some developer tools and services, like Tomcat, write multi-line log messages. These can come in various formats. For example, new log messages start with a date in a specific format. You use the multi-line-prefix() of the syslog-ng file() source to send multi-line messages as single messages instead of line by line. I must admit that I have never seen multi-line logs in production. I am not a developer, do not run Tomcat or Windows. However, recently I tested a software on Windows, which produced multi-line log messages. https://www.syslog-ng.com/community/b/blog/posts/working-with-multi-line-lo… WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0