From gguo at fortinet.com Wed Apr 3 06:19:03 2024 From: gguo at fortinet.com (Grant Guo) Date: Wed, 3 Apr 2024 06:19:03 +0000 Subject: [syslog-ng] How does syslog-ng detect the availability of the remote end Message-ID: Hi, Recently I am using syslog-ng?s docker container as syslog generator as the following: docker run -d \ --cpus=2 \ --name $name \ -v ${PWD}/syslog.log:/data/syslog.log \ --entrypoint="" \ --network=host \ balabit/syslog-ng:latest \ /usr/bin/loggen --dgram --inet --dont-parse --loop-reading --size 8192 --rate $rate --interval $duration --read-file /data/syslog.log $ip 514 I found something interesting, if my syslog server stopped, the client syslog-ng containers stopped too. So I would like to know if syslog-ng detected the availability of the remote end? If so, how? Besides, I didn?t see ICMP packets between the client and server using tcpdump. Thanks Grant *** Please note that this message and any attachments may contain confidential and proprietary material and information and are intended only for the use of the intended recipient(s). If you are not the intended recipient, you are hereby notified that any review, use, disclosure, dissemination, distribution or copying of this message and any attachments is strictly prohibited. If you have received this email in error, please immediately notify the sender and destroy this e-mail and any attachments and all copies, whether electronic or printed. Please also note that any views, opinions, conclusions or commitments expressed in this message are those of the individual sender and do not necessarily reflect the views of Fortinet, Inc., its affiliates, and emails are not binding on Fortinet and only a writing manually signed by Fortinet's General Counsel can be a binding commitment of Fortinet to Fortinet's customers or partners. Thank you. *** -------------- next part -------------- An HTML attachment was scrubbed... URL: From Peter.Czanik at oneidentity.com Thu Apr 11 11:52:01 2024 From: Peter.Czanik at oneidentity.com (Peter Czanik (pczanik)) Date: Thu, 11 Apr 2024 11:52:01 +0000 Subject: [syslog-ng] The syslog-ng Insider 2024-04: PAM Essentials; XML Eventlog; multi-line logs Message-ID: Dear syslog-ng users, This is the 119th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Collecting One Identity Cloud PAM Essentials logs using syslog-ng ----------------------------------------------- One Identity Cloud PAM Essentials is the latest security product by One Identity. It provides asset management as well as secure and monitored remote access for One Identity Cloud users to hosts on their local network. I had a chance to test PAM Essentials while still in development. While there, I also integrated it with syslog-ng. >From this blog, you can learn what PAM Essentials is, and how you can collect its logs using syslog-ng. My next blog will show you how to work with the collected log messages and create alerts when somebody connects to a host on your local network using PAM Essentials. https://www.syslog-ng.com/community/b/blog/posts/collecting-one-identity-cloud-pam-essentials-logs-using-syslog-ng Dedicated Windows XML eventlog parser in syslog-ng -------------------------------------------------- Version 4.6 of syslog-ng introduced windows-eventlog-xml-parser(), a dedicated parser for XML-formatted event logs from Windows. It makes the EventData portion of log messages more useful, as it combines two arrays into a list of name-value pairs. https://www.syslog-ng.com/community/b/blog/posts/dedicated-windows-xml-eventlog-parser-in-syslog-ng Working with multi-line logs in syslog-ng ----------------------------------------- Most log messages fit on a single line. However, Windows and some developer tools and services, like Tomcat, write multi-line log messages. These can come in various formats. For example, new log messages start with a date in a specific format. You use the multi-line-prefix() of the syslog-ng file() source to send multi-line messages as single messages instead of line by line. I must admit that I have never seen multi-line logs in production. I am not a developer, do not run Tomcat or Windows. However, recently I tested a software on Windows, which produced multi-line log messages. https://www.syslog-ng.com/community/b/blog/posts/working-with-multi-line-logs-in-syslog-ng WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik From prachix.mayekar at intel.com Tue Apr 16 10:59:53 2024 From: prachix.mayekar at intel.com (Mayekar, PrachiX) Date: Tue, 16 Apr 2024 10:59:53 +0000 Subject: [syslog-ng] we need to know if Syslog supports any of these virtualization solution - Harvester, CaaS or HyperV. If not, what other virtualization solution is supported by Syslog. Message-ID: Hi All, We are migrating the VMs from VMware to new solution (Harvester, CaaS, HyperV). Hence, we need to know if Syslog supports any of these virtualization solution - Harvester, CaaS or HyperV. If not, what other virtualization solution is supported by Syslog. Thanks & Regards, Prachi Mayekar ITI-Network Services A Contingent Worker at Intel For assistance, please visit us at https://it.intel.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From me at junc.eu Tue Apr 16 13:25:23 2024 From: me at junc.eu (Benny Pedersen) Date: Tue, 16 Apr 2024 15:25:23 +0200 Subject: [syslog-ng] we need to know if Syslog supports any of these virtualization solution - Harvester, CaaS or HyperV. If not, what other virtualization solution is supported by Syslog. In-Reply-To: References: Message-ID: <6676e1e10221d2f23a6d7c5af0bbb7a1@junc.eu> Mayekar, PrachiX skrev den 2024-04-16 12:59: > Hi All, > > We are migrating the VMs from VMware to new solution (Harvester, CaaS, > HyperV). Hence, we need to know if Syslog supports any of these > virtualization solution - Harvester, CaaS or HyperV. If not, what > other virtualization solution is supported by Syslog. syslog-ng can listen on inet port, so why is this a problem ?, just let the vm's use remote syslogs vms itself should not listen on port 514, but the syslog server should From cdukes at logzilla.net Tue Apr 16 13:44:02 2024 From: cdukes at logzilla.net (Clayton Dukes) Date: Tue, 16 Apr 2024 13:44:02 +0000 Subject: [syslog-ng] we need to know if Syslog supports any of these virtualization solution - Harvester, CaaS or HyperV. If not, what other virtualization solution is supported by Syslog. In-Reply-To: <6676e1e10221d2f23a6d7c5af0bbb7a1@junc.eu> References: <6676e1e10221d2f23a6d7c5af0bbb7a1@junc.eu> Message-ID: Mayekar, Sort of unrelated, but have you guys checked out Proxmox? It's absolutely amazing and solid. We dropped VMWare years ago for it and are very glad we did. Also, many of the extra-paid-for features that are sold with VMWare are included in Proxmox. It's also free ? they use a support model (if you want/need support, then you pay a nominal fee, which is very low priced). Note: I have no relationship with the Proxmox folks, just a very happy user of their product. From: syslog-ng on behalf of Benny Pedersen Date: Tuesday, April 16, 2024 at 9:36?AM To: syslog-ng at lists.balabit.hu Subject: Re: [syslog-ng] we need to know if Syslog supports any of these virtualization solution - Harvester, CaaS or HyperV. If not, what other virtualization solution is supported by Syslog. Mayekar, PrachiX skrev den 2024-04-16 12:59: > Hi All, > > We are migrating the VMs from VMware to new solution (Harvester, CaaS, > HyperV). Hence, we need to know if Syslog supports any of these > virtualization solution - Harvester, CaaS or HyperV. If not, what > other virtualization solution is supported by Syslog. syslog-ng can listen on inet port, so why is this a problem ?, just let the vm's use remote syslogs vms itself should not listen on port 514, but the syslog server should ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq -------------- next part -------------- An HTML attachment was scrubbed... URL: From laszlo.varady at axoflow.com Fri Apr 19 17:06:42 2024 From: laszlo.varady at axoflow.com (=?UTF-8?B?TMOhc3psw7MgVsOhcmFkeQ==?=) Date: Fri, 19 Apr 2024 19:06:42 +0200 Subject: [syslog-ng] syslog-ng 4.7.1 Message-ID: Dear syslog-ng users, We are pleased to announce the 4.7.1 version of syslog-ng, which has been released and is now available on GitHub: https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-4.7.1 Packages are also available for various platforms. The AxoSyslog project provides cloud-ready container images , Helm charts . 4.7.1 *This is the combination of the news entries of 4.7.0 and 4.7.1.* *4.7.1 hotfixed two crashes related to configuration reload.* Read Axoflow's blog post for more details. You can read more about the new features in the AxoSyslog documentation . HighlightsCollecting Jellyfin logs The new jellyfin() source, reads Jellyfin logs from its log file output. Example minimal config: source s_jellyfin { jellyfin( base-dir("/path/to/my/jellyfin/root/log/dir") filename-pattern("log_*.log") ); }; For more details about Jellyfin logging, see: - https://jellyfin.org/docs/general/administration/configuration/#main-configuration - https://jellyfin.org/docs/general/administration/configuration/#log-directory As the jellyfin() source is based on a wildcard-file() source, all of the wildcard-file() source options are applicable, too. (#4802 ) Collecting *arr logs Use the newly added *arr() sources to read various *arr logs: - lidarr() - prowlarr() - radarr() - readarr() - sonarr() - whisparr() Example minimal config: source s_radarr { radarr( dir("/path/to/my/radarr/log/dir") ); }; The logging module is stored in the name-value pair, for example: .radarr.module => ImportListSyncService. The prefix can be modified with the prefix() option. (#4803 ) Features - opentelemetry(), syslog-ng-otlp() source: Added concurrent-requests() option. This option configures the maximal number of in-flight gRPC requests per worker. Setting this value to the range of 10s or 100s is recommended when there are a high number of clients sending simultaneously. Ideally, workers() * concurrent-requests() should be greater or equal to the number of clients, but this can increase the memory usage. (#4827 ) - loki(): Support multi-tenancy with the new tenant-id() option (#4812 ) - s3(): Added support for authentication from environment. The access-key() and secret-key() options are now optional, which makes it possible to use authentication methods originated from the environment, e.g. AWS_... environment variables or credentials files from the ~/.aws/ directory. For more info, see: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html (#4881 ) - gRPC based drivers: Added channel-args() option. Affected drivers are: - bigquery() destination - loki() destination - opentelemetry() source and destination - syslog-ng-otlp() source and destination The channel-args() option accepts name-value pairs and sets channel arguments defined in https://grpc.github.io/grpc/core/group__grpc__arg__keys.html Example config: opentelemetry( channel-args( "grpc.loadreporting" => 1 "grpc.minimal_stack" => 0 ) ); (#4827 ) - ${TRANSPORT} macro: Added support for locally created logs. New values are: - "local+unix-stream" - "local+unix-dgram" - "local+file" - "local+pipe" - "local+program" - "local+devkmsg" - "local+journal" - "local+afstreams" - "local+openbsd" (#4777 ) - tags: Added new built-in tags that help identifying parse errors. New tags are: - "message.utf8_sanitized" - "message.parse_error" - "syslog.missing_pri" - "syslog.missing_timestamp" - "syslog.invalid_hostname" - "syslog.unexpected_framing" - "syslog.rfc3164_missing_header" - "syslog.rfc5424_unquoted_sdata_value" (#4804 ) - mqtt() source: Added ${MQTT_TOPIC} name-value pair. It is useful for the cases where topic() contains wildcards. Example config: log { source { mqtt(topic("#")); }; destination { stdout(template("${MQTT_TOPIC} - ${MESSAGE}\n")); }; }; (#4824 ) - template(): Added a new template function: $(tags-head) This template function accepts multiple tag names, and returns the first one that is set. Example config: # resolves to "bar" if "bar" tag is set, but "foo" is not template("$(tags-head foo bar baz)") (#4804 ) - s3(): Use default AWS URL if url() is not set. (#4813 ) - opentelemetry(), syslog-ng-otlp() source: Added log-fetch-limit() option. This option can be used to fine tune the performance. To minimize locking while moving messages between source and destination side queues, syslog-ng can move messages in batches. The log-fetch-limit() option sets the maximal size of the batch moved by a worker. By default it is equal to log-iw-size() / workers(). (#4827 ) - dqtool: add option for truncating (compacting) abandoned disk-buffers (#4875 ) Bugfixes - opentelemetry(): fix crash when an invalid configuration needs to be reverted (#4910 ) - gRPC drivers: fixed a crash when gRPC drivers were used and syslog-ng was reloaded (#4909 ) - opentelemetry(), syslog-ng-otlp() source: Fixed a crash. It occurred with multiple workers() during high load. (#4827 ) - rename(): Fixed a bug, which always converted the renamed NV pair to string type. (#4847 ) - With IPv6 disabled, there were linking errors (#4880 ) Metrics - http(): Added a new counter for HTTP requests. It is activated on stats(level(1));. Example metrics: syslogng_output_http_requests_total{url="http://localhost:8888/bar",response_code="200",driver="http",id="#anon-destination0#0"} 16 syslogng_output_http_requests_total{url="http://localhost:8888/bar",response_code="401",driver="http",id="#anon-destination0#0"} 2 syslogng_output_http_requests_total{url="http://localhost:8888/bar",response_code="502",driver="http",id="#anon-destination0#0"} 1 syslogng_output_http_requests_total{url="http://localhost:8888/foo",response_code="200",driver="http",id="#anon-destination0#0"} 24 (#4805 ) - gRPC based destination drivers: Added gRPC request related metrics. Affected drivers: - opentelemetry() - syslog-ng-otlp() - bigquery() - loki() Example metrics: syslogng_output_grpc_requests_total{driver="syslog-ng-otlp",url="localhost:12345",response_code="ok"} 49 syslogng_output_grpc_requests_total{driver="syslog-ng-otlp",url="localhost:12345",response_code="unavailable"} 11 (#4811 ) - New metric to monitor destination reachability syslogng_output_unreachable is a bool-like metric, which shows whether a destination is reachable or not. sum() can be used to count all unreachable outputs, hence the negated name. It is currently available for the network(), syslog(), unix-*() destinations, and threaded destinations (http(), opentelemetry(), redis() , mongodb(), python(), etc.). (#4876 ) - destinations: Added "syslogng_output_event_retries_total" counter. This counter is available for the following destination drivers: - amqp() - bigquery() - http() and all http based drivers - java() - kafka() - loki() - mongodb() - mqtt() - opentelemetry() - python() and all python based drivers - redis() - riemann() - smtp() - snmp() - sql() - stomp() - syslog-ng-otlp() Example metrics: syslogng_output_event_retries_total{driver="http",url="http://localhost:8888/${path}",id="#anon-destination0#0"} 5 (#4807 ) - syslogng_memory_queue_capacity Shows the capacity (maximum possible size) of each queue. Note that this metric publishes log-fifo-size(), which only limits non-flow-controlled messages. Messages coming from flow-controlled paths are not limited by log-fifo-size(), their corresponding source log-iw-size() is the upper limit. (#4831 ) Other changes - opentelemetry(), syslog-ng-otlp() source: Changed the backpressure behavior. syslog-ng no longer returns UNAVAILABLE to the gRPC request, when it cannot forward the received message because of backpressure. Instead, syslog-ng will block until the destination can accept more messages. (#4827 ) - opentelemetry(), syslog-ng-otlp() source: log-iw-size() is now split between workers. (#4827 ) - APT packages: Dropped Debian Buster support. Old packages are still available, but new syslog-ng versions will not be available on Debian Buster (#4840 ) - dbld: AlmaLinux 8 support (#4902 ) syslog-ng Discord For a bit more interactive discussion, join our Discord server: [image: Axoflow Discord Server] Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Arpad Kunszt, Attila Szakacs, Balazs Scheidler, B?lint Horv?th, Hofi, Kov?cs, Gerg? Ferenc, L?szl? V?rady, Peter Marko, shifter -------------- next part -------------- An HTML attachment was scrubbed... URL: From prachix.mayekar at intel.com Wed Apr 24 08:19:09 2024 From: prachix.mayekar at intel.com (Mayekar, PrachiX) Date: Wed, 24 Apr 2024 08:19:09 +0000 Subject: [syslog-ng] we need to know if Syslog supports any of these virtualization solution - Harvester, CaaS or HyperV. If not, what other virtualization solution is supported by Syslog. Message-ID: We need to know which virtualization solution is supported by syslog - Harvester CaaS or HyperV ? Thanks & Regards, Prachi Mayekar ITI-Network Services A Contingent Worker at Intel For assistance, please visit us at https://it.intel.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From me at junc.eu Wed Apr 24 09:04:45 2024 From: me at junc.eu (Benny Pedersen) Date: Wed, 24 Apr 2024 11:04:45 +0200 Subject: [syslog-ng] we need to know if Syslog supports any of these virtualization solution - Harvester, CaaS or HyperV. If not, what other virtualization solution is supported by Syslog. In-Reply-To: References: Message-ID: Mayekar, PrachiX skrev den 2024-04-24 10:19: > We need to know which virtualization solution is supported by syslog - > Harvester CaaS or HyperV ? 2nd time you ask imho ? you ask on syslog-ng, so does it work ?, how to setup ? syslog-ng have fine manuals https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide/45 see more info here https://www.syslog-ng.com/ more help ? From laszlo.varady at axoflow.com Wed Apr 24 09:05:34 2024 From: laszlo.varady at axoflow.com (=?UTF-8?B?TMOhc3psw7MgVsOhcmFkeQ==?=) Date: Wed, 24 Apr 2024 11:05:34 +0200 Subject: [syslog-ng] we need to know if Syslog supports any of these virtualization solution - Harvester, CaaS or HyperV. If not, what other virtualization solution is supported by Syslog. In-Reply-To: References: Message-ID: Hi, On the infrastructure level, syslog-ng works with any virtualization technology. syslog-ng is a daemon that can be installed on various operating systems (Linux, BSD, etc.); it will function properly on all supported distributions, regardless of whether it is a virtualized environment. CaaS: Axoflow, the most active contributor of syslog-ng, provides cloud-ready ARM and x86-64 container images as part of the AxoSyslog project. https://axoflow.com/cloud-ready-syslog-ng-images/ https://github.com/axoflow/axosyslog -- L?szl? V?rady -------------- next part -------------- An HTML attachment was scrubbed... URL: