- syslog-ng - lists.balabit.hu

Re: [syslog-ng] syslog-ng EPS
by Wilson, Jonathan L 12 Jun '23

12 Jun '23

We use: syslog-ng-ctl query get source.*.processed --reset In a cron job just before midnight to capture the entire day's stats. They show up in the format: source.s_local.processed=4722471 source.s_BSD_TCP.processed=6299967 source.s_RFC5424_TCP.processed=0 source.s_network_source.processed=27695900 . . . where "s_local", "s_BSD_TCP", etc. are the source names inside the syslog-ng.conf file. Note that the "-reset" on the end does just that, resets the stats to zero. A little scripting to divide the numbers above by 86400 will give you the count of messages per second by source. (We convert ours to JSON and dump it into Splunk for later viewing.) Regards, Jonathan Wilson

1 0

Can't Find Syslog-ng 4.0.0 Administration Guide Link
by Sumanta Banerjee 09 Jun '23

09 Jun '23

Hi All, Can someone please help in sending me the link for Syslog-ng 4.0.0 administration guide (if at all available). Thanks & Regards, Sumanta Banerjee Splunk Admin | CISO | Aviva Group Tel: +91-8420892593 24x7x365: +44 1603 208 582 sumanta.banerjee(a)aviva.com<mailto:sumanta.banerjee@aviva.com> GlobalCyberSecurityEngineeringTeam(a)aviva.com<mailto:GlobalCyberSecurityEngineeringTeam@aviva.com> www.aviva.com<http://www.aviva.com> Wipro Technologies - SJP2, Bangalore, India [cid:image001.gif@01D99AE4.B2631840] Aviva: Public Aviva plc, registered Office: St. Helen's, 1 Undershaft, London EC3P 3DQ. Registered in England No. 02468686. www.aviva.com This message and any attachments may be confidential or legally privileged. If you are not the intended recipient, please telephone or e-mail the sender and delete this message and any attachments from your system. Also, if you are not the intended recipient you must not copy this message or attachments or disclose the contents to any other person. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Aviva.

1 0

syslog-ng EPS
by Dragan Zecevic 08 Jun '23

08 Jun '23

Hi guys, is it possible to calculate daily events per second for all syslog messages coming to syslog-ng? Or even better separated by sources. I have syslog-ng 3 (3.34.1). Thanks. Br, Dragan

2 1

One destination, multiple TCP connections?
by Steve Bernacki 08 Jun '23

08 Jun '23

Hello, We have a syslog TCP destination that receives logs from many different destinations. During peak log volume times, this destination begins to back up and queue. We've found that if we define multiple destinations for the same IP/port and route logs through these multiple destinations, the destination is able to handle these peak log volume times much better. Our current solution of having a destination for each source is pretty kludgy. Is there a way to define a TCP destination and specify to open, say, 10 connections to the destination and load balance the traffic over those 10 connections? Thanks. Steve

2 1

Fedora / RHEL & Co: Python split -- request for testing
by Peter Czanik (pczanik) 08 Jun '23

08 Jun '23

Hi, Recently I wrote a blog about Python packaging problems and my plans to split the Python-based drivers of syslog-ng from the syslog-ng Python plugin: https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-python-packaging I implemented it for my syslog-ng git snapshot repo for Fedora / RHEL: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng-githead/ Please test it ASAP, if you got a bit of time. Originally I planned to use it only for the 4.3 release, but if I get positive responses, I'll also backport these changes to my syslog-ng42 repo, and use these changes also for the syslog-ng-stable repo. The blog post explains why is this important, but TL;DR: this way you do not have to install 50+MB of dependencies you will likely never use, and syslog-ng will not crash on start on RHEL 8 & Co. due to lack of dependencies. Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 2

uestion
by Johnson, Linwood 06 Jun '23

06 Jun '23

I get the following when I check [root@server1~]# syslog-ng WARNING: the match() filter without the use of the value() option is deprecated and hinders performance, please update your configuration; The kernel refused to set the receive buffer (SO_RCVBUF) to the requested size, you probably need to adjust buffer related kernel parameters; so_rcvbuf='536870912', so_rcvbuf_set='425984' [root@ server1~]# sysctl -a |grep rmem_max net.core.rmem_max = 212992 sysctl: reading key "net.ipv6.conf.all.stable_secret" sysctl: reading key "net.ipv6.conf.default.stable_secret" sysctl: reading key "net.ipv6.conf.eth0.stable_secret" sysctl: reading key "net.ipv6.conf.lo.stable_secret" [root@ server1 ~]# sysctl -a |grep vbuf net.ipv4.tcp_moderate_rcvbuf = 1 sysctl: reading key "net.ipv6.conf.all.stable_secret" sysctl: reading key "net.ipv6.conf.default.stable_secret" sysctl: reading key "net.ipv6.conf.eth0.stable_secret" sysctl: reading key "net.ipv6.conf.lo.stable_secret" net.sctp.rcvbuf_policy = 0 [root@ server1~ ]# cat /proc/sys/net/ipv4/tcp_rmem 4096 87380 6291456 [root@ server1~ ]# cat /proc/sys/net/core/rmem_default 212992 [root@ server1~ ]# cat /proc/sys/net/core/rmem_max 212992 [root@ server1~ ]# grep . /proc/sys/net/ipv4/tcp*mem /proc/sys/net/ipv4/tcp_mem:173202 230936 346404 /proc/sys/net/ipv4/tcp_rmem:4096 87380 6291456 /proc/sys/net/ipv4/tcp_wmem:4096 16384 4194304 [root@ server1~ hisotry]# I attempted to make the changes on the server. What can I do to resolve this issue? -- Thanks in advance, Linwood Johnson Senior Engineer | Unix Systems Administrator Site Reliability Tools [image: NewsTech] P.O Box 300 | Princeton NJ 08543 *(W)* 609-627-2179 *(C)* 732-740-0694 *Email:* Linwood.Johnson(a)dowjones.com *Team:* sitereliabilitytools(a)dowjones.com

2 1

syslog-ng 4.2.0
by Attila Szakács 17 May '23

17 May '23

Dear syslog-ng users, We are pleased to announce that the 4.2.0 version of syslog-ng has been released, and it is now available on GitHub: https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-4.2.0 4.2.0 ===== Read Axoflow's [blog post](https://axoflow.com/axosyslog-release-4-2/) for more details. ## Highlights #### Sending messages to Splunk HEC The `splunk-hec-event()` destination feeds Splunk via the [HEC events API]( https://docs.splunk.com/Documentation/Splunk/9.0.4/RESTREF/RESTinput#servic… ). Minimal config: ``` destination d_splunk_hec_event { splunk-hec-event( url("https://localhost:8088") token("70b6ae71-76b3-4c38-9597-0c5b37ad9630") ); }; ``` Additional options include: * `event()` * `index()` * `source()` * `sourcetype()` * `host()` * `time()` * `default-index()` * `default-source()` * `default-sourcetype()` * `fields()` * `extra-headers()` * `extra-queries()` * `content-type()` The `splunk-hec-raw()` destination feeds Splunk via the [HEC raw API]( https://docs.splunk.com/Documentation/Splunk/9.0.4/RESTREF/RESTinput#servic… ). Minimal config: ``` destination d_splunk_hec_raw { splunk-hec-raw( url("https://localhost:8088") token("70b6ae71-76b3-4c38-9597-0c5b37ad9630") channel("05ed4617-f186-4ccd-b4e7-08847094c8fd") ); }; ``` ([#4462](https://github.com/syslog-ng/syslog-ng/pull/4462)) #### Smart multi-line for recognizing backtraces `multi-line-mode(smart)`: With this multi-line mode, the inherently multi-line data backtrace format is recognized even if they span multiple lines in the input and are converted to a single log message for easier analysis. Backtraces for the following programming languages are recognized : Python, Java, JavaScript, PHP, Go, Ruby and Dart. The regular expressions to recognize these programming languages are specified by an external file called `/usr/share/syslog-ng/smart-multi-line.fsm` (installation path depends on configure arguments), in a format that is described in that file. `group-lines()` parser: this new parser correlates multi-line messages received as separate, but subsequent lines into a single log message. Received messages are first collected into streams related messages (using key()), then collected into correlation contexts up to timeout() seconds. The identification of multi-line messages are then performed on these message contexts within the time period. ``` group-lines(key("$FILE_NAME") multi-line-mode("smart") template("$MESSAGE") timeout(10) line-separator("\n") ); ``` ([#4225](https://github.com/syslog-ng/syslog-ng/pull/4225)) #### HYPR Audit Trail source `hypr-audit-trail()` & `hypr-app-audit-trail()` source drivers are now available to monitor the audit trails for [HYPR](https://www.hypr.com) applications. See the README.md file in the driver's directory to see usage information. ([#4175](https://github.com/syslog-ng/syslog-ng/pull/4175)) #### `ebpf()` plugin and reuseport packet randomizer A new ebpf() plugin was added as a framework to leverage the kernel's eBPF infrastructure to improve performance and scalability of syslog-ng. Example: ``` source s_udp { udp(so-reuseport(yes) port(2000) persist-name("udp1") ebpf(reuseport(sockets(4))) ); udp(so-reuseport(yes) port(2000) persist-name("udp2")); udp(so-reuseport(yes) port(2000) persist-name("udp3")); udp(so-reuseport(yes) port(2000) persist-name("udp4")); }; ``` NOTE: The `ebpf()` plugin is considered advanced usage so its compilation is disabled by default. Please don't use it unless all other avenues of configuration solutions are already tried. You will need a special toolchain and a recent kernel version to compile and run eBPF programs. ([#4365](https://github.com/syslog-ng/syslog-ng/pull/4365)) ## Features * `network` source: During a TLS handshake, syslog-ng now automatically sets the `certificate_authorities` field of the certificate request based on the `ca-file()` and `ca-dir()` options. The `pkcs12-file()` option already had this feature. ([#4412](https://github.com/syslog-ng/syslog-ng/pull/4412)) * `metrics-probe()`: Added `level()` option to set the stats level of the generated metrics. ([#4453](https://github.com/syslog-ng/syslog-ng/pull/4453)) * `metrics-probe()`: Added `increment()` option. Users can now set a template, which resolves to a number that modifies the increment of the counter. If not set, the increment is 1. ([#4447](https://github.com/syslog-ng/syslog-ng/pull/4447)) * `python`: Added support for typed custom options. This applies for `python` source, `python-fetcher` source, `python` destination, `python` parser and `python-http-header` inner destination. Example config: ``` python( class("TestClass") options( "string_option" => "example_string" "bool_option" => True # supported values are: True, False, yes, no "integer_option" => 123456789 "double_option" => 123.456789 "string_list_option" => ["string1", "string2", "string3"] "template_option" => LogTemplate("${example_template}") ) ); ``` **Breaking change! Previously values were converted to strings if possible, now they are passed to the python class with their real type. Make sure to follow up these changes in your python code!** ([#4354](https://github.com/syslog-ng/syslog-ng/pull/4354)) * `mongodb` destination: Added support for list, JSON and null types. ([#4437](https://github.com/syslog-ng/syslog-ng/pull/4437)) * `add-contextual-data()`: significantly reduce memory usage for large CSV files. ([#4444](https://github.com/syslog-ng/syslog-ng/pull/4444)) * `python()`: new LogMessage methods for querying as string and with default values - `get(key[, default])` Return the value for `key` if `key` exists, else `default`. If `default` is not given, it defaults to `None`, so that this method never raises a `KeyError`. - `get_as_str(key, default=None, encoding='utf-8', errors='strict', repr='internal')`: Return the string value for `key` if `key` exists, else `default`. If `default` is not given, it defaults to `None`, so that this method never raises a `KeyError`. The string value is decoded using the codec registered for `encoding`. `errors` may be given to set the desired error handling scheme. Note that currently `repr='internal'` is the only available representation. We may implement another more Pythonic representation in the future, so please specify the `repr` argument explicitly if you want to avoid future representation changes in your code. ([#4410](https://github.com/syslog-ng/syslog-ng/pull/4410)) * `kubernetes()` source: Added support for json-file logging driver format. ([#4419](https://github.com/syslog-ng/syslog-ng/pull/4419)) * The new `$RAWMSG_SIZE` hard macro can be used to query the original size of the incoming message in bytes. This information may not be available for all source drivers. ([#4440](https://github.com/syslog-ng/syslog-ng/pull/4440)) * syslog-ng configuration identifier A new syslog-ng configuration keyword has been added, which allows specifying a config identifier. For example: ``` @config-id: cfg-20230404-13-g02b0850fc ``` This keyword can be used for config identification in managed environments, where syslog-ng instances and their configuration are deployed/generated automatically. `syslog-ng-ctl config --id` can be used to query the active configuration ID and the SHA256 hash of the full "preprocessed" syslog-ng configuration. For example: ``` $ syslog-ng-ctl config --id cfg-20230404-13-g02b0850fc (08ddecfa52a3443b29d5d5aa3e5114e48dd465e195598062da9f5fc5a45d8a83) ``` ([#4420](https://github.com/syslog-ng/syslog-ng/pull/4420)) * `syslog-ng`: add `--config-id` command line option Similarly to `--syntax-only`, this command line option parses the configuration and then prints its ID before exiting. It can be used to query the ID of the current configuration persisted on disk. ([#4435](https://github.com/syslog-ng/syslog-ng/pull/4435)) * Health metrics and `syslog-ng-ctl healthcheck` A new `syslog-ng-ctl` command has been introduced, which can be used to query a healthcheck status from syslog-ng. Currently, only 2 basic health values are reported. `syslog-ng-ctl healthcheck --timeout <seconds>` can be specified to use it as a boolean healthy/unhealthy check. Health checks are also published as periodically updated metrics. The frequency of these checks can be configured with the `stats(healthcheck-freq())` option. The default is 5 minutes. ([#4362](https://github.com/syslog-ng/syslog-ng/pull/4362)) * `$(format-json)` and template functions which support value-pairs expressions: new key transformations upper() and lower() have been added to translate the caps of keys while formatting the output template. For example: template("$(format-json test.* --upper)\n") Would convert all keys to uppercase. Only supports US ASCII. ([#4452](https://github.com/syslog-ng/syslog-ng/pull/4452)) * `python()`, `python-fetcher()` sources: Added a mapping for the `flags()` option. The state of the `flags()` option is mapped to the `self.flags` variable, which is a `Dict[str, bool]`, for example: ```python { 'parse': True, 'check-hostname': False, 'syslog-protocol': True, 'assume-utf8': False, 'validate-utf8': False, 'sanitize-utf8': False, 'multi-line': True, 'store-legacy-msghdr': True, 'store-raw-message': False, 'expect-hostname': True, 'guess-timezone': False, 'header': True, 'rfc3164-fallback': True, } ``` ([#4455](https://github.com/syslog-ng/syslog-ng/pull/4455)) ### Metrics * `network()`, `syslog()`: TCP connection metrics ``` syslogng_socket_connections{id="tcp_src#0",driver_instance="afsocket_sd.(stream,AF_INET(0.0.0.0:5555))",direction="input"} 3 syslogng_socket_max_connections{id="tcp_src#0",driver_instance="afsocket_sd.(stream,AF_INET(0.0.0.0:5555))",direction="input"} 10 syslogng_socket_rejected_connections_total{id="tcp_src#0",driver_instance="afsocket_sd.(stream,AF_INET(0.0.0.0:5555))",direction="input"} 96 ``` `internal()`: `internal_events_queue_capacity` metric `syslog-ng-ctl healthcheck`: new healthcheck value `syslogng_internal_events_queue_usage_ratio` ([#4411](https://github.com/syslog-ng/syslog-ng/pull/4411)) * `metrics`: new network (TCP, UDP) metrics are available on stats level 1 ``` # syslog-ng-ctl stats prometheus syslogng_socket_receive_buffer_used_bytes{id="#anon-source0#3",direction="input",driver_instance="afsocket_sd.udp4"} 0 syslogng_socket_receive_buffer_max_bytes{id="#anon-source0#3",direction="input",driver_instance="afsocket_sd.udp4"} 268435456 syslogng_socket_receive_dropped_packets_total{id="#anon-source0#3",direction="input",driver_instance="afsocket_sd.udp4"} 619173 syslogng_socket_connections{id="#anon-source0#0",direction="input",driver_instance="afsocket_sd.(stream,AF_INET(0.0.0.0:2000))"} 1 ``` ([#4374](https://github.com/syslog-ng/syslog-ng/pull/4374)) * New configuration-related metrics: ``` syslogng_last_config_reload_timestamp_seconds 1681309903 syslogng_last_successful_config_reload_timestamp_seconds 1681309758 syslogng_last_config_file_modification_timestamp_seconds 1681309877 ``` ([#4420](https://github.com/syslog-ng/syslog-ng/pull/4420)) * destination: Introduced queue metrics. * The corresponding driver is identified with the "id" and "driver_instance" labels. * Available counters are "memory_usage_bytes" and "events". * Memory queue metrics are available with "syslogng_memory_queue_" prefix, `disk-buffer` metrics are available with "syslogng_disk_queue_" prefix. * `disk-buffer` metrics have an additional "path" label, pointing to the location of the disk-buffer file and a "reliable" label, which can be either "true" or "false". * Threaded destinations, like `http`, `python`, etc have an additional "worker" label. Example metrics ``` syslogng_disk_queue_events{driver_instance="http,http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00000.rqf",reliable="true",worker="0"} 80 syslogng_disk_queue_events{driver_instance="http,http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00001.rqf",reliable="true",worker="1"} 7 syslogng_disk_queue_events{driver_instance="http,http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00002.rqf",reliable="true",worker="2"} 7 syslogng_disk_queue_events{driver_instance="http,http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00003.rqf",reliable="true",worker="3"} 7 syslogng_disk_queue_events{driver_instance="tcp,localhost:1235",id="d_network_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00000.qf",reliable="false"} 101 syslogng_disk_queue_memory_usage_bytes{driver_instance="http, http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00000.rqf",reliable="true",worker="0"} 3136 syslogng_disk_queue_memory_usage_bytes{driver_instance="http, http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00001.rqf",reliable="true",worker="1"} 2776 syslogng_disk_queue_memory_usage_bytes{driver_instance="http, http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00002.rqf",reliable="true",worker="2"} 2760 syslogng_disk_queue_memory_usage_bytes{driver_instance="http, http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00003.rqf",reliable="true",worker="3"} 2776 syslogng_disk_queue_memory_usage_bytes{driver_instance="tcp,localhost:1235",id="d_network_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00000.qf",reliable="false"} 39888 syslogng_memory_queue_events{driver_instance="http,http://localhost:1236",id="d_http#0",worker="0"} 15 syslogng_memory_queue_events{driver_instance="http,http://localhost:1236",id="d_http#0",worker="1"} 14 syslogng_memory_queue_events{driver_instance="tcp,localhost:1234",id="d_network#0"} 29 syslogng_memory_queue_memory_usage_bytes{driver_instance="http, http://localhost:1236",id="d_http#0",worker="0"} 5896 syslogng_memory_queue_memory_usage_bytes{driver_instance="http, http://localhost:1236",id="d_http#0",worker="1"} 5552 syslogng_memory_queue_memory_usage_bytes{driver_instance="tcp,localhost:1234",id="d_network#0"} 11448 ``` ([#4392](https://github.com/syslog-ng/syslog-ng/pull/4392)) * `network()`, `syslog()`, `file()`, `http()`: new byte-based metrics for incoming/outgoing events These metrics show the serialized message sizes (protocol-specific header/framing/etc. length is not included). ``` syslogng_input_event_bytes_total{id="s_network#0",driver_instance="tcp,127.0.0.1"} 1925529600 syslogng_output_event_bytes_total{id="d_network#0",driver_instance="tcp, 127.0.0.1:5555"} 565215232 syslogng_output_event_bytes_total{id="d_http#0",driver_instance="http, http://127.0.0.1:8080/"} 1024 ``` ([#4440](https://github.com/syslog-ng/syslog-ng/pull/4440)) * `disk-buffer`: Added metrics for monitoring the available space in disk-buffer `dir()`s. Metrics are available from `stats(level(1))`. By default, the metrics are generated every 5 minutes, but it can be changed in the global options: ``` options { disk-buffer( stats( freq(10) ) ); }; ``` Setting `freq(0)` disabled this feature. Example metrics: ``` syslogng_disk_queue_dir_available_bytes{dir="/var/syslog-ng"} 870109413376 ``` ([#4399](https://github.com/syslog-ng/syslog-ng/pull/4399)) * `disk-buffer`: Added metrics for abandoned disk-buffer files. Availability is the same as the `disk_queue_dir_available_bytes` metric. Example metrics: ``` syslogng_disk_queue_capacity_bytes{abandoned="true",path="/var/syslog-ng/syslog-ng-00000.rqf",reliable="true"} 104853504 syslogng_disk_queue_disk_allocated_bytes{abandoned="true",path="/var/syslog-ng/syslog-ng-00000.rqf",reliable="true"} 273408 syslogng_disk_queue_disk_usage_bytes{abandoned="true",path="/var/syslog-ng/syslog-ng-00000.rqf",reliable="true"} 269312 syslogng_disk_queue_events{abandoned="true",path="/var/syslog-ng/syslog-ng-00000.rqf",reliable="true"} 860 ``` ([#4402](https://github.com/syslog-ng/syslog-ng/pull/4402)) * `disk-buffer`: Added capacity, disk_allocated and disk_usage metrics. * "capacity_bytes": The theoretical maximal useful size of the disk-buffer. This is always smaller, than `disk-buf-size()`, as there is some reserved space for metadata. The actual full disk-buffer file can be larger than this, as syslog-ng allows to write over this limit once, at the end of the file. * "disk_allocated_bytes": The current size of the disk-buffer file on the disk. Please note that the disk-buffer file size does not strictly correlate with the number of messages, as it is a ring buffer implementation, and also syslog-ng optimizes the truncation of the file for performance reasons. * "disk_usage_bytes": The serialized size of the queued messages in the disk-buffer file. This counter is useful for calculating the disk usage percentage (disk_usage_bytes / capacity_bytes) or the remaining available space (capacity_bytes - disk_usage_bytes). Example metrics: ``` syslogng_disk_queue_capacity_bytes{driver_id="d_network#0",driver_instance="tcp,localhost:1235",path="/var/syslog-ng-00000.rqf",reliable="true"} 104853504 syslogng_disk_queue_disk_allocated_bytes{driver_id="d_network#0",driver_instance="tcp,localhost:1235",path="/var/syslog-ng-00000.rqf",reliable="true"} 17284 syslogng_disk_queue_disk_usage_bytes{driver_id="d_network#0",driver_instance="tcp,localhost:1235",path="/var/syslog-ng-00000.rqf",reliable="true"} 13188 ``` ([#4356](https://github.com/syslog-ng/syslog-ng/pull/4356)) * `kubernetes()`: Added `input_events_total` and `input_event_bytes_total` metrics. ``` syslogng_input_events_total{cluster="k8s",driver="kubernetes",id="#anon-source0",namespace="default",pod="log-generator-1682517834-7797487dcc-49hqc"} 25 syslogng_input_event_bytes_total{cluster="k8s",driver="kubernetes",id="#anon-source0",namespace="default",pod="log-generator-1682517834-7797487dcc-49hqc"} 1859 ``` ([#4447](https://github.com/syslog-ng/syslog-ng/pull/4447)) ## Bugfixes * `pdbtool test`: fix two type validation bugs: 1) When `pdbtool test` validates the type information associated with a name-value pair, it was using string comparisons, which didn't take type aliases into account. This is now fixed, so that "int", "integer" or "int64" can all be used to mean the same type. 2) When type information is missing from a `<test_value/>` tag, don't validate it against "string", rather accept any extracted type. In addition to these fixes, a new alias "integer" was added to mean the same as "int", simply because syslog-ng was erroneously using this term when reporting type information in its own messages. ([#4405](https://github.com/syslog-ng/syslog-ng/pull/4405)) * `$(format-json)`: fix RFC8259 number violation `$(format-json)` produced invalid JSON output when it contained numeric values with leading zeros or + signs. This has been fixed. ([#4415](https://github.com/syslog-ng/syslog-ng/pull/4415)) * `grouping-by()`: fix `persist-name()` option not taken into account ([#4390](https://github.com/syslog-ng/syslog-ng/pull/4390)) * `python()`, `db-parser()`, `grouping-by()`, `add-contextual-data()`: fix typing compatibility with <4.0 config versions ([#4394](https://github.com/syslog-ng/syslog-ng/pull/4394)) * `python`: Fixed a crash which occurred at reloading after registering a confgen plugin. ([#4459](https://github.com/syslog-ng/syslog-ng/pull/4459)) * `date-parser()`: fix `%z` when system timezone has no daylight saving time ([#4401](https://github.com/syslog-ng/syslog-ng/pull/4401)) * Consider messages consumed into correlation states "matching": syslog-ng's correlation functionality (e.g. grouping-by() or db-parser() with such rules) drop individual messages as they are consumed into a correlation contexts and you are using `inject-mode(aggregate-only)`. This is usually happens because you are only interested in the combined message and not in those that make up the combination. However, if you are using correlation with conditional processing (e.g. if/elif/else or flags(final)), such messages were erroneously considered as unmatching, causing syslog-ng to take the alternative branch. Example: With a configuration similar to this, individual messages are consumed into a correlation state and dropped by `grouping-by()`: ``` log { source(...); if { grouping-by(... inject-mode(aggregate-only)); } else { # alternative branch }; }; ``` The bug was that these individual messages also traverse the `else` branch, even though they were successfully processed with the inclusion into the correlation context. This is not correct. The bugfix changes this behaviour. ([#4370](https://github.com/syslog-ng/syslog-ng/pull/4370)) * `netmask6()`: fix crash when user specifies too long mask ([#4429](https://github.com/syslog-ng/syslog-ng/pull/4429)) * `afprog`: Fixed possible freezing on some OSes ([#4438](https://github.com/syslog-ng/syslog-ng/pull/4438)) * `network()`, `syslog()`, `syslog-parser()`: fix null termination of SDATA param names ([#4429](https://github.com/syslog-ng/syslog-ng/pull/4429)) * `python()`: fix LogMessage subscript not raising KeyError on non-existent keys When message fields were queried (`msg["key"]`) and the given key did not exist, `None` or an empty string was returned (depending on the version of the config). Neither was correct, now a KeyError occurs in such cases. ([#4410](https://github.com/syslog-ng/syslog-ng/pull/4410)) * `$(python)`: fix template function prefix being overwritten when using datetime types ([#4410](https://github.com/syslog-ng/syslog-ng/pull/4410)) * `disk-buffer`: Fixed queued messages stats counting, when a disk-buffer became corrupted. ([#4385](https://github.com/syslog-ng/syslog-ng/pull/4385)) * `$(format-json)`: fix escaping control characters `$(format-json)` produced invalid JSON output when a string value contained control characters. ([#4417](https://github.com/syslog-ng/syslog-ng/pull/4417)) * `disk-buffer()`: fix deinitialization when starting syslog-ng with invalid configuration ([#4418](https://github.com/syslog-ng/syslog-ng/pull/4418)) * `python()`: fix exception handling when LogMessage value conversion fails ([#4410](https://github.com/syslog-ng/syslog-ng/pull/4410)) * `json-parser()`: Fixed parsing non-string arrays. syslog-ng now no longer parses non-string arrays to list of strings, losing the original type information of the array's elements. ([#4396](https://github.com/syslog-ng/syslog-ng/pull/4396)) * `disk-buffer`: Fixed a rare race condition when calculating disk-buffer filename. ([#4381](https://github.com/syslog-ng/syslog-ng/pull/4381)) * `python-persist`: fix off-by-one overflow ([#4429](https://github.com/syslog-ng/syslog-ng/pull/4429)) ## Packaging * The `--with-python-venv-dir=path` configure option can be used to modify the location of syslog-ng's venv. The default is still `${localstatedir}/python-venv`. ([#4465](https://github.com/syslog-ng/syslog-ng/pull/4465)) ## Other changes * The `sdata-prefix()` option does not accept values longer than 128 characters. ([#4429](https://github.com/syslog-ng/syslog-ng/pull/4429)) * `grouping-by()`: Remove setting of the `${.classifier.context_id}` name-value pair in all messages consumed into a correlation context. This functionality is inherited from db-parser() and has never been documented for `grouping-by()`, has of limited use, and any uses can be replaced by the use of the built-in macro named `$CONTEXT_ID`. Modifying all consumed messages this way has significant performance consequences for `grouping-by()` and removing it outweighs the small incompatibility this change introduces. The similar functionality in `db-parser()` correlation is not removed with this change. ([#4424](https://github.com/syslog-ng/syslog-ng/pull/4424)) * `config`: Added `internal()` option to `source`s, `destination`s, `parser`s and `rewrite`s. Its main usage is in SCL blocks. Drivers configured with `internal(yes)` register their metrics on level 3. This makes developers of SCLs able to create metrics manually with `metrics-probe()` and "disable" every other metrics, they do not need. ([#4451](https://github.com/syslog-ng/syslog-ng/pull/4451)) * The following Prometheus metrics have been renamed: `log_path_{in,e}gress` -> `route_{in,e}gress_total` `internal_source` -> `internal_events_total` The `internal_queue_length` stats counter has been removed. It was deprecated since syslog-ng 3.29. ([#4411](https://github.com/syslog-ng/syslog-ng/pull/4411)) ## syslog-ng Discord For a bit more interactive discussion, join our Discord server: https://discord.gg/E65kP9aZGm ## Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Alex Becker, Attila Szakacs, Balazs Scheidler, Hofi, László Várady, Muhammad Shanif, Ricfilipe, Romain Tartière

1 0

Insider 2023-05: learning; UDP; upgrading;
by Peter Czanik (pczanik) 11 May '23

11 May '23

Dear syslog-ng users, This is the 110th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Learning syslog-ng, the easier way ---------------------------------- Last year, one of the returning questions I received was how to learn syslog-ng. My answer was that read the first few chapters of the documentation, read my blogs related to your use case, and then read a few relevant parts from the rest of the documentation. Our documentation is praised by users, but it is still a reference documentation. I was asked if a less detailed, more to the point, preferably video tutorial is available. Your request was heard. In the past couple of months, I published a tutorial series in blog and video format, which brings you from basic logging concepts to using syslog-ng to collect, parse, enrich log messages and store them to Elasticsearch. Of course, these 5-10 minute videos are not enough to learn anything in depth, but they introduce you to all major syslog-ng functionalities. https://www.syslog-ng.com/community/b/blog/posts/learning-syslog-ng-the-eas… Why syslog over UDP loses messages and how to avoid that -------------------------------------------------------- Message loss related resiliency with regards to syslog over UDP has always been a problem. Users who start looking at these metrics generally report a 30-40% loss of messages for syslog over UDP, but drops of up to 90% are not unheard of. Let me get to the bottom of the reasons why messages are dropped and how these problems can be solved, or at least mitigated. https://axoflow.com/syslog-over-udp-message-loss-1/ Upgrade problems from syslog-ng 3 to 4 -------------------------------------- Version 4 of syslog-ng works perfectly well in version 3 compatibility mode. However, if you want to use the syslog-ng 4 features, you need to be aware of some significant changes. If you have a simple configuration, like those in Linux distributions, then simply rewriting the version string is most likely enough. However, if you use PatternDB or JSON parsing, any Python code, or an Elasticsearch, or MongoDB destination, you have to be aware of the changes. >From this blog you can learn about type support, how this can affect you, changes in Python support, and some tips how to prepare for the upgrade. https://www.syslog-ng.com/community/b/blog/posts/upgrade-problems-from-sysl… WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

using -dv on startup vs syslog-ng-ctl debug set=off
by Jerry Scharf 28 Apr '23

28 Apr '23

Hi, We are chasing an intermittent bug with slow reloads. We are following the debug suggestions and adding -dv to the execstart in the service file. Because of the load of messages that we push, I want to turn off the debugging after we are past the window when the slow reload happens. I naively tried doing "syslog-ng-ctl debug set=off" but it responded that the flag didn't change. Is there some way to tell syslog-ng to allow the -ctl command to make the change. I can imagine why this would be desired sometimes, but at 10k lines per minute it is going to fill up /tmp. right now I am doing ugly things to solve the problem, but really want to find a better way. thanks, jerry -- Jerry Scharf Pure Storage

2 1

What is the status of syslog-ng 4 in your environment?
by Peter Czanik (pczanik) 27 Apr '23

27 Apr '23

Hi, Version 4 of syslog-ng was released almost half a year ago. It adds type support to name-value pairs, and lots of improvements to Python and Kubernetes support. I just started a Twitter / Mastodon / LinkedIn poll to learn, what is the status of syslog-ng 4 in your environment? * https://fosstodon.org/@PCzanik/110269557855094446 * https://twitter.com/sngose/status/1651486235469201408 * https://www.linkedin.com/posts/peterczanik_poll-syslogabrng-python-activity… Of course, I'd also be happy to hear some feedback, not just a poll result. So, let me know why you are on syslog-ng 4 already, or why do you delay your upgrade to this new major version! If hearing about syslog-ng 4 completely surprised you, check https://www.syslog-ng.com/community/b/blog/posts/upgrade-problems-from-sysl… which describes some of the major changes, and what to be careful about when you upgrade! Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0