- syslog-ng - lists.balabit.hu

working with Sigma rules?
by Peter Czanik (pczanik) 11 Aug '23

11 Aug '23

Hi, Recently I was asked if Sigma rules (https://github.com/SigmaHQ/sigma) are supported by syslog-ng. Syslog-ng has message parsing, filtering, and can be used for alerting. But I'm not aware of any tools that could turn Sigma rules into PatternDB and syslog-ng.conf Syslog-ng can send logs to Splunk, ElasticSearch / OpenSearch or Graylog, all which already have sigma rules integrations. Of course, many users use/abuse syslog-ng as a kind of SIEM-lite as it is very good at real-time alerting. However, as far as I can see, Sigma rules are better suited for threat hunting on the SIEM side. If you already Sigma rules with syslog-ng or any other way: please share your experiences! Thanks, Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

2 1

Insider 2023-08: sngbench; 4.3.1 release; syslog-ng 4; Python;
by Peter Czanik (pczanik) 10 Aug '23

10 Aug '23

Dear syslog-ng users, This is the 111th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Introducing sngbench: a shell script to performance test your syslog-ng ---------------------------------- One of the returning questions I receive is how many log messages can a given hardware handle. My typical answer is that it depends on the configuration. I have now an answer, or rather a tool to answer your question sngbench.sh. It is a shell script that runs from localhost and uses loggen, the bundled benchmarking and testing tool of syslog-ng. It comes with two configurations: a performance-optimized and a realistic one. You are also free to extend sngbench with your own configurations. https://www.syslog-ng.com/community/b/blog/posts/introducing-sngbench-a-she… Version 4.3.1 of syslog-ng available ------------------------------------ Receive or send any kinds of logs, traces, and metrics with syslog-ng using OTLP/gRPC, improved performance for high-volume connections, and more! Containers and packages for major Linux distributions and FreeBSD are already available. https://axoflow.com/axosyslog-release-4-3/ For the complete release notes, check: https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-4.3.1 Syslog-ng Python packaging -------------------------- In version 4 of syslog-ng, the role of Python became even more important. Previously, all parts of syslog-ng could be extended using Python code, but no actual Python code was provided with syslog-ng. Version 4.0 added a Kubernetes module implemented in Python, while version 4.2 added support for Hypr. But how can we ensure that all Python dependencies are met? https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-python-packaging Getting syslog-ng 4 ------------------- Version 4 of syslog-ng was released last December. Quite a few people use it already in production. How can you install it for a test drive? It might be already available in your Linux distribution. There are also several unofficial repositories with the latest syslog-ng. >From this blog, you can learn how to check your syslog-ng version, where to check if it is not yet installed, and a few additional resources, if you want to install the latest version from unofficial repositories. https://www.syslog-ng.com/community/b/blog/posts/getting-syslog-ng-4 WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Need help to send Datetime in BSON ISODATE to MongoDB?
by Ibrahim Al Mahfooz 07 Aug '23

07 Aug '23

Hello, I have integration working fine between syslog-ng v3.38 and MongoDB v6. I am working to enable MongoDB Time-Series Collections on a testing DB for the purpose of having better efficiency and improved queries over time. I created the DB and collection, tested it by Mongo Compass client, then I tried to send the data from syslog-ng but it didn't work. MongoDB is throwing the following error: "17:12:57. 483 | [2023-08-06T17:12:57.355549] Failed to insert into MongoDB; time_reopen='10', reason='\'datetime\' must be present and contain a valid BSON UTC datetime value', driver='d_mongodb_ttl#0" My understanding is that it is a must to send the time/date to MongoDB with Time Series enabled DB a BSON format Date. Any suggestions? *Below is the mongoDB destination config:* destination d_mongodb_ttl { mongodb( uri("mongodb://192.168.1.19:27017/syslog-ng_ttl") collection("logs") value-pairs( scope("selected-macros") exclude("SOURCEIP") exclude("TAGS") pair("datetime" "${S_ISODATE}") pair("privateip" "${privateip}") pair("publicip" "${publicip}") pair("publicportrange" "${publicportrange}") pair("portaction" "${AI}") pair("portstatus" "${AL}") pair("host" "${HOST}")) ); }; Appreciate your help

3 4

syslog-ng 4.3.1
by László Várady 03 Aug '23

03 Aug '23

Dear syslog-ng users, We are pleased to announce the 4.3.1 version of syslog-ng, which has been released and is now available on GitHub: https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-4.3.1 Packages <https://github.com/syslog-ng/syslog-ng#installation-from-binaries> are also available for various platforms. The AxoSyslog project provides cloud-ready container images <https://axoflow.com/cloud-ready-syslog-ng-images/>, Helm charts <https://axoflow.com/axosyslog-log-collection-for-kubernetes/>, and open-source documentation at the link below: https://axoflow.com/docs/axosyslog-core/ 4.3.1 *This is the combination of the news entries of 4.3.0 and 4.3.1. 4.3.1 hotfixed* *a python-parser() related crash and a metrics related memory leak. It also* *added Ubuntu 23.04 and Debian 12 support for APT packages and the opensearch()* *destination.* Read Axoflow's blog post <https://axoflow.com/axosyslog-release-4-3/> for more details. Highlights parallelize() support for pipelines syslog-ng has traditionally performed processing of log messages arriving from a single connection sequentially. This was done to ensure message ordering as well as most efficient use of CPU on a per message basis. This mode of operation is performing well as long as we have a relatively large number of parallel connections, in which case syslog-ng would use all the CPU cores available in the system. In case only a small number of connections deliver a large number of messages, this behaviour may become a bottleneck. With the new parallelization feature, syslog-ng gained the ability to re-partition a stream of incoming messages into a set of partitions, each of which is to be processed by multiple threads in parallel. This does away with ordering guarantees and adds an extra per-message overhead. In exchange it will be able to scale the incoming load to all CPUs in the system, even if coming from a single, chatty sender. To enable this mode of execution, use the new parallelize() element in your log path: log { source { tcp( port(2000) log-iw-size(10M) max-connections(10) log-fetch-limit(100000) ); }; parallelize(partitions(4)); # from this part on, messages are processed in parallel even if # messages are originally coming from a single connection parser { ... }; destination { ... }; }; The config above will take all messages emitted by the tcp() source and push the work to 4 parallel threads of execution, regardless of how many connections were in use to deliver the stream of messages to the tcp() driver. parallelize() uses round-robin to allocate messages to partitions by default. You can however retain ordering for a subset of messages with the partition-key() option. You can use partition-key() to specify a message template. Messages that expand to the same value are guaranteed to be mapped to the same partition. For example: log { source { tcp( port(2000) log-iw-size(10M) max-connections(10) log-fetch-limit(100000) ); }; parallelize(partitions(4) partition-key("$HOST")); # from this part on, messages are processed in parallel if their # $HOST value differs. Messages with the same $HOST will be mapped # to the same partition and are processed sequentially. parser { ... }; destination { ... }; }; NOTE: parallelize() requires a patched version of libivykis that contains this PR buytenh/ivykis#25 <https://github.com/buytenh/ivykis/pull/25>. syslog-ng source releases bundle this version of ivykis in their source trees, so if you are building from source, be sure to use the internal version (--with-ivykis=internal). You can also use Axoflow's cloud native container image for syslog-ng, named AxoSyslog (https://github.com/axoflow/axosyslog-docker) which also incorporates this change. (#3966 <https://github.com/syslog-ng/syslog-ng/pull/3966>) Receiving and sending OpenTelemetry (OTLP) messages The opentelemetry() source, parser and destination are now available to receive, parse and send *OTLP/gRPC* messages. syslog-ng accepts logs <https://github.com/open-telemetry/opentelemetry-proto/blob/v0.20.0/opentele…>, metrics <https://github.com/open-telemetry/opentelemetry-proto/blob/v0.20.0/opentele…> and traces <https://github.com/open-telemetry/opentelemetry-proto/blob/v0.20.0/opentele…> . The incoming fields are not available through syslog-ng log message name-value pairs for the user by default. This is useful for forwarding functionality (the opentelemetry() destination can access and format them). If such functionality is required, you can configure the opentelemetry() parser, which maps all the fields with some limitations. The behavior of the opentelemetry() parser is the following: The name-value pairs always start with .otel. prefix. The type of the message is stored in .otel.type (possible values: log, metric and span). The resource info is mapped to .otel.resource.<...> (e.g.: .otel.resource.dropped_attributes_count, .otel.resource.schema_url ...), the scope info is mapped to .otel.scope.<...> (e.g.: .otel.scope.name, .otel.scope.schema_url, ...). The fields of log records are mapped to .otel.log.<...> (e.g. .otel.log.body, .otel.log.severity_text, ...). The fields of metrics are mapped to .otel.metric.<...> (e.g. . otel.metric.name, .otel.metric.unit, ...), the type of the metric is mapped to .otel.metric.data.type (possible values: gauge, sum, histogram, exponential_histogram, summary) with the actual data mapped to .otel.metric.data.<type>.<...> (e.g.: .otel.metric.data.gauge.data_points.0.time_unix_nano, ...). The fields of traces are mapped to .otel.span.<...> (e.g. .otel.span.name, .otel.span.trace_state, ...). repeated fields are given an index (e.g. .otel.span.events.5.time_unix_nano ). The mapping of AnyValue <https://github.com/open-telemetry/opentelemetry-proto/blob/v0.20.0/opentele…> type fields is limited. string, bool, int64, double and bytes values are mapped with the respective syslog-ng name-value type (e.g. .otel.resource.attributes.string_key => string_value), however ArrayValue and KeyValueList types are stored serialized with protobuf type. protobuf and bytes types are not directly available for the user, unless an explicit type cast is added (e.g. "bytes(${.otel.log.span_id})") or --include-bytes is passed to name-value iterating template functions (e.g. $(format-json .otel.* --include-bytes), which will base64 encode the bytes content). Three authentication methods are available in the source auth() block: insecure() (default), tls() and alts(). tls() accepts the key-file(), cert-file(), ca-file() and peer-verify() (possible values: required-trusted, required-untrusted, optional-trusted and optional-untrusted) options. ALTS <https://grpc.io/docs/languages/cpp/alts/> is a simple to use authentication, only available within Google's infrastructure. The same methods are available in the destination auth() block, with two differences: tls(peer-verify()) is not available, and there is a fourth method, called ADC <https://cloud.google.com/docs/authentication/application-default-credentials>, which accepts the target-service-account() option, where a list of service accounts can be configured to match against when authenticating the server. Example configs: log otel_forward_mode_alts { source { opentelemetry( port(12345) auth(alts()) ); }; destination { opentelemetry( url("my-otel-server:12345") auth(alts()) ); }; }; log otel_to_non_otel_insecure { source { opentelemetry( port(12345) ); }; parser { opentelemetry(); }; destination { network( "my-network-server" port(12345) template("$(format-json .otel.* --shift-levels 1 --include-bytes)\n") ); }; }; log non_otel_to_otel_tls { source { network( port(12346) ); }; destination { opentelemetry( url("my-otel-server:12346") auth( tls( ca-file("/path/to/ca.pem") key-file("/path/to/key.pem") cert-file("/path/to/cert.pem") ) ) ); }; }; (#4523 <https://github.com/syslog-ng/syslog-ng/pull/4523>) (#4510 <https://github.com/syslog-ng/syslog-ng/pull/4510>) Sending messages to CrowdStrike Falcon LogScale (Humio) The logscale() destination feeds LogScale via the Ingest API <https://library.humio.com/falcon-logscale/api-ingest.html#api-ingest-struct…> . Minimal config: destination d_logscale { logscale( token("my-token") ); }; Additional options include: - url() - rawstring() - timestamp() - timezone() - attributes() - extra-headers() - content-type() (#4472 <https://github.com/syslog-ng/syslog-ng/pull/4472>) Features - afmongodb: Bulk MongoDB insert is added via the following options - bulk (*yes*/no) turns on/off bulk insert <http://mongoc.org/libmongoc/current/bulk.html>usage, no forces the old behavior (each log is inserted one by one into the MongoDB) - bulk_unordered (yes/*no*) turns on/off unordered MongoDB bulk operations <http://mongoc.org/libmongoc/current/bulk.html#unordered-bulk-write-operatio…> - bulk_bypass_validation (yes/*no*) turns on/off MongoDB bulk operations validation <http://mongoc.org/libmongoc/1.23.3/bulk.html#bulk-operation-bypassing-docum…> - write_concern (unacked/*acked*/majority/n > 0) sets write concern mode of the MongoDB operations <http://mongoc.org/libmongoc/1.23.3/bulk.html#bulk-operation-write-concerns>, both bulk and single NOTE: Bulk sending is only efficient if the used collection is constant (e.g. not using templates) or the used template does not lead to too many collections switching within a reasonable time range. (#4483 <https://github.com/syslog-ng/syslog-ng/pull/4483>) - sql: Added 2 new options - quote_char to aid custom quoting for table and index names (e.g. MySQL needs sometimes this for certain identifiers) *NOTE*: Using a back-tick character needs a special formatting as syslog-ng uses it for configuration parameter names, so for that use: quote_char("``") (double back-tick) - dbi_driver_dir to define an optional DBI driver location for DBD initialization NOTE: libdbi and libdbi-drivers OSE forks are updated, afsql now should work nicely both on ARM and X86 macOS systems too (tested on macOS 13.3.1 and 12.6.4) Please do not use the pre-built ones (e.g. 0.9.0 from Homebrew), build from the *master* of the following - https://github.com/balabit-deps/libdbi-drivers - https://github.com/balabit-deps/libdbi (#4460 <https://github.com/syslog-ng/syslog-ng/pull/4460>) - opensearch: Added a new destination. It is similar to elasticsearch-http(), with the difference that it does not have the type() option, which is deprecated and advised not to use. (#4560 <https://github.com/syslog-ng/syslog-ng/pull/4560>) Bugfixes - network(),syslog(),tcp() destination: fix TCP keepalive tcp-keepalive-*() options were broken on the destination side since v3.34.1. (#4559 <https://github.com/syslog-ng/syslog-ng/pull/4559>) - Fixed a hang, which happend when syslog-ng received exremely low CPU time. (#4524 <https://github.com/syslog-ng/syslog-ng/pull/4524>) - $(format-json): Fixed a bug where sometimes an unnecessary comma was added in case of a type cast failure. (#4477 <https://github.com/syslog-ng/syslog-ng/pull/4477>) - Fix flow-control when fetch-limit() is set higher than 64K In high-performance use cases, users may configure log-iw-size() and fetch-limit() to be higher than 2^16, which caused flow-control issues, such as messages stuck in the queue forever or log sources not receiving messages. (#4528 <https://github.com/syslog-ng/syslog-ng/pull/4528>) - int32() and int64() type casts: accept hex numbers as proper number representations just as the @NUMBER@ parser within db-parser(). Supporting octal numbers were considered and then rejected as the canonical octal representation for numbers in C would be ambigious: a zero padded decimal number could be erroneously considered octal. I find that log messages contain zero padded decimals more often than octals. (#4535 <https://github.com/syslog-ng/syslog-ng/pull/4535>) - Fixed compilation on platforms where SO_MEMINFO is not available (#4548 <https://github.com/syslog-ng/syslog-ng/pull/4548>) - python: InstantAckTracker, ConsecutiveAckTracker and BatchedAckTracker are now called properly. Added proper fake classes for the InstantAckTracker, ConsecutiveAckTracker and BatchedAckTracker classes, and the wapper now calls the super class' constructor. Previusly the super class' constructor was not called which caused the python API to never call into the C API, which's result was that that the callback was never called. (#4549 <https://github.com/syslog-ng/syslog-ng/pull/4549>) - python: Fixed a crash when reloading with a config, which uses a python parser with multiple references. (#4552 <https://github.com/syslog-ng/syslog-ng/pull/4552>) (#4567 <https://github.com/syslog-ng/syslog-ng/pull/4567>) - mqtt(): Fixed the name of the stats instance (mqtt-source) to conform to the standard comma-separated format. (#4551 <https://github.com/syslog-ng/syslog-ng/pull/4551>) - metrics: Fixed a memory leak which happened during reload, and was introduced in 4.3.0. (#4568 <https://github.com/syslog-ng/syslog-ng/pull/4568>) Packaging - scl.conf: The scl.conf file has been moved to /share/syslog-ng/include/scl.conf (#4534 <https://github.com/syslog-ng/syslog-ng/pull/4534>) - C++ plugins: Some of syslog-ng's plugins now contain C++ code. By default they are being built if a C++ compiler is available. Disabling it is possible with --disable-cpp. Affected plugins: - lib/syslog-ng/libexamples.so - --disable-cpp will only disable the C++ part ( random-choice-generator()) - lib/syslog-ng/libotel.so (#4484 <https://github.com/syslog-ng/syslog-ng/pull/4484>) - debian: A new module is added, called syslog-ng-mod-grpc. Its dependencies are: protobuf-compiler, protobuf-compiler-grpc, libprotobuf-dev, libgrpc++-dev. Building the module can be toggled with --enable-grpc. (#4510 <https://github.com/syslog-ng/syslog-ng/pull/4510>) - pcre: syslog-ng now uses pcre2 (8 bit) as a dependency instead of pcre. The minimum pcre2 version is 10.0. (#4537 <https://github.com/syslog-ng/syslog-ng/pull/4537>) Notes to developers - lib/logmsg: Public field LogMessage::protected has been renamed to LogMessage::write_protected. Direct usage of this field is discouraged, instead use the following functions: - log_msg_is_write_protected() - log_msg_write_protect() (#4484 <https://github.com/syslog-ng/syslog-ng/pull/4484>) - lib/templates: Public field LogTemplate::template has been renamed to LogTemplate::template_str. (#4484 <https://github.com/syslog-ng/syslog-ng/pull/4484>) Other changes - syslog-ng-cfg-db: Moved to a separate repository. It is available at: https://github.com/alltilla/syslog-ng-cfg-helper (#4475 <https://github.com/syslog-ng/syslog-ng/pull/4475>) - disk-buffer: Added alternative option names disk-buf-size() -> capacity-bytes() qout-size() -> front-cache-size() mem-buf-length() -> flow-control-window-size() mem-buf-size() -> flow-control-window-bytes() Old option names are still available. Example configs: tcp( "127.0.0.1" port(2001) disk-buffer( reliable(yes) capacity-bytes(1GiB) flow-control-window-bytes(200MiB) front-cache-size(1000) ) ); tcp( "127.0.0.1" port(2001) disk-buffer( reliable(no) capacity-bytes(1GiB) flow-control-window-size(10000) front-cache-size(1000) ) ); (#4526 <https://github.com/syslog-ng/syslog-ng/pull/4526>) - selinux: Added RHEL9 support for the selinux policies Added RHEL9 support for the selinux policies at contrib/selinux (#4509 <https://github.com/syslog-ng/syslog-ng/pull/4509>) - metrics: replace driver_instance (stats_instance) with metric labels The new metric system had a label inherited from legacy: driver_instance. This non-structured label has been removed and different driver-specific labels have been added instead, for example: Before: syslogng_output_events_total{driver_instance="mongodb,localhost:27017,defaultdb,,coll",id="#anon-destination1#1",result="queued"} 4 After: syslogng_output_events_total{driver="mongodb",host="localhost:27017",database="defaultdb",collection="coll",id="#anon-destination1#1",result="queued"} 4 This change may affect legacy stats outputs (syslog-ng-ctl stats), for example, persist-name()-based naming is no longer supported in this old format. (#4551 <https://github.com/syslog-ng/syslog-ng/pull/4551>) - APT packages: Added Ubuntu Lunar Lobster and Debian Bookworm support. (#4561 <https://github.com/syslog-ng/syslog-ng/pull/4561>) syslog-ng Discord For a bit more interactive discussion, join our Discord server: [image: Axoflow Discord Server] <https://discord.gg/E65kP9aZGm> Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andreas Friedmann, Attila Szakacs, Balazs Scheidler, Bálint Horváth, Chuck Silvers, Evan Rempel, Hofi, Kovacs, Gergo Ferenc, László Várady, Romain Tartière, Ryan Faircloth, vostrelt

1 0

Properly Setting Facility and Severity into destination
by Greg Christopher 01 Aug '23

01 Aug '23

Hi All, Did my best (about a week effort) to find this information online and apologize if it's obvious. I am trying to use syslog-ng as a client to send messages to a syslog compliant server with the proper severity and facility set. I am starting with plain-text log files. The syslog rfc as well as the syslog-ng documentation are pretty clear about the syslog message format itself. There are even syslog-ng functions to substitute severity and facility using a function called "rewrite". But this doesn't seem to work if you are starting with a regular application log. In other words, there is nothing to "rewrite" since this header was never there to begin with. Although I initially configured my application log with flags(no-parse) , I attempted to remove it so that my rewrite attempt would at least have a facility and severity to "rewrite", but this also seems to have failed. Next I attempted prepending "<81>" (8*10 + 1) to the application messages to get the right priority field but this failed to change what the syslog-ng server on the other end received, as I have modified its output to include severity and facility:--- destination d_local { file("/var/log/syslog-ng/$HOST" template("Fac: $FACILITY Pri: $PRIORITY $ISODATE $HOST $(format-welf --scope all-nv-pairs)\n") frac-digits(3)); }; --- It does seem to be working properly but the above two approaches to increasing the severity and noting this is a security facility message failed. So it seems this use case where we create syslog entries from scratch sort of throws a curve ball at syslog-ng clients, at least from what I can figure. Any help on how I can set these manually into the destination would be greatly appreciated. Here is my syslog-ng.conf at the moment; note I pull log file name out of environment variable:--- @version: 4.1 # Configure the source to read from the messages log file # no-parse means whole line becomes "message" portion. # Template should indicate security alert to syslog daemon source s_APP_LOG { file(`APP_LOG`); # file(`APP_LOG` flags(no-parse)); }; rewrite set_pri_fields { set-severity("alert"); set-facility("security"); }; destination d_destination { syslog(`syslogServer`); }; # Configure the log statement to route messages from the application log file to # the syslog server specified on the command line log { source(s_APP_LOG); rewrite(set_pri_fields); destination(d_destination); }; ---Greg

2 2

parallelize: modified ivykis for Fedora / RHEL
by Peter Czanik (pczanik) 01 Aug '23

01 Aug '23

Hi, Version 4.3.1 of syslog-ng was released last weekend and it contains an interesting feature: parallelize(). TL;DR: it allows faster processing of single, high-traffic TCP connections. You can read more about it in my latest blog at: https://www.syslog-ng.com/community/b/blog/posts/accelerating-single-tcp-co… The problem is that it requires a modified ivykis library. The syslog-ng source code bundles ivykis, and has a modified version. However, all Fedora and RHEL syslog-ng packages use a separate ivykis package. I feel uncomfortable with modifying the "official" ivykis package in Fedora / RHEL with patches not yet in the upstream ivykis sources. And it would not help the majority of syslog-ng users anyway, who use RHEL 8 and compatible operating systems. I could only add it to Fedora and upcoming RHEL releases. However, I want to enable this feature for the majority of syslog-ng users. Plan: add a modified ivykis library to my unofficial repositories. The version would stay 0.42.4, but the release would be a high number, making sure that it stays larger than the "official" package. First I'd introduce it to my git snapshot repository, and once I collect some positive feedback, I implement it in other repos too for upcoming syslog-ng versions. What do you think? Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

3 2

Syslog-ng is not starting
by Gundlapally, Navanitha 11 Jul '23

11 Jul '23

Team, I am new to syslog-ng and recently installed syslog-ng v3.23 on RHEL 8. However while trying to restart syslog-ng service it is failing with below error. I read it on some blog that installing "http" should fix it but didn't find any steps on how to intall the module. I did try to do a fresh installation of syslog-ng 3.23 and no luck there. Plugin module not found in 'module-path'; module-path='/usr/lib64/syslog-ng', module='http' Thank you NG. This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system.

3 3

Re: [syslog-ng] syslog-ng EPS
by Wilson, Jonathan L 12 Jun '23

12 Jun '23

We use: syslog-ng-ctl query get source.*.processed --reset In a cron job just before midnight to capture the entire day's stats. They show up in the format: source.s_local.processed=4722471 source.s_BSD_TCP.processed=6299967 source.s_RFC5424_TCP.processed=0 source.s_network_source.processed=27695900 . . . where "s_local", "s_BSD_TCP", etc. are the source names inside the syslog-ng.conf file. Note that the "-reset" on the end does just that, resets the stats to zero. A little scripting to divide the numbers above by 86400 will give you the count of messages per second by source. (We convert ours to JSON and dump it into Splunk for later viewing.) Regards, Jonathan Wilson

1 0

Can't Find Syslog-ng 4.0.0 Administration Guide Link
by Sumanta Banerjee 09 Jun '23

09 Jun '23

Hi All, Can someone please help in sending me the link for Syslog-ng 4.0.0 administration guide (if at all available). Thanks & Regards, Sumanta Banerjee Splunk Admin | CISO | Aviva Group Tel: +91-8420892593 24x7x365: +44 1603 208 582 sumanta.banerjee(a)aviva.com<mailto:sumanta.banerjee@aviva.com> GlobalCyberSecurityEngineeringTeam(a)aviva.com<mailto:GlobalCyberSecurityEngineeringTeam@aviva.com> www.aviva.com<http://www.aviva.com> Wipro Technologies - SJP2, Bangalore, India [cid:image001.gif@01D99AE4.B2631840] Aviva: Public Aviva plc, registered Office: St. Helen's, 1 Undershaft, London EC3P 3DQ. Registered in England No. 02468686. www.aviva.com This message and any attachments may be confidential or legally privileged. If you are not the intended recipient, please telephone or e-mail the sender and delete this message and any attachments from your system. Also, if you are not the intended recipient you must not copy this message or attachments or disclose the contents to any other person. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Aviva.

1 0

syslog-ng EPS
by Dragan Zecevic 08 Jun '23

08 Jun '23

Hi guys, is it possible to calculate daily events per second for all syslog messages coming to syslog-ng? Or even better separated by sources. I have syslog-ng 3 (3.34.1). Thanks. Br, Dragan

2 1

One destination, multiple TCP connections?
by Steve Bernacki 08 Jun '23

08 Jun '23

Hello, We have a syslog TCP destination that receives logs from many different destinations. During peak log volume times, this destination begins to back up and queue. We've found that if we define multiple destinations for the same IP/port and route logs through these multiple destinations, the destination is able to handle these peak log volume times much better. Our current solution of having a destination for each source is pretty kludgy. Is there a way to define a TCP destination and specify to open, say, 10 connections to the destination and load balance the traffic over those 10 connections? Thanks. Steve

2 1

Fedora / RHEL & Co: Python split -- request for testing
by Peter Czanik (pczanik) 08 Jun '23

08 Jun '23

Hi, Recently I wrote a blog about Python packaging problems and my plans to split the Python-based drivers of syslog-ng from the syslog-ng Python plugin: https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-python-packaging I implemented it for my syslog-ng git snapshot repo for Fedora / RHEL: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng-githead/ Please test it ASAP, if you got a bit of time. Originally I planned to use it only for the 4.3 release, but if I get positive responses, I'll also backport these changes to my syslog-ng42 repo, and use these changes also for the syslog-ng-stable repo. The blog post explains why is this important, but TL;DR: this way you do not have to install 50+MB of dependencies you will likely never use, and syslog-ng will not crash on start on RHEL 8 & Co. due to lack of dependencies. Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 2

uestion
by Johnson, Linwood 06 Jun '23

06 Jun '23

I get the following when I check [root@server1~]# syslog-ng WARNING: the match() filter without the use of the value() option is deprecated and hinders performance, please update your configuration; The kernel refused to set the receive buffer (SO_RCVBUF) to the requested size, you probably need to adjust buffer related kernel parameters; so_rcvbuf='536870912', so_rcvbuf_set='425984' [root@ server1~]# sysctl -a |grep rmem_max net.core.rmem_max = 212992 sysctl: reading key "net.ipv6.conf.all.stable_secret" sysctl: reading key "net.ipv6.conf.default.stable_secret" sysctl: reading key "net.ipv6.conf.eth0.stable_secret" sysctl: reading key "net.ipv6.conf.lo.stable_secret" [root@ server1 ~]# sysctl -a |grep vbuf net.ipv4.tcp_moderate_rcvbuf = 1 sysctl: reading key "net.ipv6.conf.all.stable_secret" sysctl: reading key "net.ipv6.conf.default.stable_secret" sysctl: reading key "net.ipv6.conf.eth0.stable_secret" sysctl: reading key "net.ipv6.conf.lo.stable_secret" net.sctp.rcvbuf_policy = 0 [root@ server1~ ]# cat /proc/sys/net/ipv4/tcp_rmem 4096 87380 6291456 [root@ server1~ ]# cat /proc/sys/net/core/rmem_default 212992 [root@ server1~ ]# cat /proc/sys/net/core/rmem_max 212992 [root@ server1~ ]# grep . /proc/sys/net/ipv4/tcp*mem /proc/sys/net/ipv4/tcp_mem:173202 230936 346404 /proc/sys/net/ipv4/tcp_rmem:4096 87380 6291456 /proc/sys/net/ipv4/tcp_wmem:4096 16384 4194304 [root@ server1~ hisotry]# I attempted to make the changes on the server. What can I do to resolve this issue? -- Thanks in advance, Linwood Johnson Senior Engineer | Unix Systems Administrator Site Reliability Tools [image: NewsTech] P.O Box 300 | Princeton NJ 08543 *(W)* 609-627-2179 *(C)* 732-740-0694 *Email:* Linwood.Johnson(a)dowjones.com *Team:* sitereliabilitytools(a)dowjones.com

2 1

syslog-ng 4.2.0
by Attila Szakács 17 May '23

17 May '23

Dear syslog-ng users, We are pleased to announce that the 4.2.0 version of syslog-ng has been released, and it is now available on GitHub: https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-4.2.0 4.2.0 ===== Read Axoflow's [blog post](https://axoflow.com/axosyslog-release-4-2/) for more details. ## Highlights #### Sending messages to Splunk HEC The `splunk-hec-event()` destination feeds Splunk via the [HEC events API]( https://docs.splunk.com/Documentation/Splunk/9.0.4/RESTREF/RESTinput#servic… ). Minimal config: ``` destination d_splunk_hec_event { splunk-hec-event( url("https://localhost:8088") token("70b6ae71-76b3-4c38-9597-0c5b37ad9630") ); }; ``` Additional options include: * `event()` * `index()` * `source()` * `sourcetype()` * `host()` * `time()` * `default-index()` * `default-source()` * `default-sourcetype()` * `fields()` * `extra-headers()` * `extra-queries()` * `content-type()` The `splunk-hec-raw()` destination feeds Splunk via the [HEC raw API]( https://docs.splunk.com/Documentation/Splunk/9.0.4/RESTREF/RESTinput#servic… ). Minimal config: ``` destination d_splunk_hec_raw { splunk-hec-raw( url("https://localhost:8088") token("70b6ae71-76b3-4c38-9597-0c5b37ad9630") channel("05ed4617-f186-4ccd-b4e7-08847094c8fd") ); }; ``` ([#4462](https://github.com/syslog-ng/syslog-ng/pull/4462)) #### Smart multi-line for recognizing backtraces `multi-line-mode(smart)`: With this multi-line mode, the inherently multi-line data backtrace format is recognized even if they span multiple lines in the input and are converted to a single log message for easier analysis. Backtraces for the following programming languages are recognized : Python, Java, JavaScript, PHP, Go, Ruby and Dart. The regular expressions to recognize these programming languages are specified by an external file called `/usr/share/syslog-ng/smart-multi-line.fsm` (installation path depends on configure arguments), in a format that is described in that file. `group-lines()` parser: this new parser correlates multi-line messages received as separate, but subsequent lines into a single log message. Received messages are first collected into streams related messages (using key()), then collected into correlation contexts up to timeout() seconds. The identification of multi-line messages are then performed on these message contexts within the time period. ``` group-lines(key("$FILE_NAME") multi-line-mode("smart") template("$MESSAGE") timeout(10) line-separator("\n") ); ``` ([#4225](https://github.com/syslog-ng/syslog-ng/pull/4225)) #### HYPR Audit Trail source `hypr-audit-trail()` & `hypr-app-audit-trail()` source drivers are now available to monitor the audit trails for [HYPR](https://www.hypr.com) applications. See the README.md file in the driver's directory to see usage information. ([#4175](https://github.com/syslog-ng/syslog-ng/pull/4175)) #### `ebpf()` plugin and reuseport packet randomizer A new ebpf() plugin was added as a framework to leverage the kernel's eBPF infrastructure to improve performance and scalability of syslog-ng. Example: ``` source s_udp { udp(so-reuseport(yes) port(2000) persist-name("udp1") ebpf(reuseport(sockets(4))) ); udp(so-reuseport(yes) port(2000) persist-name("udp2")); udp(so-reuseport(yes) port(2000) persist-name("udp3")); udp(so-reuseport(yes) port(2000) persist-name("udp4")); }; ``` NOTE: The `ebpf()` plugin is considered advanced usage so its compilation is disabled by default. Please don't use it unless all other avenues of configuration solutions are already tried. You will need a special toolchain and a recent kernel version to compile and run eBPF programs. ([#4365](https://github.com/syslog-ng/syslog-ng/pull/4365)) ## Features * `network` source: During a TLS handshake, syslog-ng now automatically sets the `certificate_authorities` field of the certificate request based on the `ca-file()` and `ca-dir()` options. The `pkcs12-file()` option already had this feature. ([#4412](https://github.com/syslog-ng/syslog-ng/pull/4412)) * `metrics-probe()`: Added `level()` option to set the stats level of the generated metrics. ([#4453](https://github.com/syslog-ng/syslog-ng/pull/4453)) * `metrics-probe()`: Added `increment()` option. Users can now set a template, which resolves to a number that modifies the increment of the counter. If not set, the increment is 1. ([#4447](https://github.com/syslog-ng/syslog-ng/pull/4447)) * `python`: Added support for typed custom options. This applies for `python` source, `python-fetcher` source, `python` destination, `python` parser and `python-http-header` inner destination. Example config: ``` python( class("TestClass") options( "string_option" => "example_string" "bool_option" => True # supported values are: True, False, yes, no "integer_option" => 123456789 "double_option" => 123.456789 "string_list_option" => ["string1", "string2", "string3"] "template_option" => LogTemplate("${example_template}") ) ); ``` **Breaking change! Previously values were converted to strings if possible, now they are passed to the python class with their real type. Make sure to follow up these changes in your python code!** ([#4354](https://github.com/syslog-ng/syslog-ng/pull/4354)) * `mongodb` destination: Added support for list, JSON and null types. ([#4437](https://github.com/syslog-ng/syslog-ng/pull/4437)) * `add-contextual-data()`: significantly reduce memory usage for large CSV files. ([#4444](https://github.com/syslog-ng/syslog-ng/pull/4444)) * `python()`: new LogMessage methods for querying as string and with default values - `get(key[, default])` Return the value for `key` if `key` exists, else `default`. If `default` is not given, it defaults to `None`, so that this method never raises a `KeyError`. - `get_as_str(key, default=None, encoding='utf-8', errors='strict', repr='internal')`: Return the string value for `key` if `key` exists, else `default`. If `default` is not given, it defaults to `None`, so that this method never raises a `KeyError`. The string value is decoded using the codec registered for `encoding`. `errors` may be given to set the desired error handling scheme. Note that currently `repr='internal'` is the only available representation. We may implement another more Pythonic representation in the future, so please specify the `repr` argument explicitly if you want to avoid future representation changes in your code. ([#4410](https://github.com/syslog-ng/syslog-ng/pull/4410)) * `kubernetes()` source: Added support for json-file logging driver format. ([#4419](https://github.com/syslog-ng/syslog-ng/pull/4419)) * The new `$RAWMSG_SIZE` hard macro can be used to query the original size of the incoming message in bytes. This information may not be available for all source drivers. ([#4440](https://github.com/syslog-ng/syslog-ng/pull/4440)) * syslog-ng configuration identifier A new syslog-ng configuration keyword has been added, which allows specifying a config identifier. For example: ``` @config-id: cfg-20230404-13-g02b0850fc ``` This keyword can be used for config identification in managed environments, where syslog-ng instances and their configuration are deployed/generated automatically. `syslog-ng-ctl config --id` can be used to query the active configuration ID and the SHA256 hash of the full "preprocessed" syslog-ng configuration. For example: ``` $ syslog-ng-ctl config --id cfg-20230404-13-g02b0850fc (08ddecfa52a3443b29d5d5aa3e5114e48dd465e195598062da9f5fc5a45d8a83) ``` ([#4420](https://github.com/syslog-ng/syslog-ng/pull/4420)) * `syslog-ng`: add `--config-id` command line option Similarly to `--syntax-only`, this command line option parses the configuration and then prints its ID before exiting. It can be used to query the ID of the current configuration persisted on disk. ([#4435](https://github.com/syslog-ng/syslog-ng/pull/4435)) * Health metrics and `syslog-ng-ctl healthcheck` A new `syslog-ng-ctl` command has been introduced, which can be used to query a healthcheck status from syslog-ng. Currently, only 2 basic health values are reported. `syslog-ng-ctl healthcheck --timeout <seconds>` can be specified to use it as a boolean healthy/unhealthy check. Health checks are also published as periodically updated metrics. The frequency of these checks can be configured with the `stats(healthcheck-freq())` option. The default is 5 minutes. ([#4362](https://github.com/syslog-ng/syslog-ng/pull/4362)) * `$(format-json)` and template functions which support value-pairs expressions: new key transformations upper() and lower() have been added to translate the caps of keys while formatting the output template. For example: template("$(format-json test.* --upper)\n") Would convert all keys to uppercase. Only supports US ASCII. ([#4452](https://github.com/syslog-ng/syslog-ng/pull/4452)) * `python()`, `python-fetcher()` sources: Added a mapping for the `flags()` option. The state of the `flags()` option is mapped to the `self.flags` variable, which is a `Dict[str, bool]`, for example: ```python { 'parse': True, 'check-hostname': False, 'syslog-protocol': True, 'assume-utf8': False, 'validate-utf8': False, 'sanitize-utf8': False, 'multi-line': True, 'store-legacy-msghdr': True, 'store-raw-message': False, 'expect-hostname': True, 'guess-timezone': False, 'header': True, 'rfc3164-fallback': True, } ``` ([#4455](https://github.com/syslog-ng/syslog-ng/pull/4455)) ### Metrics * `network()`, `syslog()`: TCP connection metrics ``` syslogng_socket_connections{id="tcp_src#0",driver_instance="afsocket_sd.(stream,AF_INET(0.0.0.0:5555))",direction="input"} 3 syslogng_socket_max_connections{id="tcp_src#0",driver_instance="afsocket_sd.(stream,AF_INET(0.0.0.0:5555))",direction="input"} 10 syslogng_socket_rejected_connections_total{id="tcp_src#0",driver_instance="afsocket_sd.(stream,AF_INET(0.0.0.0:5555))",direction="input"} 96 ``` `internal()`: `internal_events_queue_capacity` metric `syslog-ng-ctl healthcheck`: new healthcheck value `syslogng_internal_events_queue_usage_ratio` ([#4411](https://github.com/syslog-ng/syslog-ng/pull/4411)) * `metrics`: new network (TCP, UDP) metrics are available on stats level 1 ``` # syslog-ng-ctl stats prometheus syslogng_socket_receive_buffer_used_bytes{id="#anon-source0#3",direction="input",driver_instance="afsocket_sd.udp4"} 0 syslogng_socket_receive_buffer_max_bytes{id="#anon-source0#3",direction="input",driver_instance="afsocket_sd.udp4"} 268435456 syslogng_socket_receive_dropped_packets_total{id="#anon-source0#3",direction="input",driver_instance="afsocket_sd.udp4"} 619173 syslogng_socket_connections{id="#anon-source0#0",direction="input",driver_instance="afsocket_sd.(stream,AF_INET(0.0.0.0:2000))"} 1 ``` ([#4374](https://github.com/syslog-ng/syslog-ng/pull/4374)) * New configuration-related metrics: ``` syslogng_last_config_reload_timestamp_seconds 1681309903 syslogng_last_successful_config_reload_timestamp_seconds 1681309758 syslogng_last_config_file_modification_timestamp_seconds 1681309877 ``` ([#4420](https://github.com/syslog-ng/syslog-ng/pull/4420)) * destination: Introduced queue metrics. * The corresponding driver is identified with the "id" and "driver_instance" labels. * Available counters are "memory_usage_bytes" and "events". * Memory queue metrics are available with "syslogng_memory_queue_" prefix, `disk-buffer` metrics are available with "syslogng_disk_queue_" prefix. * `disk-buffer` metrics have an additional "path" label, pointing to the location of the disk-buffer file and a "reliable" label, which can be either "true" or "false". * Threaded destinations, like `http`, `python`, etc have an additional "worker" label. Example metrics ``` syslogng_disk_queue_events{driver_instance="http,http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00000.rqf",reliable="true",worker="0"} 80 syslogng_disk_queue_events{driver_instance="http,http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00001.rqf",reliable="true",worker="1"} 7 syslogng_disk_queue_events{driver_instance="http,http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00002.rqf",reliable="true",worker="2"} 7 syslogng_disk_queue_events{driver_instance="http,http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00003.rqf",reliable="true",worker="3"} 7 syslogng_disk_queue_events{driver_instance="tcp,localhost:1235",id="d_network_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00000.qf",reliable="false"} 101 syslogng_disk_queue_memory_usage_bytes{driver_instance="http, http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00000.rqf",reliable="true",worker="0"} 3136 syslogng_disk_queue_memory_usage_bytes{driver_instance="http, http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00001.rqf",reliable="true",worker="1"} 2776 syslogng_disk_queue_memory_usage_bytes{driver_instance="http, http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00002.rqf",reliable="true",worker="2"} 2760 syslogng_disk_queue_memory_usage_bytes{driver_instance="http, http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00003.rqf",reliable="true",worker="3"} 2776 syslogng_disk_queue_memory_usage_bytes{driver_instance="tcp,localhost:1235",id="d_network_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00000.qf",reliable="false"} 39888 syslogng_memory_queue_events{driver_instance="http,http://localhost:1236",id="d_http#0",worker="0"} 15 syslogng_memory_queue_events{driver_instance="http,http://localhost:1236",id="d_http#0",worker="1"} 14 syslogng_memory_queue_events{driver_instance="tcp,localhost:1234",id="d_network#0"} 29 syslogng_memory_queue_memory_usage_bytes{driver_instance="http, http://localhost:1236",id="d_http#0",worker="0"} 5896 syslogng_memory_queue_memory_usage_bytes{driver_instance="http, http://localhost:1236",id="d_http#0",worker="1"} 5552 syslogng_memory_queue_memory_usage_bytes{driver_instance="tcp,localhost:1234",id="d_network#0"} 11448 ``` ([#4392](https://github.com/syslog-ng/syslog-ng/pull/4392)) * `network()`, `syslog()`, `file()`, `http()`: new byte-based metrics for incoming/outgoing events These metrics show the serialized message sizes (protocol-specific header/framing/etc. length is not included). ``` syslogng_input_event_bytes_total{id="s_network#0",driver_instance="tcp,127.0.0.1"} 1925529600 syslogng_output_event_bytes_total{id="d_network#0",driver_instance="tcp, 127.0.0.1:5555"} 565215232 syslogng_output_event_bytes_total{id="d_http#0",driver_instance="http, http://127.0.0.1:8080/"} 1024 ``` ([#4440](https://github.com/syslog-ng/syslog-ng/pull/4440)) * `disk-buffer`: Added metrics for monitoring the available space in disk-buffer `dir()`s. Metrics are available from `stats(level(1))`. By default, the metrics are generated every 5 minutes, but it can be changed in the global options: ``` options { disk-buffer( stats( freq(10) ) ); }; ``` Setting `freq(0)` disabled this feature. Example metrics: ``` syslogng_disk_queue_dir_available_bytes{dir="/var/syslog-ng"} 870109413376 ``` ([#4399](https://github.com/syslog-ng/syslog-ng/pull/4399)) * `disk-buffer`: Added metrics for abandoned disk-buffer files. Availability is the same as the `disk_queue_dir_available_bytes` metric. Example metrics: ``` syslogng_disk_queue_capacity_bytes{abandoned="true",path="/var/syslog-ng/syslog-ng-00000.rqf",reliable="true"} 104853504 syslogng_disk_queue_disk_allocated_bytes{abandoned="true",path="/var/syslog-ng/syslog-ng-00000.rqf",reliable="true"} 273408 syslogng_disk_queue_disk_usage_bytes{abandoned="true",path="/var/syslog-ng/syslog-ng-00000.rqf",reliable="true"} 269312 syslogng_disk_queue_events{abandoned="true",path="/var/syslog-ng/syslog-ng-00000.rqf",reliable="true"} 860 ``` ([#4402](https://github.com/syslog-ng/syslog-ng/pull/4402)) * `disk-buffer`: Added capacity, disk_allocated and disk_usage metrics. * "capacity_bytes": The theoretical maximal useful size of the disk-buffer. This is always smaller, than `disk-buf-size()`, as there is some reserved space for metadata. The actual full disk-buffer file can be larger than this, as syslog-ng allows to write over this limit once, at the end of the file. * "disk_allocated_bytes": The current size of the disk-buffer file on the disk. Please note that the disk-buffer file size does not strictly correlate with the number of messages, as it is a ring buffer implementation, and also syslog-ng optimizes the truncation of the file for performance reasons. * "disk_usage_bytes": The serialized size of the queued messages in the disk-buffer file. This counter is useful for calculating the disk usage percentage (disk_usage_bytes / capacity_bytes) or the remaining available space (capacity_bytes - disk_usage_bytes). Example metrics: ``` syslogng_disk_queue_capacity_bytes{driver_id="d_network#0",driver_instance="tcp,localhost:1235",path="/var/syslog-ng-00000.rqf",reliable="true"} 104853504 syslogng_disk_queue_disk_allocated_bytes{driver_id="d_network#0",driver_instance="tcp,localhost:1235",path="/var/syslog-ng-00000.rqf",reliable="true"} 17284 syslogng_disk_queue_disk_usage_bytes{driver_id="d_network#0",driver_instance="tcp,localhost:1235",path="/var/syslog-ng-00000.rqf",reliable="true"} 13188 ``` ([#4356](https://github.com/syslog-ng/syslog-ng/pull/4356)) * `kubernetes()`: Added `input_events_total` and `input_event_bytes_total` metrics. ``` syslogng_input_events_total{cluster="k8s",driver="kubernetes",id="#anon-source0",namespace="default",pod="log-generator-1682517834-7797487dcc-49hqc"} 25 syslogng_input_event_bytes_total{cluster="k8s",driver="kubernetes",id="#anon-source0",namespace="default",pod="log-generator-1682517834-7797487dcc-49hqc"} 1859 ``` ([#4447](https://github.com/syslog-ng/syslog-ng/pull/4447)) ## Bugfixes * `pdbtool test`: fix two type validation bugs: 1) When `pdbtool test` validates the type information associated with a name-value pair, it was using string comparisons, which didn't take type aliases into account. This is now fixed, so that "int", "integer" or "int64" can all be used to mean the same type. 2) When type information is missing from a `<test_value/>` tag, don't validate it against "string", rather accept any extracted type. In addition to these fixes, a new alias "integer" was added to mean the same as "int", simply because syslog-ng was erroneously using this term when reporting type information in its own messages. ([#4405](https://github.com/syslog-ng/syslog-ng/pull/4405)) * `$(format-json)`: fix RFC8259 number violation `$(format-json)` produced invalid JSON output when it contained numeric values with leading zeros or + signs. This has been fixed. ([#4415](https://github.com/syslog-ng/syslog-ng/pull/4415)) * `grouping-by()`: fix `persist-name()` option not taken into account ([#4390](https://github.com/syslog-ng/syslog-ng/pull/4390)) * `python()`, `db-parser()`, `grouping-by()`, `add-contextual-data()`: fix typing compatibility with <4.0 config versions ([#4394](https://github.com/syslog-ng/syslog-ng/pull/4394)) * `python`: Fixed a crash which occurred at reloading after registering a confgen plugin. ([#4459](https://github.com/syslog-ng/syslog-ng/pull/4459)) * `date-parser()`: fix `%z` when system timezone has no daylight saving time ([#4401](https://github.com/syslog-ng/syslog-ng/pull/4401)) * Consider messages consumed into correlation states "matching": syslog-ng's correlation functionality (e.g. grouping-by() or db-parser() with such rules) drop individual messages as they are consumed into a correlation contexts and you are using `inject-mode(aggregate-only)`. This is usually happens because you are only interested in the combined message and not in those that make up the combination. However, if you are using correlation with conditional processing (e.g. if/elif/else or flags(final)), such messages were erroneously considered as unmatching, causing syslog-ng to take the alternative branch. Example: With a configuration similar to this, individual messages are consumed into a correlation state and dropped by `grouping-by()`: ``` log { source(...); if { grouping-by(... inject-mode(aggregate-only)); } else { # alternative branch }; }; ``` The bug was that these individual messages also traverse the `else` branch, even though they were successfully processed with the inclusion into the correlation context. This is not correct. The bugfix changes this behaviour. ([#4370](https://github.com/syslog-ng/syslog-ng/pull/4370)) * `netmask6()`: fix crash when user specifies too long mask ([#4429](https://github.com/syslog-ng/syslog-ng/pull/4429)) * `afprog`: Fixed possible freezing on some OSes ([#4438](https://github.com/syslog-ng/syslog-ng/pull/4438)) * `network()`, `syslog()`, `syslog-parser()`: fix null termination of SDATA param names ([#4429](https://github.com/syslog-ng/syslog-ng/pull/4429)) * `python()`: fix LogMessage subscript not raising KeyError on non-existent keys When message fields were queried (`msg["key"]`) and the given key did not exist, `None` or an empty string was returned (depending on the version of the config). Neither was correct, now a KeyError occurs in such cases. ([#4410](https://github.com/syslog-ng/syslog-ng/pull/4410)) * `$(python)`: fix template function prefix being overwritten when using datetime types ([#4410](https://github.com/syslog-ng/syslog-ng/pull/4410)) * `disk-buffer`: Fixed queued messages stats counting, when a disk-buffer became corrupted. ([#4385](https://github.com/syslog-ng/syslog-ng/pull/4385)) * `$(format-json)`: fix escaping control characters `$(format-json)` produced invalid JSON output when a string value contained control characters. ([#4417](https://github.com/syslog-ng/syslog-ng/pull/4417)) * `disk-buffer()`: fix deinitialization when starting syslog-ng with invalid configuration ([#4418](https://github.com/syslog-ng/syslog-ng/pull/4418)) * `python()`: fix exception handling when LogMessage value conversion fails ([#4410](https://github.com/syslog-ng/syslog-ng/pull/4410)) * `json-parser()`: Fixed parsing non-string arrays. syslog-ng now no longer parses non-string arrays to list of strings, losing the original type information of the array's elements. ([#4396](https://github.com/syslog-ng/syslog-ng/pull/4396)) * `disk-buffer`: Fixed a rare race condition when calculating disk-buffer filename. ([#4381](https://github.com/syslog-ng/syslog-ng/pull/4381)) * `python-persist`: fix off-by-one overflow ([#4429](https://github.com/syslog-ng/syslog-ng/pull/4429)) ## Packaging * The `--with-python-venv-dir=path` configure option can be used to modify the location of syslog-ng's venv. The default is still `${localstatedir}/python-venv`. ([#4465](https://github.com/syslog-ng/syslog-ng/pull/4465)) ## Other changes * The `sdata-prefix()` option does not accept values longer than 128 characters. ([#4429](https://github.com/syslog-ng/syslog-ng/pull/4429)) * `grouping-by()`: Remove setting of the `${.classifier.context_id}` name-value pair in all messages consumed into a correlation context. This functionality is inherited from db-parser() and has never been documented for `grouping-by()`, has of limited use, and any uses can be replaced by the use of the built-in macro named `$CONTEXT_ID`. Modifying all consumed messages this way has significant performance consequences for `grouping-by()` and removing it outweighs the small incompatibility this change introduces. The similar functionality in `db-parser()` correlation is not removed with this change. ([#4424](https://github.com/syslog-ng/syslog-ng/pull/4424)) * `config`: Added `internal()` option to `source`s, `destination`s, `parser`s and `rewrite`s. Its main usage is in SCL blocks. Drivers configured with `internal(yes)` register their metrics on level 3. This makes developers of SCLs able to create metrics manually with `metrics-probe()` and "disable" every other metrics, they do not need. ([#4451](https://github.com/syslog-ng/syslog-ng/pull/4451)) * The following Prometheus metrics have been renamed: `log_path_{in,e}gress` -> `route_{in,e}gress_total` `internal_source` -> `internal_events_total` The `internal_queue_length` stats counter has been removed. It was deprecated since syslog-ng 3.29. ([#4411](https://github.com/syslog-ng/syslog-ng/pull/4411)) ## syslog-ng Discord For a bit more interactive discussion, join our Discord server: https://discord.gg/E65kP9aZGm ## Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Alex Becker, Attila Szakacs, Balazs Scheidler, Hofi, László Várady, Muhammad Shanif, Ricfilipe, Romain Tartière

1 0

Insider 2023-05: learning; UDP; upgrading;
by Peter Czanik (pczanik) 11 May '23

11 May '23

Dear syslog-ng users, This is the 110th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Learning syslog-ng, the easier way ---------------------------------- Last year, one of the returning questions I received was how to learn syslog-ng. My answer was that read the first few chapters of the documentation, read my blogs related to your use case, and then read a few relevant parts from the rest of the documentation. Our documentation is praised by users, but it is still a reference documentation. I was asked if a less detailed, more to the point, preferably video tutorial is available. Your request was heard. In the past couple of months, I published a tutorial series in blog and video format, which brings you from basic logging concepts to using syslog-ng to collect, parse, enrich log messages and store them to Elasticsearch. Of course, these 5-10 minute videos are not enough to learn anything in depth, but they introduce you to all major syslog-ng functionalities. https://www.syslog-ng.com/community/b/blog/posts/learning-syslog-ng-the-eas… Why syslog over UDP loses messages and how to avoid that -------------------------------------------------------- Message loss related resiliency with regards to syslog over UDP has always been a problem. Users who start looking at these metrics generally report a 30-40% loss of messages for syslog over UDP, but drops of up to 90% are not unheard of. Let me get to the bottom of the reasons why messages are dropped and how these problems can be solved, or at least mitigated. https://axoflow.com/syslog-over-udp-message-loss-1/ Upgrade problems from syslog-ng 3 to 4 -------------------------------------- Version 4 of syslog-ng works perfectly well in version 3 compatibility mode. However, if you want to use the syslog-ng 4 features, you need to be aware of some significant changes. If you have a simple configuration, like those in Linux distributions, then simply rewriting the version string is most likely enough. However, if you use PatternDB or JSON parsing, any Python code, or an Elasticsearch, or MongoDB destination, you have to be aware of the changes. >From this blog you can learn about type support, how this can affect you, changes in Python support, and some tips how to prepare for the upgrade. https://www.syslog-ng.com/community/b/blog/posts/upgrade-problems-from-sysl… WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

using -dv on startup vs syslog-ng-ctl debug set=off
by Jerry Scharf 28 Apr '23

28 Apr '23

Hi, We are chasing an intermittent bug with slow reloads. We are following the debug suggestions and adding -dv to the execstart in the service file. Because of the load of messages that we push, I want to turn off the debugging after we are past the window when the slow reload happens. I naively tried doing "syslog-ng-ctl debug set=off" but it responded that the flag didn't change. Is there some way to tell syslog-ng to allow the -ctl command to make the change. I can imagine why this would be desired sometimes, but at 10k lines per minute it is going to fill up /tmp. right now I am doing ugly things to solve the problem, but really want to find a better way. thanks, jerry -- Jerry Scharf Pure Storage

2 1

What is the status of syslog-ng 4 in your environment?
by Peter Czanik (pczanik) 27 Apr '23

27 Apr '23

Hi, Version 4 of syslog-ng was released almost half a year ago. It adds type support to name-value pairs, and lots of improvements to Python and Kubernetes support. I just started a Twitter / Mastodon / LinkedIn poll to learn, what is the status of syslog-ng 4 in your environment? * https://fosstodon.org/@PCzanik/110269557855094446 * https://twitter.com/sngose/status/1651486235469201408 * https://www.linkedin.com/posts/peterczanik_poll-syslogabrng-python-activity… Of course, I'd also be happy to hear some feedback, not just a poll result. So, let me know why you are on syslog-ng 4 already, or why do you delay your upgrade to this new major version! If hearing about syslog-ng 4 completely surprised you, check https://www.syslog-ng.com/community/b/blog/posts/upgrade-problems-from-sysl… which describes some of the major changes, and what to be careful about when you upgrade! Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Insider 2023-04: FreeBSD; Splunk; Deduplication;
by Peter Czanik (pczanik) 20 Apr '23

20 Apr '23

Dear syslog-ng users, This is the 109th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Installing a syslog-ng 4 development snapshot on FreeBSD -------------------------------------------------------- Unless there is a serious problem, FreeBSD ports usually contains the latest stable syslog-ng release. However, sometimes people want to compile a git snapshot to test a new feature or bugfix. To do that, one way is to generate a syslog-ng release tgz on FreeBSD and edit the syslog-ng port files yourself. However, this needs some practice. As such, an easier solution is to use my weekly development snapshots. https://www.syslog-ng.com/community/b/blog/posts/installing-a-syslog-ng-4-d… Getting data to Splunk ---------------------- Getting data to Splunk can be challenging. Syslog is still the most important data source, and it can provide you with hard-to-solve problems (for example, like high volume, non-compliant messages, unreliable network protocol (UDP), and more). The syslog-ng Premium Edition (PE) and syslog-ng Store Box (SSB) by One Identity can make these challenges manageable. https://www.syslog-ng.com/community/b/blog/posts/getting-data-to-splunk Streaming deduplication in syslog-ng ------------------------------------ Log volumes are growing 25% year over year, which means they are doubling every three years. Considering that SIEMs and other log processing tools are licensed based on volume, tools and mechanisms to make log storage and processing more efficient are very much sought for. Lean, how syslog-ng can help you. https://syslog-ng-future.blog/streaming-deduplication-in-syslog-ng/ WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Forwarding locally stored RAW messages
by Dragan Zecevic 13 Apr '23

13 Apr '23

Hi, we have syslog-ng Open Source Edition 3.33 and we are storing syslog messages from some systems into log files locally on a partition on syslog-ng server. Each day those log files are compressed. In order to better analyze some logs if needed we would like to extract some log files and ingest them to SIEM. Is it possible to make some forwarder that will read these RAW syslog messages from a log file and send them via syslog to SIEM? Thank you. Br, Dragan

2 1

TLS "trusted-dn" Question
by David Hauck 05 Apr '23

05 Apr '23

Hi, I'm currently using syslog-ng OSE v3.31.2 and trying to get a TLS configured endpoint to use the 'trusted-dn()' TLS option. I'm having trouble getting syslog-ng to return these DN specifiers in the Certificate Request option during the TLS negotiation so that clients can properly condition their supplied client certificates. I invariably see the following TLS negotiations (empty DNs list) in my Wireshark captures (as returned by the syslog-ng server): TLSv1.2 Record Layer: Handshake Protocol: Certificate Request Content Type: Handshake (22) Version: TLS 1.2 (0x0303) Length: 58 Handshake Protocol: Certificate Request Handshake Type: Certificate Request (13) Length: 54 Certificate types count: 3 Certificate types (3 types) Signature Hash Algorithms Length: 46 Signature Hash Algorithms (23 algorithms) Distinguished Names Length: 0 <----- always '0' In these cases my clients choose random client certificates that can't be refined to certificates signed by those expected (via the 'trusted-dn()' values) by the server and the connection is immediately closed. Here's the syslog-ng.conf entry for these sources: source s_515_tls { network( transport(tls) port(515) ip-protocol(6) tls(ca-dir("/etc/ssl/certs") key-file("/root/naservers.key") cert-file("/root/naservers.cer") peer-verify(required-trusted) trusted-dn("CN=*.netacquire.com")) ); }; I've tried several variants of the 'trusted-dn()' values, including other wildcards for country, state, etc. I always see a DNs list of zero size in the TLS Certificate Request option returned by the server. As expected switching to 'peer-verify(required-untrusted)' results in successful negotiation (with expected server-side errors indicating problems associated with the client certificates) and subsequent successful client/server logging. I figure I must be missing something obvious ;). Any ideas? Here's my syslog-ng version info: [logdest:~]# syslog-ng --version syslog-ng 3 (3.31.2) Config version: 3.29 Installer-Version: 3.31.2 Revision: Compile-Date: Nov 9 2021 12:52:59 Module-Directory: /usr/lib/syslog-ng Module-Path: /usr/lib/syslog-ng Include-Path: /usr/share/syslog-ng/include Available-Modules: tfgetent,mod-python,cryptofuncs,hook-commands,kvformat,cef,afuser,linux-kmsg-format,map-value-pairs,system-source,azure-auth-header,csvparser,affile,afprog,secure-logging,http,examples,timestamp,afsocket,confgen,stardate,dbparser,xml,disk-buffer,appmodel,afsnmp,add-contextual-data,afstomp,graphite,json-plugin,basicfuncs,syslogformat,pseudofile,tags-parser Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Systemd: off Thanks, -David

3 10

Learning syslog-ng
by Peter Czanik (pczanik) 05 Apr '23

05 Apr '23

Hi, Last year, one of the returning questions I received was how to learn syslog-ng. My answer was that read the first few chapters of the documentation, read my blogs related to your use case, and then read a few relevant parts from the rest of the documentation. Our documentation is praised by users, but it is still a reference documentation. I was asked if a less detailed, more to the point, preferably video tutorial is available. Your request was heard. In the past couple of months, I published a tutorial series in blog and video format, which brings you from basic logging concepts to using syslog-ng to collect, parse, enrich log messages and store them to Elasticsearch. Of course, these 5-10 minute videos are not enough to learn anything in depth, but they introduce you to all major syslog-ng functionalities. Even if you are a seasoned syslog-ng user, there is a good chance that you will learn something new from this introductory tutorial series: the “if” statement, in-line configuration elements, the inlist() filter or the JSON template function, just to name a few. If you have roughly 1.5 hours, then I recommend going through all the videos from the beginning to the end. You can reach the play list on YouTube at: https://www.youtube.com/playlist?list=PLoBNbOHNb0i5Pags2JY6-6wH2noLaSiTb If you would rather pick only a few topics from the tutorial series, here is a table of contents, with short summaries, pointers to the blog and video versions and the related parts from the documentation. Unfortunately, the documentation for the latest version is not available yet, pointers are included to the web version of the syslog-ng version 3.37 documentation: https://peter.czanik.hu/posts/syslog-ng-tutorial-toc/ Of course, once you read/watched my syslog-ng tutorials, reading the blogs and relevant parts of the documentation is still highly recommended. Please share this news if you know anyone who wants to learn syslog-ng! Bye, CzP Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Question on message parsing amd formats
by Generic Secure-Logging 30 Mar '23

30 Mar '23

I have a question on message parsing. From my unterstanding, if syslog-ng receives a message that is not in the correct RFC 5424 format, it will just output it as is. Does that mean that my application needs to generate the message in RFC 5424 format and syslog-ng will just process it? Or can syslog-ng be configured in such a way that an application just sends an unformatted string and syslog-ng will generate a valid RFC 5424 from it? How are RFC 5424 structured data elements treated by syslog-ng? Can it be configured using macros to generate them or does the application need to send a valid mesage containing the structured data elements in the right format? I am using syslog-ng 3.35.1. Regards, Stephan The information in this e-mail is confidential. The contents may not be disclosed or used by anyone other than the addressee. Access to this e-mail by anyone else is unauthorised. If you are not the intended recipient, please notify Airbus immediately and delete this e-mail. Airbus cannot accept any responsibility for the accuracy or completeness of this e-mail as it has been sent over public networks. If you have any concerns over the content of this message or its Accuracy or Integrity, please contact Airbus immediately. All outgoing e-mails from Airbus are checked using regularly updated virus scanning software but you should take whatever measures you deem to be appropriate to ensure that this message and any attachments are virus free.

1 0

pdbtool validate now requires type information.
by Evan Rempel 29 Mar '23

29 Mar '23

I'm working with syslog-ng-4.1.1-1 and attempting to validate a patterndb xml file. I understand that the pattern parsers will produce type aware values, but it seems that if the tests in the patterndb xml file do not contain any type information that all tests against non-string values will fail. Is that intended behavior to require type information on all tests? If that is by design I propose that this makes thing difficult because @NUMBER:var@ and @STRING:var@ will both successfully parse the same value out of the syslog message, but the test would now have to be different. Tools that programmatically produce the patterndb need to become a lot smarter. Can anyone comment on this behavior? When attempting to validate the attached patterndb.xml file I get the error Testing message: program='xinetd' message='EXIT: gsiftp status=0 pid=23133 duration=0(sec)' Wrong match name='status', value='0', type='integer', expected='0', expected_type='string' Wrong match name='sec', value='0', type='integer', expected='0', expected_type='string' Wrong match name='pid', value='23133', type='integer', expected='23133', expected_type='string' -- Evan

2 2

Single destination, multiple tcp connections?
by Steve Bernacki 25 Mar '23

25 Mar '23

I am running syslog-ng 3.38.1. I have a scenario where I am receiving logs from hundreds devices over a number of different protocols (tcp, udp, tcp/TLS). I perform some filtering on these logs and then send them to one or more destinations for further processing. Due to the volume of logs being received, the destinations are becoming saturated and logs begin queuing up on my end. I have memory and disk buffering enabled, but the receiving end isn't able to pull logs off quickly enough since they are being funneled through a single TCP connection. I'd like to be able to establish some number of concurrent tcp connections to a single destination ip:port, and balance all of the incoming logs through those connections. I'm aware of techniques to load balance to destinations using multiple channels in a single destination and filtering traffic by the R_MSEC, but this technique quickly causes simple configs to balloon to hundreds of lines. I suppose this would work even if the destinations are all the same (I'd need to use unique persist-name() labels), but is there an easier / more straightforward way of accomplishing this? Thank you. Steve

2 1

Insider 2023-03: 4.1; Homebrew; Ventura;
by Peter Czanik (pczanik) 22 Mar '23

22 Mar '23

Dear syslog-ng users, This is the 108th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Version 4.1 of syslog-ng available ---------------------------------- Version 4.1.0 of syslog-ng is now available. It brings PROXY protocol v2 support, many metrics related improvements, including initial Prometheus support, improved Kubernetes support, and many bugfixes. For a complete list of syslog-ng changes, check the release notes: https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-4.1.1 Syslog-ng is now available in Homebrew -------------------------------------- Installing syslog-ng on Mac is easy, if you use Homebrew for 3rd party packages. Previously, you had to install dependencies and then compile syslog-ng from source. Now, a single command takes care of everything: brew install syslog-ng For more details read https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-is-now-available… Syslog-ng on MacOS Ventura -------------------------- Each new MacOS release brings some surprises when it comes to compiling syslog-ng. MacOS Ventura has been released recently, while Homebrew has also been updated. So here are some updated instructions for MacOS Ventura (and also for the last MacOS minor release before Ventura). https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-on-macos-ventura WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

syslog-ng 4.1.0
by László Várady 21 Mar '23

21 Mar '23

Dear syslog-ng users, We are pleased to announce that the 4.1.0 version of syslog-ng has been released, and it is now available on GitHub: https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-4.1.0 4.1.0 Highlights PROXY protocol v2 support (#4211 <https://github.com/syslog-ng/syslog-ng/pull/4211>) We've added support for PROXY protocol v2 (transport(proxied-tcp)), a protocol used by network load balancers, such as Amazon Elastic Load Balancer and HAProxy, to carry original source/destination address information, as described in https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt Metrics revised Prometheus metric format (#4325 <https://github.com/syslog-ng/syslog-ng/pull/4325>) A new metric system has been introduced to syslog-ng, where metrics are identified by names and partitioned by labels, which is similar to the Prometheus data model <https://prometheus.io/docs/concepts/data_model/>. The syslog-ng-ctl stats prometheus command can be used to query syslog-ng metrics in a format that conforms to the Prometheus text-based exposition format. syslog-ng-ctl stats prometheus --with-legacy-metrics displays legacy metrics as well. Legacy metrics do not follow Prometheus' metric and label conventions. Classification (metadata-based metrics) (#4318 <https://github.com/syslog-ng/syslog-ng/pull/4318>) metrics-probe(), a new parser has also been added, which counts messages passing through based on the metadata of each message. The parser creates labeled metrics based on the fields of the message. Both the key and labels can be set in the config, the values of the labels can be templated. E.g.: parser p_metrics_probe { metrics-probe( key("custom_key") # adds "syslogng_" prefix => "syslogng_custom_key" labels( "custom_label_name_1" => "foobar" "custom_label_name_2" => "${.custom.field}" ) ); }; With this config, it creates counters like these: syslogng_custom_key{custom_label_name_1="foobar", custom_label_name_2="bar"} 1 syslogng_custom_key{custom_label_name_1="foobar", custom_label_name_2="foo"} 1 syslogng_custom_key{custom_label_name_1="foobar", custom_label_name_2="baz"} 3 The minimal config creates counters with the key syslogng_classified_events_total and labels app, host, program and source. E.g.: parser p_metrics_probe { metrics-probe(); }; With this config, it creates counters like these: syslogng_classified_events_total{app="example-app", host="localhost", program="baz", source="s_local_1"} 3 syslogng_classified_events_total{app="example-app", host="localhost", program="bar", source="s_local_1"} 1 syslogng_classified_events_total{app="example-app", host="localhost", program="foo", source="s_local_1"} 1 Named log paths (path ingress/egress metrics) (#4344 <https://github.com/syslog-ng/syslog-ng/pull/4344>) It is also possible to create named log paths, for example: log top-level { source(s_local); log inner-1 { filter(f_inner_1); destination(d_local_1); }; log inner-2 { filter(f_inner_2); destination(d_local_2); }; }; Each named log path counts its ingress and egress messages: syslogng_log_path_ingress{id="top-level"} 114 syslogng_log_path_ingress{id="inner-1"} 114 syslogng_log_path_ingress{id="inner-2"} 114 syslogng_log_path_egress{id="top-level"} 103 syslogng_log_path_egress{id="inner-1"} 62 syslogng_log_path_egress{id="inner-2"} 41 Note that the egress statistics only count the messages which have been have not been filtered out from the related log path, it does care about whether there are any destinations in it or that any destination delivers or drops the message. The above three features are experimental; the output of stats prometheus (names, labels, etc.) and the metrics created by metrics-probe() and named log paths may change in the next 2-3 releases. Features - $(format-date): add a new template function to format time and date values $(format-date [options] format-string [timestamp]) $(format-date) takes a timestamp in the DATETIME representation and formats it according to an strftime() format string. The DATETIME representation in syslog-ng is a UNIX timestamp formatted as a decimal number, with an optional fractional part, where the seconds and the fraction of seconds are separated by a dot. If the timestamp argument is missing, the timestamp of the message is used. Options: --time-zone <TZstring> -- override timezone of the original timestamp (#4202 <https://github.com/syslog-ng/syslog-ng/pull/4202>) - syslog-parser() and all syslog related sources: accept unquoted RFC5424 SD-PARAM-VALUEs instead of rejecting them with a parse error. sdata-parser(): this new parser allows you to parse an RFC5424 style structured data string. It can be used to parse this relatively complex format separately. (#4281 <https://github.com/syslog-ng/syslog-ng/pull/4281>) - system() source: the system() source was changed on systemd platforms to fetch journal messages that relate to the current boot only (e.g. similar to journalctl -fb) and to ignore messages generated in previous boots, even if those messages were succesfully stored in the journal and were not picked up by syslog-ng. This change was implemented as the journald access APIs work incorrectly if time goes backwards across reboots, which is an increasingly frequent event in virtualized environments and on systems that lack an RTC. If you want to retain the old behaviour, please bypass the system() source and use systemd-journal() directly, where this option can be customized. The change is not tied to @version as we deemed the new behaviour fixing an actual bug. For more information consult #2836 <https://github.com/syslog-ng/syslog-ng/issues/2836>. systemd-journald() source: add match-boot() and matches() options to allow you to constrain the collection of journal records to a subset of what is in the journal. match-boot() is a yes/no value that allows you to fetch messages that only relate to the current boot. matches() allows you to specify one or more filters on journal fields. Examples: source s_journal_current_boot_only { systemd-source(match-boot(yes)); }; source s_journal_systemd_only { systemd-source(matches( "_COMM" => "systemd" ) ); }; (#4245 <https://github.com/syslog-ng/syslog-ng/pull/4245>) - date-parser(): add value() parameter to instruct date-parser() to store the resulting timestamp in a name-value pair, instead of changing the timestamp value of the LogMessage. datetime type representation: typed values in syslog-ng are represented as strings when stored as a part of a log message. syslog-ng simply remembers the type it was stored as. Whenever the value is used as a specific type in a type-aware context where we need the value of the specific type, an automatic string parsing takes place. This parsing happens for instance whenever syslog-ng stores a datetime value in MongoDB or when $(format-date) template function takes a name-value pair as parameter. The datetime() type has stored its value as the number of milliseconds since the epoch (1970-01-01 00:00:00 GMT). This has now been enhanced by making it possible to store timestamps up to nanosecond resolutions along with an optional timezone offset. $(format-date): when applied to name-value pairs with the datetime type, use the timezone offset if one is available. (#4319 <https://github.com/syslog-ng/syslog-ng/pull/4319>) - stats: Added syslog-stats() global stats() group option. E.g.: options { stats( syslog-stats(no); ); }; It changes the behavior of counting messages based on different syslog-proto fields, like SEVERITY, FACILITY, HOST, etc... Possible values are: - yes => force enable - no => force disable - auto => let stats(level()) decide (old behavior) (#4337 <https://github.com/syslog-ng/syslog-ng/pull/4337>) - kubernetes source: Added key-delimiter() option. Some metadata fields can contain .-s in their name. This does not work with syslog-ng-s macros, which by default use . as a delimiter. The added key-delimiter() option changes this behavior by storing the parsed metadata fields with a custom delimiter. In order to reach the fields, the accessor side has to use the new delimiter format, e.g. --key-delimiter option in $(format-json). (#4213 <https://github.com/syslog-ng/syslog-ng/pull/4213>) Bugfixes - Fix conditional evaluation with a dangling filter We've fixed a bug that caused conditional evaluation (if/else/elif) and certain logpath flags (final, fallback) to occasionally malfunction. The issue only happened in certain logpath constructs; examples can be found in the PR description. (#4058 <https://github.com/syslog-ng/syslog-ng/pull/4058>) - python: Fixed a bug, where PYTHONPATH was ignored with python3.11. (#4298 <https://github.com/syslog-ng/syslog-ng/pull/4298>) - disk-buffer: Fixed disk-queue file becoming corrupt when changing disk-buf-size(). syslog-ng now continues with the originally set disk-buf-size(). Note that changing the disk-buf-size() of an existing disk-queue was never supported, but could cause errors, which are fixed now. (#4308 <https://github.com/syslog-ng/syslog-ng/pull/4308>) - dqtool: fix dqtool assign (#4355 <https://github.com/syslog-ng/syslog-ng/pull/4355>) - example-diskq-source: Fixed failing to read the disk-queue content in some cases. (#4308 <https://github.com/syslog-ng/syslog-ng/pull/4308>) - default-network-drivers(): Added support for the log-iw-size() option with a default value of 1000. Making it possible to adjust the log-iw-size() for the TCP/TLS based connections, when changing the max-connections() option. (#4328 <https://github.com/syslog-ng/syslog-ng/pull/4328>) - apache-accesslog-parser(): fix rawrequest escaping binary characters (#4303 <https://github.com/syslog-ng/syslog-ng/pull/4303>) - dqtool: Fixed dqtool cat failing to read the content in some cases. (#4308 <https://github.com/syslog-ng/syslog-ng/pull/4308>) - Fixed a rare main loop related crash on FreeBSD. (#4262 <https://github.com/syslog-ng/syslog-ng/pull/4262>) - Fix a warning message that was displayed incorrectly: "The actual number of worker threads exceeds the number of threads estimated at startup." (#4282 <https://github.com/syslog-ng/syslog-ng/pull/4282>) - Fix minor memory leak related to tznames (#4334 <https://github.com/syslog-ng/syslog-ng/pull/4334>) Packaging - dbparser: libdbparser.so has been renamed to libcorrelation.so. (#4294 <https://github.com/syslog-ng/syslog-ng/pull/4294>) - systemd-journal: Fixed a linker error, which occurred, when building with --with-systemd-journal=optional. (#4304 <https://github.com/syslog-ng/syslog-ng/pull/4304>) (#4302 <https://github.com/syslog-ng/syslog-ng/pull/4302>) Notes to developers - LogThreadedSourceDriver and Fetcher: implement source-side batching support on the input path by assigning a thread_id to dynamically spawned input threads (e.g. those spawned by LogThreadedSourceDriver) too. To actually improve performance the source driver should disable automatic closing of batches by setting auto_close_batches to FALSE and calling log_threaded_source_close_batch() explicitly. (#3969 <https://github.com/syslog-ng/syslog-ng/pull/3969>) Other changes - stats related options: The stats related options have been groupped to a new stats() block. This affects the following global options: - stats-freq() - stats-level() - stats-lifetime() - stats-max-dynamics() These options have been kept for backward compatibility, but they have been deprecated. Migrating from the old stats options to the new ones looks like this. @version: 4.0 options { stats-freq(1); stats-level(1); stats-lifetime(1000); stats-max-dynamics(10000); }; @version: 4.1 options { stats( freq(1) level(1) lifetime(1000) max-dynamics(10000) ); }; *Breaking change* For more than a decade stats() was a deprecated alias to stats-freq(), now it is used as the name of the new block. If you have been using stats(xy), use stats(freq(xy)) instead. (#4337 <https://github.com/syslog-ng/syslog-ng/pull/4337>) - kubernetes source: Improved error logging, when the pod was unreachable through the python API. (#4305 <https://github.com/syslog-ng/syslog-ng/pull/4305>) - APT repository: Added .gz, .xz and .bz2 compression to the Packages file. (#4313 <https://github.com/syslog-ng/syslog-ng/pull/4313>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Attila Szakacs, Balazs Scheidler, Bálint Horváth, Gergo Ferenc Kovacs, Hofi, László Várady, Ronny Meeus, Szilard Parrag

2 2

syslog-ng reset connection before data transfer
by Dragan Zecevic 09 Mar '23

09 Mar '23

Hi everybody, I hope you are well. I was hoping if you can help me with some issue. I am trying to collect logs from some servers via syslog. TLS session seems to be established - handshake, certs, ciphers all looking ok. But when data need to be send it seems syslog-ng server disconnects the session from some reason. In /var/log/messages I can only see start/stop messages even after turning on verbose logging: Feb 10 16:11:48 xxx syslog-ng[21262]: Syslog connection accepted; fd='175', client='AF_INET(x.y.w.z:34986)', local='AF_INET(0.0.0.0:xxx)' Feb 10 16:11:49 xxx syslog-ng[21262]: Syslog connection closed; fd='175', client='AF_INET(x.y.w.z:34986)', local='AF_INET(0.0.0.0:xxx)' This is the configuration I am using: source s_xxx { syslog( ip(0.0.0.0) port(xxx) transport("tls") tls( key-file("/etc/syslog-ng/key.d/xxx.key") cert-file("/etc/syslog-ng/cert.d/xxx.cer") ca-dir("/etc/syslog-ng/ca.d") peer-verify(required-trusted) cipher-suite("xxx...xxx") ) flags(store-raw-message) ); }; filter filter_xxx { host("xxx") ... or host("xxx"); }; destination folder_xxx { file( "/var/log/xxx/${R_YEAR}${R_MONTH}${R_DAY}/${SOURCEIP}_${HOST}_${R_HOUR}.log" template("${RAWMSG}\n") dir-group(xxx) dir-perm(0650) group(xxx) ); }; log { source(s_xxx); filter(filter_xxx); destination(folder_xxx); flags(flow-control); }; And bellow is last part of the session captured with tcpdump from source side. It is interesting to me that there is 1h offset comparing to /var/log/messages on syslog-ng. Src and dst server are in same environment. 24 2023-02-10 15:11:49,073335 x.y.w.z x.y.w.z TLSv1.2 445 Application Data Frame 24: 445 bytes on wire (3560 bits), 445 bytes captured (3560 bits) Encapsulation type: Ethernet (1) Arrival Time: Feb 10, 2023 15:11:49.073335000 Central Europe Standard Time [Time shift for this packet: 0.000000000 seconds] Epoch Time: 1676038309.073335000 seconds [Time delta from previous captured frame: 0.906559000 seconds] [Time delta from previous displayed frame: 0.906559000 seconds] [Time since reference or first frame: 6.775677000 seconds] Frame Number: 24 Frame Length: 445 bytes (3560 bits) Capture Length: 445 bytes (3560 bits) [Frame is marked: False] [Frame is ignored: False] [Protocols in frame: eth:ethertype:ip:tcp:tls] [Coloring Rule Name: TCP] [Coloring Rule String: tcp] Ethernet II, Src: xxx, Dst: xxx Destination: xxx Address: xxx .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) Source: xxx Address: xxx .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) Type: IPv4 (0x0800) Internet Protocol Version 4, Src: x.y.z.w, Dst: x.y.z.w 0100 .... = Version: 4 .... 0101 = Header Length: 20 bytes (5) Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT) 0000 00.. = Differentiated Services Codepoint: Default (0) .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0) Total Length: 431 Identification: 0xeebc (61116) 010. .... = Flags: 0x2, Don't fragment 0... .... = Reserved bit: Not set .1.. .... = Don't fragment: Set ..0. .... = More fragments: Not set ...0 0000 0000 0000 = Fragment Offset: 0 Time to Live: 64 Protocol: TCP (6) Header Checksum: 0x0a89 [validation disabled] [Header checksum status: Unverified] Source Address: x.y.z.w Destination Address: x.y.z.w Transmission Control Protocol, Src Port: 34986, Dst Port: xxx, Seq: 2380, Ack: 5750, Len: 391 Source Port: 34986 Destination Port: xxx [Stream index: 1] [Conversation completeness: Complete, WITH_DATA (63)] [TCP Segment Len: 391] Sequence Number: 2380 (relative sequence number) Sequence Number (raw): 2630184540 [Next Sequence Number: 2771 (relative sequence number)] Acknowledgment Number: 5750 (relative ack number) Acknowledgment number (raw): 2509794849 0101 .... = Header Length: 20 bytes (5) Flags: 0x018 (PSH, ACK) 000. .... .... = Reserved: Not set ...0 .... .... = Accurate ECN: Not set .... 0... .... = Congestion Window Reduced: Not set .... .0.. .... = ECN-Echo: Not set .... ..0. .... = Urgent: Not set .... ...1 .... = Acknowledgment: Set .... .... 1... = Push: Set .... .... .0.. = Reset: Not set .... .... ..0. = Syn: Not set .... .... ...0 = Fin: Not set [TCP Flags: ·······AP···] Window: 318 [Calculated window size: 40704] [Window size scaling factor: 128] Checksum: 0x41a5 [unverified] [Checksum Status: Unverified] Urgent Pointer: 0 [Timestamps] [Time since first frame in this TCP stream: 1.007330000 seconds] [Time since previous frame in this TCP stream: 0.906559000 seconds] [SEQ/ACK analysis] [iRTT: 0.003010000 seconds] [Bytes in flight: 391] [Bytes sent since last PSH flag: 391] TCP payload (391 bytes) Transport Layer Security TLSv1.2 Record Layer: Application Data Protocol: Application Data Content Type: Application Data (23) Version: TLS 1.2 (0x0303) Length: 386 Encrypted Application Data: 000000000000000146f17ff0e884c81f9317ba9e01c3b48763944e3905fb27fc96ce76fc… 25 2023-02-10 15:11:49,076214 x.y.z.w x.y.z.w TCP 60 xxx → 34986 [FIN, ACK] Seq=5750 Ack=2771 Win=65536 Len=0 Frame 25: 60 bytes on wire (480 bits), 60 bytes captured (480 bits) Encapsulation type: Ethernet (1) Arrival Time: Feb 10, 2023 15:11:49.076214000 Central Europe Standard Time [Time shift for this packet: 0.000000000 seconds] Epoch Time: 1676038309.076214000 seconds [Time delta from previous captured frame: 0.002879000 seconds] [Time delta from previous displayed frame: 0.002879000 seconds] [Time since reference or first frame: 6.778556000 seconds] Frame Number: 25 Frame Length: 60 bytes (480 bits) Capture Length: 60 bytes (480 bits) [Frame is marked: False] [Frame is ignored: False] [Protocols in frame: eth:ethertype:ip:tcp] [Coloring Rule Name: TCP SYN/FIN] [Coloring Rule String: tcp.flags & 0x02 || tcp.flags.fin == 1] Ethernet II, Src: xxx, Dst: xxx Destination: xxx Address: xxx .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) Source: xxx Address: xxx .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) Type: IPv4 (0x0800) Trailer: 000085725dce Internet Protocol Version 4, Src: x.y.z.w, Dst: x.y.z.w 0100 .... = Version: 4 .... 0101 = Header Length: 20 bytes (5) Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT) 0000 00.. = Differentiated Services Codepoint: Default (0) .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0) Total Length: 40 Identification: 0xea10 (59920) 010. .... = Flags: 0x2, Don't fragment 0... .... = Reserved bit: Not set .1.. .... = Don't fragment: Set ..0. .... = More fragments: Not set ...0 0000 0000 0000 = Fragment Offset: 0 Time to Live: 52 Protocol: TCP (6) Header Checksum: 0x1cbc [validation disabled] [Header checksum status: Unverified] Source Address: x.y.z.w Destination Address: x.y.z.w Transmission Control Protocol, Src Port: xxx, Dst Port: 34986, Seq: 5750, Ack: 2771, Len: 0 Source Port: xxx Destination Port: 34986 [Stream index: 1] [Conversation completeness: Complete, WITH_DATA (63)] [TCP Segment Len: 0] Sequence Number: 5750 (relative sequence number) Sequence Number (raw): 2509794849 [Next Sequence Number: 5751 (relative sequence number)] Acknowledgment Number: 2771 (relative ack number) Acknowledgment number (raw): 2630184931 0101 .... = Header Length: 20 bytes (5) Flags: 0x011 (FIN, ACK) 000. .... .... = Reserved: Not set ...0 .... .... = Accurate ECN: Not set .... 0... .... = Congestion Window Reduced: Not set .... .0.. .... = ECN-Echo: Not set .... ..0. .... = Urgent: Not set .... ...1 .... = Acknowledgment: Set .... .... 0... = Push: Not set .... .... .0.. = Reset: Not set .... .... ..0. = Syn: Not set .... .... ...1 = Fin: Set [TCP Flags: ·······A···F] Window: 4 [Calculated window size: 65536] [Window size scaling factor: 16384] Checksum: 0xc512 [unverified] [Checksum Status: Unverified] Urgent Pointer: 0 [Timestamps] [Time since first frame in this TCP stream: 1.010209000 seconds] [Time since previous frame in this TCP stream: 0.002879000 seconds] [SEQ/ACK analysis] [This is an ACK to the segment in frame: 24] [The RTT to ACK the segment was: 0.002879000 seconds] [iRTT: 0.003010000 seconds] 26 2023-02-10 15:11:49,117776 x.y.z.w x.y.z.w TCP 54 34986 → xxx [ACK] Seq=2771 Ack=5751 Win=40704 Len=0 Frame 26: 54 bytes on wire (432 bits), 54 bytes captured (432 bits) Encapsulation type: Ethernet (1) Arrival Time: Feb 10, 2023 15:11:49.117776000 Central Europe Standard Time [Time shift for this packet: 0.000000000 seconds] Epoch Time: 1676038309.117776000 seconds [Time delta from previous captured frame: 0.041562000 seconds] [Time delta from previous displayed frame: 0.041562000 seconds] [Time since reference or first frame: 6.820118000 seconds] Frame Number: 26 Frame Length: 54 bytes (432 bits) Capture Length: 54 bytes (432 bits) [Frame is marked: False] [Frame is ignored: False] [Protocols in frame: eth:ethertype:ip:tcp] [Coloring Rule Name: TCP] [Coloring Rule String: tcp] Ethernet II, Src: Mellanox_69:6f:47 (1c:34:da:69:6f:47), Dst: ICANNIAN_00:40:20 (00:00:5e:00:40:20) Destination: ICANNIAN_00:40:20 (00:00:5e:00:40:20) Address: ICANNIAN_00:40:20 (00:00:5e:00:40:20) .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) Source: Mellanox_69:6f:47 (1c:34:da:69:6f:47) Address: Mellanox_69:6f:47 (1c:34:da:69:6f:47) .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) Type: IPv4 (0x0800) Internet Protocol Version 4, Src: x.y.z.w, Dst: x.y.z.w 0100 .... = Version: 4 .... 0101 = Header Length: 20 bytes (5) Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT) 0000 00.. = Differentiated Services Codepoint: Default (0) .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0) Total Length: 40 Identification: 0xeebd (61117) 010. .... = Flags: 0x2, Don't fragment 0... .... = Reserved bit: Not set .1.. .... = Don't fragment: Set ..0. .... = More fragments: Not set ...0 0000 0000 0000 = Fragment Offset: 0 Time to Live: 64 Protocol: TCP (6) Header Checksum: 0x0c0f [validation disabled] [Header checksum status: Unverified] Source Address: x.y.z.w Destination Address: x.y.z.w Transmission Control Protocol, Src Port: 34986, Dst Port: xxx, Seq: 2771, Ack: 5751, Len: 0 Source Port: 34986 Destination Port: xxx [Stream index: 1] [Conversation completeness: Complete, WITH_DATA (63)] [TCP Segment Len: 0] Sequence Number: 2771 (relative sequence number) Sequence Number (raw): 2630184931 [Next Sequence Number: 2771 (relative sequence number)] Acknowledgment Number: 5751 (relative ack number) Acknowledgment number (raw): 2509794850 0101 .... = Header Length: 20 bytes (5) Flags: 0x010 (ACK) 000. .... .... = Reserved: Not set ...0 .... .... = Accurate ECN: Not set .... 0... .... = Congestion Window Reduced: Not set .... .0.. .... = ECN-Echo: Not set .... ..0. .... = Urgent: Not set .... ...1 .... = Acknowledgment: Set .... .... 0... = Push: Not set .... .... .0.. = Reset: Not set .... .... ..0. = Syn: Not set .... .... ...0 = Fin: Not set [TCP Flags: ·······A····] Window: 318 [Calculated window size: 40704] [Window size scaling factor: 128] Checksum: 0x401e [unverified] [Checksum Status: Unverified] Urgent Pointer: 0 [Timestamps] [Time since first frame in this TCP stream: 1.051771000 seconds] [Time since previous frame in this TCP stream: 0.041562000 seconds] [SEQ/ACK analysis] [This is an ACK to the segment in frame: 25] [The RTT to ACK the segment was: 0.041562000 seconds] [iRTT: 0.003010000 seconds] 27 2023-02-10 15:12:10,183395 x.y.w.z x.y.w.z TLSv1.2 457 Application Data Frame 27: 457 bytes on wire (3656 bits), 457 bytes captured (3656 bits) Encapsulation type: Ethernet (1) Arrival Time: Feb 10, 2023 15:12:10.183395000 Central Europe Standard Time [Time shift for this packet: 0.000000000 seconds] Epoch Time: 1676038330.183395000 seconds [Time delta from previous captured frame: 21.065619000 seconds] [Time delta from previous displayed frame: 21.065619000 seconds] [Time since reference or first frame: 27.885737000 seconds] Frame Number: 27 Frame Length: 457 bytes (3656 bits) Capture Length: 457 bytes (3656 bits) [Frame is marked: False] [Frame is ignored: False] [Protocols in frame: eth:ethertype:ip:tcp:tls] [Coloring Rule Name: TCP] [Coloring Rule String: tcp] Ethernet II, Src: xxx, Dst: xxx Destination: xxx Address: xxx .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) Source: xxx Address: xxx .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) Type: IPv4 (0x0800) Internet Protocol Version 4, Src: x.y.w.z, Dst: x.y.w.z 0100 .... = Version: 4 .... 0101 = Header Length: 20 bytes (5) Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT) 0000 00.. = Differentiated Services Codepoint: Default (0) .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0) Total Length: 443 Identification: 0xeebe (61118) 010. .... = Flags: 0x2, Don't fragment 0... .... = Reserved bit: Not set .1.. .... = Don't fragment: Set ..0. .... = More fragments: Not set ...0 0000 0000 0000 = Fragment Offset: 0 Time to Live: 64 Protocol: TCP (6) Header Checksum: 0x0a7b [validation disabled] [Header checksum status: Unverified] Source Address: x.y.w.z Destination Address: x.y.w.z Transmission Control Protocol, Src Port: 34986, Dst Port: xxx, Seq: 2771, Ack: 5751, Len: 403 Source Port: 34986 Destination Port: xxx [Stream index: 1] [Conversation completeness: Complete, WITH_DATA (63)] [TCP Segment Len: 403] Sequence Number: 2771 (relative sequence number) Sequence Number (raw): 2630184931 [Next Sequence Number: 3174 (relative sequence number)] Acknowledgment Number: 5751 (relative ack number) Acknowledgment number (raw): 2509794850 0101 .... = Header Length: 20 bytes (5) Flags: 0x018 (PSH, ACK) 000. .... .... = Reserved: Not set ...0 .... .... = Accurate ECN: Not set .... 0... .... = Congestion Window Reduced: Not set .... .0.. .... = ECN-Echo: Not set .... ..0. .... = Urgent: Not set .... ...1 .... = Acknowledgment: Set .... .... 1... = Push: Set .... .... .0.. = Reset: Not set .... .... ..0. = Syn: Not set .... .... ...0 = Fin: Not set [TCP Flags: ·······AP···] Window: 318 [Calculated window size: 40704] [Window size scaling factor: 128] Checksum: 0x41b1 [unverified] [Checksum Status: Unverified] Urgent Pointer: 0 [Timestamps] [Time since first frame in this TCP stream: 22.117390000 seconds] [Time since previous frame in this TCP stream: 21.065619000 seconds] [SEQ/ACK analysis] [iRTT: 0.003010000 seconds] [Bytes in flight: 403] [Bytes sent since last PSH flag: 403] TCP payload (403 bytes) Transport Layer Security TLSv1.2 Record Layer: Application Data Protocol: Application Data Content Type: Application Data (23) Version: TLS 1.2 (0x0303) Length: 398 Encrypted Application Data: 0000000000000002be4aae5822b205849734ad881717619991987bc9817d0b0417781265… 28 2023-02-10 15:12:10,185930 x.y.w.z x.y.w.z TCP 60 xxx → 34986 [RST] Seq=5751 Win=0 Len=0 Frame 28: 60 bytes on wire (480 bits), 60 bytes captured (480 bits) Encapsulation type: Ethernet (1) Arrival Time: Feb 10, 2023 15:12:10.185930000 Central Europe Standard Time [Time shift for this packet: 0.000000000 seconds] Epoch Time: 1676038330.185930000 seconds [Time delta from previous captured frame: 0.002535000 seconds] [Time delta from previous displayed frame: 0.002535000 seconds] [Time since reference or first frame: 27.888272000 seconds] Frame Number: 28 Frame Length: 60 bytes (480 bits) Capture Length: 60 bytes (480 bits) [Frame is marked: False] [Frame is ignored: False] [Protocols in frame: eth:ethertype:ip:tcp] [Coloring Rule Name: TCP RST] [Coloring Rule String: tcp.flags.reset eq 1] Ethernet II, Src: xxx, Dst: xxx Destination: xxx Address: xxx .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) Source: xxx Address: xxx .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) Type: IPv4 (0x0800) Trailer: 00003b22a540 Internet Protocol Version 4, Src: x.y.w.z, Dst: x.y.w.z 0100 .... = Version: 4 .... 0101 = Header Length: 20 bytes (5) Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT) 0000 00.. = Differentiated Services Codepoint: Default (0) .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0) Total Length: 40 Identification: 0xd52f (54575) 010. .... = Flags: 0x2, Don't fragment 0... .... = Reserved bit: Not set .1.. .... = Don't fragment: Set ..0. .... = More fragments: Not set ...0 0000 0000 0000 = Fragment Offset: 0 Time to Live: 52 Protocol: TCP (6) Header Checksum: 0x319d [validation disabled] [Header checksum status: Unverified] Source Address: x.y.w.z Destination Address: x.y.w.z Transmission Control Protocol, Src Port: xxx, Dst Port: 34986, Seq: 5751, Len: 0 Source Port: xxx Destination Port: 34986 [Stream index: 1] [Conversation completeness: Complete, WITH_DATA (63)] [TCP Segment Len: 0] Sequence Number: 5751 (relative sequence number) Sequence Number (raw): 2509794850 [Next Sequence Number: 5751 (relative sequence number)] Acknowledgment Number: 0 Acknowledgment number (raw): 0 0101 .... = Header Length: 20 bytes (5) Flags: 0x004 (RST) 000. .... .... = Reserved: Not set ...0 .... .... = Accurate ECN: Not set .... 0... .... = Congestion Window Reduced: Not set .... .0.. .... = ECN-Echo: Not set .... ..0. .... = Urgent: Not set .... ...0 .... = Acknowledgment: Not set .... .... 0... = Push: Not set .... .... .1.. = Reset: Set .... .... ..0. = Syn: Not set .... .... ...0 = Fin: Not set [TCP Flags: ·········R··] Window: 0 [Calculated window size: 0] [Window size scaling factor: 16384] Checksum: 0xd1cb [unverified] [Checksum Status: Unverified] Urgent Pointer: 0 [Timestamps] [Time since first frame in this TCP stream: 22.119925000 seconds] [Time since previous frame in this TCP stream: 0.002535000 seconds] I have other log collections working fine with TLS. Can you please give me some hint what could cause this? Thank you. Br, Dragan

4 8

Processing orphaned disk queue files
by Steve Bernacki 08 Mar '23

08 Mar '23

Hello, I am using syslog-ng OSE 3.29. I have two destinations that I recently redefined in order to spread load out across a number of receivers. However, having done this, I now have orphaned queue files of considerable size that I'd like to process through. I found one reference on how to do this with syslog-ng PE which is documented at https://support-public.cfm.quest.com/46246_syslog-ng-pe-diskbuffer-recovery…, but I'm finding the OSE tools are acting slightly differently, for example persist-tool dump prints its output as hex instead of ASCII, as is documented in the PDF I linked to above. In addition, the process is fairly involved, which makes me nervous to experiment with this. Looking at the command output for "dqtool --help", I see there are a number of commands listed that are not documented anywhere, like "relocate" and "assign", which look like they may be helpful to me. Does anyone know if these can be used to help me process through these logs? Thanks Steve

2 2

Insider 2023-02: 4.0; Azure; 101; FreeBSD;
by Peter Czanik (pczanik) 16 Feb '23

16 Feb '23

Dear syslog-ng users, This is the 107th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Version 4.0 of syslog-ng available ---------------------------------- After almost 14 years, a new major version of syslog-ng is available. Type support for name-value pairs is now available and Python support received major improvements. For a complete list of changes, check the release notes: https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-4.0.1 Version 4.0.1 is already part of some of the Linux distributions, but for most of the distributions you have to use 3rd-party packages to install syslog-ng 4.0. Debian / Ubuntu: https://www.syslog-ng.com/community/b/blog/posts/installing-the-latest-sysl… openSUSE / SLES / Fedora / RHEL: https://www.syslog-ng.com/community/b/blog/posts/installing-latest-syslog-n… FreeBSD ports are not yet updated, due to a temporary problem, but you can still compile syslog-ng 4.0 yourself: https://www.syslog-ng.com/community/b/blog/posts/installing-syslog-ng-4-0-1… Feeding events with syslog-ng PE to Azure Event Hub and Google Pub/Sub -------------------------------------------------------- The syslog-ng Premium Edition (PE) application is built on the solid foundation of syslog-ng Open Source Edition (OSE). PE inherits many of the OSE features and adds cloud-related features, among others. Our latest webinar shows you how to send log messages to Azure Event Hub and Google Pub/Sub. https://www.syslog-ng.com/community/b/blog/posts/feeding-events-with-syslog… syslog-ng 101: how to get started with learning syslog-ng? ---------------------------------------------------------- How to get started with syslog-ng? There are two main resources: the syslog-ng documentation and the syslog-ng blogs. You should learn the concepts and basics from the documentation. The blogs document use cases and you can use the docs as a reference. https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-101-how-to-get-s… Compiling syslog-ng git snapshots on FreeBSD -------------------------------------------- The syslog-ng team publishes nightly syslog-ng git snapshot builds for Debian and Ubuntu. I publish weekly snapshot builds for RPM distributions. Recently, I was asked if creating git snapshot builds for FreeBSD is also possible. Yes, it is. That is how I test syslog-ng on FreeBSD. However, it needs some extra preparations. https://www.syslog-ng.com/community/b/blog/posts/compiling-syslog-ng-git-sn… WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Syslog messages not stored in separate lines
by Dragan Zecevic 10 Feb '23

10 Feb '23

Hi, I am collecting logs from a network device. They configured syslog format on their source side to be RFC3164. On syslog-ng side I am using source and destination like this: source s_xxx { network( ip(0.0.0.0) transport(tcp) port(xxx) flags(store-raw-message) ); }; destination folder_xxx { file( "/xxx/${R_YEAR}${R_MONTH}${R_DAY}/${SOURCEIP}_${HOST}_${R_HOUR}.log" template("${RAWMSG}\n") ); }; syslog-ng version 3.34 CentOS Linux release 7.9.2009 The problem is that syslog messages are stored in raw format but not separated in different line. Parity bit of new message starts imidiatelly after previous line -without space or enter. I have the same config for some other hosts and there log files are created with separate lines. Vendor says they can't change anything on source side. Do you have any idea what is the cause of this? Thank you. Br, Dragan

3 8

Both formats (RFC3164 and 5424) via one Port
by Matthias Gruber 24 Jan '23

24 Jan '23

Hi! We have Appliances which are generating syslog-messages in both formats, one in 3164 the other in 5424. Would it be possible to put them into one source? Something like source s_remote_appl_tcp { syslog( ip("99.99.99.99") port("5152") transport("tcp") flags(no-parse) ); }; and.... (just excepts) parser p_0140_A_parser { syslog-parser(); }; parser p_0140_B_parser { syslog-parser(flags(syslog-protocol); }; log { source(s_remote_appl_tcp); parser(p_0140_A_parser); destination(d_0140_all); flags(final); }; log { source(s_remote_appl_tcp); parser(p_0140_B_parser); destination(d_0140_all); flags(final); }; Any hints are welcome... Cheers Matthias P.S. Its an OSE 3.38.1 running ------------------------------------------------------------------------------------ METZLER Informationstechnologie Matthias Gruber IT-Infrastruktur & -Betrieb B. Metzler seel. Sohn & Co. Aktiengesellschaft Untermainanlage 1 60329 Frankfurt am Main Telefon 069 21 04 - 43 30 Telefax 069 21 04 - 40 40 MGruber(a)metzler.com www.metzler.com Vorstand: Stefanie Buchmann, Kim Comperl, Mario Mattera, Franz von Metzler, Emmerich Müller, Dr. Marco Schulmerich, Gerhard Wiesheu Vorsitzender des Aufsichtsrats: Wolfgang Kirsch Sitz der Gesellschaft: Frankfurt am Main, Handelsregister-Nr. HRB 123 365 Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfänger sein, so bitten wir Sie höflich, dies unverzüglich dem Absender mitzuteilen und die Nachricht zu löschen. Es ist unzulässig, die Nachricht unbefugt weiterzuleiten oder zu kopieren. Da wir nicht die Echtheit oder Vollständigkeit der in dieser Nachricht enthaltenen Informationen garantieren oder zusichern können, sind die vorstehenden Ausführungen rechtlich nicht bindend. Eine Haftung hierfür wird ausgeschlossen. This message is confidential. If you are not the intended recipient, we kindly ask you to inform the sender and delete the information. Any unauthorised dissemination or copying hereof is prohibited. As we cannot guarantee or assure the genuineness or completeness of the information contained in this message, the statements set forth above are not legally binding. Accordingly we cannot accept any liability for their contents.

3 2

Re: [syslog-ng] Local sources seem not to be working
by Szilard Parrag (sparrag) 11 Jan '23

11 Jan '23

Hi Alex, We've checked it too and syslog-ng does not release the file descriptor of journald even with flow-control enabled. Also, your internal logs seem rather terse, maybe there is a filter which filters out the important parts. Could you please check it? Szilard

4 12

syslog-ng 4.0, new major release
by Balazs Scheidler 22 Dec '22

22 Dec '22

Dear syslog-ng users, I am pleased to announce that a new major release of syslog-ng, has been released and is now available on our GitHub releases page: https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-4.0.1 The release is a culmination of changes incrementally introduced in the past year since 3.36.1, with new behaviours being enabled if you use '@version: 4.0' at the top of your configuration file. The NEWS file is included below. It's been a while since I personally announced a syslog-ng release myself. I am very proud having been part of this major milestone, further extending syslog-ng's capabilities to cover a lot more use cases, just as syslog-ng 3.0 did that in 2009. Happy logging! Bazsi 4.0.1 ===== _This is the combination of the news entries of 4.0.0 and 4.0.1._ This is a new major version of syslog-ng, ending the 3.x series which started roughly 13 years ago, on 17th February 2009. Like all releases in the 3.x series, 4.0.0 is not a breaking change either. Long-term compatibility has been and continues to be an essential objective of syslog-ng; thus, you can still run unchanged configurations that were originally created for syslog-ng 3.0.0. You can safely upgrade to 4.0.0 if you followed along 3.x, and you should probably also consider upgrading if you are stuck with an older 3.x release. The new version number primarily indicates that this version of syslog-ng is much more than the software we released 13 years ago. While it does have certain "big-bang" items in its feature list, new features were continuously introduced throughout our 3.x series as well. Our engineering practices have not changed simply because we were working on a new major release: this is the continuation of our previous releases in every respect, produced in the same manner, just with a more catchy version number. For this reason, there is no separate deprecation or support period for 3.x releases, similarly with our existing practice. We support earlier syslog-ng releases by providing maintenance and fixes in the new release track. Fixes to problems are not backported to earlier releases by the syslog-ng project. ## Highlights ### Introduce runtime type information to name-value pairs syslog-ng uses a data model where a log message contains an unordered set of name-value pairs. The values stored in these name-value pairs are usually textual, so syslog-ng has traditionally stored these values in text format. With the increase of JSON-based message sources and destinations, types became more important. If we encounter a message where a name-value pair originates from a JSON document, and this document contains a member that is numeric, we may want to reproduce that as we send this data to a consumer. For example, sometimes we extract a numerical metric from a log message, and we need to send this to a consumer, again with the correct type. To be able to do this, we added runtime type information to the syslog-ng message model: each name-value pair becomes a (name, type, value) triplet. We introduced the following types: - string: simple textual data, mostly utf8 (but not always) - int: an integer representable by a 64 bit signed value - double: a double precision floating point number - boolean: true or false - datetime: Date and Time represented by the milliseconds since epoch - list: list of strings - json: JSON snippet - null: an unset value Apart from the syslog-ng core supporting the notion of types, its use is up to the sources, filters, rewrite rules, parsers and destinations that set or make use of them in any way it makes the most sense for the component in question. ### Type-aware comparisons syslog-ng uses filter expressions to make routing decisions and during the transformation of messages. These filter expressions are used in filter {} or if {} statements, for example. In these expressions, you can use comparison operators. This example, for instance, uses the '>' operator to check for HTTP response codes greater-or-equal than 500: ``` if ("${apache.response}" >= 500) { }; ``` Earlier, we had two sets of operators, one for numeric (==, !=, <, >) and the other for string-based comparisons (eq, ne, gt, lt). The separate operators were cumbersome to use. Users often forgot which operator was the right one for a specific case. Typing allows us to do the right thing in most cases automatically, and a syntax that allows the user to override the automatic decisions in the rare case. With that, starting with 4.0, the old-numeric operators have been converted to be type-aware operators. It would compare as strings if both sides of the comparisons are strings. It would compare numerically if at least one side is numeric. A great deal of inspiration was taken from JavaScript, which was considered to be a good model, since the problem space is similar. See this blog post for more details: https://syslog-ng-future.blog/syslog-ng-4-progress-3-38-1-release/ ### Capture type information from JSON When using json-parser(), syslog-ng converts all members of a JSON object to syslog-ng name-value pairs. Prior to the introduction of type support, these name-value pairs were all stored as strings. Any type information originally present in the incoming JSON object was lost. This meant that if you regenerated the JSON from the name-value pairs using the $(format-json) template function, all numbers, booleans and other types became strings in the output. There has been a feature in syslog-ng that alleviated the loss of types. This feature was called "type-hints". Type-hints tell $(format-json) to use a specific type on output, independently of a name-value pair's original type, but this type conversion needed to be explicit in the configuration. An example configuration that parses JSON on input and produces a JSON on output: ``` log { source { ... }; parser { json-parser(prefix('.json.')); }; destination { file(... template("$(format-json .json.*)\n")); }; }; ``` To augment the above with type hinting, you could use: ``` log { source { ... }; parser { json-parser(prefix('.json.')); }; destination { file(... template("$(format-json .json.* .json.value=int64(${.json.value})\n")); }; }; ``` NOTE the presence of the int64() type hint in the 2nd example. The new feature introduced with typing is that syslog-ng would automatically store the JSON type information as a syslog-ng type, thus it will transparently carry over types from inputs to output, without having to be explicit about them. ### Typing support for various components in syslog-ng Typing is a feature throughout syslog-ng, and although the gust of it has been explained in the highlights section, some further details are documented in the list down below: * type-aware comparisons in filter expressions: as detailed above, the previously numeric operators become type-aware, and the exact comparison performed will be based on types associated with the values we compare. * json-parser() and $(format-json): JSON support is massively improved with the introduction of types. For one: type information is retained across input parsing->transformation->output formatting. JSON lists (arrays) are now supported and are converted to syslog-ng lists so they can be manipulated using the $(list-*) template functions. There are other important improvements in how we support JSON. * set(), groupset(): in any case where we allow the use of templates, support for type-casting was added, and the type information is properly promoted. * db-parser() type support: db-parser() gets support for type casts, <value> assignments within db-parser() rules can associate types with values using the "type" attribute, e.g. `<value name="foobar" type="integer">$PID</value>`. The “integer” is a type-cast that associates $foobar with an integer type. db-parser()’s internal parsers (e.g. @NUMBER@) will also associate type information with a name-value pair automatically. * add-contextual-data() type support: any new name-value pair that is populated using add-contextual-data() will propagate type information, similarly to db-parser(). * map-value-pairs() type support: propagate type information * SQL type support: the sql() driver gained support for types, so that columns with specific types will be stored as those types. * template type support: templates can now be casted explicitly to a specific type, but they also propagate type information from macros/template functions and values in the template string * value-pairs type support: value-pairs form the backbone of specifying a set of name-value pairs and associated transformations to generate JSON or a key-value pair format. It also gained support for types, the existing type-hinting feature that was already part of value-pairs was adapted and expanded to other parts of syslog-ng. * `python()` typing: support for typing was added to all Python components (sources, destinations, parsers and template functions), along with more documentation & examples on how the Python bindings work. All types except json() are supported as they are queried- or changed by Python code. * on-disk serialized formats (e.g. disk buffer/logstore): we remain compatible with messages serialized with an earlier version of syslog-ng, and the format we choose remains compatible for “downgrades” as well. E.g. even if a new version of syslog-ng serialized a message, the old syslog-ng and associated tools will be able to read it (sans type information of course) ### Improved support for lists (arrays) For syslog-ng, everything is traditionally a string. A convention was started with syslog-ng in v3.10, where a comma-separated format could be used as a kind of array using the `$(list-*)` family of template functions. For example, $(list-head) takes off the first element in a list, while $(list-tail) takes the last. You can index and slice list elements using the $(list-slice) and $(list-nth) functions and so on. syslog-ng has started to return such lists in various cases, so they can be manipulated using these list-specific template functions. These include the xml-parser(), or the $(explode) template function, but there are others. Here is an example that has worked since syslog-ng 3.10: ``` # MSG contains foo:bar:baz # - the $(list-head) takes off the first element of a list # - the $(explode) expression splits a string at the specified separator, ':' in this case. $(list-head $(explode : $MSG)) ``` New functions that improve these features: - JSON arrays are converted to lists, making it a lot easier to slice and extract information from JSON arrays. Of course, $(format-json) will take lists and convert them back to arrays. - The $* is a new macro that converts the internal list of match variables ($1, $2, $3 and so on) to a list, usable with $(list-*) template functions. These match variables have traditionally been filled by regular expressions when a capture group in a regexp matches. - The set-matches() rewrite operation performs the reverse; it assigns the match variables to list elements, making it easier to use list elements in template expressions by assigning them to $1, $2, $3 and so on. - Top-level JSON arrays (e.g. ones where the incoming JSON data is an array and not an object) are now accepted, and the array elements are assigned to the match variables. ### Python support syslog-ng has had support for Python-based processing elements since 3.7, released in 2015, which was greatly expanded early 2017 (3.9, LogParser) and late 2018 (3.18, LogSource and LogFetcher). This support has now been improved in a number of ways to make its use both easier and its potential more powerful. A framework was added to syslog-ng that allows seamless implementation of syslog-ng features in Python, with a look and feel of that of a native implementation. An example for using this framework is available in the `modules/python-modules/example` directory, as well as detailed documentation in the form of modules/python-modules/README.md that is installed to /etc/syslog-ng/python. The framework consists of these changes: * `syslogng` Python package: native code provided by the syslog-ng core has traditionally been exported in the `syslogng` Python module. An effort was made to make these native classes exported by the C layer more discoverable and more intuitive. As a part of this effort, the interfaces for all key Python components (LogSource, LogFetcher, LogDestination, LogParser) were exposed in the syslogng module, along with in-line documentation. * `/etc/syslog-ng/python`: syslog-ng now automatically adds this directory to the PYTHONPATH so that you have an easy place to add Python modules required by your configuration. * Python virtualenv support for production use: more sophisticated Python modules usually have 3rd party dependencies, which either needed to be installed from the OS repositories (using the apt-get or yum/dnf tools) or PyPI (using the pip tool). syslog-ng now acquired support for an embedded Python virtualenv (/var/lib/syslog-ng/python-venv or similar, depending on the installation layout), meaning that these requirements can be installed privately, without deploying them in the system PYTHONPATH where it might collide with other applications. The base set of requirements that syslog-ng relies on can be installed via the `syslog-ng-update-virtualenv` script, which has been added to our rpm/deb postinst scripts. Our mod-python module validates this virtualenv at startup and activates it automatically if the validation is successful. You can disable this behaviour by loading the Python module explicitly with the following configuration statement: @module mod-python use-virtualenv(no) You can force syslog-ng to use a specific virtualenv by activating it first, prior to executing syslog-ng. In this case, syslog-ng will not try to use its private virtualenv, rather it would use the one activated when it was started. It assumes that any requirements needed for syslog-ng functionality implemented in Python are deployed by the user. These requirements are listed in the /usr/lib/syslog-ng/python/requirements.txt file. * SCL snippets in Python plugins: by adding an `scl/whatever.conf` file to your Python-based syslog-ng plugin, you can easily wrap a Python-based log processing functionality with a syslog-ng block {}, so the user can use a syntax very similar to native plugins in their main configuration. * `confgen` in Python: should a simple block {} statement not be enough to wrap the functionality implemented in Python, the mod-python module now supports confgen functions to be implemented in Python. confgen has been a feature in syslog-ng for a long time that allows you to generate configuration snippets dynamically by executing an external program or script. This has now been ported to Python, e.g. syslog-ng can invoke a Python function to generate parts of its configuration. Example: ``` @version: 4.0 python { from syslogng import register_config_generator def generate_foobar(args): print(args) return "tcp(port(2000))" # # this registers a plugin in the "source" context named "foobar" # which would invoke the generate_foobar() function when a foobar() source # reference is encountered. # register_config_generator("source", "foobar", generate_foobar) }; log { # we are actually calling the generate_foobar() function in this # source, passing all parameters as values in the "args" dictionary source { foobar(this(is) a(value)); }; destination { file("logfile"); }; }; ``` ## Features * `kubernetes()` source and `kubernetes-metadata-parser()`: these two components gained the ability to enrich log messages with Kubernetes metadata. When reading container logs, syslog-ng would query the Kubernetes API for the following fields and add them to the log-message. The returned meta-data is cached in memory, so not all log messages trigger a new query. .k8s.pod_uuid .k8s.labels.<label_name> .k8s.annotations.<annotation_name> .k8s.namespace_name .k8s.pod_name .k8s.container_name .k8s.container_image .k8s.container_hash .k8s.docker_id * `java()` destinations: fixed compatibility with newer Java versions, syslog-ng is now able to compile up to Java 18. * `disk-buffer`: Added `prealloc()` option to preallocate new disk-buffer files. ([#4056](https://github.com/syslog-ng/syslog-ng/pull/4056)) * `disk-buffer`: The default value of `truncate-size-ratio()` has been changed to 1, which means truncation is disabled by default. This means that by default, the disk-buffer files will gradually become larger and will never reduce in size. This improves performance. ([#4056](https://github.com/syslog-ng/syslog-ng/pull/4056)) * `log-level()`: added a new global option to control syslog-ng's own internal log level. This augments the existing support for doing the same via the command line (via -d, -v and -t options) and via syslog-ng-ctl. This change also causes higher log-levels to include messages from lower log-levels, e.g. "trace" also implies "debug" and "verbose". By adding this capability to the configuration, it becomes easier to control logging in containerized environments where changing command line options is more challenging. `syslog-ng-ctl log-level`: this new subcommand in syslog-ng-ctl allows setting the log level in a more intuitive way, compared to the existing `syslog-ng-ctl verbose|debug|trace -s` syntax. `syslog-ng --log-level`: this new command line option for the syslog-ng main binary allows you to set the desired log-level similar to how you can control it from the configuration or through `syslog-ng-ctl`. ([#4091](https://github.com/syslog-ng/syslog-ng/pull/4091)) * `network`/`syslog`/`tls` context options: SSL_CONF_cmd support SSL_CONF_cmd TLS configuration support for `network()` and `syslog()` driver has been added. OpenSSL offers an alternative, software-independent configuration mechanism through the SSL_CONF_cmd interface to support a common solution for setting the so many various SSL_CTX and SSL options that can be set earlier via multiple, separated openssl function calls only. This update implements that similar to the mod_ssl in Apache. IMPORTANT: The newly introduced `openssl-conf-cmds` always has the highest priority, its content parsed last, so it will override any other options that can be found in the `tls()` section, does not matter if they appear before or after `openssl-conf-cmds`. As described in the SSL_CONF_cmd documentation, the order of operations within openssl-conf-cmds() is significant and the commands are executed in top-down order. This means that if there are multiple occurrences of setting the same option then the 'last wins'. This is also true for options that can be set multiple ways (e.g. used cipher suites and/or protocols). Example config: ``` source source_name { network ( ip(0.0.0.0) port(6666) transport("tls") tls( ca-dir("/etc/ca.d") key-file("/etc/cert.d/serverkey.pem") cert-file("/etc/cert.d/servercert.pem") peer-verify(yes) openssl-conf-cmds( # For system wide available cipher suites use: /usr/bin/openssl ciphers -v # For formatting rules see: https://www.openssl.org/docs/man1.1.1/man3/SSL_CONF_cmd.html # For quick and dirty testing try: https://github.com/rbsec/sslscan # "CipherString" => "ECDHE-RSA-AES128-SHA", # TLSv1.2 and bellow "CipherSuites" => "TLS_CHACHA20_POLY1305_SHA256:TLS_AES_256_GCM_SHA384", # TLSv1.3+ (OpenSSl 1.1.1+) "Options" => "PrioritizeChaCha", "Protocol" => "-ALL,TLSv1.3", ) ) ); }; ``` * `network`/`syslog`/`http` destination: OCSP stapling support OCSP stapling support for network destinations and for the `http()` module has been added. When OCSP stapling verification is enabled, the server will be requested to send back OCSP status responses. This status response will be verified using the trust store configured by the user (`ca-file()`, `ca-dir()`, `pkcs12-file()`). Note: RFC 6961 multi-stapling and TLS 1.3-provided multiple responses are currently not validated, only the peer certificate is verified. Example config: ``` destination { network("test.tld" transport(tls) tls( pkcs12-file("/path/to/test.p12") peer-verify(yes) ocsp-stapling-verify(yes) ) ); http(url("https://test.tld") method("POST") tls(peer-verify(yes) ocsp-stapling-verify(yes))); }; ``` ([#4082](https://github.com/syslog-ng/syslog-ng/pull/4082)) * Python `LogMessage` class: get_pri() and get_timestamp() methods were added that allow the query of the syslog-style priority and the message timestamp, respectively. The return value of get_pri() is an integer, while get_timestamp() returns a Python datetime.datetime instance. Some macros that were previously unavailable from Python (e.g. the STAMP, R_STAMP and C_STAMP macros) are now made available. * Python `Logger`: the low-level Logger class exported by syslog-ng was wrapped by a logging.LogHandler class so that normal Python APIs for logging can now be used. * `db-parser()` and `grouping-by()`: added a `prefix()` option to both `db-parser()` and `grouping-by()` that allows specifying an extra prefix to be prepended to all name-value pairs that get extracted from messages using patterns or <value> tags. * `csv-parser()`: add a new dialect, called escape-backslash-with-sequences which uses "\" as an escape character but also supports C-style escape sequences, like "\n" or "\r". ## Bugfixes * `tcp()`, `network()` or `syslog()` destinations: fixed a crash that could happen after reload when a kept-alive connection is terminated, in case the target server is configured using a hostname (and not an IP address) and that name becomes unresolvable (e.g. dropped from DNS or /etc/hosts) ([#4044](https://github.com/syslog-ng/syslog-ng/pull/4044)) * `python()` destination: Fixed a crash, when trying to resolve the "R_STAMP", "P_STAMP" or "STAMP" macros from Python code. ([#4057](https://github.com/syslog-ng/syslog-ng/pull/4057)) * Python `LogSource` & `LogFetcher`: a potential deadlock was fixed in acknowledgement tracking. * Python `LogTemplate`: the use of template functions in templates instantiated from Python caused a crash, which has been fixed. * `grouping-by()` persist-name() option: fixed a segmentation fault in the grammar. ([#4180](https://github.com/syslog-ng/syslog-ng/pull/4180)) * `$(format-json)`: fix a bug in the --key-delimiter option introduced in 3.38, which causes the generated JSON to contain multiple values for the same key in case the key in question contains a nested object and key-delimiter specified is not the dot character. ([#4127](https://github.com/syslog-ng/syslog-ng/pull/4127)) * `add-contextual-data()`: add compatibility warnings and update advise in case of the value field of the add-contextual-data() database contains an expression that resembles the new type-hinting syntax: type(value). * `syslog-ng --help` screen: the output for the --help command line option has included sample paths to various files that contained autoconf style directory references (e.g. ${prefix}/etc for instance). This is now fixed, these paths will contain the expanded path. Fixes Debian Bug report #962839: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=962839 ([#4143](https://github.com/syslog-ng/syslog-ng/pull/4143)) * `csv-parser()`: fixed the processing of the dialect() parameter, which was not taken into consideration. * `apache-accesslog-parser()`: Apache may use backslash-style escapes in the `request` field, so support it by setting the csv-parser() dialect to `escape-backslash-with-sequences`. Also added validation that the `rawrequest` field contains a valid HTTP request and only extract `verb`, `request` and `httpversion` if this is the case. * `riemann`: fixed severity levels of Riemann diagnostics messages, the error returned by riemann_communicate() was previously only logged at the trace level and was even incomplete: not covering the case where riemann_communicate() returns NULL. ([#4238](https://github.com/syslog-ng/syslog-ng/pull/4238)) ## Packaging * `python`: python2 support is now completely removed. `syslog-ng` can no longer be configured with `--with-python=2`. ([#4057](https://github.com/syslog-ng/syslog-ng/pull/4057)) * `python`: Python 2 support is now completely removed from the syslog-ng functional test framework, called Light, too. Light will support only Python 3 from now. ([#4174](https://github.com/syslog-ng/syslog-ng/pull/4174)) * Python virtualenv support for development use: syslog-ng is now capable of using a build-time virtualenv, where all Python development tools are automatically deployed by the build system. You can control if you want to use this using the --with-python-packages configure option. There are three possible values for this parameter: * `venv`: denoting that you want to use the virtualenv and install all these requirements automatically using pip, into the venv. * `system`: meaning that you want to rely on the system Python without using a virtualenv. syslog-ng build scripts would install requirements automatically to the system Python path usually `/usr/local/lib/pythonX.Y` * `none`: disable deploying packages automatically. All dependencies are assumed to be present in the system Python before running the syslog-ng build process. Please note that syslog-ng has acquired quite a number of these development time dependencies with the growing number of functionality the Python binding offers, so using the `system` or `none` settings are considered advanced usage, meant to be used for distro packaging. * `make dist`: fixed make dist of FreeBSD so that source tarballs can easily be produced even if running on FreeBSD. ([#4163](https://github.com/syslog-ng/syslog-ng/pull/4163)) * Debian and derivatives: The `syslog-ng-mod-python` package is now built with `python3` on the following platforms: * `debian-stretch` * `debian-buster` * `ubuntu-bionic` ([#4057](https://github.com/syslog-ng/syslog-ng/pull/4057)) * `dbld`: Removed support for `ubuntu-xenial`. ([#4057](https://github.com/syslog-ng/syslog-ng/pull/4057)) * `dbld`: Updated support from Fedora 35 to Fedora 37 * Leaner production docker image: the balabit/syslog-ng docker image stops pulling in logrotate and its dependencies into the image. logrotate recursively pulled in cron and exim4 which are inoperable within the image anyway and causes the image to be larger as well as increasing the potential attack surface. * Debian packaging: logrotate became Suggested instead of Recommended to avoid installing logrotate by default. * `scl`: To match the way `scl`s are packaged in debian, we have added a `syslog-ng-scl` package. This makes it possible to upgrade from the official debian `syslog-ng` package to the ose-repo provided one. ([#4252](https://github.com/syslog-ng/syslog-ng/pull/4252)) ([#4256](https://github.com/syslog-ng/syslog-ng/pull/4256)) ## Other changes * `sumologic-http()` improvements Improved defaults: `sumologic-http()` originally sent incomplete messages (only the `$MESSAGE` part) to Sumo Logic by default. The new default is a JSON object, containing all name-value pairs. This is a breaking change if you used the default value as it was, but this is not really anticipated. To override the new message format or revert to the old default, the `template()` option can be used. `sumologic-http()` enables batching by default to significantly increase the destination's performance. The `tls()` block has become optional, Sumo Logic servers will be verified using the system's certificate store by default. ([#4124](https://github.com/syslog-ng/syslog-ng/pull/4124)) ### Installation packages * Debian/Ubuntu Packages https://github.com/syslog-ng/syslog-ng#debianubuntu * RedHat, CentOS and Fedora Packages https://www.syslog-ng.com/community/b/blog/posts/rpm-packages-from-syslog-n… * docker image: Nightly production docker images are now available as `balabit/syslog-ng:nightly` ([#4117](https://github.com/syslog-ng/syslog-ng/pull/4117)) * docker image: added jemalloc to the production image, which improves performance, decreases memory fragmentation and makes syslog-ng to return memory to the system much more aggressively. * Removed support for Debian stretch. ## Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Attila Szakacs, Attila Szalay, Balazs Scheidler, Bálint Horváth, Gabor Nagy, István Hoffmann, Joshua Root, László Várady, Szilárd Parrag

1 0

Two questions about syslog-ng
by Mik J 08 Dec '22

08 Dec '22

Hello, I would like to know if syslog-ng is able to collect netflow and/or sflow flows. Logstash used to do it through the netflow module. Is it possible to start a script when receiving a specific syslog ? Log received Dec 7 22:36:10 myserver sshd[46926]: somemessage from 192.168.2.201 port 59489 ssh2 Action /somewhere/script_ban_ip 192.168.2.201 Thank you

1 0

Match logs with variable lengths
by Mik J 07 Dec '22

07 Dec '22

Hello, I'm new to syslog-ng and I'm trying to match my DNS logs. I have either Dec 6 01:44:49 mydnsserver named[66669]: client @0x53f33c122d0 1.2.3.4#56610 (my.original.query.com) query: my.qname.com IN A -E(0)DC (1.1.1.1) or Dec 6 01:44:49 mydnsserver named[66669]: client @0x53f33c122d0 1.2.3.4#56610 (my.original.query.com) query: my.qname.com IN A -E(0)DC (1.1.1.1) [ECS 192.168.0.0/24/0] So the field [ECS 192.168.0.0/24/0] is optional and either exist or not To match fields I use this pattern <pattern>client @@@ESTRING:dns_clientid: @@IPvANY:src_ip@#@NUMBER:src_port@ (@ESTRING:dns_original_query:)@: query: @ESTRING:dns_qname: @IN @ESTRING:dns_type: @@ESTRING:dns_record_flags: @(@IPvANY:dns_server@)</pattern> and it matches both lines for the existing fields But how can I match this optional part at the end of the line: [ECS 192.168.0.0/24/0] Should I use @PCRE@ ? Or is it possible with other pattern parsers If yes why use other pattern parsers and not build the matching around @PCRE@ for the entire line which would make think simple to translate from grok or other tools. Thank you

2 2

syslog-ng-3.38.1 causing type hint errors in patterndb - bug?
by Evan Rempel 22 Nov '22

22 Nov '22

We have a patterndb file that contains both patterns and values with fixed text that include a strings of the form "xxxx (yyy)" which now can not be loaded by the patterndb. Syslog-ng throws the error Error parsing pattern database file; ... Error compiling value template, rule=FLARE-3543, name=AUTHPROGRAM, value=su(pam_unix), error=Unknown type specified in type hinting: su' I have not enabled the version 4.0 testing of type hinting. The version configuration is @version: 3.36 I am unable to find documentation for the full syntax of the patterndb file (seems to have been lost from the docs since perhaps 3.16?) Is there a patterndb syntax specification document? I would expect that even when type hinting is enabled in the patterndb file, given that it is an XML document, that the type hinting would be part of the XML tag metadata, rather than part of the XML static data. -- Evan

2 6

https://www.syslog-ng.com/community/b/blog/posts/installing-the-latest-syslog-ng-on-ubuntu-and-other-deb-distributions
by Burgett, Michael 18 Nov '22

18 Nov '22

Hello, I am attempting to follow the instructions at this blog link, but get an error trying to download the public key: # wget -qO - https://ose-repo.syslog-ng.com/apt/syslog-ng-ose-pub.asc | sudo apt-key add - gpg: no valid OpenPGP data found. I've tried searching later blog post, without success. I'd like to try a later version (3.25.1-3 from ubuntu-20.04 packages) since we seem to randomly have this version hang, which eventually seems to block journald as well. Thanks, Mike Burgett

2 2

Insider 2022-11: 4.0; OIDC; nightly; sudo;
by Peter Czanik (pczanik) 17 Nov '22

17 Nov '22

Dear syslog-ng users, This is the 106th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Testing syslog-ng 4.0 --------------------- This syslog-ng blog does not demonstrate any new syslog-ng features or integrations. Instead of those, it is a request for testing syslog-ng 4.0 and points you at a few related blogs and resources. https://www.syslog-ng.com/community/b/blog/posts/testing-syslog-ng-4-0 syslog-ng Store Box federated single sign-on support via OpenID Connect -------------------------------------------------------- The syslog-ng Store Box (SSB) appliance is built upon syslog-ng Premium Edition (PE). SSB inherits most of syslog-ng PE’s features and makes them available with an easy-to-use graphical user interface. There are multiple ways how users can authenticate when using SSB. Recent versions also introduced federated single sign-on (SSO) via OpenID Connect (OIDC). https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-store-box-federa… Nightly syslog-ng container images ---------------------------------- The syslog-ng team started publishing container images many years ago. For quite a while, it was a manual process, however, a few releases ago, publishing a container image became part of the release process. Recently, nightly container images have also become available, so you can test the latest features and bug fixes easily. https://www.syslog-ng.com/community/b/blog/posts/nightly-syslog-ng-containe… Type support: working with sudo logs in syslog-ng 4.0 ----------------------------------------------------- Last week I gave you a quick introduction to a major syslog-ng 4.0 feature: type support. I mentioned that it also works nicely for JSON-formatted sudo logs. I have been asked to share a working syslog-ng configuration. From this blog, you can learn how to turn on JSON-formatted logging in sudo, and how to work with those logs in syslog-ng. https://www.syslog-ng.com/community/b/blog/posts/type-support-working-with-… WEBINARS * Register for our latest webinars at https://www.syslog-ng.com/event/syslogng-webcast-series-2022/ * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Syslog-ng Message Format
by Maurya, Shivani 07 Nov '22

07 Nov '22

Hi All, With the older Syslog-ng RFC format (rfc 3164), the devices used to send the syslog messages in the below format - %TAG TIMESTAMP FULLHOSTFROM SEVERITY MSG Example - %SYSMGR-2-NON_VOLATILE_DB_INODE_FULL:2022 Nov 7 03:54:30 MST: SYSMGR-2-NON_VOLATILE_DB_INODE_FULL: System non-volatile inode storage usage is unexpectedly high at 96 But with the new syslog-ng RFC format (rfc 5425), the devices send the syslog messages in below format - TIMESTAMP FULLHOSTFROM SEVERITY MSG Is there a way to update/format the messages (rfc 5424) to be pre-fixed with %TAG ? Regards, Shivani Maurya

2 3

Insider 2022-10: type support; Splunk/Sentinel; MongoDB; Photon;
by Peter Czanik (pczanik) 20 Oct '22

20 Oct '22

Dear syslog-ng users, This is the 105th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Type support: getting started with syslog-ng 4.0 ------------------------------------------------ Version 4.0 of syslog-ng is right around the corner. It hasn’tyet been released; however, you can already try some of its features. The largest and most interesting change is type support. Right now, name-value pairs within syslog-ng are represented as text, even if the PatternDB or JSON parsers could see the actual type of the incoming data. This does not change, but starting with 4.0, syslog-ng will keep the type information, and use it correctly on the destination side. This makes your life easier, for example when you store numbers to Elasticsearch or to other type-aware storage. >From this blog, you can learn how type support makes your life easier and helps you to give it a testdrive on your own hosts. https://www.syslog-ng.com/community/b/blog/posts/type-support-getting-start… syslog-ng Store Box Splunk/HEC and Sentinel destinations -------------------------------------------------------- The syslog-ng Store Box (SSB) appliance is built on syslog-ng Premium Edition (PE). SSB inherits most of syslog-ng PE’s features and makes them available with an easy-to-use graphical user interface. One of the typical use cases for SSB (and syslog-ng PE) is optimizing the logging infrastructure for SIEM / log analysis. Two recently introduced SSB destinations for log analytics are Splunk HEC (HTTP Event Collector) and Microsoft Sentinel. https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-store-box-splunk… A quick test of MongoDB 6.0 with syslog-ng ------------------------------------------ Any time I see that one of the syslog-ng destinations has a major new version, I'm a bit scared, as it is not uncommon to introduce breaking changes with them. MongoDB 6.0, however, was a pleasant surprise. I gave it a quick try, and everything worked as expected. Along the way, I even learned about MongoDB Compass, an easy-to-use GUI for MongoDB databases. https://www.syslog-ng.com/community/b/blog/posts/a-quick-test-of-mongodb-6-… Vmware Photon OS 4.0: an interesting syslog-ng package ------------------------------------------------------ A few weeks ago I wrote about syslog-ng in Microsoft’s own Linux distribution, CBL-Mariner. Vmware Photon OS 4.0 is another niche Linux distribution. It not only features syslog-ng, but syslog-ng is also up-to-date. The syslog-ng package in Photon, though, is also the strangest one I have ever encountered. https://www.syslog-ng.com/community/b/blog/posts/vmware-photon-os-4-0-an-in… WEBINARS * Register for our latest webinars at https://www.syslog-ng.com/event/syslogng-webcast-series-2022/ * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

syslog-ng version
by Mario Benoun 11 Oct '22

11 Oct '22

Please provide steps to update to the latest version running on Ubuntu 22.04.1 LTS Thanks! Mario Benoun Field Application Engineer CCNA, CWNA, CWSP, MCTS Moxa Americas, Inc. Western Region USA 714.577.6225<tel:714.577.6225> mario.benoun(a)moxa.com<mailto:mario.benoun@moxa.com>

2 1

UDP drops with syslog-ng 3.24.1-1
by brian hoffman 05 Oct '22

05 Oct '22

We've been centrally logging with syslog-ng for about 5 years now. Over that time, the number of sources has grown significantly, and at some point we crossed a line where drops were happening (a quick survey of 3 million syslog packets yielded 420 unique currently sending hosts). After much research and experimentation, we've been able to get to the point where throughout the day there are 0 drops for the most part. This was achieved by installing the latest syslog-ng (not the RedHat packaged one) and creating a source for each CPU. Occasionally, though, we still have periods of drops so I'm trying to eliminate these last few. Here are some relevant configuration items:2 RedHat 7.8 VMs (load balanced via an F5) with 16GB memory and 4 CPUs each running syslog-ng-3.24.1-1.el7.x86_64.net.core.rmem_default = 212992net.core.rmem_max = 268435456 log_fifo_size(268435456); source s_network {network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp1"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp2"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp3"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp4"));network(ip("0.0.0.0") port(514) transport("tcp") max_connections(200) keep_alive(yes) so_rcvbuf(67108864));}; We are limited to UDP, unfortunately, because we do not have control over the devices/networks/etc. that are sending to us, but we have changed as many of the internal senders and destinations to TCP as we can. With a script I created to view the packets, including drops, as well as the individual RECVQs, the issue can be illustrated. Here's what things look like normally:Thu May 7 10:48:15 EDT 2020, 27003 IP pkts rcvd,26980 IP pkts sent,24951 UDP pkts rcvd, 28075 UDP pkts sent,0 UDP pkt rcv err RECVQ-1=2176RECVQ-2=0RECVQ-3=0RECVQ-4=0Thu May 7 10:48:16 EDT 2020, 28453 IP pkts rcvd,28426 IP pkts sent,26185 UDP pkts rcvd, 29180 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=0RECVQ-2=0RECVQ-3=4352RECVQ-4=0Thu May 7 10:48:17 EDT 2020, 28294 IP pkts rcvd,28276 IP pkts sent,26277 UDP pkts rcvd, 28709 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=2176RECVQ-2=0RECVQ-3=0RECVQ-4=0 The RECVQs are sparsely used, and there are no errors. Around 9pm every night, the packet counts go up significantly (probably due to backup related logs):Wed May 6 21:00:08 EDT 2020, 66382 IP pkts rcvd,66366 IP pkts sent,39405 UDP pkts rcvd, 67592 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=1595008RECVQ-2=106217088RECVQ-3=53694976RECVQ-4=31858816Wed May 6 21:00:09 EDT 2020, 69317 IP pkts rcvd,69338 IP pkts sent,44446 UDP pkts rcvd, 75958 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=13056RECVQ-2=126397312RECVQ-3=75568128RECVQ-4=41626880Wed May 6 21:00:10 EDT 2020, 71205 IP pkts rcvd,71227 IP pkts sent,43657 UDP pkts rcvd, 74603 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=920448RECVQ-2=146122752RECVQ-3=100951168RECVQ-4=52622208Wed May 6 21:00:12 EDT 2020, 69578 IP pkts rcvd,69454 IP pkts sent,124465 UDP pkts rcvd, 163367 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=13140864RECVQ-2=44494848RECVQ-3=125579136RECVQ-4=0 Still, though, it's handling it with no errors. But then at some point a threshold is reached and errors start piling up:Wed May 6 21:00:20 EDT 2020, 63177 IP pkts rcvd,63291 IP pkts sent,0 UDP pkts rcvd, 0 UDP pkts sent,38011 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=200357376RECVQ-3=292948352RECVQ-4=28890752Wed May 6 21:00:21 EDT 2020, 69501 IP pkts rcvd,69464 IP pkts sent,0 UDP pkts rcvd, 1 UDP pkts sent,42158 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=223551360RECVQ-3=314995584RECVQ-4=41735680Wed May 6 21:00:23 EDT 2020, 69962 IP pkts rcvd,69978 IP pkts sent,0 UDP pkts rcvd, 2 UDP pkts sent,43775 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=244732544RECVQ-3=338239616RECVQ-4=53858176Wed May 6 21:00:24 EDT 2020, 68266 IP pkts rcvd,68216 IP pkts sent,0 UDP pkts rcvd, 0 UDP pkts sent,43118 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=265258752RECVQ-3=360643712RECVQ-4=65362688 The common denominator I've found is that one of the RECVQs hits 536,871,424. This number seems to be almost exactly double (512 difference) the rmem.max/log_fifo_size (268,435,456). Even though there seems to be capacity in the other RECVQs, just one of those hitting that magic number seems to be enough to throw things out of whack. During this time, the CPU usage for syslog-ng also drops: top - 21:00:11 up 2 days, 8:40, 2 users, load average: 1.09, 1.41, 1.32 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 123.6 10.9 3080:46 syslog-ng top - 21:00:14 up 2 days, 8:40, 2 users, load average: 1.00, 1.39, 1.31 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 24.6 10.9 3080:46 syslog-ng top - 21:00:17 up 2 days, 8:40, 2 users, load average: 1.00, 1.39, 1.31 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 7.6 10.9 3080:47 syslog-ng As best I can tell, based on the reading I've done, the message counts we're getting should be doable, but it's still not clear exactly how to size some of these options as they relate to UDP (like log_fetch_limit, which we do not have set). Anything else I should try?

2 4

Sending json logs with syslog-ng to Splunk via HEC
by Carlos Lopez 28 Sep '22

28 Sep '22

Hi all, I am trying to configure syslog-ng to send events in json format to our internal Splunk (free version) server. Searching info regarding how to accomplish this config, I see these entries in Balabit’s blog: https://www.syslog-ng.com/community/b/blog/posts/optimize-your-splunk-infra… https://www.syslog-ng.com/community/b/blog/posts/sending-logs-splunk-http Are these entries accurate to accomplish this config? Somebody can share any sample? I am using syslog-ng’ OSS under FreeBSD 13.1 hosts … Best regards, C. L. Martinez

1 0

Insider 2022-09: 3.38; SQL; disk-buffer; nightly;
by Peter Czanik (pczanik) 08 Sep '22

08 Sep '22

Dear syslog-ng users, This is the 104th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS 3.38.1 released, 4.0 almost feature complete -------------------------------------------- syslog-ng 4 is right around the corner. Instead of a pile of breaking changes, we choose to improve syslog-ng in an evolutionary manner: providing fine grained compatibility with older versions along the way, so that syslog-ng 4 remains a drop-in replacement for any earlier release in the past 15 years. You can test many 4.0 features already in the fresh 3.38.1 release: https://syslog-ng-future.blog/syslog-ng-4-progress-3-38-1-release/ syslog-ng Store Box SQL source ------------------------------ The syslog-ng Store Box (SSB) is an appliance built on syslog-ng Premium Edition (PE). SSB inherits most of syslog-ng PE’s features and makes them available with an easy-to-use graphical user interface. One of these features is the SQL source. Many applications log their messages to SQL databases. The SQL source of syslog-ng PE and SSB allows you to collect these log messages in real-time and process and store them with the rest of your logs. https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-store-box-sql-so… Why is my syslog-ng disk-buffer file so huge even when it is empty? --------------------------------------- Most people expect to see how many log messages are waiting in the disk-buffer from the size of the syslog-ng disk-buffer file.. While it was mostly true for earlier syslog-ng releases, for recent syslog-ng releases (3.34+) the disk-buffer file can stay large even when it is empty. This is a side effect of a recent syslog-ng performance tuning. https://www.syslog-ng.com/community/b/blog/posts/why-is-my-syslog-ng-disk-b… Nightly syslog-ng builds for Debian and Ubuntu ---------------------------------------------- The syslog-ng team made nightly builds available for Debian and Ubuntu running on the AMD64 architecture. These packages are not intended to be used in production, they are only for testing fixes and new features. https://www.syslog-ng.com/community/b/blog/posts/nightly-syslog-ng-builds-f… WEBINARS * Register for our latest webinars at https://www.syslog-ng.com/event/syslogng-webcast-series-2022/ * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

syslog-ng-3.38.1
by Laszlo Varady (lvarady) 29 Aug '22

29 Aug '22

3.38.1 <https://github.com/syslog-ng/syslog-ng/blob/master/NEWS.md#highlights> Highlights <https://github.com/syslog-ng/syslog-ng/blob/master/NEWS.md#sneak-peek-into-…> Sneak peek into syslog-ng v4.0 syslog-ng v4.0 is right around the corner. This release (v3.38.1) contains all major changes, however, they are currently all hidden behind a feature flag. To enable and try those features, you need to specify @version: 4.0 at the top of the configuration file. You can find out more about the 4.0 changes and features here<https://github.com/syslog-ng/syslog-ng/blob/master/NEWS-4.0.md>. Read our practical introduction to typing at syslog-ng-future.blog<https://syslog-ng-future.blog/syslog-ng-4-progress-3-38-1-release/>. <https://github.com/syslog-ng/syslog-ng/blob/master/NEWS.md#features> Features * grouping-by(): added inject-mode(aggregate-only) This inject mode will drop individual messages that make up the correlation context (key() groups) and would only yield the aggregate messages (e.g. the results of the correlation). (#3998<https://github.com/syslog-ng/syslog-ng/pull/3998>) * add-contextual-data(): add support for type propagation, e.g. set the type of name-value pairs as they are created/updated to the value returned by the template expression that we use to set the value. The 3rd column in the CSV file (e.g. the template expression) now supports specifying a type-hint, in the format of "type-hint(template-expr)". Example line in the CSV database: selector-value,name-value-pair-to-be-created,list(foo,bar,baz) (#4051<https://github.com/syslog-ng/syslog-ng/pull/4051>) * $(format-json): add --key-delimiter option to reconstruct JSON objects using an alternative structure separator, that was created using the key-delimiter() option of json-parser(). (#4093<https://github.com/syslog-ng/syslog-ng/pull/4093>) * json-parser(): add key-delimiter() option to extract JSON structure members into name-value pairs, so that the names are flattened using the character specified, instead of dot. Example: Input: {"foo":{"key":"value"}} Using json-parser() without key-delimiter() this is extracted to: foo.key="value" Using json-parser(key-delimiter("~")) this is extracted to: foo~key="value" This feature is useful in case the JSON keys contain dots themselves, in those cases the syslog-ng representation is ambigious. (#4093<https://github.com/syslog-ng/syslog-ng/pull/4093>) <https://github.com/syslog-ng/syslog-ng/blob/master/NEWS.md#bugfixes> Bugfixes * Fixed buffer handling of syslog and timestamp parsers Multiple buffer out-of-bounds issues have been fixed, which could cause hangs, high CPU usage, or other undefined behavior. (#4110<https://github.com/syslog-ng/syslog-ng/pull/4110>) * Fixed building with LibreSSL (#4081<https://github.com/syslog-ng/syslog-ng/pull/4081>) * network(): Fixed a bug, where syslog-ng halted the input instead of skipping a character in case of a character conversion error. (#4084<https://github.com/syslog-ng/syslog-ng/pull/4084>) * redis(): Fixed bug where using redis driver without the batch-lines option caused program crash. (#4114<https://github.com/syslog-ng/syslog-ng/pull/4114>) * pdbtool: fix a SIGABRT on FreeBSD that was triggered right before pdbtool exits. Apart from being an ugly crash that produces a core file, functionally the tool behaved correctly and this case does not affect syslog-ng itself. (#4037<https://github.com/syslog-ng/syslog-ng/pull/4037>) * regexp-parser(): due to a change introduced in 3.37, named capture groups are stored indirectly in the LogMessage to avoid copying of the value. In this case the name-value pair created with the regexp is only stored as a reference (name + length of the original value), which improves performance and makes such name-value pairs use less memory. One omission in the original change in 3.37 is that syslog-ng does not allow builtin values to be stored indirectly (e.g. $MESSAGE and a few of others) and this case causes an assertion to fail and syslog-ng to crash with a SIGABRT. This abort is now fixed. Here's a sample config that reproduces the issue: regexp-parser(patterns('(?<MESSAGE>.*)')); (#4043<https://github.com/syslog-ng/syslog-ng/pull/4043>) * set-tag: fix cloning issue when string literal were used (see #4062) (#4065<https://github.com/syslog-ng/syslog-ng/pull/4065>) * add-contextual-data(): fix high memory usage when using large CSV files (#4067<https://github.com/syslog-ng/syslog-ng/pull/4067>) <https://github.com/syslog-ng/syslog-ng/blob/master/NEWS.md#other-changes> Other changes * The json-c library is no longer bundled in the syslog-ng source tarball Since all known OS package managers provide json-c packages nowadays, the json-c submodule has been removed from the source tarball. The --with-jsonc=internal option of the configure script has been removed accordingly, system libraries will be used instead. For special cases, the JSON support can be disabled by specifying --with-jsonc=no. (#4078<https://github.com/syslog-ng/syslog-ng/pull/4078>) * platforms: Dropped support for ubuntu-impish as it became EOL (#4088<https://github.com/syslog-ng/syslog-ng/pull/4088>) <https://github.com/syslog-ng/syslog-ng/blob/master/NEWS.md#credits> Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Alvin Šipraga, Andras Mitzki, Attila Szakacs, Balazs Scheidler, Bálint Horváth, Daniel Klauer, Fabrice Fontaine, Gabor Nagy, HenryTheSir, László Várady, Parrag Szilárd, Peter Kokai, Shikhar Vashistha, Szilárd Parrag, Vivin Peris Installation from binaries: https://github.com/syslog-ng/syslog-ng#installation-from-binaries<https://github.com/syslog-ng/syslog-ng#installation-from-binaries<https://github.com/syslog-ng/syslog-ng#installation-from-binaries>

1 0

Potential issue with line breaking not working correctly when writing logs to disk
by John Law 23 Aug '22

23 Aug '22

Hi, We have a number of network devices sending syslog traffic to syslog-ng OSE 3.25 installed on RHEL 7.9. We are having an issue where multiple events are being written as the same line. The config file is template-function adm1 "${HOST} message-length=$(length \"${MSG}\") ${MESSAGE}\n"; source s_adm1 { udp(ip(0.0.0.0) port(5527) ); tcp(ip(0.0.0.0) port(5527) ); }; destination d_adm1 { file( "/Data/syslog_data/$LOGHOST/$R_YEAR-$R_MONTH-$R_DAY/adm1/$HOST_FROM/$HOST/$FACILITY.local" template("$(adm1-function)\n") ); }; log { source(s_adm1); destination(d_adm1); flags(final); }; A small extract of one of the files is, naturally I have changed the IPs 192.1.1.1/31181->192.1.3.1/135 0x0 source rule r12 N/A N/A 6 ACME-CNB010 ACME CNB 1727847 N/A(N/A) reth3.860 UNKNOWN UNKNOWN UNKNOWN275 <14>Aug 19 11:38:50 sunny-fwl29 RT_FLOW: RT_FLOW_SESSION_CREATE: session created 192.6.20.1/47250->192.104.20.1/443 0x0 junos-https 192.6.20.1/47250->192.104.20.1/443 0x0 N/A N/A N/A N/A 6 ACME-BWC002 ACME BWC_EDESIX_AWS 917879 N/A(N/A) reth3.860 UNKNOWN UNKNOWN UNKNOWN275 <14>Aug 19 11:38:50 sunny-fwl29 RT_FLOW: RT_FLOW_SESSION_CREATE: session created 192.6.20.1/47252->192.104.20.1/443 0x0 junos-https 192.6.20.1/47252->192.104.20.1/443 0x0 N/A N/A N/A N/A 6 ACME-BWC002 ACME BWC_EDESIX_AWS 595069 N/A(N/A) reth3.860 UNKNOWN UNKNOWN UNKNOWN285 <14>Aug 19 11:38:50 sunny-fwl29 RT_FLOW: RT_FLOW_SESSION_CREATE: session created 192.1.57.1/54205->192.1.7.1/137 0x0 junos-nbname 192.1.57.1/54205->192.1.7.1/137 0x0 N/A N/A N/A N/A 17 ACME-COMP007 ACME CCA 1735324 N/A(N/A) reth3.860 UNKNOWN UNKNOWN UNKNOWN288 <14>Aug 19 11:38:50 sunny-fwl29 RT_FLOW: RT_FLOW_SESSION_CREATE: session created 192.1.57.1/62486->192.1.7.1/135 0x0 junos-ms-rpc-tcp 192.1.57.1/62486->192.1.7.1/135 0x0 N/A N/A N/A N/A 6 ACME-COMP007 ACME CCA 1027474 N/A(N/A) reth3.860 UNKNOWN UNKNOWN UNKNOWN249 I suspect it might be something to do with the amount of logs that are received in very quick succession, but I have no idea how to address this. One thing I have noticed is that it always appears to be the same device that is causing the issue. When I look at other devices coming in on the same port, they appear to write out correctly. Any suggestion would be really welcome. Thanks John This message has been scanned for malware by Forcepoint. www.forcepoint.com

2 4

Multiple timezone seen on messages originating from local machine
by Francois Marier 16 Aug '22

16 Aug '22

I'm trying to understand timezones in syslog-ng, but I think there's something I'm missing. If I do the following on my OpenWRT machine: /etc/init.d/syslog-ng restart logger TestA I see the following in /var/log/messages: Aug 14 20:39:35 hostname syslog-ng[9860]: syslog-ng shutting down; version='3.37.1' Aug 14 20:39:36 hostname syslog-ng[10024]: syslog-ng starting up; version='3.37.1' Aug 14 20:39:36 hostname syslog-ng[10024]: Syslog connection established; fd='15', server='AF_INET(192.168.1.10:514)', local='AF_INET(0.0.0.0:0)' Aug 15 03:39:49 hostname root: TestA The correct timezone is the one in the first three lines. Other daemon messages are displayed using the incorrect timezone like `logger`. I've attached my syslog-ng.conf file. So my question is why is the time information correct in the first three messages and then incorrect from there on? Francois -- https://fmarier.org/

3 4

drop log lines having keyword "INTERNET"
by Sachchidanand Upadhyay 04 Aug '22

04 Aug '22

Hi, I want to drop all log lines having keyword "INTERNET", I tried following scenario: scenario-1 filter f_log { facility(local3); }; filter f_nointernet { not message("INTERNET"); }; log { source(s_sys); filter(f_nointernet); filter(f_log); destination(d_log); }; scenario-2 filter f_internet { message("INTERNET"); }; filter f_log { facility(local3); and not filter(f_internet); }; log { source(s_sys); filter(f_log); destination(d_log); }; scenario-3 filter f_log { facility(local3); }; filter f_internet { message("INTERNET"); }; filter f_nointernet { not filter(f_internet); }; log { source(s_sys); filter(f_nointernet); filter(f_log); destination(d_log); }; But none of them is working. Please help. Regards, Sachchidanand

1 0

Kafka-c issues
by Sanjay Patel 15 Jul '22

15 Jul '22

I am trying to setup Syslog-NG t send output to Kakka. I am running ubuntu 18.04, Syslog-NG version 3.37. This is my syslog-ng.conf ( I have #scl.conf because if I don't it tries to install kafka-java) @version: 3.37 @define kafka-implementation kafka-c #@include "scl.conf" destination d_kafka { kafka( bootstrap-servers("127.0.0.1:9092") topic("{incoming}") ); }; This is the error when I try and run sudo syslog-ng Error parsing config, syntax error, unexpected LL_IDENTIFIER, expecting '}' in /etc/syslog-ng/syslog-ng.conf:6:3-6:8: 1 @version: 3.37 2 @define kafka-implementation kafka-c 3 #@include "scl.conf" 4 5 destination d_kafka { 6-----> kafka( 6-----> ^^^^^ 7 bootstrap-servers("127.0.0.1:9092") 8 topic("{incoming}") 9 ); 10 }; Any help will be appreciated. SKP

4 8

Insider 2022-06: RHEL 9; disk-buffer; Microsoft Linux;
by Peter Czanik (pczanik) 14 Jul '22

14 Jul '22

Dear syslog-ng users, This is the 103rd issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS RHEL 9 syslog-ng news --------------------- Red Hat Enterprise Linux 9 became generally available recently. Version 3.35 of syslog-ng has been part of EPEL 9 (the semi-official extra software repo for RHEL maintained by Fedora packagers) for a while and now I enabled a few more destination drivers. I also enabled RHEL 9 support in my unofficial Git snapshot packages, so I can support RHEL 9 together with other RHEL and Fedora versions on the next syslog-ng release. https://www.syslog-ng.com/community/b/blog/posts/rhel-9-syslog-ng-news How does the syslog-ng disk-buffer work? ---------------------------------------- Last time, we had an overview of the syslog-ng disk-buffer. This time, we dig a bit deeper and take a quick look at how it works, and a recent major change that helped speed up the reliable disk-buffer considerably. https://www.syslog-ng.com/community/b/blog/posts/how-does-the-syslog-ng-dis… Installing syslog-ng on Microsoft Linux --------------------------------------- Yes, Microsoft has its own Linux distribution, called CBL-Mariner. It is an internal Linux distribution by Microsoft used for cloud infrastructure and edge products and services. And even if it is not installed in the OS by default, CBL-Mariner also includes syslog-ng. https://www.syslog-ng.com/community/b/blog/posts/installing-syslog-ng-on-mi… WEBINARS * Register for our latest webinars at https://www.syslog-ng.com/event/syslogng-webcast-series-2022/ * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0