- syslog-ng - lists.balabit.hu

How to enable Syslog-ng authentication with MongoDB
by Ibrahim Al Mahfooz 20 Aug '23

20 Aug '23

Hi, I would like to harden the security by enabling authentication between syslog-ng and MongoDB. I'm using syslog-ng v3.38 and MongoDB v6.0.8 At the destination I tried the below two methods but none of themt worked, am I doing something wrong? Or the authentication is not supported? *---> First method:* destination d_mongodb_01 { mongodb( uri("mongodb://SampleUser:SamplePass@172.16.17.16:27017/syslog-ng_01") collection("logs") value-pairs( pair("datetime" "${S_ISODATE}") pair("privateip" "${privateip}") pair("portstatus" "${PORTSTATUS}") pair("host" "${HOST}")) ); }; *---> Second method:* destination d_mongodb_01 { mongodb( uri("mongodb://172.16.17.16:27017/syslog-ng_01") username("SampleUser") password("SamplePass") collection("logs") value-pairs( pair("datetime" "${S_ISODATE}") pair("privateip" "${privateip}") pair("portstatus" "${PORTSTATUS}") pair("host" "${HOST}")) ); }; Appreciate your help Best regards

2 2

discussion: Should ptbtool test validate types
by Evan Rempel 19 Aug '23

19 Aug '23

Now that syslog-ng has type hinting should the pdbtool test also validate the types of any values specified for a pattern. <rule id="pid1" class="myclass" provider="me"> <patterns> <pattern>some pattern @NUMBER:mynum@ with number</pattern> </patterns> <tags> <tag>rate</tag> </tags> <values> <value name="more" type="integer">100A</value> </values> <examples> <example> <test_message program="some pattern 123 with number</test_message> <test_values> <test_value name="mynum" type="integer">123</test_value> </test_values> </example> </examples> </rule> Currently the test_values are verified for both the value, and the type that the pattern parses, however, none of the values that are used to enrich this message are verified for type. This results in a patterndb that passes validation, yet will fail at the time the pattern matches a log message. comments? -- Evan

2 1

working with Sigma rules?
by Peter Czanik (pczanik) 11 Aug '23

11 Aug '23

Hi, Recently I was asked if Sigma rules (https://github.com/SigmaHQ/sigma) are supported by syslog-ng. Syslog-ng has message parsing, filtering, and can be used for alerting. But I'm not aware of any tools that could turn Sigma rules into PatternDB and syslog-ng.conf Syslog-ng can send logs to Splunk, ElasticSearch / OpenSearch or Graylog, all which already have sigma rules integrations. Of course, many users use/abuse syslog-ng as a kind of SIEM-lite as it is very good at real-time alerting. However, as far as I can see, Sigma rules are better suited for threat hunting on the SIEM side. If you already Sigma rules with syslog-ng or any other way: please share your experiences! Thanks, Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

2 1

Insider 2023-08: sngbench; 4.3.1 release; syslog-ng 4; Python;
by Peter Czanik (pczanik) 10 Aug '23

10 Aug '23

Dear syslog-ng users, This is the 111th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Introducing sngbench: a shell script to performance test your syslog-ng ---------------------------------- One of the returning questions I receive is how many log messages can a given hardware handle. My typical answer is that it depends on the configuration. I have now an answer, or rather a tool to answer your question sngbench.sh. It is a shell script that runs from localhost and uses loggen, the bundled benchmarking and testing tool of syslog-ng. It comes with two configurations: a performance-optimized and a realistic one. You are also free to extend sngbench with your own configurations. https://www.syslog-ng.com/community/b/blog/posts/introducing-sngbench-a-she… Version 4.3.1 of syslog-ng available ------------------------------------ Receive or send any kinds of logs, traces, and metrics with syslog-ng using OTLP/gRPC, improved performance for high-volume connections, and more! Containers and packages for major Linux distributions and FreeBSD are already available. https://axoflow.com/axosyslog-release-4-3/ For the complete release notes, check: https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-4.3.1 Syslog-ng Python packaging -------------------------- In version 4 of syslog-ng, the role of Python became even more important. Previously, all parts of syslog-ng could be extended using Python code, but no actual Python code was provided with syslog-ng. Version 4.0 added a Kubernetes module implemented in Python, while version 4.2 added support for Hypr. But how can we ensure that all Python dependencies are met? https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-python-packaging Getting syslog-ng 4 ------------------- Version 4 of syslog-ng was released last December. Quite a few people use it already in production. How can you install it for a test drive? It might be already available in your Linux distribution. There are also several unofficial repositories with the latest syslog-ng. >From this blog, you can learn how to check your syslog-ng version, where to check if it is not yet installed, and a few additional resources, if you want to install the latest version from unofficial repositories. https://www.syslog-ng.com/community/b/blog/posts/getting-syslog-ng-4 WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Need help to send Datetime in BSON ISODATE to MongoDB?
by Ibrahim Al Mahfooz 07 Aug '23

07 Aug '23

Hello, I have integration working fine between syslog-ng v3.38 and MongoDB v6. I am working to enable MongoDB Time-Series Collections on a testing DB for the purpose of having better efficiency and improved queries over time. I created the DB and collection, tested it by Mongo Compass client, then I tried to send the data from syslog-ng but it didn't work. MongoDB is throwing the following error: "17:12:57. 483 | [2023-08-06T17:12:57.355549] Failed to insert into MongoDB; time_reopen='10', reason='\'datetime\' must be present and contain a valid BSON UTC datetime value', driver='d_mongodb_ttl#0" My understanding is that it is a must to send the time/date to MongoDB with Time Series enabled DB a BSON format Date. Any suggestions? *Below is the mongoDB destination config:* destination d_mongodb_ttl { mongodb( uri("mongodb://192.168.1.19:27017/syslog-ng_ttl") collection("logs") value-pairs( scope("selected-macros") exclude("SOURCEIP") exclude("TAGS") pair("datetime" "${S_ISODATE}") pair("privateip" "${privateip}") pair("publicip" "${publicip}") pair("publicportrange" "${publicportrange}") pair("portaction" "${AI}") pair("portstatus" "${AL}") pair("host" "${HOST}")) ); }; Appreciate your help

3 4

syslog-ng 4.3.1
by László Várady 03 Aug '23

03 Aug '23

Dear syslog-ng users, We are pleased to announce the 4.3.1 version of syslog-ng, which has been released and is now available on GitHub: https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-4.3.1 Packages <https://github.com/syslog-ng/syslog-ng#installation-from-binaries> are also available for various platforms. The AxoSyslog project provides cloud-ready container images <https://axoflow.com/cloud-ready-syslog-ng-images/>, Helm charts <https://axoflow.com/axosyslog-log-collection-for-kubernetes/>, and open-source documentation at the link below: https://axoflow.com/docs/axosyslog-core/ 4.3.1 *This is the combination of the news entries of 4.3.0 and 4.3.1. 4.3.1 hotfixed* *a python-parser() related crash and a metrics related memory leak. It also* *added Ubuntu 23.04 and Debian 12 support for APT packages and the opensearch()* *destination.* Read Axoflow's blog post <https://axoflow.com/axosyslog-release-4-3/> for more details. Highlights parallelize() support for pipelines syslog-ng has traditionally performed processing of log messages arriving from a single connection sequentially. This was done to ensure message ordering as well as most efficient use of CPU on a per message basis. This mode of operation is performing well as long as we have a relatively large number of parallel connections, in which case syslog-ng would use all the CPU cores available in the system. In case only a small number of connections deliver a large number of messages, this behaviour may become a bottleneck. With the new parallelization feature, syslog-ng gained the ability to re-partition a stream of incoming messages into a set of partitions, each of which is to be processed by multiple threads in parallel. This does away with ordering guarantees and adds an extra per-message overhead. In exchange it will be able to scale the incoming load to all CPUs in the system, even if coming from a single, chatty sender. To enable this mode of execution, use the new parallelize() element in your log path: log { source { tcp( port(2000) log-iw-size(10M) max-connections(10) log-fetch-limit(100000) ); }; parallelize(partitions(4)); # from this part on, messages are processed in parallel even if # messages are originally coming from a single connection parser { ... }; destination { ... }; }; The config above will take all messages emitted by the tcp() source and push the work to 4 parallel threads of execution, regardless of how many connections were in use to deliver the stream of messages to the tcp() driver. parallelize() uses round-robin to allocate messages to partitions by default. You can however retain ordering for a subset of messages with the partition-key() option. You can use partition-key() to specify a message template. Messages that expand to the same value are guaranteed to be mapped to the same partition. For example: log { source { tcp( port(2000) log-iw-size(10M) max-connections(10) log-fetch-limit(100000) ); }; parallelize(partitions(4) partition-key("$HOST")); # from this part on, messages are processed in parallel if their # $HOST value differs. Messages with the same $HOST will be mapped # to the same partition and are processed sequentially. parser { ... }; destination { ... }; }; NOTE: parallelize() requires a patched version of libivykis that contains this PR buytenh/ivykis#25 <https://github.com/buytenh/ivykis/pull/25>. syslog-ng source releases bundle this version of ivykis in their source trees, so if you are building from source, be sure to use the internal version (--with-ivykis=internal). You can also use Axoflow's cloud native container image for syslog-ng, named AxoSyslog (https://github.com/axoflow/axosyslog-docker) which also incorporates this change. (#3966 <https://github.com/syslog-ng/syslog-ng/pull/3966>) Receiving and sending OpenTelemetry (OTLP) messages The opentelemetry() source, parser and destination are now available to receive, parse and send *OTLP/gRPC* messages. syslog-ng accepts logs <https://github.com/open-telemetry/opentelemetry-proto/blob/v0.20.0/opentele…>, metrics <https://github.com/open-telemetry/opentelemetry-proto/blob/v0.20.0/opentele…> and traces <https://github.com/open-telemetry/opentelemetry-proto/blob/v0.20.0/opentele…> . The incoming fields are not available through syslog-ng log message name-value pairs for the user by default. This is useful for forwarding functionality (the opentelemetry() destination can access and format them). If such functionality is required, you can configure the opentelemetry() parser, which maps all the fields with some limitations. The behavior of the opentelemetry() parser is the following: The name-value pairs always start with .otel. prefix. The type of the message is stored in .otel.type (possible values: log, metric and span). The resource info is mapped to .otel.resource.<...> (e.g.: .otel.resource.dropped_attributes_count, .otel.resource.schema_url ...), the scope info is mapped to .otel.scope.<...> (e.g.: .otel.scope.name, .otel.scope.schema_url, ...). The fields of log records are mapped to .otel.log.<...> (e.g. .otel.log.body, .otel.log.severity_text, ...). The fields of metrics are mapped to .otel.metric.<...> (e.g. . otel.metric.name, .otel.metric.unit, ...), the type of the metric is mapped to .otel.metric.data.type (possible values: gauge, sum, histogram, exponential_histogram, summary) with the actual data mapped to .otel.metric.data.<type>.<...> (e.g.: .otel.metric.data.gauge.data_points.0.time_unix_nano, ...). The fields of traces are mapped to .otel.span.<...> (e.g. .otel.span.name, .otel.span.trace_state, ...). repeated fields are given an index (e.g. .otel.span.events.5.time_unix_nano ). The mapping of AnyValue <https://github.com/open-telemetry/opentelemetry-proto/blob/v0.20.0/opentele…> type fields is limited. string, bool, int64, double and bytes values are mapped with the respective syslog-ng name-value type (e.g. .otel.resource.attributes.string_key => string_value), however ArrayValue and KeyValueList types are stored serialized with protobuf type. protobuf and bytes types are not directly available for the user, unless an explicit type cast is added (e.g. "bytes(${.otel.log.span_id})") or --include-bytes is passed to name-value iterating template functions (e.g. $(format-json .otel.* --include-bytes), which will base64 encode the bytes content). Three authentication methods are available in the source auth() block: insecure() (default), tls() and alts(). tls() accepts the key-file(), cert-file(), ca-file() and peer-verify() (possible values: required-trusted, required-untrusted, optional-trusted and optional-untrusted) options. ALTS <https://grpc.io/docs/languages/cpp/alts/> is a simple to use authentication, only available within Google's infrastructure. The same methods are available in the destination auth() block, with two differences: tls(peer-verify()) is not available, and there is a fourth method, called ADC <https://cloud.google.com/docs/authentication/application-default-credentials>, which accepts the target-service-account() option, where a list of service accounts can be configured to match against when authenticating the server. Example configs: log otel_forward_mode_alts { source { opentelemetry( port(12345) auth(alts()) ); }; destination { opentelemetry( url("my-otel-server:12345") auth(alts()) ); }; }; log otel_to_non_otel_insecure { source { opentelemetry( port(12345) ); }; parser { opentelemetry(); }; destination { network( "my-network-server" port(12345) template("$(format-json .otel.* --shift-levels 1 --include-bytes)\n") ); }; }; log non_otel_to_otel_tls { source { network( port(12346) ); }; destination { opentelemetry( url("my-otel-server:12346") auth( tls( ca-file("/path/to/ca.pem") key-file("/path/to/key.pem") cert-file("/path/to/cert.pem") ) ) ); }; }; (#4523 <https://github.com/syslog-ng/syslog-ng/pull/4523>) (#4510 <https://github.com/syslog-ng/syslog-ng/pull/4510>) Sending messages to CrowdStrike Falcon LogScale (Humio) The logscale() destination feeds LogScale via the Ingest API <https://library.humio.com/falcon-logscale/api-ingest.html#api-ingest-struct…> . Minimal config: destination d_logscale { logscale( token("my-token") ); }; Additional options include: - url() - rawstring() - timestamp() - timezone() - attributes() - extra-headers() - content-type() (#4472 <https://github.com/syslog-ng/syslog-ng/pull/4472>) Features - afmongodb: Bulk MongoDB insert is added via the following options - bulk (*yes*/no) turns on/off bulk insert <http://mongoc.org/libmongoc/current/bulk.html>usage, no forces the old behavior (each log is inserted one by one into the MongoDB) - bulk_unordered (yes/*no*) turns on/off unordered MongoDB bulk operations <http://mongoc.org/libmongoc/current/bulk.html#unordered-bulk-write-operatio…> - bulk_bypass_validation (yes/*no*) turns on/off MongoDB bulk operations validation <http://mongoc.org/libmongoc/1.23.3/bulk.html#bulk-operation-bypassing-docum…> - write_concern (unacked/*acked*/majority/n > 0) sets write concern mode of the MongoDB operations <http://mongoc.org/libmongoc/1.23.3/bulk.html#bulk-operation-write-concerns>, both bulk and single NOTE: Bulk sending is only efficient if the used collection is constant (e.g. not using templates) or the used template does not lead to too many collections switching within a reasonable time range. (#4483 <https://github.com/syslog-ng/syslog-ng/pull/4483>) - sql: Added 2 new options - quote_char to aid custom quoting for table and index names (e.g. MySQL needs sometimes this for certain identifiers) *NOTE*: Using a back-tick character needs a special formatting as syslog-ng uses it for configuration parameter names, so for that use: quote_char("``") (double back-tick) - dbi_driver_dir to define an optional DBI driver location for DBD initialization NOTE: libdbi and libdbi-drivers OSE forks are updated, afsql now should work nicely both on ARM and X86 macOS systems too (tested on macOS 13.3.1 and 12.6.4) Please do not use the pre-built ones (e.g. 0.9.0 from Homebrew), build from the *master* of the following - https://github.com/balabit-deps/libdbi-drivers - https://github.com/balabit-deps/libdbi (#4460 <https://github.com/syslog-ng/syslog-ng/pull/4460>) - opensearch: Added a new destination. It is similar to elasticsearch-http(), with the difference that it does not have the type() option, which is deprecated and advised not to use. (#4560 <https://github.com/syslog-ng/syslog-ng/pull/4560>) Bugfixes - network(),syslog(),tcp() destination: fix TCP keepalive tcp-keepalive-*() options were broken on the destination side since v3.34.1. (#4559 <https://github.com/syslog-ng/syslog-ng/pull/4559>) - Fixed a hang, which happend when syslog-ng received exremely low CPU time. (#4524 <https://github.com/syslog-ng/syslog-ng/pull/4524>) - $(format-json): Fixed a bug where sometimes an unnecessary comma was added in case of a type cast failure. (#4477 <https://github.com/syslog-ng/syslog-ng/pull/4477>) - Fix flow-control when fetch-limit() is set higher than 64K In high-performance use cases, users may configure log-iw-size() and fetch-limit() to be higher than 2^16, which caused flow-control issues, such as messages stuck in the queue forever or log sources not receiving messages. (#4528 <https://github.com/syslog-ng/syslog-ng/pull/4528>) - int32() and int64() type casts: accept hex numbers as proper number representations just as the @NUMBER@ parser within db-parser(). Supporting octal numbers were considered and then rejected as the canonical octal representation for numbers in C would be ambigious: a zero padded decimal number could be erroneously considered octal. I find that log messages contain zero padded decimals more often than octals. (#4535 <https://github.com/syslog-ng/syslog-ng/pull/4535>) - Fixed compilation on platforms where SO_MEMINFO is not available (#4548 <https://github.com/syslog-ng/syslog-ng/pull/4548>) - python: InstantAckTracker, ConsecutiveAckTracker and BatchedAckTracker are now called properly. Added proper fake classes for the InstantAckTracker, ConsecutiveAckTracker and BatchedAckTracker classes, and the wapper now calls the super class' constructor. Previusly the super class' constructor was not called which caused the python API to never call into the C API, which's result was that that the callback was never called. (#4549 <https://github.com/syslog-ng/syslog-ng/pull/4549>) - python: Fixed a crash when reloading with a config, which uses a python parser with multiple references. (#4552 <https://github.com/syslog-ng/syslog-ng/pull/4552>) (#4567 <https://github.com/syslog-ng/syslog-ng/pull/4567>) - mqtt(): Fixed the name of the stats instance (mqtt-source) to conform to the standard comma-separated format. (#4551 <https://github.com/syslog-ng/syslog-ng/pull/4551>) - metrics: Fixed a memory leak which happened during reload, and was introduced in 4.3.0. (#4568 <https://github.com/syslog-ng/syslog-ng/pull/4568>) Packaging - scl.conf: The scl.conf file has been moved to /share/syslog-ng/include/scl.conf (#4534 <https://github.com/syslog-ng/syslog-ng/pull/4534>) - C++ plugins: Some of syslog-ng's plugins now contain C++ code. By default they are being built if a C++ compiler is available. Disabling it is possible with --disable-cpp. Affected plugins: - lib/syslog-ng/libexamples.so - --disable-cpp will only disable the C++ part ( random-choice-generator()) - lib/syslog-ng/libotel.so (#4484 <https://github.com/syslog-ng/syslog-ng/pull/4484>) - debian: A new module is added, called syslog-ng-mod-grpc. Its dependencies are: protobuf-compiler, protobuf-compiler-grpc, libprotobuf-dev, libgrpc++-dev. Building the module can be toggled with --enable-grpc. (#4510 <https://github.com/syslog-ng/syslog-ng/pull/4510>) - pcre: syslog-ng now uses pcre2 (8 bit) as a dependency instead of pcre. The minimum pcre2 version is 10.0. (#4537 <https://github.com/syslog-ng/syslog-ng/pull/4537>) Notes to developers - lib/logmsg: Public field LogMessage::protected has been renamed to LogMessage::write_protected. Direct usage of this field is discouraged, instead use the following functions: - log_msg_is_write_protected() - log_msg_write_protect() (#4484 <https://github.com/syslog-ng/syslog-ng/pull/4484>) - lib/templates: Public field LogTemplate::template has been renamed to LogTemplate::template_str. (#4484 <https://github.com/syslog-ng/syslog-ng/pull/4484>) Other changes - syslog-ng-cfg-db: Moved to a separate repository. It is available at: https://github.com/alltilla/syslog-ng-cfg-helper (#4475 <https://github.com/syslog-ng/syslog-ng/pull/4475>) - disk-buffer: Added alternative option names disk-buf-size() -> capacity-bytes() qout-size() -> front-cache-size() mem-buf-length() -> flow-control-window-size() mem-buf-size() -> flow-control-window-bytes() Old option names are still available. Example configs: tcp( "127.0.0.1" port(2001) disk-buffer( reliable(yes) capacity-bytes(1GiB) flow-control-window-bytes(200MiB) front-cache-size(1000) ) ); tcp( "127.0.0.1" port(2001) disk-buffer( reliable(no) capacity-bytes(1GiB) flow-control-window-size(10000) front-cache-size(1000) ) ); (#4526 <https://github.com/syslog-ng/syslog-ng/pull/4526>) - selinux: Added RHEL9 support for the selinux policies Added RHEL9 support for the selinux policies at contrib/selinux (#4509 <https://github.com/syslog-ng/syslog-ng/pull/4509>) - metrics: replace driver_instance (stats_instance) with metric labels The new metric system had a label inherited from legacy: driver_instance. This non-structured label has been removed and different driver-specific labels have been added instead, for example: Before: syslogng_output_events_total{driver_instance="mongodb,localhost:27017,defaultdb,,coll",id="#anon-destination1#1",result="queued"} 4 After: syslogng_output_events_total{driver="mongodb",host="localhost:27017",database="defaultdb",collection="coll",id="#anon-destination1#1",result="queued"} 4 This change may affect legacy stats outputs (syslog-ng-ctl stats), for example, persist-name()-based naming is no longer supported in this old format. (#4551 <https://github.com/syslog-ng/syslog-ng/pull/4551>) - APT packages: Added Ubuntu Lunar Lobster and Debian Bookworm support. (#4561 <https://github.com/syslog-ng/syslog-ng/pull/4561>) syslog-ng Discord For a bit more interactive discussion, join our Discord server: [image: Axoflow Discord Server] <https://discord.gg/E65kP9aZGm> Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andreas Friedmann, Attila Szakacs, Balazs Scheidler, Bálint Horváth, Chuck Silvers, Evan Rempel, Hofi, Kovacs, Gergo Ferenc, László Várady, Romain Tartière, Ryan Faircloth, vostrelt

1 0

Properly Setting Facility and Severity into destination
by Greg Christopher 01 Aug '23

01 Aug '23

Hi All, Did my best (about a week effort) to find this information online and apologize if it's obvious. I am trying to use syslog-ng as a client to send messages to a syslog compliant server with the proper severity and facility set. I am starting with plain-text log files. The syslog rfc as well as the syslog-ng documentation are pretty clear about the syslog message format itself. There are even syslog-ng functions to substitute severity and facility using a function called "rewrite". But this doesn't seem to work if you are starting with a regular application log. In other words, there is nothing to "rewrite" since this header was never there to begin with. Although I initially configured my application log with flags(no-parse) , I attempted to remove it so that my rewrite attempt would at least have a facility and severity to "rewrite", but this also seems to have failed. Next I attempted prepending "<81>" (8*10 + 1) to the application messages to get the right priority field but this failed to change what the syslog-ng server on the other end received, as I have modified its output to include severity and facility:--- destination d_local { file("/var/log/syslog-ng/$HOST" template("Fac: $FACILITY Pri: $PRIORITY $ISODATE $HOST $(format-welf --scope all-nv-pairs)\n") frac-digits(3)); }; --- It does seem to be working properly but the above two approaches to increasing the severity and noting this is a security facility message failed. So it seems this use case where we create syslog entries from scratch sort of throws a curve ball at syslog-ng clients, at least from what I can figure. Any help on how I can set these manually into the destination would be greatly appreciated. Here is my syslog-ng.conf at the moment; note I pull log file name out of environment variable:--- @version: 4.1 # Configure the source to read from the messages log file # no-parse means whole line becomes "message" portion. # Template should indicate security alert to syslog daemon source s_APP_LOG { file(`APP_LOG`); # file(`APP_LOG` flags(no-parse)); }; rewrite set_pri_fields { set-severity("alert"); set-facility("security"); }; destination d_destination { syslog(`syslogServer`); }; # Configure the log statement to route messages from the application log file to # the syslog server specified on the command line log { source(s_APP_LOG); rewrite(set_pri_fields); destination(d_destination); }; ---Greg

2 2

parallelize: modified ivykis for Fedora / RHEL
by Peter Czanik (pczanik) 01 Aug '23

01 Aug '23

Hi, Version 4.3.1 of syslog-ng was released last weekend and it contains an interesting feature: parallelize(). TL;DR: it allows faster processing of single, high-traffic TCP connections. You can read more about it in my latest blog at: https://www.syslog-ng.com/community/b/blog/posts/accelerating-single-tcp-co… The problem is that it requires a modified ivykis library. The syslog-ng source code bundles ivykis, and has a modified version. However, all Fedora and RHEL syslog-ng packages use a separate ivykis package. I feel uncomfortable with modifying the "official" ivykis package in Fedora / RHEL with patches not yet in the upstream ivykis sources. And it would not help the majority of syslog-ng users anyway, who use RHEL 8 and compatible operating systems. I could only add it to Fedora and upcoming RHEL releases. However, I want to enable this feature for the majority of syslog-ng users. Plan: add a modified ivykis library to my unofficial repositories. The version would stay 0.42.4, but the release would be a high number, making sure that it stays larger than the "official" package. First I'd introduce it to my git snapshot repository, and once I collect some positive feedback, I implement it in other repos too for upcoming syslog-ng versions. What do you think? Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

3 2

Syslog-ng is not starting
by Gundlapally, Navanitha 11 Jul '23

11 Jul '23

Team, I am new to syslog-ng and recently installed syslog-ng v3.23 on RHEL 8. However while trying to restart syslog-ng service it is failing with below error. I read it on some blog that installing "http" should fix it but didn't find any steps on how to intall the module. I did try to do a fresh installation of syslog-ng 3.23 and no luck there. Plugin module not found in 'module-path'; module-path='/usr/lib64/syslog-ng', module='http' Thank you NG. This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system.

3 3

Re: [syslog-ng] syslog-ng EPS
by Wilson, Jonathan L 12 Jun '23

12 Jun '23

We use: syslog-ng-ctl query get source.*.processed --reset In a cron job just before midnight to capture the entire day's stats. They show up in the format: source.s_local.processed=4722471 source.s_BSD_TCP.processed=6299967 source.s_RFC5424_TCP.processed=0 source.s_network_source.processed=27695900 . . . where "s_local", "s_BSD_TCP", etc. are the source names inside the syslog-ng.conf file. Note that the "-reset" on the end does just that, resets the stats to zero. A little scripting to divide the numbers above by 86400 will give you the count of messages per second by source. (We convert ours to JSON and dump it into Splunk for later viewing.) Regards, Jonathan Wilson

1 0

Can't Find Syslog-ng 4.0.0 Administration Guide Link
by Sumanta Banerjee 09 Jun '23

09 Jun '23

Hi All, Can someone please help in sending me the link for Syslog-ng 4.0.0 administration guide (if at all available). Thanks & Regards, Sumanta Banerjee Splunk Admin | CISO | Aviva Group Tel: +91-8420892593 24x7x365: +44 1603 208 582 sumanta.banerjee(a)aviva.com<mailto:sumanta.banerjee@aviva.com> GlobalCyberSecurityEngineeringTeam(a)aviva.com<mailto:GlobalCyberSecurityEngineeringTeam@aviva.com> www.aviva.com<http://www.aviva.com> Wipro Technologies - SJP2, Bangalore, India [cid:image001.gif@01D99AE4.B2631840] Aviva: Public Aviva plc, registered Office: St. Helen's, 1 Undershaft, London EC3P 3DQ. Registered in England No. 02468686. www.aviva.com This message and any attachments may be confidential or legally privileged. If you are not the intended recipient, please telephone or e-mail the sender and delete this message and any attachments from your system. Also, if you are not the intended recipient you must not copy this message or attachments or disclose the contents to any other person. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Aviva.

1 0

syslog-ng EPS
by Dragan Zecevic 08 Jun '23

08 Jun '23

Hi guys, is it possible to calculate daily events per second for all syslog messages coming to syslog-ng? Or even better separated by sources. I have syslog-ng 3 (3.34.1). Thanks. Br, Dragan

2 1

One destination, multiple TCP connections?
by Steve Bernacki 08 Jun '23

08 Jun '23

Hello, We have a syslog TCP destination that receives logs from many different destinations. During peak log volume times, this destination begins to back up and queue. We've found that if we define multiple destinations for the same IP/port and route logs through these multiple destinations, the destination is able to handle these peak log volume times much better. Our current solution of having a destination for each source is pretty kludgy. Is there a way to define a TCP destination and specify to open, say, 10 connections to the destination and load balance the traffic over those 10 connections? Thanks. Steve

2 1

Fedora / RHEL & Co: Python split -- request for testing
by Peter Czanik (pczanik) 08 Jun '23

08 Jun '23

Hi, Recently I wrote a blog about Python packaging problems and my plans to split the Python-based drivers of syslog-ng from the syslog-ng Python plugin: https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-python-packaging I implemented it for my syslog-ng git snapshot repo for Fedora / RHEL: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng-githead/ Please test it ASAP, if you got a bit of time. Originally I planned to use it only for the 4.3 release, but if I get positive responses, I'll also backport these changes to my syslog-ng42 repo, and use these changes also for the syslog-ng-stable repo. The blog post explains why is this important, but TL;DR: this way you do not have to install 50+MB of dependencies you will likely never use, and syslog-ng will not crash on start on RHEL 8 & Co. due to lack of dependencies. Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 2

uestion
by Johnson, Linwood 06 Jun '23

06 Jun '23

I get the following when I check [root@server1~]# syslog-ng WARNING: the match() filter without the use of the value() option is deprecated and hinders performance, please update your configuration; The kernel refused to set the receive buffer (SO_RCVBUF) to the requested size, you probably need to adjust buffer related kernel parameters; so_rcvbuf='536870912', so_rcvbuf_set='425984' [root@ server1~]# sysctl -a |grep rmem_max net.core.rmem_max = 212992 sysctl: reading key "net.ipv6.conf.all.stable_secret" sysctl: reading key "net.ipv6.conf.default.stable_secret" sysctl: reading key "net.ipv6.conf.eth0.stable_secret" sysctl: reading key "net.ipv6.conf.lo.stable_secret" [root@ server1 ~]# sysctl -a |grep vbuf net.ipv4.tcp_moderate_rcvbuf = 1 sysctl: reading key "net.ipv6.conf.all.stable_secret" sysctl: reading key "net.ipv6.conf.default.stable_secret" sysctl: reading key "net.ipv6.conf.eth0.stable_secret" sysctl: reading key "net.ipv6.conf.lo.stable_secret" net.sctp.rcvbuf_policy = 0 [root@ server1~ ]# cat /proc/sys/net/ipv4/tcp_rmem 4096 87380 6291456 [root@ server1~ ]# cat /proc/sys/net/core/rmem_default 212992 [root@ server1~ ]# cat /proc/sys/net/core/rmem_max 212992 [root@ server1~ ]# grep . /proc/sys/net/ipv4/tcp*mem /proc/sys/net/ipv4/tcp_mem:173202 230936 346404 /proc/sys/net/ipv4/tcp_rmem:4096 87380 6291456 /proc/sys/net/ipv4/tcp_wmem:4096 16384 4194304 [root@ server1~ hisotry]# I attempted to make the changes on the server. What can I do to resolve this issue? -- Thanks in advance, Linwood Johnson Senior Engineer | Unix Systems Administrator Site Reliability Tools [image: NewsTech] P.O Box 300 | Princeton NJ 08543 *(W)* 609-627-2179 *(C)* 732-740-0694 *Email:* Linwood.Johnson(a)dowjones.com *Team:* sitereliabilitytools(a)dowjones.com

2 1

syslog-ng 4.2.0
by Attila Szakács 17 May '23

17 May '23

Dear syslog-ng users, We are pleased to announce that the 4.2.0 version of syslog-ng has been released, and it is now available on GitHub: https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-4.2.0 4.2.0 ===== Read Axoflow's [blog post](https://axoflow.com/axosyslog-release-4-2/) for more details. ## Highlights #### Sending messages to Splunk HEC The `splunk-hec-event()` destination feeds Splunk via the [HEC events API]( https://docs.splunk.com/Documentation/Splunk/9.0.4/RESTREF/RESTinput#servic… ). Minimal config: ``` destination d_splunk_hec_event { splunk-hec-event( url("https://localhost:8088") token("70b6ae71-76b3-4c38-9597-0c5b37ad9630") ); }; ``` Additional options include: * `event()` * `index()` * `source()` * `sourcetype()` * `host()` * `time()` * `default-index()` * `default-source()` * `default-sourcetype()` * `fields()` * `extra-headers()` * `extra-queries()` * `content-type()` The `splunk-hec-raw()` destination feeds Splunk via the [HEC raw API]( https://docs.splunk.com/Documentation/Splunk/9.0.4/RESTREF/RESTinput#servic… ). Minimal config: ``` destination d_splunk_hec_raw { splunk-hec-raw( url("https://localhost:8088") token("70b6ae71-76b3-4c38-9597-0c5b37ad9630") channel("05ed4617-f186-4ccd-b4e7-08847094c8fd") ); }; ``` ([#4462](https://github.com/syslog-ng/syslog-ng/pull/4462)) #### Smart multi-line for recognizing backtraces `multi-line-mode(smart)`: With this multi-line mode, the inherently multi-line data backtrace format is recognized even if they span multiple lines in the input and are converted to a single log message for easier analysis. Backtraces for the following programming languages are recognized : Python, Java, JavaScript, PHP, Go, Ruby and Dart. The regular expressions to recognize these programming languages are specified by an external file called `/usr/share/syslog-ng/smart-multi-line.fsm` (installation path depends on configure arguments), in a format that is described in that file. `group-lines()` parser: this new parser correlates multi-line messages received as separate, but subsequent lines into a single log message. Received messages are first collected into streams related messages (using key()), then collected into correlation contexts up to timeout() seconds. The identification of multi-line messages are then performed on these message contexts within the time period. ``` group-lines(key("$FILE_NAME") multi-line-mode("smart") template("$MESSAGE") timeout(10) line-separator("\n") ); ``` ([#4225](https://github.com/syslog-ng/syslog-ng/pull/4225)) #### HYPR Audit Trail source `hypr-audit-trail()` & `hypr-app-audit-trail()` source drivers are now available to monitor the audit trails for [HYPR](https://www.hypr.com) applications. See the README.md file in the driver's directory to see usage information. ([#4175](https://github.com/syslog-ng/syslog-ng/pull/4175)) #### `ebpf()` plugin and reuseport packet randomizer A new ebpf() plugin was added as a framework to leverage the kernel's eBPF infrastructure to improve performance and scalability of syslog-ng. Example: ``` source s_udp { udp(so-reuseport(yes) port(2000) persist-name("udp1") ebpf(reuseport(sockets(4))) ); udp(so-reuseport(yes) port(2000) persist-name("udp2")); udp(so-reuseport(yes) port(2000) persist-name("udp3")); udp(so-reuseport(yes) port(2000) persist-name("udp4")); }; ``` NOTE: The `ebpf()` plugin is considered advanced usage so its compilation is disabled by default. Please don't use it unless all other avenues of configuration solutions are already tried. You will need a special toolchain and a recent kernel version to compile and run eBPF programs. ([#4365](https://github.com/syslog-ng/syslog-ng/pull/4365)) ## Features * `network` source: During a TLS handshake, syslog-ng now automatically sets the `certificate_authorities` field of the certificate request based on the `ca-file()` and `ca-dir()` options. The `pkcs12-file()` option already had this feature. ([#4412](https://github.com/syslog-ng/syslog-ng/pull/4412)) * `metrics-probe()`: Added `level()` option to set the stats level of the generated metrics. ([#4453](https://github.com/syslog-ng/syslog-ng/pull/4453)) * `metrics-probe()`: Added `increment()` option. Users can now set a template, which resolves to a number that modifies the increment of the counter. If not set, the increment is 1. ([#4447](https://github.com/syslog-ng/syslog-ng/pull/4447)) * `python`: Added support for typed custom options. This applies for `python` source, `python-fetcher` source, `python` destination, `python` parser and `python-http-header` inner destination. Example config: ``` python( class("TestClass") options( "string_option" => "example_string" "bool_option" => True # supported values are: True, False, yes, no "integer_option" => 123456789 "double_option" => 123.456789 "string_list_option" => ["string1", "string2", "string3"] "template_option" => LogTemplate("${example_template}") ) ); ``` **Breaking change! Previously values were converted to strings if possible, now they are passed to the python class with their real type. Make sure to follow up these changes in your python code!** ([#4354](https://github.com/syslog-ng/syslog-ng/pull/4354)) * `mongodb` destination: Added support for list, JSON and null types. ([#4437](https://github.com/syslog-ng/syslog-ng/pull/4437)) * `add-contextual-data()`: significantly reduce memory usage for large CSV files. ([#4444](https://github.com/syslog-ng/syslog-ng/pull/4444)) * `python()`: new LogMessage methods for querying as string and with default values - `get(key[, default])` Return the value for `key` if `key` exists, else `default`. If `default` is not given, it defaults to `None`, so that this method never raises a `KeyError`. - `get_as_str(key, default=None, encoding='utf-8', errors='strict', repr='internal')`: Return the string value for `key` if `key` exists, else `default`. If `default` is not given, it defaults to `None`, so that this method never raises a `KeyError`. The string value is decoded using the codec registered for `encoding`. `errors` may be given to set the desired error handling scheme. Note that currently `repr='internal'` is the only available representation. We may implement another more Pythonic representation in the future, so please specify the `repr` argument explicitly if you want to avoid future representation changes in your code. ([#4410](https://github.com/syslog-ng/syslog-ng/pull/4410)) * `kubernetes()` source: Added support for json-file logging driver format. ([#4419](https://github.com/syslog-ng/syslog-ng/pull/4419)) * The new `$RAWMSG_SIZE` hard macro can be used to query the original size of the incoming message in bytes. This information may not be available for all source drivers. ([#4440](https://github.com/syslog-ng/syslog-ng/pull/4440)) * syslog-ng configuration identifier A new syslog-ng configuration keyword has been added, which allows specifying a config identifier. For example: ``` @config-id: cfg-20230404-13-g02b0850fc ``` This keyword can be used for config identification in managed environments, where syslog-ng instances and their configuration are deployed/generated automatically. `syslog-ng-ctl config --id` can be used to query the active configuration ID and the SHA256 hash of the full "preprocessed" syslog-ng configuration. For example: ``` $ syslog-ng-ctl config --id cfg-20230404-13-g02b0850fc (08ddecfa52a3443b29d5d5aa3e5114e48dd465e195598062da9f5fc5a45d8a83) ``` ([#4420](https://github.com/syslog-ng/syslog-ng/pull/4420)) * `syslog-ng`: add `--config-id` command line option Similarly to `--syntax-only`, this command line option parses the configuration and then prints its ID before exiting. It can be used to query the ID of the current configuration persisted on disk. ([#4435](https://github.com/syslog-ng/syslog-ng/pull/4435)) * Health metrics and `syslog-ng-ctl healthcheck` A new `syslog-ng-ctl` command has been introduced, which can be used to query a healthcheck status from syslog-ng. Currently, only 2 basic health values are reported. `syslog-ng-ctl healthcheck --timeout <seconds>` can be specified to use it as a boolean healthy/unhealthy check. Health checks are also published as periodically updated metrics. The frequency of these checks can be configured with the `stats(healthcheck-freq())` option. The default is 5 minutes. ([#4362](https://github.com/syslog-ng/syslog-ng/pull/4362)) * `$(format-json)` and template functions which support value-pairs expressions: new key transformations upper() and lower() have been added to translate the caps of keys while formatting the output template. For example: template("$(format-json test.* --upper)\n") Would convert all keys to uppercase. Only supports US ASCII. ([#4452](https://github.com/syslog-ng/syslog-ng/pull/4452)) * `python()`, `python-fetcher()` sources: Added a mapping for the `flags()` option. The state of the `flags()` option is mapped to the `self.flags` variable, which is a `Dict[str, bool]`, for example: ```python { 'parse': True, 'check-hostname': False, 'syslog-protocol': True, 'assume-utf8': False, 'validate-utf8': False, 'sanitize-utf8': False, 'multi-line': True, 'store-legacy-msghdr': True, 'store-raw-message': False, 'expect-hostname': True, 'guess-timezone': False, 'header': True, 'rfc3164-fallback': True, } ``` ([#4455](https://github.com/syslog-ng/syslog-ng/pull/4455)) ### Metrics * `network()`, `syslog()`: TCP connection metrics ``` syslogng_socket_connections{id="tcp_src#0",driver_instance="afsocket_sd.(stream,AF_INET(0.0.0.0:5555))",direction="input"} 3 syslogng_socket_max_connections{id="tcp_src#0",driver_instance="afsocket_sd.(stream,AF_INET(0.0.0.0:5555))",direction="input"} 10 syslogng_socket_rejected_connections_total{id="tcp_src#0",driver_instance="afsocket_sd.(stream,AF_INET(0.0.0.0:5555))",direction="input"} 96 ``` `internal()`: `internal_events_queue_capacity` metric `syslog-ng-ctl healthcheck`: new healthcheck value `syslogng_internal_events_queue_usage_ratio` ([#4411](https://github.com/syslog-ng/syslog-ng/pull/4411)) * `metrics`: new network (TCP, UDP) metrics are available on stats level 1 ``` # syslog-ng-ctl stats prometheus syslogng_socket_receive_buffer_used_bytes{id="#anon-source0#3",direction="input",driver_instance="afsocket_sd.udp4"} 0 syslogng_socket_receive_buffer_max_bytes{id="#anon-source0#3",direction="input",driver_instance="afsocket_sd.udp4"} 268435456 syslogng_socket_receive_dropped_packets_total{id="#anon-source0#3",direction="input",driver_instance="afsocket_sd.udp4"} 619173 syslogng_socket_connections{id="#anon-source0#0",direction="input",driver_instance="afsocket_sd.(stream,AF_INET(0.0.0.0:2000))"} 1 ``` ([#4374](https://github.com/syslog-ng/syslog-ng/pull/4374)) * New configuration-related metrics: ``` syslogng_last_config_reload_timestamp_seconds 1681309903 syslogng_last_successful_config_reload_timestamp_seconds 1681309758 syslogng_last_config_file_modification_timestamp_seconds 1681309877 ``` ([#4420](https://github.com/syslog-ng/syslog-ng/pull/4420)) * destination: Introduced queue metrics. * The corresponding driver is identified with the "id" and "driver_instance" labels. * Available counters are "memory_usage_bytes" and "events". * Memory queue metrics are available with "syslogng_memory_queue_" prefix, `disk-buffer` metrics are available with "syslogng_disk_queue_" prefix. * `disk-buffer` metrics have an additional "path" label, pointing to the location of the disk-buffer file and a "reliable" label, which can be either "true" or "false". * Threaded destinations, like `http`, `python`, etc have an additional "worker" label. Example metrics ``` syslogng_disk_queue_events{driver_instance="http,http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00000.rqf",reliable="true",worker="0"} 80 syslogng_disk_queue_events{driver_instance="http,http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00001.rqf",reliable="true",worker="1"} 7 syslogng_disk_queue_events{driver_instance="http,http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00002.rqf",reliable="true",worker="2"} 7 syslogng_disk_queue_events{driver_instance="http,http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00003.rqf",reliable="true",worker="3"} 7 syslogng_disk_queue_events{driver_instance="tcp,localhost:1235",id="d_network_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00000.qf",reliable="false"} 101 syslogng_disk_queue_memory_usage_bytes{driver_instance="http, http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00000.rqf",reliable="true",worker="0"} 3136 syslogng_disk_queue_memory_usage_bytes{driver_instance="http, http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00001.rqf",reliable="true",worker="1"} 2776 syslogng_disk_queue_memory_usage_bytes{driver_instance="http, http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00002.rqf",reliable="true",worker="2"} 2760 syslogng_disk_queue_memory_usage_bytes{driver_instance="http, http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00003.rqf",reliable="true",worker="3"} 2776 syslogng_disk_queue_memory_usage_bytes{driver_instance="tcp,localhost:1235",id="d_network_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00000.qf",reliable="false"} 39888 syslogng_memory_queue_events{driver_instance="http,http://localhost:1236",id="d_http#0",worker="0"} 15 syslogng_memory_queue_events{driver_instance="http,http://localhost:1236",id="d_http#0",worker="1"} 14 syslogng_memory_queue_events{driver_instance="tcp,localhost:1234",id="d_network#0"} 29 syslogng_memory_queue_memory_usage_bytes{driver_instance="http, http://localhost:1236",id="d_http#0",worker="0"} 5896 syslogng_memory_queue_memory_usage_bytes{driver_instance="http, http://localhost:1236",id="d_http#0",worker="1"} 5552 syslogng_memory_queue_memory_usage_bytes{driver_instance="tcp,localhost:1234",id="d_network#0"} 11448 ``` ([#4392](https://github.com/syslog-ng/syslog-ng/pull/4392)) * `network()`, `syslog()`, `file()`, `http()`: new byte-based metrics for incoming/outgoing events These metrics show the serialized message sizes (protocol-specific header/framing/etc. length is not included). ``` syslogng_input_event_bytes_total{id="s_network#0",driver_instance="tcp,127.0.0.1"} 1925529600 syslogng_output_event_bytes_total{id="d_network#0",driver_instance="tcp, 127.0.0.1:5555"} 565215232 syslogng_output_event_bytes_total{id="d_http#0",driver_instance="http, http://127.0.0.1:8080/"} 1024 ``` ([#4440](https://github.com/syslog-ng/syslog-ng/pull/4440)) * `disk-buffer`: Added metrics for monitoring the available space in disk-buffer `dir()`s. Metrics are available from `stats(level(1))`. By default, the metrics are generated every 5 minutes, but it can be changed in the global options: ``` options { disk-buffer( stats( freq(10) ) ); }; ``` Setting `freq(0)` disabled this feature. Example metrics: ``` syslogng_disk_queue_dir_available_bytes{dir="/var/syslog-ng"} 870109413376 ``` ([#4399](https://github.com/syslog-ng/syslog-ng/pull/4399)) * `disk-buffer`: Added metrics for abandoned disk-buffer files. Availability is the same as the `disk_queue_dir_available_bytes` metric. Example metrics: ``` syslogng_disk_queue_capacity_bytes{abandoned="true",path="/var/syslog-ng/syslog-ng-00000.rqf",reliable="true"} 104853504 syslogng_disk_queue_disk_allocated_bytes{abandoned="true",path="/var/syslog-ng/syslog-ng-00000.rqf",reliable="true"} 273408 syslogng_disk_queue_disk_usage_bytes{abandoned="true",path="/var/syslog-ng/syslog-ng-00000.rqf",reliable="true"} 269312 syslogng_disk_queue_events{abandoned="true",path="/var/syslog-ng/syslog-ng-00000.rqf",reliable="true"} 860 ``` ([#4402](https://github.com/syslog-ng/syslog-ng/pull/4402)) * `disk-buffer`: Added capacity, disk_allocated and disk_usage metrics. * "capacity_bytes": The theoretical maximal useful size of the disk-buffer. This is always smaller, than `disk-buf-size()`, as there is some reserved space for metadata. The actual full disk-buffer file can be larger than this, as syslog-ng allows to write over this limit once, at the end of the file. * "disk_allocated_bytes": The current size of the disk-buffer file on the disk. Please note that the disk-buffer file size does not strictly correlate with the number of messages, as it is a ring buffer implementation, and also syslog-ng optimizes the truncation of the file for performance reasons. * "disk_usage_bytes": The serialized size of the queued messages in the disk-buffer file. This counter is useful for calculating the disk usage percentage (disk_usage_bytes / capacity_bytes) or the remaining available space (capacity_bytes - disk_usage_bytes). Example metrics: ``` syslogng_disk_queue_capacity_bytes{driver_id="d_network#0",driver_instance="tcp,localhost:1235",path="/var/syslog-ng-00000.rqf",reliable="true"} 104853504 syslogng_disk_queue_disk_allocated_bytes{driver_id="d_network#0",driver_instance="tcp,localhost:1235",path="/var/syslog-ng-00000.rqf",reliable="true"} 17284 syslogng_disk_queue_disk_usage_bytes{driver_id="d_network#0",driver_instance="tcp,localhost:1235",path="/var/syslog-ng-00000.rqf",reliable="true"} 13188 ``` ([#4356](https://github.com/syslog-ng/syslog-ng/pull/4356)) * `kubernetes()`: Added `input_events_total` and `input_event_bytes_total` metrics. ``` syslogng_input_events_total{cluster="k8s",driver="kubernetes",id="#anon-source0",namespace="default",pod="log-generator-1682517834-7797487dcc-49hqc"} 25 syslogng_input_event_bytes_total{cluster="k8s",driver="kubernetes",id="#anon-source0",namespace="default",pod="log-generator-1682517834-7797487dcc-49hqc"} 1859 ``` ([#4447](https://github.com/syslog-ng/syslog-ng/pull/4447)) ## Bugfixes * `pdbtool test`: fix two type validation bugs: 1) When `pdbtool test` validates the type information associated with a name-value pair, it was using string comparisons, which didn't take type aliases into account. This is now fixed, so that "int", "integer" or "int64" can all be used to mean the same type. 2) When type information is missing from a `<test_value/>` tag, don't validate it against "string", rather accept any extracted type. In addition to these fixes, a new alias "integer" was added to mean the same as "int", simply because syslog-ng was erroneously using this term when reporting type information in its own messages. ([#4405](https://github.com/syslog-ng/syslog-ng/pull/4405)) * `$(format-json)`: fix RFC8259 number violation `$(format-json)` produced invalid JSON output when it contained numeric values with leading zeros or + signs. This has been fixed. ([#4415](https://github.com/syslog-ng/syslog-ng/pull/4415)) * `grouping-by()`: fix `persist-name()` option not taken into account ([#4390](https://github.com/syslog-ng/syslog-ng/pull/4390)) * `python()`, `db-parser()`, `grouping-by()`, `add-contextual-data()`: fix typing compatibility with <4.0 config versions ([#4394](https://github.com/syslog-ng/syslog-ng/pull/4394)) * `python`: Fixed a crash which occurred at reloading after registering a confgen plugin. ([#4459](https://github.com/syslog-ng/syslog-ng/pull/4459)) * `date-parser()`: fix `%z` when system timezone has no daylight saving time ([#4401](https://github.com/syslog-ng/syslog-ng/pull/4401)) * Consider messages consumed into correlation states "matching": syslog-ng's correlation functionality (e.g. grouping-by() or db-parser() with such rules) drop individual messages as they are consumed into a correlation contexts and you are using `inject-mode(aggregate-only)`. This is usually happens because you are only interested in the combined message and not in those that make up the combination. However, if you are using correlation with conditional processing (e.g. if/elif/else or flags(final)), such messages were erroneously considered as unmatching, causing syslog-ng to take the alternative branch. Example: With a configuration similar to this, individual messages are consumed into a correlation state and dropped by `grouping-by()`: ``` log { source(...); if { grouping-by(... inject-mode(aggregate-only)); } else { # alternative branch }; }; ``` The bug was that these individual messages also traverse the `else` branch, even though they were successfully processed with the inclusion into the correlation context. This is not correct. The bugfix changes this behaviour. ([#4370](https://github.com/syslog-ng/syslog-ng/pull/4370)) * `netmask6()`: fix crash when user specifies too long mask ([#4429](https://github.com/syslog-ng/syslog-ng/pull/4429)) * `afprog`: Fixed possible freezing on some OSes ([#4438](https://github.com/syslog-ng/syslog-ng/pull/4438)) * `network()`, `syslog()`, `syslog-parser()`: fix null termination of SDATA param names ([#4429](https://github.com/syslog-ng/syslog-ng/pull/4429)) * `python()`: fix LogMessage subscript not raising KeyError on non-existent keys When message fields were queried (`msg["key"]`) and the given key did not exist, `None` or an empty string was returned (depending on the version of the config). Neither was correct, now a KeyError occurs in such cases. ([#4410](https://github.com/syslog-ng/syslog-ng/pull/4410)) * `$(python)`: fix template function prefix being overwritten when using datetime types ([#4410](https://github.com/syslog-ng/syslog-ng/pull/4410)) * `disk-buffer`: Fixed queued messages stats counting, when a disk-buffer became corrupted. ([#4385](https://github.com/syslog-ng/syslog-ng/pull/4385)) * `$(format-json)`: fix escaping control characters `$(format-json)` produced invalid JSON output when a string value contained control characters. ([#4417](https://github.com/syslog-ng/syslog-ng/pull/4417)) * `disk-buffer()`: fix deinitialization when starting syslog-ng with invalid configuration ([#4418](https://github.com/syslog-ng/syslog-ng/pull/4418)) * `python()`: fix exception handling when LogMessage value conversion fails ([#4410](https://github.com/syslog-ng/syslog-ng/pull/4410)) * `json-parser()`: Fixed parsing non-string arrays. syslog-ng now no longer parses non-string arrays to list of strings, losing the original type information of the array's elements. ([#4396](https://github.com/syslog-ng/syslog-ng/pull/4396)) * `disk-buffer`: Fixed a rare race condition when calculating disk-buffer filename. ([#4381](https://github.com/syslog-ng/syslog-ng/pull/4381)) * `python-persist`: fix off-by-one overflow ([#4429](https://github.com/syslog-ng/syslog-ng/pull/4429)) ## Packaging * The `--with-python-venv-dir=path` configure option can be used to modify the location of syslog-ng's venv. The default is still `${localstatedir}/python-venv`. ([#4465](https://github.com/syslog-ng/syslog-ng/pull/4465)) ## Other changes * The `sdata-prefix()` option does not accept values longer than 128 characters. ([#4429](https://github.com/syslog-ng/syslog-ng/pull/4429)) * `grouping-by()`: Remove setting of the `${.classifier.context_id}` name-value pair in all messages consumed into a correlation context. This functionality is inherited from db-parser() and has never been documented for `grouping-by()`, has of limited use, and any uses can be replaced by the use of the built-in macro named `$CONTEXT_ID`. Modifying all consumed messages this way has significant performance consequences for `grouping-by()` and removing it outweighs the small incompatibility this change introduces. The similar functionality in `db-parser()` correlation is not removed with this change. ([#4424](https://github.com/syslog-ng/syslog-ng/pull/4424)) * `config`: Added `internal()` option to `source`s, `destination`s, `parser`s and `rewrite`s. Its main usage is in SCL blocks. Drivers configured with `internal(yes)` register their metrics on level 3. This makes developers of SCLs able to create metrics manually with `metrics-probe()` and "disable" every other metrics, they do not need. ([#4451](https://github.com/syslog-ng/syslog-ng/pull/4451)) * The following Prometheus metrics have been renamed: `log_path_{in,e}gress` -> `route_{in,e}gress_total` `internal_source` -> `internal_events_total` The `internal_queue_length` stats counter has been removed. It was deprecated since syslog-ng 3.29. ([#4411](https://github.com/syslog-ng/syslog-ng/pull/4411)) ## syslog-ng Discord For a bit more interactive discussion, join our Discord server: https://discord.gg/E65kP9aZGm ## Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Alex Becker, Attila Szakacs, Balazs Scheidler, Hofi, László Várady, Muhammad Shanif, Ricfilipe, Romain Tartière

1 0

Insider 2023-05: learning; UDP; upgrading;
by Peter Czanik (pczanik) 11 May '23

11 May '23

Dear syslog-ng users, This is the 110th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Learning syslog-ng, the easier way ---------------------------------- Last year, one of the returning questions I received was how to learn syslog-ng. My answer was that read the first few chapters of the documentation, read my blogs related to your use case, and then read a few relevant parts from the rest of the documentation. Our documentation is praised by users, but it is still a reference documentation. I was asked if a less detailed, more to the point, preferably video tutorial is available. Your request was heard. In the past couple of months, I published a tutorial series in blog and video format, which brings you from basic logging concepts to using syslog-ng to collect, parse, enrich log messages and store them to Elasticsearch. Of course, these 5-10 minute videos are not enough to learn anything in depth, but they introduce you to all major syslog-ng functionalities. https://www.syslog-ng.com/community/b/blog/posts/learning-syslog-ng-the-eas… Why syslog over UDP loses messages and how to avoid that -------------------------------------------------------- Message loss related resiliency with regards to syslog over UDP has always been a problem. Users who start looking at these metrics generally report a 30-40% loss of messages for syslog over UDP, but drops of up to 90% are not unheard of. Let me get to the bottom of the reasons why messages are dropped and how these problems can be solved, or at least mitigated. https://axoflow.com/syslog-over-udp-message-loss-1/ Upgrade problems from syslog-ng 3 to 4 -------------------------------------- Version 4 of syslog-ng works perfectly well in version 3 compatibility mode. However, if you want to use the syslog-ng 4 features, you need to be aware of some significant changes. If you have a simple configuration, like those in Linux distributions, then simply rewriting the version string is most likely enough. However, if you use PatternDB or JSON parsing, any Python code, or an Elasticsearch, or MongoDB destination, you have to be aware of the changes. >From this blog you can learn about type support, how this can affect you, changes in Python support, and some tips how to prepare for the upgrade. https://www.syslog-ng.com/community/b/blog/posts/upgrade-problems-from-sysl… WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

using -dv on startup vs syslog-ng-ctl debug set=off
by Jerry Scharf 28 Apr '23

28 Apr '23

Hi, We are chasing an intermittent bug with slow reloads. We are following the debug suggestions and adding -dv to the execstart in the service file. Because of the load of messages that we push, I want to turn off the debugging after we are past the window when the slow reload happens. I naively tried doing "syslog-ng-ctl debug set=off" but it responded that the flag didn't change. Is there some way to tell syslog-ng to allow the -ctl command to make the change. I can imagine why this would be desired sometimes, but at 10k lines per minute it is going to fill up /tmp. right now I am doing ugly things to solve the problem, but really want to find a better way. thanks, jerry -- Jerry Scharf Pure Storage

2 1

What is the status of syslog-ng 4 in your environment?
by Peter Czanik (pczanik) 27 Apr '23

27 Apr '23

Hi, Version 4 of syslog-ng was released almost half a year ago. It adds type support to name-value pairs, and lots of improvements to Python and Kubernetes support. I just started a Twitter / Mastodon / LinkedIn poll to learn, what is the status of syslog-ng 4 in your environment? * https://fosstodon.org/@PCzanik/110269557855094446 * https://twitter.com/sngose/status/1651486235469201408 * https://www.linkedin.com/posts/peterczanik_poll-syslogabrng-python-activity… Of course, I'd also be happy to hear some feedback, not just a poll result. So, let me know why you are on syslog-ng 4 already, or why do you delay your upgrade to this new major version! If hearing about syslog-ng 4 completely surprised you, check https://www.syslog-ng.com/community/b/blog/posts/upgrade-problems-from-sysl… which describes some of the major changes, and what to be careful about when you upgrade! Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Insider 2023-04: FreeBSD; Splunk; Deduplication;
by Peter Czanik (pczanik) 20 Apr '23

20 Apr '23

Dear syslog-ng users, This is the 109th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Installing a syslog-ng 4 development snapshot on FreeBSD -------------------------------------------------------- Unless there is a serious problem, FreeBSD ports usually contains the latest stable syslog-ng release. However, sometimes people want to compile a git snapshot to test a new feature or bugfix. To do that, one way is to generate a syslog-ng release tgz on FreeBSD and edit the syslog-ng port files yourself. However, this needs some practice. As such, an easier solution is to use my weekly development snapshots. https://www.syslog-ng.com/community/b/blog/posts/installing-a-syslog-ng-4-d… Getting data to Splunk ---------------------- Getting data to Splunk can be challenging. Syslog is still the most important data source, and it can provide you with hard-to-solve problems (for example, like high volume, non-compliant messages, unreliable network protocol (UDP), and more). The syslog-ng Premium Edition (PE) and syslog-ng Store Box (SSB) by One Identity can make these challenges manageable. https://www.syslog-ng.com/community/b/blog/posts/getting-data-to-splunk Streaming deduplication in syslog-ng ------------------------------------ Log volumes are growing 25% year over year, which means they are doubling every three years. Considering that SIEMs and other log processing tools are licensed based on volume, tools and mechanisms to make log storage and processing more efficient are very much sought for. Lean, how syslog-ng can help you. https://syslog-ng-future.blog/streaming-deduplication-in-syslog-ng/ WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Forwarding locally stored RAW messages
by Dragan Zecevic 13 Apr '23

13 Apr '23

Hi, we have syslog-ng Open Source Edition 3.33 and we are storing syslog messages from some systems into log files locally on a partition on syslog-ng server. Each day those log files are compressed. In order to better analyze some logs if needed we would like to extract some log files and ingest them to SIEM. Is it possible to make some forwarder that will read these RAW syslog messages from a log file and send them via syslog to SIEM? Thank you. Br, Dragan

2 1

TLS "trusted-dn" Question
by David Hauck 05 Apr '23

05 Apr '23

Hi, I'm currently using syslog-ng OSE v3.31.2 and trying to get a TLS configured endpoint to use the 'trusted-dn()' TLS option. I'm having trouble getting syslog-ng to return these DN specifiers in the Certificate Request option during the TLS negotiation so that clients can properly condition their supplied client certificates. I invariably see the following TLS negotiations (empty DNs list) in my Wireshark captures (as returned by the syslog-ng server): TLSv1.2 Record Layer: Handshake Protocol: Certificate Request Content Type: Handshake (22) Version: TLS 1.2 (0x0303) Length: 58 Handshake Protocol: Certificate Request Handshake Type: Certificate Request (13) Length: 54 Certificate types count: 3 Certificate types (3 types) Signature Hash Algorithms Length: 46 Signature Hash Algorithms (23 algorithms) Distinguished Names Length: 0 <----- always '0' In these cases my clients choose random client certificates that can't be refined to certificates signed by those expected (via the 'trusted-dn()' values) by the server and the connection is immediately closed. Here's the syslog-ng.conf entry for these sources: source s_515_tls { network( transport(tls) port(515) ip-protocol(6) tls(ca-dir("/etc/ssl/certs") key-file("/root/naservers.key") cert-file("/root/naservers.cer") peer-verify(required-trusted) trusted-dn("CN=*.netacquire.com")) ); }; I've tried several variants of the 'trusted-dn()' values, including other wildcards for country, state, etc. I always see a DNs list of zero size in the TLS Certificate Request option returned by the server. As expected switching to 'peer-verify(required-untrusted)' results in successful negotiation (with expected server-side errors indicating problems associated with the client certificates) and subsequent successful client/server logging. I figure I must be missing something obvious ;). Any ideas? Here's my syslog-ng version info: [logdest:~]# syslog-ng --version syslog-ng 3 (3.31.2) Config version: 3.29 Installer-Version: 3.31.2 Revision: Compile-Date: Nov 9 2021 12:52:59 Module-Directory: /usr/lib/syslog-ng Module-Path: /usr/lib/syslog-ng Include-Path: /usr/share/syslog-ng/include Available-Modules: tfgetent,mod-python,cryptofuncs,hook-commands,kvformat,cef,afuser,linux-kmsg-format,map-value-pairs,system-source,azure-auth-header,csvparser,affile,afprog,secure-logging,http,examples,timestamp,afsocket,confgen,stardate,dbparser,xml,disk-buffer,appmodel,afsnmp,add-contextual-data,afstomp,graphite,json-plugin,basicfuncs,syslogformat,pseudofile,tags-parser Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Systemd: off Thanks, -David

3 10

Learning syslog-ng
by Peter Czanik (pczanik) 05 Apr '23

05 Apr '23

Hi, Last year, one of the returning questions I received was how to learn syslog-ng. My answer was that read the first few chapters of the documentation, read my blogs related to your use case, and then read a few relevant parts from the rest of the documentation. Our documentation is praised by users, but it is still a reference documentation. I was asked if a less detailed, more to the point, preferably video tutorial is available. Your request was heard. In the past couple of months, I published a tutorial series in blog and video format, which brings you from basic logging concepts to using syslog-ng to collect, parse, enrich log messages and store them to Elasticsearch. Of course, these 5-10 minute videos are not enough to learn anything in depth, but they introduce you to all major syslog-ng functionalities. Even if you are a seasoned syslog-ng user, there is a good chance that you will learn something new from this introductory tutorial series: the “if” statement, in-line configuration elements, the inlist() filter or the JSON template function, just to name a few. If you have roughly 1.5 hours, then I recommend going through all the videos from the beginning to the end. You can reach the play list on YouTube at: https://www.youtube.com/playlist?list=PLoBNbOHNb0i5Pags2JY6-6wH2noLaSiTb If you would rather pick only a few topics from the tutorial series, here is a table of contents, with short summaries, pointers to the blog and video versions and the related parts from the documentation. Unfortunately, the documentation for the latest version is not available yet, pointers are included to the web version of the syslog-ng version 3.37 documentation: https://peter.czanik.hu/posts/syslog-ng-tutorial-toc/ Of course, once you read/watched my syslog-ng tutorials, reading the blogs and relevant parts of the documentation is still highly recommended. Please share this news if you know anyone who wants to learn syslog-ng! Bye, CzP Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Question on message parsing amd formats
by Generic Secure-Logging 30 Mar '23

30 Mar '23

I have a question on message parsing. From my unterstanding, if syslog-ng receives a message that is not in the correct RFC 5424 format, it will just output it as is. Does that mean that my application needs to generate the message in RFC 5424 format and syslog-ng will just process it? Or can syslog-ng be configured in such a way that an application just sends an unformatted string and syslog-ng will generate a valid RFC 5424 from it? How are RFC 5424 structured data elements treated by syslog-ng? Can it be configured using macros to generate them or does the application need to send a valid mesage containing the structured data elements in the right format? I am using syslog-ng 3.35.1. Regards, Stephan The information in this e-mail is confidential. The contents may not be disclosed or used by anyone other than the addressee. Access to this e-mail by anyone else is unauthorised. If you are not the intended recipient, please notify Airbus immediately and delete this e-mail. Airbus cannot accept any responsibility for the accuracy or completeness of this e-mail as it has been sent over public networks. If you have any concerns over the content of this message or its Accuracy or Integrity, please contact Airbus immediately. All outgoing e-mails from Airbus are checked using regularly updated virus scanning software but you should take whatever measures you deem to be appropriate to ensure that this message and any attachments are virus free.

1 0

pdbtool validate now requires type information.
by Evan Rempel 29 Mar '23

29 Mar '23

I'm working with syslog-ng-4.1.1-1 and attempting to validate a patterndb xml file. I understand that the pattern parsers will produce type aware values, but it seems that if the tests in the patterndb xml file do not contain any type information that all tests against non-string values will fail. Is that intended behavior to require type information on all tests? If that is by design I propose that this makes thing difficult because @NUMBER:var@ and @STRING:var@ will both successfully parse the same value out of the syslog message, but the test would now have to be different. Tools that programmatically produce the patterndb need to become a lot smarter. Can anyone comment on this behavior? When attempting to validate the attached patterndb.xml file I get the error Testing message: program='xinetd' message='EXIT: gsiftp status=0 pid=23133 duration=0(sec)' Wrong match name='status', value='0', type='integer', expected='0', expected_type='string' Wrong match name='sec', value='0', type='integer', expected='0', expected_type='string' Wrong match name='pid', value='23133', type='integer', expected='23133', expected_type='string' -- Evan

2 2