August 2023 - syslog-ng - lists.balabit.hu

Syslog-ng Not Working properly
by Sumanta Banerjee 15 Jan '24

15 Jan '24

Hi Team, I am trying to configure syslog-ng in one our linux instance to get NGIPS/FMC data via udp connection on its default port (514). I have configured syslog-ng.conf under /etc/syslog-ng and then we have set SE Linux as Permissive. I am using RHEL 8.7 and syslog version 4.0. Apparently all looked good to me however while checking in the destination path that is mentioned I don't see any directory or logfile from for the said udp connection got created. Below is our observation and steps that we executed, can any of you please help me telling where I went wrong or if I am missing something, there is another testing in pipeline that is stalled for this - 1. Define source, destination and log_file in syslog-ng.conf (file attached). 2. Run the below SELinux command - # ausearch -c 'syslog-ng' --raw | audit2allow -M my-syslogng # semodule -X 300 -i my-syslogng.pp 1. Restart syslog-ng service - # systemctl restart syslog-ng.service (no error message received) 1. Check if the service is running - [cid:image002.png@01D99309.5147A9A0] 1. Check if syslog-ng is listening to udp port 514 - [cid:image003.png@01D99309.5147A9A0] 1. Checked and we have incoming data stream from source using the below command - tcpdump -i any -c10 -nn -A port 514 1. I have went through the syslog-ng troubleshooting steps mentioned in the link - https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edi… [cid:image004.png@01D9930B.D0B70940] syslog-ng -Fdev command output is also attached. 1. While running the following command got the below output - # watch '/usr/sbin/syslog-ng-ctl stats | grep "^center"' [cid:image005.png@01D99311.15E497D0] 1. # journaltctl command output (first 500 lines) attached 1. Current SE Linux status : [cid:image006.png@01D9931C.B8765370] 1. Our syslog-ng is logging to /var/log/messages and we are getting this message in /var/log/messages - [cid:image007.png@01D9931E.592725A0] Thanks & Regards, Sumanta Banerjee Splunk Admin | CISO | Aviva Group Tel: +91-8420892593 24x7x365: +44 1603 208 582 sumanta.banerjee(a)aviva.com<mailto:sumanta.banerjee@aviva.com> GlobalCyberSecurityEngineeringTeam(a)aviva.com<mailto:GlobalCyberSecurityEngineeringTeam@aviva.com> www.aviva.com<http://www.aviva.com> Wipro Technologies - SJP2, Bangalore, India [cid:image001.gif@01D99303.20AF3AC0] Aviva: Internal Aviva plc, registered Office: St. Helen's, 1 Undershaft, London EC3P 3DQ. Registered in England No. 02468686. www.aviva.com This message and any attachments may be confidential or legally privileged. If you are not the intended recipient, please telephone or e-mail the sender and delete this message and any attachments from your system. Also, if you are not the intended recipient you must not copy this message or attachments or disclose the contents to any other person. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Aviva.

2 4

How to enable Syslog-ng authentication with MongoDB
by Ibrahim Al Mahfooz 20 Aug '23

20 Aug '23

Hi, I would like to harden the security by enabling authentication between syslog-ng and MongoDB. I'm using syslog-ng v3.38 and MongoDB v6.0.8 At the destination I tried the below two methods but none of themt worked, am I doing something wrong? Or the authentication is not supported? *---> First method:* destination d_mongodb_01 { mongodb( uri("mongodb://SampleUser:SamplePass@172.16.17.16:27017/syslog-ng_01") collection("logs") value-pairs( pair("datetime" "${S_ISODATE}") pair("privateip" "${privateip}") pair("portstatus" "${PORTSTATUS}") pair("host" "${HOST}")) ); }; *---> Second method:* destination d_mongodb_01 { mongodb( uri("mongodb://172.16.17.16:27017/syslog-ng_01") username("SampleUser") password("SamplePass") collection("logs") value-pairs( pair("datetime" "${S_ISODATE}") pair("privateip" "${privateip}") pair("portstatus" "${PORTSTATUS}") pair("host" "${HOST}")) ); }; Appreciate your help Best regards

2 2

discussion: Should ptbtool test validate types
by Evan Rempel 19 Aug '23

19 Aug '23

Now that syslog-ng has type hinting should the pdbtool test also validate the types of any values specified for a pattern. <rule id="pid1" class="myclass" provider="me"> <patterns> <pattern>some pattern @NUMBER:mynum@ with number</pattern> </patterns> <tags> <tag>rate</tag> </tags> <values> <value name="more" type="integer">100A</value> </values> <examples> <example> <test_message program="some pattern 123 with number</test_message> <test_values> <test_value name="mynum" type="integer">123</test_value> </test_values> </example> </examples> </rule> Currently the test_values are verified for both the value, and the type that the pattern parses, however, none of the values that are used to enrich this message are verified for type. This results in a patterndb that passes validation, yet will fail at the time the pattern matches a log message. comments? -- Evan

2 1

working with Sigma rules?
by Peter Czanik (pczanik) 11 Aug '23

11 Aug '23

Hi, Recently I was asked if Sigma rules (https://github.com/SigmaHQ/sigma) are supported by syslog-ng. Syslog-ng has message parsing, filtering, and can be used for alerting. But I'm not aware of any tools that could turn Sigma rules into PatternDB and syslog-ng.conf Syslog-ng can send logs to Splunk, ElasticSearch / OpenSearch or Graylog, all which already have sigma rules integrations. Of course, many users use/abuse syslog-ng as a kind of SIEM-lite as it is very good at real-time alerting. However, as far as I can see, Sigma rules are better suited for threat hunting on the SIEM side. If you already Sigma rules with syslog-ng or any other way: please share your experiences! Thanks, Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

2 1

Insider 2023-08: sngbench; 4.3.1 release; syslog-ng 4; Python;
by Peter Czanik (pczanik) 10 Aug '23

10 Aug '23

Dear syslog-ng users, This is the 111th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Introducing sngbench: a shell script to performance test your syslog-ng ---------------------------------- One of the returning questions I receive is how many log messages can a given hardware handle. My typical answer is that it depends on the configuration. I have now an answer, or rather a tool to answer your question sngbench.sh. It is a shell script that runs from localhost and uses loggen, the bundled benchmarking and testing tool of syslog-ng. It comes with two configurations: a performance-optimized and a realistic one. You are also free to extend sngbench with your own configurations. https://www.syslog-ng.com/community/b/blog/posts/introducing-sngbench-a-she… Version 4.3.1 of syslog-ng available ------------------------------------ Receive or send any kinds of logs, traces, and metrics with syslog-ng using OTLP/gRPC, improved performance for high-volume connections, and more! Containers and packages for major Linux distributions and FreeBSD are already available. https://axoflow.com/axosyslog-release-4-3/ For the complete release notes, check: https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-4.3.1 Syslog-ng Python packaging -------------------------- In version 4 of syslog-ng, the role of Python became even more important. Previously, all parts of syslog-ng could be extended using Python code, but no actual Python code was provided with syslog-ng. Version 4.0 added a Kubernetes module implemented in Python, while version 4.2 added support for Hypr. But how can we ensure that all Python dependencies are met? https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-python-packaging Getting syslog-ng 4 ------------------- Version 4 of syslog-ng was released last December. Quite a few people use it already in production. How can you install it for a test drive? It might be already available in your Linux distribution. There are also several unofficial repositories with the latest syslog-ng. >From this blog, you can learn how to check your syslog-ng version, where to check if it is not yet installed, and a few additional resources, if you want to install the latest version from unofficial repositories. https://www.syslog-ng.com/community/b/blog/posts/getting-syslog-ng-4 WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Need help to send Datetime in BSON ISODATE to MongoDB?
by Ibrahim Al Mahfooz 07 Aug '23

07 Aug '23

Hello, I have integration working fine between syslog-ng v3.38 and MongoDB v6. I am working to enable MongoDB Time-Series Collections on a testing DB for the purpose of having better efficiency and improved queries over time. I created the DB and collection, tested it by Mongo Compass client, then I tried to send the data from syslog-ng but it didn't work. MongoDB is throwing the following error: "17:12:57. 483 | [2023-08-06T17:12:57.355549] Failed to insert into MongoDB; time_reopen='10', reason='\'datetime\' must be present and contain a valid BSON UTC datetime value', driver='d_mongodb_ttl#0" My understanding is that it is a must to send the time/date to MongoDB with Time Series enabled DB a BSON format Date. Any suggestions? *Below is the mongoDB destination config:* destination d_mongodb_ttl { mongodb( uri("mongodb://192.168.1.19:27017/syslog-ng_ttl") collection("logs") value-pairs( scope("selected-macros") exclude("SOURCEIP") exclude("TAGS") pair("datetime" "${S_ISODATE}") pair("privateip" "${privateip}") pair("publicip" "${publicip}") pair("publicportrange" "${publicportrange}") pair("portaction" "${AI}") pair("portstatus" "${AL}") pair("host" "${HOST}")) ); }; Appreciate your help

3 4

syslog-ng 4.3.1
by László Várady 03 Aug '23

03 Aug '23

Dear syslog-ng users, We are pleased to announce the 4.3.1 version of syslog-ng, which has been released and is now available on GitHub: https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-4.3.1 Packages <https://github.com/syslog-ng/syslog-ng#installation-from-binaries> are also available for various platforms. The AxoSyslog project provides cloud-ready container images <https://axoflow.com/cloud-ready-syslog-ng-images/>, Helm charts <https://axoflow.com/axosyslog-log-collection-for-kubernetes/>, and open-source documentation at the link below: https://axoflow.com/docs/axosyslog-core/ 4.3.1 *This is the combination of the news entries of 4.3.0 and 4.3.1. 4.3.1 hotfixed* *a python-parser() related crash and a metrics related memory leak. It also* *added Ubuntu 23.04 and Debian 12 support for APT packages and the opensearch()* *destination.* Read Axoflow's blog post <https://axoflow.com/axosyslog-release-4-3/> for more details. Highlights parallelize() support for pipelines syslog-ng has traditionally performed processing of log messages arriving from a single connection sequentially. This was done to ensure message ordering as well as most efficient use of CPU on a per message basis. This mode of operation is performing well as long as we have a relatively large number of parallel connections, in which case syslog-ng would use all the CPU cores available in the system. In case only a small number of connections deliver a large number of messages, this behaviour may become a bottleneck. With the new parallelization feature, syslog-ng gained the ability to re-partition a stream of incoming messages into a set of partitions, each of which is to be processed by multiple threads in parallel. This does away with ordering guarantees and adds an extra per-message overhead. In exchange it will be able to scale the incoming load to all CPUs in the system, even if coming from a single, chatty sender. To enable this mode of execution, use the new parallelize() element in your log path: log { source { tcp( port(2000) log-iw-size(10M) max-connections(10) log-fetch-limit(100000) ); }; parallelize(partitions(4)); # from this part on, messages are processed in parallel even if # messages are originally coming from a single connection parser { ... }; destination { ... }; }; The config above will take all messages emitted by the tcp() source and push the work to 4 parallel threads of execution, regardless of how many connections were in use to deliver the stream of messages to the tcp() driver. parallelize() uses round-robin to allocate messages to partitions by default. You can however retain ordering for a subset of messages with the partition-key() option. You can use partition-key() to specify a message template. Messages that expand to the same value are guaranteed to be mapped to the same partition. For example: log { source { tcp( port(2000) log-iw-size(10M) max-connections(10) log-fetch-limit(100000) ); }; parallelize(partitions(4) partition-key("$HOST")); # from this part on, messages are processed in parallel if their # $HOST value differs. Messages with the same $HOST will be mapped # to the same partition and are processed sequentially. parser { ... }; destination { ... }; }; NOTE: parallelize() requires a patched version of libivykis that contains this PR buytenh/ivykis#25 <https://github.com/buytenh/ivykis/pull/25>. syslog-ng source releases bundle this version of ivykis in their source trees, so if you are building from source, be sure to use the internal version (--with-ivykis=internal). You can also use Axoflow's cloud native container image for syslog-ng, named AxoSyslog (https://github.com/axoflow/axosyslog-docker) which also incorporates this change. (#3966 <https://github.com/syslog-ng/syslog-ng/pull/3966>) Receiving and sending OpenTelemetry (OTLP) messages The opentelemetry() source, parser and destination are now available to receive, parse and send *OTLP/gRPC* messages. syslog-ng accepts logs <https://github.com/open-telemetry/opentelemetry-proto/blob/v0.20.0/opentele…>, metrics <https://github.com/open-telemetry/opentelemetry-proto/blob/v0.20.0/opentele…> and traces <https://github.com/open-telemetry/opentelemetry-proto/blob/v0.20.0/opentele…> . The incoming fields are not available through syslog-ng log message name-value pairs for the user by default. This is useful for forwarding functionality (the opentelemetry() destination can access and format them). If such functionality is required, you can configure the opentelemetry() parser, which maps all the fields with some limitations. The behavior of the opentelemetry() parser is the following: The name-value pairs always start with .otel. prefix. The type of the message is stored in .otel.type (possible values: log, metric and span). The resource info is mapped to .otel.resource.<...> (e.g.: .otel.resource.dropped_attributes_count, .otel.resource.schema_url ...), the scope info is mapped to .otel.scope.<...> (e.g.: .otel.scope.name, .otel.scope.schema_url, ...). The fields of log records are mapped to .otel.log.<...> (e.g. .otel.log.body, .otel.log.severity_text, ...). The fields of metrics are mapped to .otel.metric.<...> (e.g. . otel.metric.name, .otel.metric.unit, ...), the type of the metric is mapped to .otel.metric.data.type (possible values: gauge, sum, histogram, exponential_histogram, summary) with the actual data mapped to .otel.metric.data.<type>.<...> (e.g.: .otel.metric.data.gauge.data_points.0.time_unix_nano, ...). The fields of traces are mapped to .otel.span.<...> (e.g. .otel.span.name, .otel.span.trace_state, ...). repeated fields are given an index (e.g. .otel.span.events.5.time_unix_nano ). The mapping of AnyValue <https://github.com/open-telemetry/opentelemetry-proto/blob/v0.20.0/opentele…> type fields is limited. string, bool, int64, double and bytes values are mapped with the respective syslog-ng name-value type (e.g. .otel.resource.attributes.string_key => string_value), however ArrayValue and KeyValueList types are stored serialized with protobuf type. protobuf and bytes types are not directly available for the user, unless an explicit type cast is added (e.g. "bytes(${.otel.log.span_id})") or --include-bytes is passed to name-value iterating template functions (e.g. $(format-json .otel.* --include-bytes), which will base64 encode the bytes content). Three authentication methods are available in the source auth() block: insecure() (default), tls() and alts(). tls() accepts the key-file(), cert-file(), ca-file() and peer-verify() (possible values: required-trusted, required-untrusted, optional-trusted and optional-untrusted) options. ALTS <https://grpc.io/docs/languages/cpp/alts/> is a simple to use authentication, only available within Google's infrastructure. The same methods are available in the destination auth() block, with two differences: tls(peer-verify()) is not available, and there is a fourth method, called ADC <https://cloud.google.com/docs/authentication/application-default-credentials>, which accepts the target-service-account() option, where a list of service accounts can be configured to match against when authenticating the server. Example configs: log otel_forward_mode_alts { source { opentelemetry( port(12345) auth(alts()) ); }; destination { opentelemetry( url("my-otel-server:12345") auth(alts()) ); }; }; log otel_to_non_otel_insecure { source { opentelemetry( port(12345) ); }; parser { opentelemetry(); }; destination { network( "my-network-server" port(12345) template("$(format-json .otel.* --shift-levels 1 --include-bytes)\n") ); }; }; log non_otel_to_otel_tls { source { network( port(12346) ); }; destination { opentelemetry( url("my-otel-server:12346") auth( tls( ca-file("/path/to/ca.pem") key-file("/path/to/key.pem") cert-file("/path/to/cert.pem") ) ) ); }; }; (#4523 <https://github.com/syslog-ng/syslog-ng/pull/4523>) (#4510 <https://github.com/syslog-ng/syslog-ng/pull/4510>) Sending messages to CrowdStrike Falcon LogScale (Humio) The logscale() destination feeds LogScale via the Ingest API <https://library.humio.com/falcon-logscale/api-ingest.html#api-ingest-struct…> . Minimal config: destination d_logscale { logscale( token("my-token") ); }; Additional options include: - url() - rawstring() - timestamp() - timezone() - attributes() - extra-headers() - content-type() (#4472 <https://github.com/syslog-ng/syslog-ng/pull/4472>) Features - afmongodb: Bulk MongoDB insert is added via the following options - bulk (*yes*/no) turns on/off bulk insert <http://mongoc.org/libmongoc/current/bulk.html>usage, no forces the old behavior (each log is inserted one by one into the MongoDB) - bulk_unordered (yes/*no*) turns on/off unordered MongoDB bulk operations <http://mongoc.org/libmongoc/current/bulk.html#unordered-bulk-write-operatio…> - bulk_bypass_validation (yes/*no*) turns on/off MongoDB bulk operations validation <http://mongoc.org/libmongoc/1.23.3/bulk.html#bulk-operation-bypassing-docum…> - write_concern (unacked/*acked*/majority/n > 0) sets write concern mode of the MongoDB operations <http://mongoc.org/libmongoc/1.23.3/bulk.html#bulk-operation-write-concerns>, both bulk and single NOTE: Bulk sending is only efficient if the used collection is constant (e.g. not using templates) or the used template does not lead to too many collections switching within a reasonable time range. (#4483 <https://github.com/syslog-ng/syslog-ng/pull/4483>) - sql: Added 2 new options - quote_char to aid custom quoting for table and index names (e.g. MySQL needs sometimes this for certain identifiers) *NOTE*: Using a back-tick character needs a special formatting as syslog-ng uses it for configuration parameter names, so for that use: quote_char("``") (double back-tick) - dbi_driver_dir to define an optional DBI driver location for DBD initialization NOTE: libdbi and libdbi-drivers OSE forks are updated, afsql now should work nicely both on ARM and X86 macOS systems too (tested on macOS 13.3.1 and 12.6.4) Please do not use the pre-built ones (e.g. 0.9.0 from Homebrew), build from the *master* of the following - https://github.com/balabit-deps/libdbi-drivers - https://github.com/balabit-deps/libdbi (#4460 <https://github.com/syslog-ng/syslog-ng/pull/4460>) - opensearch: Added a new destination. It is similar to elasticsearch-http(), with the difference that it does not have the type() option, which is deprecated and advised not to use. (#4560 <https://github.com/syslog-ng/syslog-ng/pull/4560>) Bugfixes - network(),syslog(),tcp() destination: fix TCP keepalive tcp-keepalive-*() options were broken on the destination side since v3.34.1. (#4559 <https://github.com/syslog-ng/syslog-ng/pull/4559>) - Fixed a hang, which happend when syslog-ng received exremely low CPU time. (#4524 <https://github.com/syslog-ng/syslog-ng/pull/4524>) - $(format-json): Fixed a bug where sometimes an unnecessary comma was added in case of a type cast failure. (#4477 <https://github.com/syslog-ng/syslog-ng/pull/4477>) - Fix flow-control when fetch-limit() is set higher than 64K In high-performance use cases, users may configure log-iw-size() and fetch-limit() to be higher than 2^16, which caused flow-control issues, such as messages stuck in the queue forever or log sources not receiving messages. (#4528 <https://github.com/syslog-ng/syslog-ng/pull/4528>) - int32() and int64() type casts: accept hex numbers as proper number representations just as the @NUMBER@ parser within db-parser(). Supporting octal numbers were considered and then rejected as the canonical octal representation for numbers in C would be ambigious: a zero padded decimal number could be erroneously considered octal. I find that log messages contain zero padded decimals more often than octals. (#4535 <https://github.com/syslog-ng/syslog-ng/pull/4535>) - Fixed compilation on platforms where SO_MEMINFO is not available (#4548 <https://github.com/syslog-ng/syslog-ng/pull/4548>) - python: InstantAckTracker, ConsecutiveAckTracker and BatchedAckTracker are now called properly. Added proper fake classes for the InstantAckTracker, ConsecutiveAckTracker and BatchedAckTracker classes, and the wapper now calls the super class' constructor. Previusly the super class' constructor was not called which caused the python API to never call into the C API, which's result was that that the callback was never called. (#4549 <https://github.com/syslog-ng/syslog-ng/pull/4549>) - python: Fixed a crash when reloading with a config, which uses a python parser with multiple references. (#4552 <https://github.com/syslog-ng/syslog-ng/pull/4552>) (#4567 <https://github.com/syslog-ng/syslog-ng/pull/4567>) - mqtt(): Fixed the name of the stats instance (mqtt-source) to conform to the standard comma-separated format. (#4551 <https://github.com/syslog-ng/syslog-ng/pull/4551>) - metrics: Fixed a memory leak which happened during reload, and was introduced in 4.3.0. (#4568 <https://github.com/syslog-ng/syslog-ng/pull/4568>) Packaging - scl.conf: The scl.conf file has been moved to /share/syslog-ng/include/scl.conf (#4534 <https://github.com/syslog-ng/syslog-ng/pull/4534>) - C++ plugins: Some of syslog-ng's plugins now contain C++ code. By default they are being built if a C++ compiler is available. Disabling it is possible with --disable-cpp. Affected plugins: - lib/syslog-ng/libexamples.so - --disable-cpp will only disable the C++ part ( random-choice-generator()) - lib/syslog-ng/libotel.so (#4484 <https://github.com/syslog-ng/syslog-ng/pull/4484>) - debian: A new module is added, called syslog-ng-mod-grpc. Its dependencies are: protobuf-compiler, protobuf-compiler-grpc, libprotobuf-dev, libgrpc++-dev. Building the module can be toggled with --enable-grpc. (#4510 <https://github.com/syslog-ng/syslog-ng/pull/4510>) - pcre: syslog-ng now uses pcre2 (8 bit) as a dependency instead of pcre. The minimum pcre2 version is 10.0. (#4537 <https://github.com/syslog-ng/syslog-ng/pull/4537>) Notes to developers - lib/logmsg: Public field LogMessage::protected has been renamed to LogMessage::write_protected. Direct usage of this field is discouraged, instead use the following functions: - log_msg_is_write_protected() - log_msg_write_protect() (#4484 <https://github.com/syslog-ng/syslog-ng/pull/4484>) - lib/templates: Public field LogTemplate::template has been renamed to LogTemplate::template_str. (#4484 <https://github.com/syslog-ng/syslog-ng/pull/4484>) Other changes - syslog-ng-cfg-db: Moved to a separate repository. It is available at: https://github.com/alltilla/syslog-ng-cfg-helper (#4475 <https://github.com/syslog-ng/syslog-ng/pull/4475>) - disk-buffer: Added alternative option names disk-buf-size() -> capacity-bytes() qout-size() -> front-cache-size() mem-buf-length() -> flow-control-window-size() mem-buf-size() -> flow-control-window-bytes() Old option names are still available. Example configs: tcp( "127.0.0.1" port(2001) disk-buffer( reliable(yes) capacity-bytes(1GiB) flow-control-window-bytes(200MiB) front-cache-size(1000) ) ); tcp( "127.0.0.1" port(2001) disk-buffer( reliable(no) capacity-bytes(1GiB) flow-control-window-size(10000) front-cache-size(1000) ) ); (#4526 <https://github.com/syslog-ng/syslog-ng/pull/4526>) - selinux: Added RHEL9 support for the selinux policies Added RHEL9 support for the selinux policies at contrib/selinux (#4509 <https://github.com/syslog-ng/syslog-ng/pull/4509>) - metrics: replace driver_instance (stats_instance) with metric labels The new metric system had a label inherited from legacy: driver_instance. This non-structured label has been removed and different driver-specific labels have been added instead, for example: Before: syslogng_output_events_total{driver_instance="mongodb,localhost:27017,defaultdb,,coll",id="#anon-destination1#1",result="queued"} 4 After: syslogng_output_events_total{driver="mongodb",host="localhost:27017",database="defaultdb",collection="coll",id="#anon-destination1#1",result="queued"} 4 This change may affect legacy stats outputs (syslog-ng-ctl stats), for example, persist-name()-based naming is no longer supported in this old format. (#4551 <https://github.com/syslog-ng/syslog-ng/pull/4551>) - APT packages: Added Ubuntu Lunar Lobster and Debian Bookworm support. (#4561 <https://github.com/syslog-ng/syslog-ng/pull/4561>) syslog-ng Discord For a bit more interactive discussion, join our Discord server: [image: Axoflow Discord Server] <https://discord.gg/E65kP9aZGm> Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andreas Friedmann, Attila Szakacs, Balazs Scheidler, Bálint Horváth, Chuck Silvers, Evan Rempel, Hofi, Kovacs, Gergo Ferenc, László Várady, Romain Tartière, Ryan Faircloth, vostrelt

1 0

Properly Setting Facility and Severity into destination
by Greg Christopher 01 Aug '23

01 Aug '23

Hi All, Did my best (about a week effort) to find this information online and apologize if it's obvious. I am trying to use syslog-ng as a client to send messages to a syslog compliant server with the proper severity and facility set. I am starting with plain-text log files. The syslog rfc as well as the syslog-ng documentation are pretty clear about the syslog message format itself. There are even syslog-ng functions to substitute severity and facility using a function called "rewrite". But this doesn't seem to work if you are starting with a regular application log. In other words, there is nothing to "rewrite" since this header was never there to begin with. Although I initially configured my application log with flags(no-parse) , I attempted to remove it so that my rewrite attempt would at least have a facility and severity to "rewrite", but this also seems to have failed. Next I attempted prepending "<81>" (8*10 + 1) to the application messages to get the right priority field but this failed to change what the syslog-ng server on the other end received, as I have modified its output to include severity and facility:--- destination d_local { file("/var/log/syslog-ng/$HOST" template("Fac: $FACILITY Pri: $PRIORITY $ISODATE $HOST $(format-welf --scope all-nv-pairs)\n") frac-digits(3)); }; --- It does seem to be working properly but the above two approaches to increasing the severity and noting this is a security facility message failed. So it seems this use case where we create syslog entries from scratch sort of throws a curve ball at syslog-ng clients, at least from what I can figure. Any help on how I can set these manually into the destination would be greatly appreciated. Here is my syslog-ng.conf at the moment; note I pull log file name out of environment variable:--- @version: 4.1 # Configure the source to read from the messages log file # no-parse means whole line becomes "message" portion. # Template should indicate security alert to syslog daemon source s_APP_LOG { file(`APP_LOG`); # file(`APP_LOG` flags(no-parse)); }; rewrite set_pri_fields { set-severity("alert"); set-facility("security"); }; destination d_destination { syslog(`syslogServer`); }; # Configure the log statement to route messages from the application log file to # the syslog server specified on the command line log { source(s_APP_LOG); rewrite(set_pri_fields); destination(d_destination); }; ---Greg

2 2

parallelize: modified ivykis for Fedora / RHEL
by Peter Czanik (pczanik) 01 Aug '23

01 Aug '23

Hi, Version 4.3.1 of syslog-ng was released last weekend and it contains an interesting feature: parallelize(). TL;DR: it allows faster processing of single, high-traffic TCP connections. You can read more about it in my latest blog at: https://www.syslog-ng.com/community/b/blog/posts/accelerating-single-tcp-co… The problem is that it requires a modified ivykis library. The syslog-ng source code bundles ivykis, and has a modified version. However, all Fedora and RHEL syslog-ng packages use a separate ivykis package. I feel uncomfortable with modifying the "official" ivykis package in Fedora / RHEL with patches not yet in the upstream ivykis sources. And it would not help the majority of syslog-ng users anyway, who use RHEL 8 and compatible operating systems. I could only add it to Fedora and upcoming RHEL releases. However, I want to enable this feature for the majority of syslog-ng users. Plan: add a modified ivykis library to my unofficial repositories. The version would stay 0.42.4, but the release would be a high number, making sure that it stays larger than the "official" package. First I'd introduce it to my git snapshot repository, and once I collect some positive feedback, I implement it in other repos too for upcoming syslog-ng versions. What do you think? Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

3 2