- syslog-ng - lists.balabit.hu

Sending json logs with syslog-ng to Splunk via HEC
by Carlos Lopez 28 Sep '22

28 Sep '22

Hi all, I am trying to configure syslog-ng to send events in json format to our internal Splunk (free version) server. Searching info regarding how to accomplish this config, I see these entries in Balabit’s blog: https://www.syslog-ng.com/community/b/blog/posts/optimize-your-splunk-infra… https://www.syslog-ng.com/community/b/blog/posts/sending-logs-splunk-http Are these entries accurate to accomplish this config? Somebody can share any sample? I am using syslog-ng’ OSS under FreeBSD 13.1 hosts … Best regards, C. L. Martinez

1 0

Insider 2022-09: 3.38; SQL; disk-buffer; nightly;
by Peter Czanik (pczanik) 08 Sep '22

08 Sep '22

Dear syslog-ng users, This is the 104th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS 3.38.1 released, 4.0 almost feature complete -------------------------------------------- syslog-ng 4 is right around the corner. Instead of a pile of breaking changes, we choose to improve syslog-ng in an evolutionary manner: providing fine grained compatibility with older versions along the way, so that syslog-ng 4 remains a drop-in replacement for any earlier release in the past 15 years. You can test many 4.0 features already in the fresh 3.38.1 release: https://syslog-ng-future.blog/syslog-ng-4-progress-3-38-1-release/ syslog-ng Store Box SQL source ------------------------------ The syslog-ng Store Box (SSB) is an appliance built on syslog-ng Premium Edition (PE). SSB inherits most of syslog-ng PE’s features and makes them available with an easy-to-use graphical user interface. One of these features is the SQL source. Many applications log their messages to SQL databases. The SQL source of syslog-ng PE and SSB allows you to collect these log messages in real-time and process and store them with the rest of your logs. https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-store-box-sql-so… Why is my syslog-ng disk-buffer file so huge even when it is empty? --------------------------------------- Most people expect to see how many log messages are waiting in the disk-buffer from the size of the syslog-ng disk-buffer file.. While it was mostly true for earlier syslog-ng releases, for recent syslog-ng releases (3.34+) the disk-buffer file can stay large even when it is empty. This is a side effect of a recent syslog-ng performance tuning. https://www.syslog-ng.com/community/b/blog/posts/why-is-my-syslog-ng-disk-b… Nightly syslog-ng builds for Debian and Ubuntu ---------------------------------------------- The syslog-ng team made nightly builds available for Debian and Ubuntu running on the AMD64 architecture. These packages are not intended to be used in production, they are only for testing fixes and new features. https://www.syslog-ng.com/community/b/blog/posts/nightly-syslog-ng-builds-f… WEBINARS * Register for our latest webinars at https://www.syslog-ng.com/event/syslogng-webcast-series-2022/ * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

syslog-ng-3.38.1
by Laszlo Varady (lvarady) 29 Aug '22

29 Aug '22

3.38.1 <https://github.com/syslog-ng/syslog-ng/blob/master/NEWS.md#highlights> Highlights <https://github.com/syslog-ng/syslog-ng/blob/master/NEWS.md#sneak-peek-into-…> Sneak peek into syslog-ng v4.0 syslog-ng v4.0 is right around the corner. This release (v3.38.1) contains all major changes, however, they are currently all hidden behind a feature flag. To enable and try those features, you need to specify @version: 4.0 at the top of the configuration file. You can find out more about the 4.0 changes and features here<https://github.com/syslog-ng/syslog-ng/blob/master/NEWS-4.0.md>. Read our practical introduction to typing at syslog-ng-future.blog<https://syslog-ng-future.blog/syslog-ng-4-progress-3-38-1-release/>. <https://github.com/syslog-ng/syslog-ng/blob/master/NEWS.md#features> Features * grouping-by(): added inject-mode(aggregate-only) This inject mode will drop individual messages that make up the correlation context (key() groups) and would only yield the aggregate messages (e.g. the results of the correlation). (#3998<https://github.com/syslog-ng/syslog-ng/pull/3998>) * add-contextual-data(): add support for type propagation, e.g. set the type of name-value pairs as they are created/updated to the value returned by the template expression that we use to set the value. The 3rd column in the CSV file (e.g. the template expression) now supports specifying a type-hint, in the format of "type-hint(template-expr)". Example line in the CSV database: selector-value,name-value-pair-to-be-created,list(foo,bar,baz) (#4051<https://github.com/syslog-ng/syslog-ng/pull/4051>) * $(format-json): add --key-delimiter option to reconstruct JSON objects using an alternative structure separator, that was created using the key-delimiter() option of json-parser(). (#4093<https://github.com/syslog-ng/syslog-ng/pull/4093>) * json-parser(): add key-delimiter() option to extract JSON structure members into name-value pairs, so that the names are flattened using the character specified, instead of dot. Example: Input: {"foo":{"key":"value"}} Using json-parser() without key-delimiter() this is extracted to: foo.key="value" Using json-parser(key-delimiter("~")) this is extracted to: foo~key="value" This feature is useful in case the JSON keys contain dots themselves, in those cases the syslog-ng representation is ambigious. (#4093<https://github.com/syslog-ng/syslog-ng/pull/4093>) <https://github.com/syslog-ng/syslog-ng/blob/master/NEWS.md#bugfixes> Bugfixes * Fixed buffer handling of syslog and timestamp parsers Multiple buffer out-of-bounds issues have been fixed, which could cause hangs, high CPU usage, or other undefined behavior. (#4110<https://github.com/syslog-ng/syslog-ng/pull/4110>) * Fixed building with LibreSSL (#4081<https://github.com/syslog-ng/syslog-ng/pull/4081>) * network(): Fixed a bug, where syslog-ng halted the input instead of skipping a character in case of a character conversion error. (#4084<https://github.com/syslog-ng/syslog-ng/pull/4084>) * redis(): Fixed bug where using redis driver without the batch-lines option caused program crash. (#4114<https://github.com/syslog-ng/syslog-ng/pull/4114>) * pdbtool: fix a SIGABRT on FreeBSD that was triggered right before pdbtool exits. Apart from being an ugly crash that produces a core file, functionally the tool behaved correctly and this case does not affect syslog-ng itself. (#4037<https://github.com/syslog-ng/syslog-ng/pull/4037>) * regexp-parser(): due to a change introduced in 3.37, named capture groups are stored indirectly in the LogMessage to avoid copying of the value. In this case the name-value pair created with the regexp is only stored as a reference (name + length of the original value), which improves performance and makes such name-value pairs use less memory. One omission in the original change in 3.37 is that syslog-ng does not allow builtin values to be stored indirectly (e.g. $MESSAGE and a few of others) and this case causes an assertion to fail and syslog-ng to crash with a SIGABRT. This abort is now fixed. Here's a sample config that reproduces the issue: regexp-parser(patterns('(?<MESSAGE>.*)')); (#4043<https://github.com/syslog-ng/syslog-ng/pull/4043>) * set-tag: fix cloning issue when string literal were used (see #4062) (#4065<https://github.com/syslog-ng/syslog-ng/pull/4065>) * add-contextual-data(): fix high memory usage when using large CSV files (#4067<https://github.com/syslog-ng/syslog-ng/pull/4067>) <https://github.com/syslog-ng/syslog-ng/blob/master/NEWS.md#other-changes> Other changes * The json-c library is no longer bundled in the syslog-ng source tarball Since all known OS package managers provide json-c packages nowadays, the json-c submodule has been removed from the source tarball. The --with-jsonc=internal option of the configure script has been removed accordingly, system libraries will be used instead. For special cases, the JSON support can be disabled by specifying --with-jsonc=no. (#4078<https://github.com/syslog-ng/syslog-ng/pull/4078>) * platforms: Dropped support for ubuntu-impish as it became EOL (#4088<https://github.com/syslog-ng/syslog-ng/pull/4088>) <https://github.com/syslog-ng/syslog-ng/blob/master/NEWS.md#credits> Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Alvin Šipraga, Andras Mitzki, Attila Szakacs, Balazs Scheidler, Bálint Horváth, Daniel Klauer, Fabrice Fontaine, Gabor Nagy, HenryTheSir, László Várady, Parrag Szilárd, Peter Kokai, Shikhar Vashistha, Szilárd Parrag, Vivin Peris Installation from binaries: https://github.com/syslog-ng/syslog-ng#installation-from-binaries<https://github.com/syslog-ng/syslog-ng#installation-from-binaries<https://github.com/syslog-ng/syslog-ng#installation-from-binaries>

1 0

Potential issue with line breaking not working correctly when writing logs to disk
by John Law 23 Aug '22

23 Aug '22

Hi, We have a number of network devices sending syslog traffic to syslog-ng OSE 3.25 installed on RHEL 7.9. We are having an issue where multiple events are being written as the same line. The config file is template-function adm1 "${HOST} message-length=$(length \"${MSG}\") ${MESSAGE}\n"; source s_adm1 { udp(ip(0.0.0.0) port(5527) ); tcp(ip(0.0.0.0) port(5527) ); }; destination d_adm1 { file( "/Data/syslog_data/$LOGHOST/$R_YEAR-$R_MONTH-$R_DAY/adm1/$HOST_FROM/$HOST/$FACILITY.local" template("$(adm1-function)\n") ); }; log { source(s_adm1); destination(d_adm1); flags(final); }; A small extract of one of the files is, naturally I have changed the IPs 192.1.1.1/31181->192.1.3.1/135 0x0 source rule r12 N/A N/A 6 ACME-CNB010 ACME CNB 1727847 N/A(N/A) reth3.860 UNKNOWN UNKNOWN UNKNOWN275 <14>Aug 19 11:38:50 sunny-fwl29 RT_FLOW: RT_FLOW_SESSION_CREATE: session created 192.6.20.1/47250->192.104.20.1/443 0x0 junos-https 192.6.20.1/47250->192.104.20.1/443 0x0 N/A N/A N/A N/A 6 ACME-BWC002 ACME BWC_EDESIX_AWS 917879 N/A(N/A) reth3.860 UNKNOWN UNKNOWN UNKNOWN275 <14>Aug 19 11:38:50 sunny-fwl29 RT_FLOW: RT_FLOW_SESSION_CREATE: session created 192.6.20.1/47252->192.104.20.1/443 0x0 junos-https 192.6.20.1/47252->192.104.20.1/443 0x0 N/A N/A N/A N/A 6 ACME-BWC002 ACME BWC_EDESIX_AWS 595069 N/A(N/A) reth3.860 UNKNOWN UNKNOWN UNKNOWN285 <14>Aug 19 11:38:50 sunny-fwl29 RT_FLOW: RT_FLOW_SESSION_CREATE: session created 192.1.57.1/54205->192.1.7.1/137 0x0 junos-nbname 192.1.57.1/54205->192.1.7.1/137 0x0 N/A N/A N/A N/A 17 ACME-COMP007 ACME CCA 1735324 N/A(N/A) reth3.860 UNKNOWN UNKNOWN UNKNOWN288 <14>Aug 19 11:38:50 sunny-fwl29 RT_FLOW: RT_FLOW_SESSION_CREATE: session created 192.1.57.1/62486->192.1.7.1/135 0x0 junos-ms-rpc-tcp 192.1.57.1/62486->192.1.7.1/135 0x0 N/A N/A N/A N/A 6 ACME-COMP007 ACME CCA 1027474 N/A(N/A) reth3.860 UNKNOWN UNKNOWN UNKNOWN249 I suspect it might be something to do with the amount of logs that are received in very quick succession, but I have no idea how to address this. One thing I have noticed is that it always appears to be the same device that is causing the issue. When I look at other devices coming in on the same port, they appear to write out correctly. Any suggestion would be really welcome. Thanks John This message has been scanned for malware by Forcepoint. www.forcepoint.com

2 4

Multiple timezone seen on messages originating from local machine
by Francois Marier 16 Aug '22

16 Aug '22

I'm trying to understand timezones in syslog-ng, but I think there's something I'm missing. If I do the following on my OpenWRT machine: /etc/init.d/syslog-ng restart logger TestA I see the following in /var/log/messages: Aug 14 20:39:35 hostname syslog-ng[9860]: syslog-ng shutting down; version='3.37.1' Aug 14 20:39:36 hostname syslog-ng[10024]: syslog-ng starting up; version='3.37.1' Aug 14 20:39:36 hostname syslog-ng[10024]: Syslog connection established; fd='15', server='AF_INET(192.168.1.10:514)', local='AF_INET(0.0.0.0:0)' Aug 15 03:39:49 hostname root: TestA The correct timezone is the one in the first three lines. Other daemon messages are displayed using the incorrect timezone like `logger`. I've attached my syslog-ng.conf file. So my question is why is the time information correct in the first three messages and then incorrect from there on? Francois -- https://fmarier.org/

3 4

drop log lines having keyword "INTERNET"
by Sachchidanand Upadhyay 04 Aug '22

04 Aug '22

Hi, I want to drop all log lines having keyword "INTERNET", I tried following scenario: scenario-1 filter f_log { facility(local3); }; filter f_nointernet { not message("INTERNET"); }; log { source(s_sys); filter(f_nointernet); filter(f_log); destination(d_log); }; scenario-2 filter f_internet { message("INTERNET"); }; filter f_log { facility(local3); and not filter(f_internet); }; log { source(s_sys); filter(f_log); destination(d_log); }; scenario-3 filter f_log { facility(local3); }; filter f_internet { message("INTERNET"); }; filter f_nointernet { not filter(f_internet); }; log { source(s_sys); filter(f_nointernet); filter(f_log); destination(d_log); }; But none of them is working. Please help. Regards, Sachchidanand

1 0

Kafka-c issues
by Sanjay Patel 15 Jul '22

15 Jul '22

I am trying to setup Syslog-NG t send output to Kakka. I am running ubuntu 18.04, Syslog-NG version 3.37. This is my syslog-ng.conf ( I have #scl.conf because if I don't it tries to install kafka-java) @version: 3.37 @define kafka-implementation kafka-c #@include "scl.conf" destination d_kafka { kafka( bootstrap-servers("127.0.0.1:9092") topic("{incoming}") ); }; This is the error when I try and run sudo syslog-ng Error parsing config, syntax error, unexpected LL_IDENTIFIER, expecting '}' in /etc/syslog-ng/syslog-ng.conf:6:3-6:8: 1 @version: 3.37 2 @define kafka-implementation kafka-c 3 #@include "scl.conf" 4 5 destination d_kafka { 6-----> kafka( 6-----> ^^^^^ 7 bootstrap-servers("127.0.0.1:9092") 8 topic("{incoming}") 9 ); 10 }; Any help will be appreciated. SKP

4 8

Insider 2022-06: RHEL 9; disk-buffer; Microsoft Linux;
by Peter Czanik (pczanik) 14 Jul '22

14 Jul '22

Dear syslog-ng users, This is the 103rd issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS RHEL 9 syslog-ng news --------------------- Red Hat Enterprise Linux 9 became generally available recently. Version 3.35 of syslog-ng has been part of EPEL 9 (the semi-official extra software repo for RHEL maintained by Fedora packagers) for a while and now I enabled a few more destination drivers. I also enabled RHEL 9 support in my unofficial Git snapshot packages, so I can support RHEL 9 together with other RHEL and Fedora versions on the next syslog-ng release. https://www.syslog-ng.com/community/b/blog/posts/rhel-9-syslog-ng-news How does the syslog-ng disk-buffer work? ---------------------------------------- Last time, we had an overview of the syslog-ng disk-buffer. This time, we dig a bit deeper and take a quick look at how it works, and a recent major change that helped speed up the reliable disk-buffer considerably. https://www.syslog-ng.com/community/b/blog/posts/how-does-the-syslog-ng-dis… Installing syslog-ng on Microsoft Linux --------------------------------------- Yes, Microsoft has its own Linux distribution, called CBL-Mariner. It is an internal Linux distribution by Microsoft used for cloud infrastructure and edge products and services. And even if it is not installed in the OS by default, CBL-Mariner also includes syslog-ng. https://www.syslog-ng.com/community/b/blog/posts/installing-syslog-ng-on-mi… WEBINARS * Register for our latest webinars at https://www.syslog-ng.com/event/syslogng-webcast-series-2022/ * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Missing Depends libjson-c4
by Joe Craig 06 Jul '22

06 Jul '22

The following packages have unmet dependencies: syslog-ng-core : Depends: libjson-c4 (>= 0.13.1) but it is not installable Depends: libssl1.1 (>= 1.1.1) but it is not installable >From research it looks like libjson-c4 has been replaced in the Debian repositories by libjson-c5 is there a way to utilize the libjson-c5 version or how should I resolve this issue? FYI this was working on Ubuntu 22.04 Jammy up until the past week. I ran updates on the system to resolve another issue and after that I'm getting an error starting the daemon × syslog-ng.service - System Logger Daemon Loaded: loaded (/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Tue 2022-07-05 15:39:48 UTC; 4s ago Docs: man:syslog-ng(8) Process: 1342 ExecStart=/usr/sbin/syslog-ng -F $SYSLOGNG_OPTS (code=exited, status=2) Main PID: 1342 (code=exited, status=2) Status: "Starting up... (Tue Jul 5 15:39:48 2022" CPU: 29ms Jul 05 15:39:48 <hostname> systemd[1]: syslog-ng.service: Main process exited, code=exited, status=2/INVALIDARGUMENT Jul 05 15:39:48 <hostname> systemd[1]: syslog-ng.service: Failed with result 'exit-code'. Jul 05 15:39:48 <hostname> systemd[1]: Failed to start System Logger Daemon. Jul 05 15:39:48 <hostname> systemd[1]: syslog-ng.service: Scheduled restart job, restart counter is at 5. Jul 05 15:39:48 <hostname> systemd[1]: Stopped System Logger Daemon. Jul 05 15:39:48 <hostname> systemd[1]: syslog-ng.service: Start request repeated too quickly. Jul 05 15:39:48 <hostname> systemd[1]: syslog-ng.service: Failed with result 'exit-code'. Jul 05 15:39:48 <hostname> systemd[1]: Failed to start System Logger Daemon. The rabbit hole lead me to this current state. Any assistance would be appreciated. Joseph Craig

2 2

Environment variables in syslog
by Faisal Chishti 28 Jun '22

28 Jun '22

Hellow, Please see below the snippet from my syslog-ng.conf file. ABC_LOG_PATH is the environment variable on the host that contains the path. I am trying to access the env variable within the config. It does not throw any error and the configuration is valid but it does not work properly. I don't see the file agent.log being used at all. source s_agent { file("`ABC_LOG_PATH`/agent.log" follow-freq(1) flags(no-parse)); }; root@syslog-ng-784fdc497-pgdzr:/# syslog-ng --version syslog-ng 3 (3.30.1) Config version: 3.29 Installer-Version: 3.30.1 Revision: Compile-Date: Apr 12 2022 20:15:37 Module-Directory: /usr/lib/syslog-ng Module-Path: /usr/lib/syslog-ng Include-Path: /usr/share/syslog-ng/include Available-Modules: add-contextual-data,afamqp,affile,afprog,afsocket,afsql,afstomp,afuser,appmodel,azure-auth-header,basicfuncs,cef,confgen,cryptofuncs,csvparser,dbparser,disk-buffer,graphite,hook-commands,http,json-plugin,kvformat,linux-kmsg-format,map-value-pairs,mod-python,pseudofile,redis,secure-logging,stardate,syslogformat,system-source,tags-parser,timestamp,xml Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: off Enable-Systemd: off root@syslog-ng-784fdc497-pgdzr:/# uname -a Linux syslog-ng-784fdc497-pgdzr 4.18.0-147.3.1.el8_1.x86_64 #1 SMP Fri Jan 3 23:55:26 UTC 2020 x86_64 GNU/Linux root@syslog-ng-784fdc497-pgdzr:/# -- Regards, Faisal

2 1