- syslog-ng - lists.balabit.hu

syslog-ng-3.37.1
by Laszlo Varady (lvarady) 08 Jun '22

08 Jun '22

3.37.1 Highlights * kubernetes source: A new source for Kubernetes CRI (Container Runtime Interface) format. By default it tails the /var/log/containers folder which can be overriden with the base-dir() parameter. Example configuration: source { kubernetes(); # or specifying the directory: # kubernetes(base-dir("/dir/to/tail")); }; (#4015<https://github.com/syslog-ng/syslog-ng/pull/4015>) * mariadb-audit-parser: A new parser for mariadb/mysql audit plugin logs have been added. The parser supports the syslog output type's format, see mariadb page<https://mariadb.com/kb/en/mariadb-audit-plugin> for details. (#3947<https://github.com/syslog-ng/syslog-ng/pull/3947>) Features * internal(): add rcptid tag to all trace messages that relate to incoming log messages. This makes it easier to correlate parsing, rewriting and routing actions with incoming log messages. (#3972<https://github.com/syslog-ng/syslog-ng/pull/3972>) * syslog-parser(): allow comma (e.g. ',') to separate the seconds and the fraction of a second part as some devices use that character. This change applies to both to syslog-parser() and the builtin syslog parsing functionality of network source drivers (e.g. udp(), tcp(), network() and syslog()). (#3949<https://github.com/syslog-ng/syslog-ng/pull/3949>) * cisco-parser: add ISO 8601 timestamp support (#3934<https://github.com/syslog-ng/pull/3934>) * network(), syslog() sources and destinations: added new TLS options sigalgs() and client-sigalgs() They can be used to restrict which signature/hash pairs can be used in digital signatures. It sets the "signature_algorithms" extension specified in RFC5246 and RFC8446. Example configuration: destination { network("test.host" port(4444) transport(tls) tls( pkcs12-file("/path/to/tls/test.p12") peer-verify(yes) sigalgs("RSA-PSS+SHA256:ed25519") ) ); }; (#4000<https://github.com/syslog-ng/syslog-ng/pull/4000>) * set-matches() and unset-matches(): these new rewrite operations allow the setting of match variables ($1, $2, ...) in a single operation, based on a syslog-ng list expression. Example: # set $1, $2 and $3 respectively set-matches("foo,bar,baz"); # likewise, but using a list function set-matches("$(explode ':' 'foo:bar:baz')"); (#3948<https://github.com/syslog-ng/syslog-ng/pull/3948>) * $* macro: the $* macro in template expressions convert the match variables (e.g. $1, $2, ...) into a syslog-ng list that can be further manipulated using the list template functions, or turned into a list in type-aware destinations. (#3948<https://github.com/syslog-ng/syslog-ng/pull/3948>) * set-tag(): add support for using template expressions in set-tag() rewrite operations, which makes it possible to use tag names that include macro references. (#3962<https://github.com/syslog-ng/syslog-ng/pull/3962>) Bugfixes * http() and other threaded destinations: fix $SEQNUM processing so that only local messages get an associated $SEQNUM, just like normal syslog()-like destinations. This avoids a [meta sequenceId="XXX"] SD-PARAM being added to $SDATA for non-local messages. (#3928<https://github.com/syslog-ng/syslog-ng/pull/3928>) * grouping-by(): fix grouping-by() use through parser references. Originally if a grouping-by() was part of a named parser statement and was referenced from multiple log statements, only the first grouping-by() instance behaved properly, 2nd and subsequent references were ignoring all configuration options and have reverted using defaults instead. (#3957<https://github.com/syslog-ng/syslog-ng/pull/3957>) * db-parser(): similarly to grouping-by(), db-parser() also had issues propagating some of its options to 2nd and subsequent references of a parser statement. This includes drop-unmatched(), program-template() and template() options. (#3957<https://github.com/syslog-ng/syslog-ng/pull/3957>) * match(), subst() and regexp-parser(): fixed storing of numbered (e.g. $1,$2, $3 and so on) and named capture groups in regular expressions in case the input of the regexp is the same as one of the match variables being stored. In some cases the output of the regexp was clobbered and an invalid value stored. (#3948<https://github.com/syslog-ng/syslog-ng/pull/3948>) * fix threaded(no) related crash: if threaded mode is disabled for asynchronous sources and destinations (all syslog-like drivers such as tcp/udp/syslog/network qualify), a use-after-free condition can happen due to a reference counting bug in the non-threaded code path. The threaded(yes) setting has been the default since 3.6.1 so if you are using a more recent version, you are most probably unaffected. If you are using threaded(no) a use-after-free condition happens as the connection closes. The problem is more likely to surface on 32 bit platforms due to pointer sizes and struct layouts where this causes a NULL pointer dereference. (#3997<https://github.com/syslog-ng/syslog-ng/pull/3997>) * set(): make sure that template formatting options (such as time-zone() or frac-digits()) are propagated to all references of the rewrite rule containing a set(). Previously the clone() operation used to implement multiple references missed the template related options while cloning set(), causing template formatting options to be set differently, depending on where the set() was referenced from. (#3962<https://github.com/syslog-ng/syslog-ng/pull/3962>) * csv-parser(): fix flags(strip-whitespace) and null-value handling for greedy column (#4028<https://github.com/syslog-ng/syslog-ng/pull/4028>) Other changes * java()/python() destinations: the $SEQNUM macro (and "seqnum" attribute in Python) was erroneously for both local and non-local logs, while it should have had a value only in case of local logs to match RFC5424 behavior (section 7.3.1). This bug is now fixed, but that means that all non-local logs will have $SEQNUM set to zero from this version on, e.g. the $SEQNUM macro would expand to an string, to match the syslog() driver behaviour. (#3928<https://github.com/syslog-ng/syslog-ng/pull/3928>) * dbld: add support for Fedora 35 in favour of Fedora 33 (#3933<https://github.com/syslog-ng/syslog-ng/pull/3933>) * debian: fix logrotate file not doing the file rotation. (The path and command was invalid.) (#4031<https://github.com/syslog-ng/syslog-ng/pull/4031>) * OpenSSL: add support for OpenSSL 3.0 (#4012<https://github.com/syslog-ng/syslog-ng/pull/4012>) * The MD4 hash function ($(md4)) is no longer available when compiling syslog-ng with OpenSSL v3.0. MD4 is now deprecated, it will be removed completely in future versions. (#4012<https://github.com/syslog-ng/syslog-ng/pull/4012>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Attila Szakacs, Balazs Scheidler, Ben Burrows, Fᴀʙɪᴇɴ Wᴇʀɴʟɪ, Gabor Nagy, László Várady, mohitvaid, Parrag Szilárd, Peter Kokai, Peter Viskup, Roffild, Ryan Faircloth, Scott Parlane, Zoltan Pallagi Installation from binaries: https://github.com/syslog-ng/syslog-ng#installation-from-binaries<https://github.com/syslog-ng/syslog-ng#installation-from-binaries>

1 0

rhel 9 / centos 9
by Peter Czanik (pczanik) 19 May '22

19 May '22

Hi, Testing and feedback needed! RHEL9 was released recently. My syslog-ng git snapshot repo now also supports RHEL 9. I tested my syslog-ng packages on CentOS Stream 9. Let me know, how it works for you! It is available at: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng-githead/ Note, that even if SQL support (libdbi) is enabled, the drivers are not there, I could not get them compiled. I plan to remove SQL support next week, as without drivers it's not much use. Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

HUP or reload
by Faine, Mark R. (MSFC-IS40)[AEGIS] 19 May '22

19 May '22

Are there certain circumstances where I should prefer doing a HUP on the syslog-ng process instead of syslog-ng-ctl reload? My use case specifically is for postrotate but I'm also asking generally. Thanks, Mark

2 1

Insider 2022-05: directions; EPEL 9; throttle; regexp-parser;
by Peter Czanik (pczanik) 18 May '22

18 May '22

Dear syslog-ng users, This is the 101st issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS syslog-ng on the long term: a draft on strategic directions ----------------------------------------------------------- Balázs Scheidler, founder of the syslog-ng project, describes five possible strategic directions: “In the past few weeks I performed a round of discussions/interviews with syslog-ng users. I also spent time looking at other products and analyst reports on the market. Based on all this information I’ve come up with a list of potential strategic directions for syslog-ng to tackle. Focusing on these and prioritizing features that fall into one of these directions ensures that syslog-ng indeed moves ahead.” Read the rest of the blog at https://syslog-ng-future.blog/syslog-ng-on-the-long-term-a-draft-on-strateg… A minimalist syslog-ng package is heading to EPEL 9 --------------------------------------------------- Last week, the ivykis library, the most important core dependency of syslog-ng landed in EPEL 9 successfully. There are still plenty of dependencies missing, but this way, I could submit a slightly cut down version of syslog-ng to EPEL 9. Hopefully the rest of the dependencies will arrive in EPEL 9 as well. I plan to update the syslog-ng package as soon as the dependencies arrive. Luckily, these are only needed to enable some less frequently used syslog-ng destination drivers, no core functionality is affected. https://www.syslog-ng.com/community/b/blog/posts/a-minimalist-syslog-ng-pac… The difference between throttle() and rate-limit() in syslog-ng --------------------------------------------------------------- There are multiple ways in syslog-ng to limit message rate. The throttle() option of syslog-ng destinations tries to make sure that all messages are delivered without exceeding a specified message rate. The rate-limit() filter introduced in syslog-ng 3.36 drops surplus log messages, making sure that a processing pipeline or destination is not overloaded with log messages. https://www.syslog-ng.com/community/b/blog/posts/the-difference-between-thr… Using the regexp-parser of syslog-ng ------------------------------------ For many years, you could use the match() filter of syslog-ng to parse log messages with regular expressions. However, the primary function of match() is filtering. Recent syslog-ng versions now have a dedicated regular expression parser, the regexp-parser(). So, you should use match() only if your primary use case is filtering. Otherwise, use the regexp-parser for parsing, as it is a lot more flexible. https://www.syslog-ng.com/community/b/blog/posts/using-the-regexp-parser-of… WEBINARS * Register for our latest webinars at https://www.syslog-ng.com/event/syslogng-webcast-series-2022/ * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Re: [syslog-ng] Not logging postfix logs in mail.info mail.warn (SysAdmin EM)
by Szilard Parrag (sparrag) 17 May '22

17 May '22

Hi emawata, Checking your configuration snippet it seems that `d_mailwarn` and `d_mailinfo` are not connected to a logpath, which is why syslog-ng doesn't write into them. If you wish to use them create a `log{};` block, which fetches data from source and writes it into `destination(d_mailwarn);` or `destination(d_mailinfo);` Hope it helped, Szilard

1 0

Re: [syslog-ng] Local sources seem not to be working
by Szilard Parrag (sparrag) 16 May '22

16 May '22

Hi Alex, After checking the stats, you have sent we can see that there had been some writes: from dst.program;d_localfile_linecard#0;/opt/machine/local/bin/write_with_rotation.sh /var/log/linecard.log 10 10;a;written;4518 to dst.program;d_localfile_linecard#0;/opt/machine/local/bin/write_with_rotation.sh /var/log/linecard.log 10 10;a;written;4549 * we do not see increase in the counters of /var/log destinatons, but only on one destination * we could only see an increase in syslog-udp processed counters * there are no dropped/queued counters We would guess this could be due to flow-control, but for that we would need to see non-zero queued counter values, which is not the case. It could happen that one destination hangs/can't send messages out, which leads to suspended sources due to flow-control, but the syslog() source is not affected since it doesn't send messages to the hanged destination(s). Based on the stats, only "d_localfile_linecard" is active (~30 messages in 15 minutes), maybe the syslog() source would be affected too without the filtering. We should see more internal logs, which is problematic, since internal() source seems to be stopped too. For that I would recommend extracting internal() source from the s_src statement and putting it in a separate log path with a simple file destination. Also, if possible, could you please share your `write_with_rotation.sh` script? It is unlikely that it interferes with syslog-ng, but a double check would be nice. 🙂

2 2

Not logging postfix logs in mail.info mail.warn
by SysAdmin EM 16 May '22

16 May '22

I cannot configure Postfix logs correctly, the mail.warn and mail.info logs are empty. Any helps?? Here is my configuration: options { chain_hostnames(off); flush_lines(0); use_dns(no); use_fqdn(no); owner("root"); group("adm"); perm(0640); stats_freq(0); bad_hostname("^gconfd$"); }; source s_src { system(); internal(); }; destination d_auth { file("/var/log/auth.log"); }; destination d_cron { file("/var/log/cron.log"); }; destination d_daemon { file("/var/log/daemon.log"); }; destination d_kern { file("/var/log/kern.log"); }; destination d_lpr { file("/var/log/lpr.log"); }; destination d_mail { file("/var/log/mail.log"); }; destination d_syslog { file("/var/log/syslog"); }; destination d_user { file("/var/log/user.log"); }; destination d_uucp { file("/var/log/uucp.log"); }; destination d_mailinfo { file("/var/log/mail.info"); }; destination d_mailwarn { file("/var/log/mail.warn"); }; destination d_mailerr { file("/var/log/mail.err"); }; destination d_newscrit { file("/var/log/news/news.crit"); }; destination d_newserr { file("/var/log/news/news.err"); }; destination d_newsnotice { file("/var/log/news/news.notice"); }; destination d_debug { file("/var/log/debug"); }; destination d_error { file("/var/log/error"); }; destination d_messages { file("/var/log/messages"); }; destination d_console { usertty("root"); }; destination d_console_all { file(`tty10`); }; destination d_xconsole { pipe("/dev/xconsole"); }; log_fifo_size(1000)); }; destination d_ppp { file("/var/log/ppp.log"); }; filter f_dbg { level(debug); }; filter f_info { level(info); }; filter f_notice { level(notice); }; filter f_warn { level(warn); }; filter f_err { level(err); }; filter f_crit { level(crit .. emerg); }; filter f_debug { level(debug) and not facility(auth, authpriv, news, mail); }; filter f_error { level(err .. emerg) ; }; filter f_messages { level(info,notice,warn) and not facility(auth,authpriv,cron,daemon,mail,news); }; filter f_auth { facility(auth, authpriv) and not filter(f_debug); }; filter f_cron { facility(cron) and not filter(f_debug); }; filter f_daemon { facility(daemon) and not filter(f_debug); }; filter f_kern { facility(kern) and not filter(f_debug); }; filter f_lpr { facility(lpr) and not filter(f_debug); }; filter f_local { facility(local0, local1, local3, local4, local5, local6, local7) and not filter(f_debug); }; filter f_mail { facility(mail) and not filter(f_debug); }; filter f_news { facility(news) and not filter(f_debug); }; filter f_syslog3 { not facility(auth, authpriv, mail) and not filter(f_debug); }; filter f_user { facility(user) and not filter(f_debug); }; filter f_uucp { facility(uucp) and not filter(f_debug); }; filter f_cnews { level(notice, err, crit) and facility(news); }; filter f_cother { level(debug, info, notice, warn) or facility(daemon, mail); }; filter f_ppp { facility(local2) and not filter(f_debug); }; filter f_console { level(warn .. emerg); }; ######################## # Log paths ######################## log { source(s_src); filter(f_auth); destination(d_auth); }; log { source(s_src); filter(f_cron); destination(d_cron); }; log { source(s_src); filter(f_daemon); destination(d_daemon); }; log { source(s_src); filter(f_kern); destination(d_kern); }; log { source(s_src); filter(f_lpr); destination(d_lpr); }; log { source(s_src); filter(f_syslog3); destination(d_syslog); }; log { source(s_src); filter(f_user); destination(d_user); }; log { source(s_src); filter(f_uucp); destination(d_uucp); }; log { source(s_src); filter(f_mail); destination(d_mail); }; log { source(s_src); filter(f_news); filter(f_crit); destination(d_newscrit); }; log { source(s_src); filter(f_news); filter(f_err); destination(d_newserr); }; log { source(s_src); filter(f_news); filter(f_notice); destination(d_newsnotice); }; log { source(s_src); filter(f_debug); destination(d_debug); }; log { source(s_src); filter(f_error); destination(d_error); }; log { source(s_src); filter(f_messages); destination(d_messages); }; log { source(s_src); filter(f_console); destination(d_console_all); destination(d_xconsole); }; log { source(s_src); filter(f_crit); destination(d_console); };

1 0

Filtering all systemd user service logs
by Thomas Preisner 02 May '22

02 May '22

Hello all, I've been using syslog-ng for quite a while now to centrally collect logs of a few computers running debian and it worked great. However, a few weeks ago I enabled systemd user services on those machines and since then, my 'daemon' logs have been basically completely flooded with (in my opinion) useless log messages created by the aforementioned user services. So far, I've been using the following source configuration: source src { internal(); system(); udp(); udp6(); }; Until now, I've been filtering out the user service logs with my log-analysis tool using some regexes but due to the logs being 'user-controlled' (users can create arbitrary logs by simply creating and starting custom systemd user services) this is obviously no permanent solution. Even worse, the user-created logs end up in the 'daemon'-facility and not in the 'user'-facility as mentioned before. Therefore, here my question: Is there any clean way to detect/filter out these user service logs? I couldn't find anything relevant in the documentation so far. After all, these logs are collected separately by systemd-journald and then merged with the remaining system logs by syslog-ng (at least as far as I see it). Thank you for your response in advance! Best regards, Thomas Preisner

2 1

Local sources seem not to be working
by Alexandre Santos 28 Apr '22

28 Apr '22

Hi, I have syslog-ng 3.32.1 running in a Debian GNU/Linux 10 (buster) with the configuration in the attachement. After sometime running, syslog-ng seems be unable to read from system() and internal() sources. Log messages from syslog(ip(10.20.30.40) transport("udp") port(514) keep-alive(no)); are seen in the output folders. Also journald logs are working fine. After a reload of configuration in which what changes is this line: rewrite r_host { set("MACHINE-${HOST}", value("HOST")); }; logging is resumed. Here is the time gap for logs: <43>1 2022-03-11T11:55:23.802+00:00 xmm4-1-1 syslog-ng 8283 - [meta sequenceId="767"] Last message 'Destination reliable' repeated 8933 times, suppressed by syslog-ng on xmm4-1-1 <46>1 2022-03-14T07:19:01.817+00:00 xmm4-1-1 syslog-ng 8283 - [meta sequenceId="1"] Module loaded and initialized successfully; module='syslogformat' Do you know why this is happening? Thanks & Regards, Alex

2 13

Insider 2022-04: typing; sudo; Zinc; Elastic Cloud; 3.36;
by Peter Czanik (pczanik) 21 Apr '22

21 Apr '22

Dear syslog-ng users, This is the 100th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS syslog-ng 4 theme: typing ------------------------- Balázs Scheidler, founder of the syslog-ng project, describes a major new syslog-ng version 4.0 feature: typing. “syslog-ng traditionally assumes that log data, even if it comes in a structured form (like RFC5424 structured data or JSON) is primarily textual in nature. For this reason, name-value pairs in syslog-ng are text values just as the log message as a whole.” Read the rest of the blog to learn how typing helps not just when forwarding logs to Elasticsearch and others, but also when doing routing decisions: https://syslog-ng-future.blog/syslog-ng-4-theme-typing/ Working with JSON logs from sudo in syslog-ng --------------------------------------------- >From this blog you will learn how to work with JSON formatted logs in syslog-ng and also about new sudo features along the way: JSON logging in sudo, chroot support, logging sub-commands, and how to work with these logs in syslog-ng. https://www.syslog-ng.com/community/b/blog/posts/working-with-json-logs-fro… Elasticsearch 8 and syslog-ng ----------------------------- General availability of Elasticsearch 8 was announced recently. There were quite a few rumors that it will break compatibility with third party tools. I tested it as soon as I had a little time: I am happy to share that anything I tested with the elasticsearch-http() destination of syslog-ng still seems to work perfectly well with the latest version of Elasticsearch. https://www.syslog-ng.com/community/b/blog/posts/elasticsearch-8-and-syslog… Contacting the syslog-ng team: reporting problems, asking questions --------------------------------------------- Recently I got some complaints that it is difficult to figure out how to contact the syslog-ng team to get help or report problems. Most of this information is available both on the syslog-ng website and at the syslog-ng repository on GitHub, but collecting here all information might be still useful for some people. https://www.syslog-ng.com/community/b/blog/posts/contacting-the-syslog-ng-t… WEBINARS * Register for our latest webinars at https://www.syslog-ng.com/event/syslogng-webcast-series-2022/ * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0