- syslog-ng - lists.balabit.hu

laszlo_budai's GPG key expired
by Szmytka, Piotr 03 Jan '22

03 Jan '22

Hi, It seems the GPG key used to sign Debian 10 packages has expired. The deb repository is therefore unusable. Can this be fixed please? $ curl -s http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | gpg - gpg: WARNING: no command supplied. Trying to guess what you mean ... pub rsa2048 2017-08-25 [SC] [expired: 2021-12-31] F20F51628D04901AD01175013B92A8D27CFDAEDD uid home:laszlo_budai:syslog-ng OBS Project <home:laszlo_budai:syslog-ng@build.opensuse.org> Thanks! --- PIOTR SZMYTKA Senior Systems Engineer, Cloud Engineering piotr.szmytka(a)vimn.com<mailto:piotr.szmytka@vimn.com> VIMN Poland sp. z o.o. ul. Mokotowska 19, 00-560 Warsaw, POLAND [signature_1140942668]

2 1

Insider 2021-12: Humio; Log Management; Panther;
by Peter Czanik (pczanik) 16 Dec '21

16 Dec '21

Dear syslog-ng users, This is the 97th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Sending logs to Panther using syslog-ng --------------------------------------- Panther is an open-source log management system, which is also available as a service for a time-limited trial. It is still in beta phase, but it looks promising. You can see the “beta” sign on its opening page: https://app.panther.support/ I tested the time-limited cloud service version, but you can also install it locally, either from Dockerhub, or you can build the containers locally from the source. Even if it is still in beta phase, Panther comes with detailed documentation. There is a notable exception: while syslog-ng is shown on some of the figures, documenting it is still to be done. This blog helps you to get started with sending logs to Panther, using syslog-ng. You can use either legacy syslog with TLS encryption (still a bit problematic) or the http() destination to send logs to the Panther HTTP API. https://www.syslog-ng.com/community/b/blog/posts/sending-logs-to-panther-us… Reducing the complexity of log management ----------------------------------------- It is easy to over-complicate log management. Almost all departments in a company need to log messages for their daily activities. However, installing several different log management and analysis systems in parallel is a nightmare both from a security and an operations perspective and wastes many resources. You cannot always reduce the number of log analysis systems, but you can reduce the complexity of log management. Let me show you, how. https://www.syslog-ng.com/community/b/blog/posts/reducing-the-complexity-of… Sending logs to Humio using the elasticsearch-http() destination of syslog-ng ------------- One of the most popular syslog-ng destinations is Elasticsearch. Humio, a log management provider, supports a broad range of ingest options and interfaces, including an Elasticsearch-compatible API. Last week, Humio announced Humio Community Edition, which provides the full Humio experience for free, with some limitations on daily ingestion and retention time. I tested the Community Edition, and it works perfectly well with syslog-ng. If you come from the Humio side, you might wonder what syslog-ng is. It is an application for high performance central log collection. Traditionally, syslog messages were collected centrally and saved to text files. Nowadays, syslog-ng acts more like a log management layer: collects log messages from hosts, saves them for long term storage, but also forwards them to multiple destinations, like SIEMs and other log analysis solutions. This way, it is enough to collect log messages only once, and syslog-ng delivers the right log messages to the right destinations in the right format, after some initial processing. Humio is available as a cloud service or self-hosted, where you can send all your logs for storage and analysis. It has an easy-to-use interface to query log messages which can be extended with further analytics possibilities from the Humio marketplace. >From this blog, you can learn how to get started with Humio and syslog-ng. While Humio provides many other APIs for log ingestion, I focus on the elasticsearch-http() destination of syslog-ng, demonstrating that there is no vendor lock-in: the same driver works equally well for Elastic’s Elasticsearch, AWS’s OpenSearch and for Humio. https://www.syslog-ng.com/community/b/blog/posts/sending-logs-to-humio-usin… WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

syslog-ng no longer receiving kernel messages
by John Covici 13 Dec '21

13 Dec '21

Hi folks. I am having a problem that syslog-ng no longer gets any kernel messages. It seems to get other messages, but not kernel messages. Here is what happened. In an update from kernel 5.10.70 to 5.10.82, I enabled the kernel race condition sanitizer. I looked at its output and decided that it would not do me any good, so I took that out and recompiled the kernel. However after rebooting the system, syslog-ng only gives me a few messages from when the kernel sanitizer was active in my /var/log/messages and the same for /var/log/kernel. I am using systemd, so I have the jernal and it is OK, all messages are posted correctly there. Here is my /etc/syslog-ng/syslog-ng.conf . @version: 3.34 # $Header: /var/cvsroot/gentoo-x86/app-admin/syslog-ng/files/3.4/syslog-ng.conf.gentoo,v 1.2 2013/06/02 01:18:35 mr_bones_ Exp $ # # Syslog-ng default configuration file for Gentoo Linux # https://bugs.gentoo.org/show_bug.cgi?id=426814 @include "scl.conf" options { dir_perm(0755); perm(0644); chain_hostnames(no); threaded(yes); chain_hostnames(no); # The default action of syslog-ng is to log a STATS line # to the file every 10 minutes. That's pretty ugly after a while. # Change it to every 12 hours so you get a nice daily update of # how many messages syslog-ng missed (0). stats_freq(43200); # The default action of syslog-ng is to log a MARK line # to the file every 20 minutes. That's seems high for most # people so turn it down to once an hour. Set it to zero # if you don't want the functionality at all. mark_freq(0); keep_hostname(yes); }; source local { system() ; internal(); }; # *.emerg * filter f_9 { level(emerg); }; destination d_6 { usertty("*"); }; log { source(local); filter(f_9); destination(d_6); }; # kern.=debug /var/log/kernel filter f_1 { facility(kern) and level(debug..emerg); }; destination d_1 { file("/var/log/kernel" create_dirs(yes)); }; log { source(local); filter(f_1); destination(d_1); }; # kern.err /dev/console filter f_2 { facility(kern) and level(err..emerg); }; destination d_2 { file("/dev/console" create_dirs(yes)); }; log { source(local); filter(f_2); destination(d_2); }; # *.debug;mail.none;news.none;authpriv.none /var/log/messages filter f_3 { level(debug..emerg); }; filter f_4 { not facility(mail); }; filter f_5 { not facility(news); }; filter f_6 { not facility(authpriv); }; destination d_3 { file("/var/log/messages" create_dirs(yes)); }; log { source(local); filter(f_3); filter(f_4); filter(f_5); filter(f_6); destination(d_3); flags(final flow-control); }; # authpriv.* /var/log/secure filter f_7 { facility(authpriv) and level(debug..emerg); }; destination d_4 { file("/var/log/secure" create_dirs(yes)); }; log { source(local); filter(f_7); destination(d_4); }; # mail.* /var/log/maillog filter f_8 { facility(mail) and level(debug..emerg); }; destination d_5 { file("/var/log/maillog" create_dirs(yes)); }; log { source(local); filter(f_8); destination(d_5); }; # news.=crit /var/log/news/news.crit filter f_10 { facility(news) and level(crit..emerg); }; destination d_7 { file("/var/log/news/news.crit" create_dirs(yes)); }; log { source(local); filter(f_10); destination(d_7); }; # news.=err /var/log/news/news.err filter f_11 { facility(news) and level(err..emerg); }; destination d_8 { file("/var/log/news/news.err" create_dirs(yes)); }; log { source(local); filter(f_11); destination(d_8); }; # news.notice /var/log/news/news.notice filter f_12 { facility(news) and level(notice..emerg); }; destination d_9 { file("/var/log/news/news.notice" create_dirs(yes)); }; log { source(local); filter(f_12); destination(d_9); }; # local7.* /var/log/boot.log filter f_13 { facility(local7) and level(debug..emerg); }; destination d_10 { file("/var/log/boot.log" create_dirs(yes)); }; log { source(local); filter(f_13); destination(d_10); }; # local0.* /var/log/dhcpcd.log filter f_14 { facility(local0) and level(debug..emerg); }; So, what have I been doing wrong, or is it something else? Thanks in advance for any suggestions. destination d_11 { file("/var/log/dhcpcd.log" create_dirs(yes)); }; log { source(local); filter(f_14); destination(d_11); }; -- Your life is like a penny. You're going to lose it. The question is: How do you spend it? John Covici wb2una covici(a)ccs.covici.com

2 4

[patch] new feature for file destination: persistent symlinks
by a.biardi＠tiscali.it 07 Dec '21

07 Dec '21

Hi all, I'm new here. I love syslog-ng: it's well written and documented and each time I need some new feature, I dig up the documentation and I find it's already implemented! Like many others, I use macros in file destinations (say, "/var/log/cron.${YEAR}${MONTH}"), but I also want to maintain a persistent symlink to the current logfile (that would be something like "/var/log/cron") in order to have "easier" access if I'm grepping for something manually. I couldn't find this feature in syslog-ng, so I wrote the code myself; in the configuration file, it looks like this: destination d_file_cron { file("/var/log/cron.${YEAR}${MONTH}" symlink_as("/var/log/cron")); }; It works for me on my Slackware boxes. I'm sharing the patch I put together, hoping that it may be useful to others and perhaps even considered for inclusion in mainstream. From a functional perspective, the "symlink_as" file inherits both "create-dirs" and file ownership from its file destination (permissions are not applicable to symlinks, at least on linux). Patch applies cleanly to latest production release (3.33). I tried to keep the original coding style for easier review (admittedly, the code could have gone into the function that opens the file destination, but I didn't want to change too much). Thanks for the good work and happy logging! Andrea.

3 4

Re: [syslog-ng] [patch] new feature for file destination: persistent symlinks
by a.biardi＠tiscali.it 06 Dec '21

06 Dec '21

On Mon, 6 Dec 2021 15:28:18 +0000, "Gabor Nagy (gnagy)" <Gabor.Nagy(a)oneidentity.com> wrote: > Dear Andrea! > > I'm forwarding my answer to make sure you get it anyway. > Looking forward to hearing from you! > > Best Regards, > Gábor > ________________________________ > From: Gabor Nagy (gnagy) <Gabor.Nagy(a)oneidentity.com> > Sent: Sunday, December 5, 2021 17:49 > To: Syslog-ng users' and developers' mailing list > <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] [patch] new > feature for file destination: persistent symlinks > > Hello Andrea! > > Thank you for your contribution! We always welcome feedback and > contribution from the community! > > I've quickly checked your patch, and it looks good. Maybe minor > things could be changed, but we will see this during a thorough > review. > > We integrate patches via pull requests on GitHub. [1] > How would you like to continue this contribution? > Would you like to open a pull request on GitHub? > > I see a corner case of the feature: if the filename template is not > time-based, like e.g. $PROGRAM or $HOST, it can happen that the > symlink is changed several times in a short time. This is not an > argument against the feature, it's just a behaviour that should be > considered and documented. As I see, the feature in general creates a > symlink with the name given in "symlink_as" option when a new file is > opened. In other words: the configured symlink always points to the > latest file that were opened. > > > Best Regards, > Gábor > > [1] https://github.com/syslog-ng/syslog-ng > > ________________________________ > From: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu> on behalf of > a.biardi(a)tiscali.it <a.biardi(a)tiscali.it> Sent: Saturday, December 4, > 2021 15:29 To: syslog-ng(a)lists.balabit.hu <syslog-ng(a)lists.balabit.hu> > Subject: [syslog-ng] [patch] new feature for file destination: > persistent symlinks > > CAUTION: This email originated from outside of the organization. Do > not follow guidance, click links, or open attachments unless you > recognize the sender and know the content is safe. > > > Hi all, > > I'm new here. I love syslog-ng: it's well written and documented and > each time I need some new feature, I dig up the documentation and I > find it's already implemented! > > Like many others, I use macros in file destinations (say, > "/var/log/cron.${YEAR}${MONTH}"), but I also want to maintain a > persistent symlink to the current logfile (that would be > something like "/var/log/cron") in order to have "easier" access if > I'm grepping for something manually. > > I couldn't find this feature in syslog-ng, so I wrote the code myself; > in the configuration file, it looks like this: > > destination d_file_cron { > file("/var/log/cron.${YEAR}${MONTH}" symlink_as("/var/log/cron")); > }; > > It works for me on my Slackware boxes. I'm sharing the patch I put > together, hoping that it may be useful to others and perhaps even > considered for inclusion in mainstream. > > From a functional perspective, the "symlink_as" file inherits both > "create-dirs" and file ownership from its file destination > (permissions are not applicable to symlinks, at least on linux). > > Patch applies cleanly to latest production release (3.33). I tried to > keep the original coding style for easier review (admittedly, the code > could have gone into the function that opens the file destination, but > I didn't want to change too much). > > Thanks for the good work and happy logging! > > Andrea. Hi Gábor, Sorry for the delay, daytime job keeps me busy :) Will submit a PR through github asap. I did not consider the case where the template contains something like $PROGRAM or $HOST. As you point out, the code wouldn't fail but you'd get an ever-changing symlink of dubious usefulness. Worth documenting. Cheers, Andrea.

1 0

Selective application of keep-hostname()?
by Steve Bernacki 29 Nov '21

29 Nov '21

I have a use case where I need to selectively apply keep-hostname(yes) to messages from certain source IPs, while defaulting to keep-hostname(no) for the rest of my sources. I know I can apply this option on a per-source basis, but in this case I want to be able to selectively apply this option based on the source IP of the message. I'm trying to avoid having to set up an alternate port if I can. Thanks Steve

2 1

Insider 2021-11: 3.35; MacOS; mqtt() destination updates;
by Peter Czanik (pczanik) 24 Nov '21

24 Nov '21

Dear syslog-ng users, This is the 96th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS syslog-ng 3.35.1 available -------------------------- Version 3.35.1 of syslog-ng is released. New features include: - APT repo for x86 Debian and Ubuntu - Throttle filter - MQTT source - PROXY header before TLS payload You can read more details and download the syslog-ng source code at https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-3.35.1 Sending logs from syslog-ng store box to Splunk ----------------------------------------------- One of the most popular applications to feed Splunk with syslog messages is syslog-ng. However not everyone is happy to work on the command line anymore. This is where syslog-ng store box (SSB), an appliance built around syslog-ng, can help. The SSB GUI provides you not only with an easyto-use interface to configure most syslog-ng features, but also a search interface and complete log life cycle management. It can forward log messages to several destinations, recently also to Splunk’s HTTP Event Collector (HEC). From this blog you can learn about how SSB fits into your logging infrastructure and how to configure SSB for Splunk. https://www.syslog-ng.com/community/b/blog/posts/sending-logs-from-syslog-n… MacOS support ------------- For the past couple of months, Yash Mathne has been working on testing syslog-ng on MacOS as a GSoC (Google Summer of Code) student. He worked both on x86 and on the freshly released ARM hardware. And we have some good news here to share: while there is still room for improvement, most of syslog-ng works perfectly well on MacOS. You can read the two parts of his report at: Overview: https://www.syslog-ng.com/community/b/blog/posts/gsoc-report-syslog-ng-maco… Technical details: https://www.syslog-ng.com/community/b/blog/posts/status-of-syslog-ng-on-mac… Syslog-ng 3.34: MQTT destination with TLS and WebSocket support --------------------------------------------------------------- Version 3.33 of syslog-ng arrived with basic MQTT support. Version 3.34 has added many important features to it: user authentication, TLS support and WebSocket support. These features give you both security and flexibility while sending log messages to an MQTT broker. This blog helps you to make your first steps securing your MQTT connection. https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-3-34-mqtt-destin… WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

syslog-ng-stable rpm repos
by Peter Czanik (pczanik) 22 Nov '21

22 Nov '21

Hi, I just updated my syslog-ng-stable rpm repos to syslog-ng version 3.35. It was released a week ago, quite a few people already use it and nobody reported any problems. So, it should be a safe update. For new features and changes of 3.35, check: https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-3.35.1 For an overview of my unofficial RPM repositories for openSUSE / SLES, Fedora / RHEL (and compatibles), check: https://www.syslog-ng.com/community/b/blog/posts/overview-of-syslog-ng-rpm-… Bye, CzP Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 1

CentOS EoL is near
by Peter Czanik (pczanik) 16 Nov '21

16 Nov '21

Hi, For many years CentOS Linux was the most popular platform to run syslog-ng. Unfortunately, CentOS 8 will be end of life by the end of the year: https://www.centos.org/centos-linux-eol/ I wonder if you already made the switch to something else, RHEL, one of the old or new RHEL clones, CentOS Stream or something completely different. Or are you still in the planning phase? Or completely unaware of the CentOS Linux EoL? I put together a poll on Twitter: https://twitter.com/PCzanik/status/1460610601848299525 and I would appreciate voting and retweets. And of course, I'm very interested in your experiences: where did you switch, why, how did it go. Either here on the mailing list, or directly to me if you do not want to share it publicly. Your answers help me to decide where should I put more effort while packaging and testing syslog-ng. And of course, also what OS should I use more often in my blogs. Previously the focus was CentOS with everything else appearing a lot less, now BSD variants and openSUSE also appear frequently. Thanks, Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

syslog-ng-3.35.1
by Attila Szakacs (aszakacs) 15 Nov '21

15 Nov '21

3.35.1 syslog-ng OSE APT repository >From now on, Ubuntu and Debian packages will be published with every syslog-ng release in the form of an APT repository. We, syslog-ng developers, provide these packages and the APT repository "as is" without warranty of any kind, on a best-effort level. Currently, syslog-ng packages are released for the following distribution versions (x86-64): * Debian: bullseye, buster, stretch, sid, testing * Ubuntu: Impish, Focal, Bionic, Xenial For instructions on how to install syslog-ng on Debian/Ubuntu distributions, see the README<https://github.com/syslog-ng/syslog-ng/blob/master/README.md#debianubuntu>. Highlights * throttle(): added a new filter that allows rate limiting messages based on arbitrary keys in each message. Note: messages over the rate limit are dropped (just like in any other filter). filter f_throttle { throttle( template("$HOST") rate(5000) ); }; (#3781<https://github.com/syslog-ng/syslog-ng/pull/3781>) * mqtt(): added a new source that can be used to receive messages using the MQTT protocol. Supported transports: tcp, ws, ssl, wss Example config: source { mqtt{ topic("sub1"), address("tcp://localhost:4445") }; }; (#3809<https://github.com/syslog-ng/syslog-ng/pull/3809>) Features * afsocket: Socket options, such as ip-ttl() or tcp-keepalive-time(), are traditionally named by their identifier defined in socket(7) and unix(7) man pages. This was not the case with the pass-unix-credentials() option, which - unlike other similar options - was also possible to set globally. A new option called so-passcred() is now introduced, which works similarly how other socket related options do, which also made possible a nice code cleanup in the related code sections. Of course the old name remains supported in compatibility modes. The PR also implements a new source flag ignore-aux-data, which causes syslog-ng not to propagate transport-level auxiliary information to log messages. Auxiliary information includes for example the pid/uid of the sending process in the case of UNIX based transports, OR the X.509 certificate information in case of SSL/TLS encrypted data streams. By setting flags(ignore-aux-data) one can improve performance at the cost of making this information unavailable in the log messages received through affected sources. (#3670<https://github.com/syslog-ng/syslog-ng/pull/3670>) * network: add support for PROXY header before TLS payload This new transport method called proxied-tls-passthrough is capable of detecting the PROXY header<http://www.haproxy.org/download/1.8/doc/proxy-protocol.txt> before the TLS payload. Loggen has been updated with the--proxied-tls-passthrough option for testing purposes. source s_proxied_tls_passthrough{ network( port(1234) transport("proxied-tls-passthrough"), tls( key-file("/path/to/server_key.pem"), cert-file("/path/to/server_cert.pem"), ca-dir("/path/to/ca/") ) ); }; (#3770<https://github.com/syslog-ng/syslog-ng/pull/3770>) * mqtt() destination: added client-id option. It specifies the unique client ID sent to the broker. (#3809<https://github.com/syslog-ng/syslog-ng/pull/3809>) Bugfixes * unset(), groupunset(): fix unwanted removal of values on different log paths Due to a copy-on-write bug, unset() and groupunset() not only removed values from the appropriate log paths, but from all the others where the same message went through. This has been fixed. (#3803<https://github.com/syslog-ng/syslog-ng/pull/3803>) * regexp-parser(): fix storing unnamed capture groups under prefix() (#3810<https://github.com/syslog-ng/syslog-ng/pull/3810>) * loggen: cannot detect plugins on platforms with non .so shared libs (osx) (#3832<https://github.com/syslog-ng/syslog-ng/pull/3832>) Packaging * debian/control: Added libcriterion-dev as a build dependency, where it is available from APT. (debian-bullseye, debian-testing, debian-sid) (#3794<https://github.com/syslog-ng/syslog-ng/pull/3794>) * centos-7: kafka and mqtt modules are now packaged. The following packages are used as dependencies: * librdkafka-devel from EPEL 7 * paho-c-devel from copr:copr.fedorainfracloud.org:czanik:syslog-ng-githead (#3797<https://github.com/syslog-ng/syslog-ng/pull/3797>) * debian: Added bullseye support. (#3794<https://github.com/syslog-ng/syslog-ng/pull/3794>) * bison: support build with bison 3.8 (#3784<https://github.com/syslog-ng/syslog-ng/pull/3784>) Notes to developers * dbld: As new distributions use python3 by default it makes sense to explicitly state older platforms which use python2 instead of the other way around, so it is not necessary to add that new platform to the python3 case. (#3780<https://github.com/syslog-ng/syslog-ng/pull/3780>) * dbld: move dbld image cache from DockerHub to GitHub In 2021, GitHub introduced the GitHub Packages service. Among other repositories - it provides a standard Docker registry. DBLD uses this registry, to avoid unnecessary rebuilding of the images. (#3782<https://github.com/syslog-ng/syslog-ng/pull/3782>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler, Balázs Barkó, Benedek Cserhati, Colin Douch, Gabor Nagy, Laszlo Szemere, László Várady, Norbert Takacs, Parrag Szilárd, Peter Czanik (CzP), Peter Kokai, Robert Paschedag, Ryan Faircloth, Szilárd Parrag, Thomas Klausner, Zoltan Pallagi

1 0