- syslog-ng - lists.balabit.hu

Sequence – making PatternDB creation for syslog-ng easier
by Peter Czanik (pczanik) 09 Nov '21

09 Nov '21

Hi, I try not to spam the mailing list about each of my blogs separately. However, with all its problems I love sysog-ng's PatternDB technology for message parsing. Thanks to the hard work of Fabien Wernli and his colleagues, creating patterns for PatternDB just became a whole lot more easier! "We are well into the 21st century, but most of the log messages still arrive in an unstructured format. For well over a decade, syslog-ng had a solution to turn unstructured messages into name-value pairs, called PatternDB. However, creating a pattern database for PatternDB from scratch is a source of major pain. Or rather, it was: sequence-rtg – a fork of the sequence log analyzer – provides a new hope! It can easily create ready-to-use patterns for your most frequent log messages. Sequence-rtg is still in beta phase, and therefore is a bit rough around the edges. However, once you deal with the initial struggles of creating the database, it works just fine. Especially if you have lots of log messages. My experience was that the more log messages and larger batch sizes I had, the better quality patterns were generated." Read my blog at https://www.syslog-ng.com/community/b/blog/posts/sequence-making-patterndb-… Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Insider 2021-10: OpenSearch; udp-balancer(); mqtt() destination; process accounting;
by Peter Czanik (pczanik) 21 Oct '21

21 Oct '21

Dear syslog-ng users, This is the 95th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Elasticsearch 7.14 and OpenSearch 1.0 are available – and work fine with syslog-ng ------------------------ One of the most popular destinations in syslog-ng is Elasticsearch. Due to the license change of the Elastic stack, some people changed quickly to Grafana/Loki and other technologies. However, most syslog-ng users decided to wait and see. Version 1.0.0 of OpenSearch, a fork of the Elastic code base from before the license change is now available. Elastic also published a new release last week. For this blog, I tested the latest and greatest from both product lines and I’m sharing my experiences. For the impatient: both work perfectly well. https://www.syslog-ng.com/community/b/blog/posts/elasticsearch-7-14-and-ope… Using the udp-balancer() source of syslog-ng PE ----------------------------------------------- UDP-based log collection is so last century. We had TCP-based log collection for decades and TLS encryption to secure connections. Still, UDP is in wide use, especially at large companies and industrial automation, where every change is slow. In most cases, UDP logging is used by networking devices, but sometimes it is just left there from ancient times and people are reluctant to change it. In either case, at higher message rates it can lead to performance problems and thus to message loss. Originally, the udp() source of syslog-ng was single-threaded. That does not scale well with typical multi-core CPUs with slower cores. There are many tricks to enhance UDP performance in syslog-ng. Combining those with the udp-balancer() source of syslog-ng PE gives the most reliable solution. https://www.syslog-ng.com/community/b/blog/posts/using-the-udp-balancer-sou… Syslog-ng 3.33: the MQTT destination ------------------------------------ Version 3.33 of syslog-ng introduced an MQTT destination. It uses the paho-c client library to send log messages to an MQTT broker. The current implementation supports version 3.1 and 3.1.1 of the protocol over non-encrypted connections, but this is only a first step. From this blog, you can learn how to configure and test the mqtt() destination in syslog-ng. Note: syslog-ng 3.34 enhanced the mqtt() destination in many ways. https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-3-33-the-mqtt-de… Collecting process accounting logs on Linux with syslog-ng ---------------------------------------------------------- Process accounting logs are collected into binary log files on Linux. You can turn them into human readable format locally, using various tools. You can also use syslog-ng to read those files. Syslog-ng can parse those binary logs, create name-value pairs from them and store the results. https://www.syslog-ng.com/community/b/blog/posts/collecting-process-account… WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

syslog-ng-stable updated
by Peter Czanik (pczanik) 29 Sep '21

29 Sep '21

Hi, My unofficial syslog-ng-stable repos are now updated to syslog-ng version 3.34. Along the way unsupported openSUSE releases were removed and new releases were added. Note, that if you use Kafka: librdkafka was updated from version 1.0 to 1.5. You can read more about the unofficial syslog-ng-stable repo at https://www.syslog-ng.com/community/b/blog/posts/introducing-the-syslog-ng-… Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Debian 11 repo
by Pal, Laszlo 27 Sep '21

27 Sep '21

Hi, One of my customers started to upgrade Debian 11. Is there any repo for this? Thx L:

2 1

LF Rhel8 Repo
by Pal, Laszlo 27 Sep '21

27 Sep '21

Hi, It seems Rhel8 repo a bit abandoned at https://copr-be.cloud.fedoraproject.org/results/czanik/syslog-ng-stable/. The latest version is 3.31. Is there any other alternative? Thx L:

3 2

Insider 2021-09: 3.34; OpenBSD; OpenSearch; http() destination;
by Peter Czanik (pczanik) 17 Sep '21

17 Sep '21

Dear syslog-ng users, This is the 94th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Version 3.34.1 available ------------------------ Version 3.34.1 of syslog-ng has been released with many interesting new features. There is now a new parser that can parse messages with regular expressions. The throughput of the Redis destination driver has increased drastically. TLS and WebSocket Secure support have been added to the MQTT destination. Performance of the disk-buffer has improved considerably. For a full list of changes and source code, check https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-3.34.1 Syslog-ng updated in OpenBSD ports ---------------------------------- Recently I have found that the number of syslog-ng users on OpenBSD is growing, even with an ancient syslog-ng version in OpenBSD ports that is unable to collect local log messages. Then I remembered that Todd Miller – maintainer of sudo, and my colleague at One Identity – is also an OpenBSD user and developer. I asked him for a little help, which turned out to be quite a lot in the end, but syslog-ng is now updated to the latest version in OpenBSD ports! https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-updated-in-openb… Opensearch and syslog-ng ------------------------ Opensearch is a fork of the Elastic stack code base, made right before the license change. The first release candidate (RC1) has been released recently. Next to plain text files, Elasticsearch is one of the most popular destinations in syslog-ng, but after the license change people started to look for alternatives. I did some quick tests and using the elasticsearch-http() destination, syslog-ng seems to work fine with Opensearch as well. You can learn from this blog how to get started with Opensearch, dashboards and syslog-ng. Another alternative that syslog-ng users explored is Grafana Loki. https://www.syslog-ng.com/community/b/blog/posts/opensearch-and-syslog-ng Creating a new http()-based syslog-ng destination: Seq ------------------------------------------------------ Recently, many services provide an HTTP-based API to send messages. With a bit of luck, the given service is already supported directly by syslog-ng, or by using the Apprise Python library from the syslog-ng Python destination. In other cases, you need to do the research yourself on how the given HTTP-based service works. It might be scary at first, but often, it just takes a bit of experimenting and reading the documentation. In this blog, I’m showing you how to send log messages to Seq, a container-based log management software for application logs. The focus of this blog is to understand what to look for in the documentation of software to create an http()-based destination in syslog-ng. You can install Seq in a container, it is easy, but not necessary. https://www.syslog-ng.com/community/b/blog/posts/creating-a-new-http--based… WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

syslog-ng on NetBSD
by Peter Czanik (pczanik) 14 Sep '21

14 Sep '21

Hi, I just published updated syslog-ng pkgsrc for NetBSD on GitHub for testing. From the README: " syslog-ng pkgsrc for testing Thanks to the syslog-ng team, syslog-ng now works fine on NetBSD. This is my attempt to update syslog-ng in pkgsrc from version 3.17 to 3.34. It works on my machine (AMD64, NetBSD 9.2), but before trying to submit it to pkgsrc I'd love to see, that it also works for others :-) Rename /usr/pkgsrc/sysutils/syslog-ng to something else, copy the syslog-ng directory from this repo to replace it. Now you are ready to compile syslog-ng. " It is available at: https://github.com/czanik/netbsd Any feedback is welcome! Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

syslog-ng-3.34.1
by Attila Szakacs (aszakacs) 14 Sep '21

14 Sep '21

3.34.1 Highlights * regexp-parser(): new parser that can parse messages with regular expressions Example: regexp-parser( template("${MESSAGE}") prefix(".regexp.") patterns("(?<DN>foo)", "(?<DN>ball)") ); regexp-parser() can be used as an intuitive replacement for regexp filters that had their store-matches flag set in order to save those matches. (#3702<https://github.com/syslog-ng/syslog-ng/pull/3702>) * redis(): workers() and batching support The Redis driver now support the workers() option, which specifies the number of parallel workers, and the batch-lines() option. This could drastically increase the throughput of the Redis destination driver. Example: redis( host("localhost") port(6379) command("HINCRBY", "hosts", "$HOST", "1") workers(8) batch-lines(100) batch-timeout(10000) log-fifo-size(100000) ); (#3732<https://github.com/syslog-ng/syslog-ng/pull/3732>, #3745<https://github.com/syslog-ng/syslog-ng/pull/3745>) * mqtt(): TLS and WebSocket Secure support The MQTT destination now supports TLS and WSS. Example config: mqtt( address("ssl://localhost:8883") topic("syslog/$HOST") fallback-topic("syslog/fallback") tls( ca-file("/path/to/ca.crt") key-file("/path/to/client.key") cert-file("/path/to/client.crt") peer-verify(yes) ) ); (#3747<https://github.com/syslog-ng/syslog-ng/pull/3747>) Features * system() source: added support for NetBSD (#3761<https://github.com/syslog-ng/syslog-ng/pull/3761>) * stats: new statistics counter The following statistics are now available for the HTTP destination, and other file and network based sources/destinations: * msg_size_max/msg_size_avg: Shows the largest/average message size of the given source/destination that has been measured so far. * batch_size_max/batch_size_avg: When batching is enabled, then this shows the largest/average batch size of the given source/destination that has been measured so far. * eps_last_1h, eps_last_24h, eps_since_start: Events per second, measured for the last hour, for the last 24 hours, and since syslog-ng startup, respectively. Notes: * Message sizes are calculated from the incoming raw message length on the source side, and from the outgoing formatted message length on the destination side. * EPS counters are just approximate values, they are updated every minute. (#3753<https://github.com/syslog-ng/syslog-ng/pull/3753>) * mqtt(): username/password authentication Example config: mqtt( address("tcp://localhost:1883") topic("syslog/messages") username("user") password("passwd") ); Note: The password is transmitted in cleartext without using ssl:// or wss://. (#3747<https://github.com/syslog-ng/syslog-ng/pull/3747>) * mqtt(): new option http-proxy() for specifying HTTP/HTTPS proxy for WebSocket connections (#3747<https://github.com/syslog-ng/syslog-ng/pull/3747>) * syslog-ng-ctl: new flag for pruning statistics syslog-ng-ctl stats --remove-orphans can be used to remove "orphaned" statistic counters. It is useful when, for example, a templated file destination ($YEAR.$MONTH.$DAY) produces a lot of stats, and one wants to remove those abandoned counters occasionally/conditionally. (#3760<https://github.com/syslog-ng/syslog-ng/pull/3760>) * disk-buffer(): added a new option to reliable disk-buffer: qout-size(). This option sets the number of messages that are stored in the memory in addition to storing them on disk. The default value is 1000. This serves performance purposes and offers the same no-message-loss guarantees as before. It can be used to maintain a higher throughput when only a small number of messages are waiting in the disk-buffer. (#3754<https://github.com/syslog-ng/syslog-ng/pull/3754>) Bugfixes * network(), syslog(): fixed network sources on NetBSD On NetBSD, TCP-based network sources closed their listeners shortly after startup due to a non-portable TCP keepalive setting. This has been fixed. (#3751<https://github.com/syslog-ng/syslog-ng/pull/3751>) * disk-buffer(): fixed a very rare case, where the reliable disk-buffer never resumed after triggering flow-control. (#3752<https://github.com/syslog-ng/syslog-ng/pull/3752>) * disk-buffer(): fixed a rare memory leak that occurred when mem-buf-length() or mem-buf-size() was configured incorrectly (#3750<https://github.com/syslog-ng/syslog-ng/pull/3750>) * redis(): fixed command errors that were not detected and marked as successful delivery (#3748<https://github.com/syslog-ng/syslog-ng/pull/3748>) Notes to developers * Light framework: new proxy-related options are supported with loggen: --proxy-src-ip, --proxy-dst-ip, --proxy-src-port, --proxy-dst-port (#3766<https://github.com/syslog-ng/syslog-ng/pull/3766>) * log-threaded-dest: descendant drivers from LogThreadedDest no longer inherit batch-lines() and batch-timeout() automatically. Each driver have to opt-in for these options with log_threaded_dest_driver_batch_option. log_threaded_dest_driver_option has been renamed to log_threaded_dest_driver_general_option, and log_threaded_dest_driver_workers_option have been added similarly to the batch-related options. (#3741<https://github.com/syslog-ng/syslog-ng/pull/3741>) Other changes * disk-buffer(): performance improvements Based on our measurements, the following can be expected compared to the previous syslog-ng release (v3.33.1): * non-reliable disk buffer: up to 50% performance gain; * reliable disk buffer: up to 80% increase in performance. (#3743<https://github.com/syslog-ng/syslog-ng/pull/3743>, #3746<https://github.com/syslog-ng/syslog-ng/pull/3746>, #3754<https://github.com/syslog-ng/syslog-ng/pull/3754>, #3756<https://github.com/syslog-ng/syslog-ng/pull/3756>, #3757<https://github.com/syslog-ng/syslog-ng/pull/3757>) * disk-buffer(): the default value of the following options has been changed for performance reasons: * truncate-size-ratio(): from 0.01 to 0.1 (from 1% to 10%) * qout-size(): from 64 to 1000 (this affects only the non-reliable disk buffer) (#3757<https://github.com/syslog-ng/syslog-ng/pull/3757>) * kafka-c(): properties-file() option is removed Please list librdkafka properties in the config() option in syslog-ng's configuration. See librdkafka configuration here<https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md>. (#3704<https://github.com/syslog-ng/syslog-ng/pull/3704>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Attila Szakacs, Balazs Scheidler, Balázs Barkó, Benedek Cserhati, Fabrice Fontaine, Gabor Nagy, Laszlo Szemere, LittleFish33, László Várady, Norbert Takacs, Parrag Szilárd, Peter Czanik, Peter Kokai, Zoltan Pallagi

2 1

librdkafka config property error on startup
by Sanjay Patel 09 Sep '21

09 Sep '21

I am trying to use Kafka-c with Syslog-ng. I have version Kafka 3.31 of Kafka installed, and I have syslog-ng-mod-rdkafka installed. When I start Syslog-ng, I get the following error. I can't figure out where it is getting this setting from. Syslog starts, but it does not send any logs to Kafka. kafka: error setting librdkafka config property; name='topic.partitioner', value='murmur2_random', error='No such configuration property: My Kafka config looks as follows: @include "scl.conf" destination d_kafka { kafka-c( bootstrap-servers("10.254.164.15:9092") topic("incoming") ); }; log { source(s_network); destination(d_kafka); }; Any help will be deeply appreciated. SKP <https://t.sidekickopen07.com/s3t/o/5/f18dQhb0S7n28cFBYDW5xwfpz2zGCvGW40Fv_6 2SXRqbW4NBnl23Lrg1WW5vbH2n1cd60nf197v5Y04?si=7000000000968799&pi=6743fafd-f6 76-41ef-ba50-d690c0278659>

3 3

Re: [syslog-ng] FW: Resource temporarily unavailable while reading from system.journal
by Alexandre Santos 02 Sep '21

02 Sep '21

Hi Laszlo, I am running a debian10 with syslog-ng 3.32. If using the default syslog-ng configuration for Debian 10, the error is reproducible. It happened to me when stimulating the syslog-ng using logger command. Sorry for the delay in reply. Thanks and regards, Alex On Wed, Aug 18, 2021 at 9:39 AM Alexandre Santos <ASantos(a)infinera.com> wrote: > > > > > *From:* syslog-ng <syslog-ng-bounces(a)lists.balabit.hu> * On Behalf Of *Laszlo > Szemere (lszemere) > *Sent:* 17 de agosto de 2021 15:20 > *To:* Syslog-ng users' and developers' mailing list < > syslog-ng(a)lists.balabit.hu> > *Subject:* Re: [syslog-ng] Resource temporarily unavailable while reading > from system.journal > > > > Hello Alex, > > > > sorry for not responding sooner. > > > > From my experience, the most common reasons behind this error messages are: > > > > 1) Syslog-ng do not have the proper permissions to read /dev/log > > 2) /dev/log is already opened by an another process. (Maybe Syslog-ng > started multiple times.) > > > > > > If the problem still exists, please share the details of your platform and > I will try to reproduce your problem with the attached configuration. > > > > > > Best regards, > > Laszlo > > > ------------------------------ > > *From:* syslog-ng <syslog-ng-bounces(a)lists.balabit.hu> on behalf of > Alexandre Santos <alexandre.rosas.santos(a)gmail.com> > *Sent:* Thursday, August 12, 2021 15:40 > *To:* Syslog-ng users' and developers' mailing list < > syslog-ng(a)lists.balabit.hu> > *Subject:* [syslog-ng] Resource temporarily unavailable while reading > from system.journal > > > > Hi, > > > > I am seeing the following strace log when running the syslog-ng 3.32 with > the configuration in the attachment: > > root@myhost:/home/shared# sudo strace -f -yyy -s 256 -e trace=read -p 2539 > strace: Process 2539 attached with 4 threads > [pid 2539] read(48<anon_inode:[timerfd]>, "\1\0\0\0\0\0\0\0", 8) = 8 > [pid 2539] read(11<anon_inode:inotify>, > "\2\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 > [pid 2539] read(11<anon_inode:inotify>, 0x7ffda019f930, 272) = -1 EAGAIN > (Resource temporarily unavailable) > [pid 2539] read(11<anon_inode:inotify>, > "\2\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 > [pid 2539] read(11<anon_inode:inotify>, 0x7ffda019f930, 272) = -1 EAGAIN > (Resource temporarily unavailable) > [pid 2539] read(11<anon_inode:inotify>, > "\2\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 > [pid 2539] read(11<anon_inode:inotify>, 0x7ffda019f930, 272) = -1 EAGAIN > (Resource temporarily unavailable) > [pid 2539] read(11<anon_inode:inotify>, > "\2\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 > [pid 2539] read(11<anon_inode:inotify>, 0x7ffda019f930, 272) = -1 EAGAIN > (Resource temporarily unavailable) > [pid 2539] read(11<anon_inode:inotify>, > "\2\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 > [pid 2539] read(11<anon_inode:inotify>, 0x7ffda019f930, 272) = -1 EAGAIN > (Resource temporarily unavailable) > [pid 2539] read(11<anon_inode:inotify>, > "\2\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 > [pid 2539] read(11<anon_inode:inotify>, 0x7ffda019f930, 272) = -1 EAGAIN > (Resource temporarily unavailable) > [pid 2539] read(11<anon_inode:inotify>, > "\2\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 > [pid 2539] read(11<anon_inode:inotify>, 0x7ffda019f930, 272) = -1 EAGAIN > (Resource temporarily unavailable) > [pid 2539] read(11<anon_inode:inotify>, > "\2\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 > [pid 2539] read(11<anon_inode:inotify>, 0x7ffda019f930, 272) = -1 EAGAIN > (Resource temporarily unavailable) > [pid 2539] read(11<anon_inode:inotify>, > "\2\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 > [pid 2539] read(11<anon_inode:inotify>, 0x7ffda019f930, 272) = -1 EAGAIN > (Resource temporarily unavailable) > ^Cstrace: Process 2539 detached > strace: Process 11623 detached > strace: Process 20518 detached > strace: Process 22517 detached > > > > This normal behaviour? > > > > Thanks & Regards, > > Alex >

1 0