- syslog-ng - lists.balabit.hu

Insider 2021-11: 3.35; MacOS; mqtt() destination updates;
by Peter Czanik (pczanik) 24 Nov '21

24 Nov '21

Dear syslog-ng users, This is the 96th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS syslog-ng 3.35.1 available -------------------------- Version 3.35.1 of syslog-ng is released. New features include: - APT repo for x86 Debian and Ubuntu - Throttle filter - MQTT source - PROXY header before TLS payload You can read more details and download the syslog-ng source code at https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-3.35.1 Sending logs from syslog-ng store box to Splunk ----------------------------------------------- One of the most popular applications to feed Splunk with syslog messages is syslog-ng. However not everyone is happy to work on the command line anymore. This is where syslog-ng store box (SSB), an appliance built around syslog-ng, can help. The SSB GUI provides you not only with an easyto-use interface to configure most syslog-ng features, but also a search interface and complete log life cycle management. It can forward log messages to several destinations, recently also to Splunk’s HTTP Event Collector (HEC). From this blog you can learn about how SSB fits into your logging infrastructure and how to configure SSB for Splunk. https://www.syslog-ng.com/community/b/blog/posts/sending-logs-from-syslog-n… MacOS support ------------- For the past couple of months, Yash Mathne has been working on testing syslog-ng on MacOS as a GSoC (Google Summer of Code) student. He worked both on x86 and on the freshly released ARM hardware. And we have some good news here to share: while there is still room for improvement, most of syslog-ng works perfectly well on MacOS. You can read the two parts of his report at: Overview: https://www.syslog-ng.com/community/b/blog/posts/gsoc-report-syslog-ng-maco… Technical details: https://www.syslog-ng.com/community/b/blog/posts/status-of-syslog-ng-on-mac… Syslog-ng 3.34: MQTT destination with TLS and WebSocket support --------------------------------------------------------------- Version 3.33 of syslog-ng arrived with basic MQTT support. Version 3.34 has added many important features to it: user authentication, TLS support and WebSocket support. These features give you both security and flexibility while sending log messages to an MQTT broker. This blog helps you to make your first steps securing your MQTT connection. https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-3-34-mqtt-destin… WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

syslog-ng-stable rpm repos
by Peter Czanik (pczanik) 22 Nov '21

22 Nov '21

Hi, I just updated my syslog-ng-stable rpm repos to syslog-ng version 3.35. It was released a week ago, quite a few people already use it and nobody reported any problems. So, it should be a safe update. For new features and changes of 3.35, check: https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-3.35.1 For an overview of my unofficial RPM repositories for openSUSE / SLES, Fedora / RHEL (and compatibles), check: https://www.syslog-ng.com/community/b/blog/posts/overview-of-syslog-ng-rpm-… Bye, CzP Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 1

CentOS EoL is near
by Peter Czanik (pczanik) 16 Nov '21

16 Nov '21

Hi, For many years CentOS Linux was the most popular platform to run syslog-ng. Unfortunately, CentOS 8 will be end of life by the end of the year: https://www.centos.org/centos-linux-eol/ I wonder if you already made the switch to something else, RHEL, one of the old or new RHEL clones, CentOS Stream or something completely different. Or are you still in the planning phase? Or completely unaware of the CentOS Linux EoL? I put together a poll on Twitter: https://twitter.com/PCzanik/status/1460610601848299525 and I would appreciate voting and retweets. And of course, I'm very interested in your experiences: where did you switch, why, how did it go. Either here on the mailing list, or directly to me if you do not want to share it publicly. Your answers help me to decide where should I put more effort while packaging and testing syslog-ng. And of course, also what OS should I use more often in my blogs. Previously the focus was CentOS with everything else appearing a lot less, now BSD variants and openSUSE also appear frequently. Thanks, Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

syslog-ng-3.35.1
by Attila Szakacs (aszakacs) 15 Nov '21

15 Nov '21

3.35.1 syslog-ng OSE APT repository >From now on, Ubuntu and Debian packages will be published with every syslog-ng release in the form of an APT repository. We, syslog-ng developers, provide these packages and the APT repository "as is" without warranty of any kind, on a best-effort level. Currently, syslog-ng packages are released for the following distribution versions (x86-64): * Debian: bullseye, buster, stretch, sid, testing * Ubuntu: Impish, Focal, Bionic, Xenial For instructions on how to install syslog-ng on Debian/Ubuntu distributions, see the README<https://github.com/syslog-ng/syslog-ng/blob/master/README.md#debianubuntu>. Highlights * throttle(): added a new filter that allows rate limiting messages based on arbitrary keys in each message. Note: messages over the rate limit are dropped (just like in any other filter). filter f_throttle { throttle( template("$HOST") rate(5000) ); }; (#3781<https://github.com/syslog-ng/syslog-ng/pull/3781>) * mqtt(): added a new source that can be used to receive messages using the MQTT protocol. Supported transports: tcp, ws, ssl, wss Example config: source { mqtt{ topic("sub1"), address("tcp://localhost:4445") }; }; (#3809<https://github.com/syslog-ng/syslog-ng/pull/3809>) Features * afsocket: Socket options, such as ip-ttl() or tcp-keepalive-time(), are traditionally named by their identifier defined in socket(7) and unix(7) man pages. This was not the case with the pass-unix-credentials() option, which - unlike other similar options - was also possible to set globally. A new option called so-passcred() is now introduced, which works similarly how other socket related options do, which also made possible a nice code cleanup in the related code sections. Of course the old name remains supported in compatibility modes. The PR also implements a new source flag ignore-aux-data, which causes syslog-ng not to propagate transport-level auxiliary information to log messages. Auxiliary information includes for example the pid/uid of the sending process in the case of UNIX based transports, OR the X.509 certificate information in case of SSL/TLS encrypted data streams. By setting flags(ignore-aux-data) one can improve performance at the cost of making this information unavailable in the log messages received through affected sources. (#3670<https://github.com/syslog-ng/syslog-ng/pull/3670>) * network: add support for PROXY header before TLS payload This new transport method called proxied-tls-passthrough is capable of detecting the PROXY header<http://www.haproxy.org/download/1.8/doc/proxy-protocol.txt> before the TLS payload. Loggen has been updated with the--proxied-tls-passthrough option for testing purposes. source s_proxied_tls_passthrough{ network( port(1234) transport("proxied-tls-passthrough"), tls( key-file("/path/to/server_key.pem"), cert-file("/path/to/server_cert.pem"), ca-dir("/path/to/ca/") ) ); }; (#3770<https://github.com/syslog-ng/syslog-ng/pull/3770>) * mqtt() destination: added client-id option. It specifies the unique client ID sent to the broker. (#3809<https://github.com/syslog-ng/syslog-ng/pull/3809>) Bugfixes * unset(), groupunset(): fix unwanted removal of values on different log paths Due to a copy-on-write bug, unset() and groupunset() not only removed values from the appropriate log paths, but from all the others where the same message went through. This has been fixed. (#3803<https://github.com/syslog-ng/syslog-ng/pull/3803>) * regexp-parser(): fix storing unnamed capture groups under prefix() (#3810<https://github.com/syslog-ng/syslog-ng/pull/3810>) * loggen: cannot detect plugins on platforms with non .so shared libs (osx) (#3832<https://github.com/syslog-ng/syslog-ng/pull/3832>) Packaging * debian/control: Added libcriterion-dev as a build dependency, where it is available from APT. (debian-bullseye, debian-testing, debian-sid) (#3794<https://github.com/syslog-ng/syslog-ng/pull/3794>) * centos-7: kafka and mqtt modules are now packaged. The following packages are used as dependencies: * librdkafka-devel from EPEL 7 * paho-c-devel from copr:copr.fedorainfracloud.org:czanik:syslog-ng-githead (#3797<https://github.com/syslog-ng/syslog-ng/pull/3797>) * debian: Added bullseye support. (#3794<https://github.com/syslog-ng/syslog-ng/pull/3794>) * bison: support build with bison 3.8 (#3784<https://github.com/syslog-ng/syslog-ng/pull/3784>) Notes to developers * dbld: As new distributions use python3 by default it makes sense to explicitly state older platforms which use python2 instead of the other way around, so it is not necessary to add that new platform to the python3 case. (#3780<https://github.com/syslog-ng/syslog-ng/pull/3780>) * dbld: move dbld image cache from DockerHub to GitHub In 2021, GitHub introduced the GitHub Packages service. Among other repositories - it provides a standard Docker registry. DBLD uses this registry, to avoid unnecessary rebuilding of the images. (#3782<https://github.com/syslog-ng/syslog-ng/pull/3782>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler, Balázs Barkó, Benedek Cserhati, Colin Douch, Gabor Nagy, Laszlo Szemere, László Várady, Norbert Takacs, Parrag Szilárd, Peter Czanik (CzP), Peter Kokai, Robert Paschedag, Ryan Faircloth, Szilárd Parrag, Thomas Klausner, Zoltan Pallagi

1 0

Sequence – making PatternDB creation for syslog-ng easier
by Peter Czanik (pczanik) 09 Nov '21

09 Nov '21

Hi, I try not to spam the mailing list about each of my blogs separately. However, with all its problems I love sysog-ng's PatternDB technology for message parsing. Thanks to the hard work of Fabien Wernli and his colleagues, creating patterns for PatternDB just became a whole lot more easier! "We are well into the 21st century, but most of the log messages still arrive in an unstructured format. For well over a decade, syslog-ng had a solution to turn unstructured messages into name-value pairs, called PatternDB. However, creating a pattern database for PatternDB from scratch is a source of major pain. Or rather, it was: sequence-rtg – a fork of the sequence log analyzer – provides a new hope! It can easily create ready-to-use patterns for your most frequent log messages. Sequence-rtg is still in beta phase, and therefore is a bit rough around the edges. However, once you deal with the initial struggles of creating the database, it works just fine. Especially if you have lots of log messages. My experience was that the more log messages and larger batch sizes I had, the better quality patterns were generated." Read my blog at https://www.syslog-ng.com/community/b/blog/posts/sequence-making-patterndb-… Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Insider 2021-10: OpenSearch; udp-balancer(); mqtt() destination; process accounting;
by Peter Czanik (pczanik) 21 Oct '21

21 Oct '21

Dear syslog-ng users, This is the 95th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Elasticsearch 7.14 and OpenSearch 1.0 are available – and work fine with syslog-ng ------------------------ One of the most popular destinations in syslog-ng is Elasticsearch. Due to the license change of the Elastic stack, some people changed quickly to Grafana/Loki and other technologies. However, most syslog-ng users decided to wait and see. Version 1.0.0 of OpenSearch, a fork of the Elastic code base from before the license change is now available. Elastic also published a new release last week. For this blog, I tested the latest and greatest from both product lines and I’m sharing my experiences. For the impatient: both work perfectly well. https://www.syslog-ng.com/community/b/blog/posts/elasticsearch-7-14-and-ope… Using the udp-balancer() source of syslog-ng PE ----------------------------------------------- UDP-based log collection is so last century. We had TCP-based log collection for decades and TLS encryption to secure connections. Still, UDP is in wide use, especially at large companies and industrial automation, where every change is slow. In most cases, UDP logging is used by networking devices, but sometimes it is just left there from ancient times and people are reluctant to change it. In either case, at higher message rates it can lead to performance problems and thus to message loss. Originally, the udp() source of syslog-ng was single-threaded. That does not scale well with typical multi-core CPUs with slower cores. There are many tricks to enhance UDP performance in syslog-ng. Combining those with the udp-balancer() source of syslog-ng PE gives the most reliable solution. https://www.syslog-ng.com/community/b/blog/posts/using-the-udp-balancer-sou… Syslog-ng 3.33: the MQTT destination ------------------------------------ Version 3.33 of syslog-ng introduced an MQTT destination. It uses the paho-c client library to send log messages to an MQTT broker. The current implementation supports version 3.1 and 3.1.1 of the protocol over non-encrypted connections, but this is only a first step. From this blog, you can learn how to configure and test the mqtt() destination in syslog-ng. Note: syslog-ng 3.34 enhanced the mqtt() destination in many ways. https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-3-33-the-mqtt-de… Collecting process accounting logs on Linux with syslog-ng ---------------------------------------------------------- Process accounting logs are collected into binary log files on Linux. You can turn them into human readable format locally, using various tools. You can also use syslog-ng to read those files. Syslog-ng can parse those binary logs, create name-value pairs from them and store the results. https://www.syslog-ng.com/community/b/blog/posts/collecting-process-account… WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

syslog-ng-stable updated
by Peter Czanik (pczanik) 29 Sep '21

29 Sep '21

Hi, My unofficial syslog-ng-stable repos are now updated to syslog-ng version 3.34. Along the way unsupported openSUSE releases were removed and new releases were added. Note, that if you use Kafka: librdkafka was updated from version 1.0 to 1.5. You can read more about the unofficial syslog-ng-stable repo at https://www.syslog-ng.com/community/b/blog/posts/introducing-the-syslog-ng-… Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Debian 11 repo
by Pal, Laszlo 27 Sep '21

27 Sep '21

Hi, One of my customers started to upgrade Debian 11. Is there any repo for this? Thx L:

2 1

LF Rhel8 Repo
by Pal, Laszlo 27 Sep '21

27 Sep '21

Hi, It seems Rhel8 repo a bit abandoned at https://copr-be.cloud.fedoraproject.org/results/czanik/syslog-ng-stable/. The latest version is 3.31. Is there any other alternative? Thx L:

3 2

Insider 2021-09: 3.34; OpenBSD; OpenSearch; http() destination;
by Peter Czanik (pczanik) 17 Sep '21

17 Sep '21

Dear syslog-ng users, This is the 94th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Version 3.34.1 available ------------------------ Version 3.34.1 of syslog-ng has been released with many interesting new features. There is now a new parser that can parse messages with regular expressions. The throughput of the Redis destination driver has increased drastically. TLS and WebSocket Secure support have been added to the MQTT destination. Performance of the disk-buffer has improved considerably. For a full list of changes and source code, check https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-3.34.1 Syslog-ng updated in OpenBSD ports ---------------------------------- Recently I have found that the number of syslog-ng users on OpenBSD is growing, even with an ancient syslog-ng version in OpenBSD ports that is unable to collect local log messages. Then I remembered that Todd Miller – maintainer of sudo, and my colleague at One Identity – is also an OpenBSD user and developer. I asked him for a little help, which turned out to be quite a lot in the end, but syslog-ng is now updated to the latest version in OpenBSD ports! https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-updated-in-openb… Opensearch and syslog-ng ------------------------ Opensearch is a fork of the Elastic stack code base, made right before the license change. The first release candidate (RC1) has been released recently. Next to plain text files, Elasticsearch is one of the most popular destinations in syslog-ng, but after the license change people started to look for alternatives. I did some quick tests and using the elasticsearch-http() destination, syslog-ng seems to work fine with Opensearch as well. You can learn from this blog how to get started with Opensearch, dashboards and syslog-ng. Another alternative that syslog-ng users explored is Grafana Loki. https://www.syslog-ng.com/community/b/blog/posts/opensearch-and-syslog-ng Creating a new http()-based syslog-ng destination: Seq ------------------------------------------------------ Recently, many services provide an HTTP-based API to send messages. With a bit of luck, the given service is already supported directly by syslog-ng, or by using the Apprise Python library from the syslog-ng Python destination. In other cases, you need to do the research yourself on how the given HTTP-based service works. It might be scary at first, but often, it just takes a bit of experimenting and reading the documentation. In this blog, I’m showing you how to send log messages to Seq, a container-based log management software for application logs. The focus of this blog is to understand what to look for in the documentation of software to create an http()-based destination in syslog-ng. You can install Seq in a container, it is easy, but not necessary. https://www.syslog-ng.com/community/b/blog/posts/creating-a-new-http--based… WEBINARS * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

syslog-ng on NetBSD
by Peter Czanik (pczanik) 14 Sep '21

14 Sep '21

Hi, I just published updated syslog-ng pkgsrc for NetBSD on GitHub for testing. From the README: " syslog-ng pkgsrc for testing Thanks to the syslog-ng team, syslog-ng now works fine on NetBSD. This is my attempt to update syslog-ng in pkgsrc from version 3.17 to 3.34. It works on my machine (AMD64, NetBSD 9.2), but before trying to submit it to pkgsrc I'd love to see, that it also works for others :-) Rename /usr/pkgsrc/sysutils/syslog-ng to something else, copy the syslog-ng directory from this repo to replace it. Now you are ready to compile syslog-ng. " It is available at: https://github.com/czanik/netbsd Any feedback is welcome! Peter Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

syslog-ng-3.34.1
by Attila Szakacs (aszakacs) 14 Sep '21

14 Sep '21

3.34.1 Highlights * regexp-parser(): new parser that can parse messages with regular expressions Example: regexp-parser( template("${MESSAGE}") prefix(".regexp.") patterns("(?<DN>foo)", "(?<DN>ball)") ); regexp-parser() can be used as an intuitive replacement for regexp filters that had their store-matches flag set in order to save those matches. (#3702<https://github.com/syslog-ng/syslog-ng/pull/3702>) * redis(): workers() and batching support The Redis driver now support the workers() option, which specifies the number of parallel workers, and the batch-lines() option. This could drastically increase the throughput of the Redis destination driver. Example: redis( host("localhost") port(6379) command("HINCRBY", "hosts", "$HOST", "1") workers(8) batch-lines(100) batch-timeout(10000) log-fifo-size(100000) ); (#3732<https://github.com/syslog-ng/syslog-ng/pull/3732>, #3745<https://github.com/syslog-ng/syslog-ng/pull/3745>) * mqtt(): TLS and WebSocket Secure support The MQTT destination now supports TLS and WSS. Example config: mqtt( address("ssl://localhost:8883") topic("syslog/$HOST") fallback-topic("syslog/fallback") tls( ca-file("/path/to/ca.crt") key-file("/path/to/client.key") cert-file("/path/to/client.crt") peer-verify(yes) ) ); (#3747<https://github.com/syslog-ng/syslog-ng/pull/3747>) Features * system() source: added support for NetBSD (#3761<https://github.com/syslog-ng/syslog-ng/pull/3761>) * stats: new statistics counter The following statistics are now available for the HTTP destination, and other file and network based sources/destinations: * msg_size_max/msg_size_avg: Shows the largest/average message size of the given source/destination that has been measured so far. * batch_size_max/batch_size_avg: When batching is enabled, then this shows the largest/average batch size of the given source/destination that has been measured so far. * eps_last_1h, eps_last_24h, eps_since_start: Events per second, measured for the last hour, for the last 24 hours, and since syslog-ng startup, respectively. Notes: * Message sizes are calculated from the incoming raw message length on the source side, and from the outgoing formatted message length on the destination side. * EPS counters are just approximate values, they are updated every minute. (#3753<https://github.com/syslog-ng/syslog-ng/pull/3753>) * mqtt(): username/password authentication Example config: mqtt( address("tcp://localhost:1883") topic("syslog/messages") username("user") password("passwd") ); Note: The password is transmitted in cleartext without using ssl:// or wss://. (#3747<https://github.com/syslog-ng/syslog-ng/pull/3747>) * mqtt(): new option http-proxy() for specifying HTTP/HTTPS proxy for WebSocket connections (#3747<https://github.com/syslog-ng/syslog-ng/pull/3747>) * syslog-ng-ctl: new flag for pruning statistics syslog-ng-ctl stats --remove-orphans can be used to remove "orphaned" statistic counters. It is useful when, for example, a templated file destination ($YEAR.$MONTH.$DAY) produces a lot of stats, and one wants to remove those abandoned counters occasionally/conditionally. (#3760<https://github.com/syslog-ng/syslog-ng/pull/3760>) * disk-buffer(): added a new option to reliable disk-buffer: qout-size(). This option sets the number of messages that are stored in the memory in addition to storing them on disk. The default value is 1000. This serves performance purposes and offers the same no-message-loss guarantees as before. It can be used to maintain a higher throughput when only a small number of messages are waiting in the disk-buffer. (#3754<https://github.com/syslog-ng/syslog-ng/pull/3754>) Bugfixes * network(), syslog(): fixed network sources on NetBSD On NetBSD, TCP-based network sources closed their listeners shortly after startup due to a non-portable TCP keepalive setting. This has been fixed. (#3751<https://github.com/syslog-ng/syslog-ng/pull/3751>) * disk-buffer(): fixed a very rare case, where the reliable disk-buffer never resumed after triggering flow-control. (#3752<https://github.com/syslog-ng/syslog-ng/pull/3752>) * disk-buffer(): fixed a rare memory leak that occurred when mem-buf-length() or mem-buf-size() was configured incorrectly (#3750<https://github.com/syslog-ng/syslog-ng/pull/3750>) * redis(): fixed command errors that were not detected and marked as successful delivery (#3748<https://github.com/syslog-ng/syslog-ng/pull/3748>) Notes to developers * Light framework: new proxy-related options are supported with loggen: --proxy-src-ip, --proxy-dst-ip, --proxy-src-port, --proxy-dst-port (#3766<https://github.com/syslog-ng/syslog-ng/pull/3766>) * log-threaded-dest: descendant drivers from LogThreadedDest no longer inherit batch-lines() and batch-timeout() automatically. Each driver have to opt-in for these options with log_threaded_dest_driver_batch_option. log_threaded_dest_driver_option has been renamed to log_threaded_dest_driver_general_option, and log_threaded_dest_driver_workers_option have been added similarly to the batch-related options. (#3741<https://github.com/syslog-ng/syslog-ng/pull/3741>) Other changes * disk-buffer(): performance improvements Based on our measurements, the following can be expected compared to the previous syslog-ng release (v3.33.1): * non-reliable disk buffer: up to 50% performance gain; * reliable disk buffer: up to 80% increase in performance. (#3743<https://github.com/syslog-ng/syslog-ng/pull/3743>, #3746<https://github.com/syslog-ng/syslog-ng/pull/3746>, #3754<https://github.com/syslog-ng/syslog-ng/pull/3754>, #3756<https://github.com/syslog-ng/syslog-ng/pull/3756>, #3757<https://github.com/syslog-ng/syslog-ng/pull/3757>) * disk-buffer(): the default value of the following options has been changed for performance reasons: * truncate-size-ratio(): from 0.01 to 0.1 (from 1% to 10%) * qout-size(): from 64 to 1000 (this affects only the non-reliable disk buffer) (#3757<https://github.com/syslog-ng/syslog-ng/pull/3757>) * kafka-c(): properties-file() option is removed Please list librdkafka properties in the config() option in syslog-ng's configuration. See librdkafka configuration here<https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md>. (#3704<https://github.com/syslog-ng/syslog-ng/pull/3704>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Attila Szakacs, Balazs Scheidler, Balázs Barkó, Benedek Cserhati, Fabrice Fontaine, Gabor Nagy, Laszlo Szemere, LittleFish33, László Várady, Norbert Takacs, Parrag Szilárd, Peter Czanik, Peter Kokai, Zoltan Pallagi

2 1

librdkafka config property error on startup
by Sanjay Patel 09 Sep '21

09 Sep '21

I am trying to use Kafka-c with Syslog-ng. I have version Kafka 3.31 of Kafka installed, and I have syslog-ng-mod-rdkafka installed. When I start Syslog-ng, I get the following error. I can't figure out where it is getting this setting from. Syslog starts, but it does not send any logs to Kafka. kafka: error setting librdkafka config property; name='topic.partitioner', value='murmur2_random', error='No such configuration property: My Kafka config looks as follows: @include "scl.conf" destination d_kafka { kafka-c( bootstrap-servers("10.254.164.15:9092") topic("incoming") ); }; log { source(s_network); destination(d_kafka); }; Any help will be deeply appreciated. SKP <https://t.sidekickopen07.com/s3t/o/5/f18dQhb0S7n28cFBYDW5xwfpz2zGCvGW40Fv_6 2SXRqbW4NBnl23Lrg1WW5vbH2n1cd60nf197v5Y04?si=7000000000968799&pi=6743fafd-f6 76-41ef-ba50-d690c0278659>

3 3

Re: [syslog-ng] FW: Resource temporarily unavailable while reading from system.journal
by Alexandre Santos 02 Sep '21

02 Sep '21

Hi Laszlo, I am running a debian10 with syslog-ng 3.32. If using the default syslog-ng configuration for Debian 10, the error is reproducible. It happened to me when stimulating the syslog-ng using logger command. Sorry for the delay in reply. Thanks and regards, Alex On Wed, Aug 18, 2021 at 9:39 AM Alexandre Santos <ASantos(a)infinera.com> wrote: > > > > > *From:* syslog-ng <syslog-ng-bounces(a)lists.balabit.hu> * On Behalf Of *Laszlo > Szemere (lszemere) > *Sent:* 17 de agosto de 2021 15:20 > *To:* Syslog-ng users' and developers' mailing list < > syslog-ng(a)lists.balabit.hu> > *Subject:* Re: [syslog-ng] Resource temporarily unavailable while reading > from system.journal > > > > Hello Alex, > > > > sorry for not responding sooner. > > > > From my experience, the most common reasons behind this error messages are: > > > > 1) Syslog-ng do not have the proper permissions to read /dev/log > > 2) /dev/log is already opened by an another process. (Maybe Syslog-ng > started multiple times.) > > > > > > If the problem still exists, please share the details of your platform and > I will try to reproduce your problem with the attached configuration. > > > > > > Best regards, > > Laszlo > > > ------------------------------ > > *From:* syslog-ng <syslog-ng-bounces(a)lists.balabit.hu> on behalf of > Alexandre Santos <alexandre.rosas.santos(a)gmail.com> > *Sent:* Thursday, August 12, 2021 15:40 > *To:* Syslog-ng users' and developers' mailing list < > syslog-ng(a)lists.balabit.hu> > *Subject:* [syslog-ng] Resource temporarily unavailable while reading > from system.journal > > > > Hi, > > > > I am seeing the following strace log when running the syslog-ng 3.32 with > the configuration in the attachment: > > root@myhost:/home/shared# sudo strace -f -yyy -s 256 -e trace=read -p 2539 > strace: Process 2539 attached with 4 threads > [pid 2539] read(48<anon_inode:[timerfd]>, "\1\0\0\0\0\0\0\0", 8) = 8 > [pid 2539] read(11<anon_inode:inotify>, > "\2\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 > [pid 2539] read(11<anon_inode:inotify>, 0x7ffda019f930, 272) = -1 EAGAIN > (Resource temporarily unavailable) > [pid 2539] read(11<anon_inode:inotify>, > "\2\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 > [pid 2539] read(11<anon_inode:inotify>, 0x7ffda019f930, 272) = -1 EAGAIN > (Resource temporarily unavailable) > [pid 2539] read(11<anon_inode:inotify>, > "\2\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 > [pid 2539] read(11<anon_inode:inotify>, 0x7ffda019f930, 272) = -1 EAGAIN > (Resource temporarily unavailable) > [pid 2539] read(11<anon_inode:inotify>, > "\2\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 > [pid 2539] read(11<anon_inode:inotify>, 0x7ffda019f930, 272) = -1 EAGAIN > (Resource temporarily unavailable) > [pid 2539] read(11<anon_inode:inotify>, > "\2\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 > [pid 2539] read(11<anon_inode:inotify>, 0x7ffda019f930, 272) = -1 EAGAIN > (Resource temporarily unavailable) > [pid 2539] read(11<anon_inode:inotify>, > "\2\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 > [pid 2539] read(11<anon_inode:inotify>, 0x7ffda019f930, 272) = -1 EAGAIN > (Resource temporarily unavailable) > [pid 2539] read(11<anon_inode:inotify>, > "\2\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 > [pid 2539] read(11<anon_inode:inotify>, 0x7ffda019f930, 272) = -1 EAGAIN > (Resource temporarily unavailable) > [pid 2539] read(11<anon_inode:inotify>, > "\2\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 > [pid 2539] read(11<anon_inode:inotify>, 0x7ffda019f930, 272) = -1 EAGAIN > (Resource temporarily unavailable) > [pid 2539] read(11<anon_inode:inotify>, > "\2\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 > [pid 2539] read(11<anon_inode:inotify>, 0x7ffda019f930, 272) = -1 EAGAIN > (Resource temporarily unavailable) > ^Cstrace: Process 2539 detached > strace: Process 11623 detached > strace: Process 20518 detached > strace: Process 22517 detached > > > > This normal behaviour? > > > > Thanks & Regards, > > Alex >

1 0

FW: librdkafka config property error on startup
by Sanjay Patel 02 Sep '21

02 Sep '21

I am trying to use Kafka-c with Syslog-ng. I have version Kafka 3.31 of Kafka installed, and I have syslog-ng-mod-rdkafka installed. When I start Syslog-ng, I get the following error. I can't figure out where it is getting this setting from. Syslog starts, but it does not send any logs to Kafka. kafka: error setting librdkafka config property; name='topic.partitioner', value='murmur2_random', error='No such configuration property: My Kafka config looks as follows: @include "scl.conf" destination d_kafka { kafka-c( bootstrap-servers("10.254.164.15:9092") topic("incoming") ); }; log { source(s_network); destination(d_kafka); }; Any help will be deeply appreciated. SKP <https://t.sidekickopen07.com/s3t/o/5/f18dQhb0S7n28cFBYDW5xwfpz2zGCvGW40Fv_6 2SXRqbW4NBnl23Lrg1WW5vbH2n1cd60nf197v5Y04?si=7000000000968799&pi=e03e7148-99 d8-470a-8205-9df1e2d92622>

1 0

No remote logging
by Alexandre Santos 01 Sep '21

01 Sep '21

Hi, I tying to do some remote logging from a Debian10 with syslog-ng 3.19. I have 2 syslog-ng's running, one in the default VRF which receives the logs from the applications, and another syslog-ng in an EXTERNAL VRF which send the logs to log-servers. The default VRF sends the logs to the EXTERNAL VRF via a unix domain. The problem is that sometimes is not streaming anything to remote log servers after configuring it. I have to restart the syslog-ng processes to start the remote logging. Local logging is working fine. Have you come across this issue before? Thanks and Regards, Alex

1 0

Resource temporarily unavailable while reading from system.journal
by Alexandre Santos 17 Aug '21

17 Aug '21

Hi, I am seeing the following strace log when running the syslog-ng 3.32 with the configuration in the attachment: root@myhost:/home/shared# sudo strace -f -yyy -s 256 -e trace=read -p 2539 strace: Process 2539 attached with 4 threads [pid 2539] read(48<anon_inode:[timerfd]>, "\1\0\0\0\0\0\0\0", 8) = 8 [pid 2539] read(11<anon_inode:inotify>, "\2\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 [pid 2539] read(11<anon_inode:inotify>, 0x7ffda019f930, 272) = -1 EAGAIN (Resource temporarily unavailable) [pid 2539] read(11<anon_inode:inotify>, "\2\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 [pid 2539] read(11<anon_inode:inotify>, 0x7ffda019f930, 272) = -1 EAGAIN (Resource temporarily unavailable) [pid 2539] read(11<anon_inode:inotify>, "\2\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 [pid 2539] read(11<anon_inode:inotify>, 0x7ffda019f930, 272) = -1 EAGAIN (Resource temporarily unavailable) [pid 2539] read(11<anon_inode:inotify>, "\2\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 [pid 2539] read(11<anon_inode:inotify>, 0x7ffda019f930, 272) = -1 EAGAIN (Resource temporarily unavailable) [pid 2539] read(11<anon_inode:inotify>, "\2\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 [pid 2539] read(11<anon_inode:inotify>, 0x7ffda019f930, 272) = -1 EAGAIN (Resource temporarily unavailable) [pid 2539] read(11<anon_inode:inotify>, "\2\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 [pid 2539] read(11<anon_inode:inotify>, 0x7ffda019f930, 272) = -1 EAGAIN (Resource temporarily unavailable) [pid 2539] read(11<anon_inode:inotify>, "\2\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 [pid 2539] read(11<anon_inode:inotify>, 0x7ffda019f930, 272) = -1 EAGAIN (Resource temporarily unavailable) [pid 2539] read(11<anon_inode:inotify>, "\2\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 [pid 2539] read(11<anon_inode:inotify>, 0x7ffda019f930, 272) = -1 EAGAIN (Resource temporarily unavailable) [pid 2539] read(11<anon_inode:inotify>, "\2\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 [pid 2539] read(11<anon_inode:inotify>, 0x7ffda019f930, 272) = -1 EAGAIN (Resource temporarily unavailable) ^Cstrace: Process 2539 detached strace: Process 11623 detached strace: Process 20518 detached strace: Process 22517 detached This normal behaviour? Thanks & Regards, Alex

2 1

syslong-ng-ctl stats
by Faine, Mark R. (MSFC-IS40)[NICS] 16 Aug '21

16 Aug '21

Can syslog-ng-ctl stats show me the not just the number of messages processed but size transferred and/or on disk? Thanks, -Mark

2 1

Re: [syslog-ng] syslog-ng Digest, Vol 195, Issue 9
by Linn, William 23 Jul '21

23 Jul '21

Laci, Thanks for your response, I figured out my problem. Thanks again -----Original Message----- From: syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of syslog-ng-request(a)lists.balabit.hu Sent: Friday, July 23, 2021 4:08 AM To: syslog-ng(a)lists.balabit.hu Subject: syslog-ng Digest, Vol 195, Issue 9 Send syslog-ng mailing list submissions to syslog-ng(a)lists.balabit.hu To subscribe or unsubscribe via the World Wide Web, visit https://urldefense.com/v3/__https://lists.balabit.hu/mailman/listinfo/syslo… or, via email, send a message with subject or body 'help' to syslog-ng-request(a)lists.balabit.hu You can reach the person managing the list at syslog-ng-owner(a)lists.balabit.hu When replying, please edit your Subject line so it is more specific than "Re: Contents of syslog-ng digest..." Today's Topics: 1. Re: syslog-ng Digest, Vol 195, Issue 7 (Laszlo Szemere (lszemere)) ---------------------------------------------------------------------- Message: 1 Date: Fri, 23 Jul 2021 08:08:10 +0000 From: "Laszlo Szemere (lszemere)" <Laszlo.Szemere(a)oneidentity.com> To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] syslog-ng Digest, Vol 195, Issue 7 Message-ID: <DM6PR19MB346532CA52F4D51920BFFC529DE59(a)DM6PR19MB3465.namprd19.prod.outlook.com> Content-Type: text/plain; charset="iso-8859-1" Hello Linn, I am sorry, but I haven't noticed your message before. (My email client mixed it because of the subject.) Could you please elaborate on the statement "fails to work"? What are the symptoms? Syslog-ng fails to start, or it is working differently than what you expect? Starting Syslog-ng with debugging enabled, might help to pinpoint the problem. ./syslog-ng -Fdevt -F, --foreground Do not go into the background after initialization -v, --verbose Be a bit more verbose -d, --debug Enable debug messages -t, --trace Enable trace messages -e, --stderr Log messages to stderr Best regards, Laci ________________________________ From: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu> on behalf of Linn, William <wlinn(a)bofa.com> Sent: Tuesday, July 20, 2021 16:08 To: syslog-ng(a)lists.balabit.hu <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] syslog-ng Digest, Vol 195, Issue 7 CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. I am having issues with a previous configuration set up in 2012. How to re-write it to work, here is the configuration that fails to work. destination d_apps { file("/syslog-ng/app/${HOST}/${HOST}-application.log.${DAY}" owner("root") group("root") perm(0644) dir_perm(0755) create_dirs(yes) overwrite_if_older(604800) ); }; destination d_systems {file("/syslog-ng/sys/${HOST}/${HOST}-system.log.${DAY}" owner("root") group("root") perm(0644) dir_perm(0755) create_dirs(yes) overwrite_if_older(604800) ); }; destination d_auth {file("/syslog-ng/auth/auth.log.${MONTH}" owner("root") group("root") perm(0640) dir_perm(0750) create_dirs(yes) overwrite_if_older(604800) ); }; -----Original Message----- From: syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of syslog-ng-request(a)lists.balabit.hu Sent: Tuesday, July 20, 2021 8:00 AM To: syslog-ng(a)lists.balabit.hu Subject: syslog-ng Digest, Vol 195, Issue 7 Send syslog-ng mailing list submissions to syslog-ng(a)lists.balabit.hu To subscribe or unsubscribe via the World Wide Web, visit https://urldefense.com/v3/__https://nam12.safelinks.protection.outlook.com/… or, via email, send a message with subject or body 'help' to syslog-ng-request(a)lists.balabit.hu You can reach the person managing the list at syslog-ng-owner(a)lists.balabit.hu When replying, please edit your Subject line so it is more specific than "Re: Contents of syslog-ng digest..." Today's Topics: 1. syslog-ng-3.33.2 (Laszlo Szemere (lszemere)) ---------------------------------------------------------------------- Message: 1 Date: Mon, 19 Jul 2021 13:38:23 +0000 From: "Laszlo Szemere (lszemere)" <Laszlo.Szemere(a)oneidentity.com> To: "syslog-ng(a)lists.balabit.hu" <syslog-ng(a)lists.balabit.hu> Subject: [syslog-ng] syslog-ng-3.33.2 Message-ID: <DM6PR19MB346594FBE1EDA3F73ACFEB699DE19(a)DM6PR19MB3465.namprd19.prod.outlook.com> Content-Type: text/plain; charset="iso-8859-1" 3.33.2 Bugfixes * disk-buffer: fixed a bug, which was introduced in 3.33.1, where we sometimes corrupted the disk-buffer file when it reached full size. (#3726<https://urldefense.com/v3/__https://nam12.safelinks.protection.outlook.com/… >) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Attila Szakacs, Balázs Barkó, Balazs Scheidler, Benedek Cserhati, Gabor Nagy, Josef Schlehofer, Laszlo Budai, Laszlo Szemere, László Várady, Norbert Takacs, Parrag Szilárd, Peter Kokai, Zoltan Pallagi -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://urldefense.com/v3/__https://nam12.safelinks.protection.outlook.com/… > ------------------------------ Subject: Digest Footer _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://urldefense.com/v3/__https://nam12.safelinks.protection.outlook.com/… ------------------------------ End of syslog-ng Digest, Vol 195, Issue 7 ***************************************** ---------------------------------------------------------------------- This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at https://urldefense.com/v3/__https://nam12.safelinks.protection.outlook.com/… . If you are not the intended recipient, please delete this message. ______________________________________________________________________________ Member info: https://urldefense.com/v3/__https://nam12.safelinks.protection.outlook.com/… Documentation: https://urldefense.com/v3/__https://nam12.safelinks.protection.outlook.com/… FAQ: https://urldefense.com/v3/__https://nam12.safelinks.protection.outlook.com/… -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://urldefense.com/v3/__http://lists.balabit.hu/pipermail/syslog-ng/att… > ------------------------------ Subject: Digest Footer _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://urldefense.com/v3/__https://lists.balabit.hu/mailman/listinfo/syslo… ------------------------------ End of syslog-ng Digest, Vol 195, Issue 9 ***************************************** ---------------------------------------------------------------------- This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.

1 0

Re: [syslog-ng] syslog-ng Digest, Vol 195, Issue 7
by Linn, William 23 Jul '21

23 Jul '21

I am having issues with a previous configuration set up in 2012. How to re-write it to work, here is the configuration that fails to work. destination d_apps { file("/syslog-ng/app/${HOST}/${HOST}-application.log.${DAY}" owner("root") group("root") perm(0644) dir_perm(0755) create_dirs(yes) overwrite_if_older(604800) ); }; destination d_systems {file("/syslog-ng/sys/${HOST}/${HOST}-system.log.${DAY}" owner("root") group("root") perm(0644) dir_perm(0755) create_dirs(yes) overwrite_if_older(604800) ); }; destination d_auth {file("/syslog-ng/auth/auth.log.${MONTH}" owner("root") group("root") perm(0640) dir_perm(0750) create_dirs(yes) overwrite_if_older(604800) ); }; -----Original Message----- From: syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of syslog-ng-request(a)lists.balabit.hu Sent: Tuesday, July 20, 2021 8:00 AM To: syslog-ng(a)lists.balabit.hu Subject: syslog-ng Digest, Vol 195, Issue 7 Send syslog-ng mailing list submissions to syslog-ng(a)lists.balabit.hu To subscribe or unsubscribe via the World Wide Web, visit https://urldefense.com/v3/__https://lists.balabit.hu/mailman/listinfo/syslo… or, via email, send a message with subject or body 'help' to syslog-ng-request(a)lists.balabit.hu You can reach the person managing the list at syslog-ng-owner(a)lists.balabit.hu When replying, please edit your Subject line so it is more specific than "Re: Contents of syslog-ng digest..." Today's Topics: 1. syslog-ng-3.33.2 (Laszlo Szemere (lszemere)) ---------------------------------------------------------------------- Message: 1 Date: Mon, 19 Jul 2021 13:38:23 +0000 From: "Laszlo Szemere (lszemere)" <Laszlo.Szemere(a)oneidentity.com> To: "syslog-ng(a)lists.balabit.hu" <syslog-ng(a)lists.balabit.hu> Subject: [syslog-ng] syslog-ng-3.33.2 Message-ID: <DM6PR19MB346594FBE1EDA3F73ACFEB699DE19(a)DM6PR19MB3465.namprd19.prod.outlook.com> Content-Type: text/plain; charset="iso-8859-1" 3.33.2 Bugfixes * disk-buffer: fixed a bug, which was introduced in 3.33.1, where we sometimes corrupted the disk-buffer file when it reached full size. (#3726<https://urldefense.com/v3/__https://github.com/syslog-ng/syslog-ng/pull/372… >) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Attila Szakacs, Balázs Barkó, Balazs Scheidler, Benedek Cserhati, Gabor Nagy, Josef Schlehofer, Laszlo Budai, Laszlo Szemere, László Várady, Norbert Takacs, Parrag Szilárd, Peter Kokai, Zoltan Pallagi -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://urldefense.com/v3/__http://lists.balabit.hu/pipermail/syslog-ng/att… > ------------------------------ Subject: Digest Footer _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://urldefense.com/v3/__https://lists.balabit.hu/mailman/listinfo/syslo… ------------------------------ End of syslog-ng Digest, Vol 195, Issue 7 ***************************************** ---------------------------------------------------------------------- This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.

2 1

syslog-ng-3.33.2
by Laszlo Szemere (lszemere) 19 Jul '21

19 Jul '21

3.33.2 Bugfixes * disk-buffer: fixed a bug, which was introduced in 3.33.1, where we sometimes corrupted the disk-buffer file when it reached full size. (#3726<https://github.com/syslog-ng/syslog-ng/pull/3726>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Attila Szakacs, Balázs Barkó, Balazs Scheidler, Benedek Cserhati, Gabor Nagy, Josef Schlehofer, Laszlo Budai, Laszlo Szemere, László Várady, Norbert Takacs, Parrag Szilárd, Peter Kokai, Zoltan Pallagi

1 0

Re: [syslog-ng] Logging to postgresql - error creating table with IP
by Herda, Martin 14 Jul '21

14 Jul '21

Thanks for your reply, I did indeed apply the single quotes and also tried with escaped double quotes. In either case the SQL statment string gets malformed to this: CREATE TABLE _logs_192.168.75.130_ Which will result in the same error as before: Error running SQL query; type='pgsql', host='127.0.0.1', port='5432', user='syslog', database='db_syslog', error='6819553: FEHLER: ERROR: Syntax error at >.168<\x0aLINE 1: CREATE TABLE _logs_192.168.75.130_ (datetime varchar(16), ho...\x0a ^\x0a', query='CREATE TABLE _logs_192.168.75.130_ (datetime varchar(16), host varchar(32), program varchar(20), pid varchar(10), message varchar(800))' Error creating table, giving up; table='_logs_192.168.75.130_' Br, MH Von: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu> Im Auftrag von Laszlo Szemere (lszemere) Gesendet: Dienstag, 13. Juli 2021 19:24 An: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Betreff: Re: [syslog-ng] Logging to postgresql - error creating table with IP Hello, as a quick and dirty test I found an online Postgress test env. https://extendsclass.com/postgresql-online.html I made some tests with the following commands: -- create table scientist (id integer, firstname varchar(100), lastname varchar(100)); -- insert into scientist (id, firstname, lastname) values (1, 'albert', 'einstein'); -- insert into scientist (id, firstname, lastname) values (2, 'isaac', 'newton'); -- insert into scientist (id, firstname, lastname) values (3, 'marie', 'curie'); -- select * from scientist; DROP TABLE "logs_192.168.75.102"; CREATE TABLE "logs_192.168.75.102" (datetime varchar(16), host varchar(32), program varchar(20), pid varchar(10), message varchar(800)); insert into "logs_192.168.75.102" (datetime, host, program, pid, message) values ('datetime', 'host', 'program', 'pid', 'message'); insert into "logs_192.168.75.102" (datetime, host, program, pid, message) values ('datetime', 'host', 'program', 'pid', 'message'); insert into "logs_192.168.75.102" (datetime, host, program, pid, message) values ('datetime', 'host', 'program', 'pid', 'message'); select * from "logs_192.168.75.102"; As it turned out, Postgres indeed accepts dots in the table name, however the SQL statement tourns out to be syntactically invalid without quoting it. (Please confirm: Did you apply quotes during the manual test?) Syslog-ng do not apply any (unnecessary?) quotes to the table name: https://github.com/syslog-ng/syslog-ng/blob/master/modules/afsql/afsql.c#L5… It looks like, simply adding it to the template of the table option: table("logs_${HOST}") --> table("'logs_${HOST}'") should do the trick. However, I cannot test it right now. Sorry. (Please inform me if the problem persists, and I will investigate it further when I am back to my machine.) Br, Laci ________________________________ From: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> on behalf of Herda, Martin <Martin.Herda(a)fritz-edv.de<mailto:Martin.Herda@fritz-edv.de>> Sent: Tuesday, July 13, 2021 17:15 To: 'syslog-ng(a)lists.balabit.hu' <syslog-ng(a)lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: [syslog-ng] Logging to postgresql - error creating table with IP CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Greetings to all, Im on Debian 4.19.194-2 using syslog-ng 3.19.1 The .conf is set to send all incoming syslog packages into pg database: destination d_pgsql { sql(type(pgsql) host("127.0.0.1") username("syslog") password("syslog") port("5432") database("db_syslog") table("logs_${HOST}") columns("datetime varchar(16)", "host varchar(32)", "program varchar(20)", "pid varchar(10)", "message varchar(800)") values("$R_DATE", "${HOST}", "$PROGRAM", "$PID", "$MSG") indexes("datetime", "host", "program", "pid", "message")); }; When there is a hostname available, the table is created successfully. When hostname can not be resolved, IP Address is used instead. But in this case syslog-ng is unable to create a new table: syslog-ng[499]: Error running SQL query; type='pgsql', host='127.0.0.1', port='5432', user='syslog', database='db_syslog', error='6819553: ERROR: Syntax error at >.168<\x0aLINE 1: CREATE TABLE logs_192.168.75.102 (datetime varchar(16), ho...\x0a ^\x0a', query='CREATE TABLE logs_192.168.75.102 (datetime varchar(16), host varchar(32), program varchar(20), pid varchar(10), message varchar(800))' Error creating table, giving up; table='logs_192.168.75.102' The issue seems to be the dot of the IP address. But dots in table names are possible AFAIK. Im able to manually create this table via HeidiSQL with no issues. Does the CREATE TABLE statement need to be escaped in some way? Or is there a way to replace the dots from the $HOST macro with underscores?

2 1

Logging to postgresql - error creating table with IP
by Herda, Martin 13 Jul '21

13 Jul '21

Greetings to all, Im on Debian 4.19.194-2 using syslog-ng 3.19.1 The .conf is set to send all incoming syslog packages into pg database: destination d_pgsql { sql(type(pgsql) host("127.0.0.1") username("syslog") password("syslog") port("5432") database("db_syslog") table("logs_${HOST}") columns("datetime varchar(16)", "host varchar(32)", "program varchar(20)", "pid varchar(10)", "message varchar(800)") values("$R_DATE", "${HOST}", "$PROGRAM", "$PID", "$MSG") indexes("datetime", "host", "program", "pid", "message")); }; When there is a hostname available, the table is created successfully. When hostname can not be resolved, IP Address is used instead. But in this case syslog-ng is unable to create a new table: syslog-ng[499]: Error running SQL query; type='pgsql', host='127.0.0.1', port='5432', user='syslog', database='db_syslog', error='6819553: ERROR: Syntax error at >.168<\x0aLINE 1: CREATE TABLE logs_192.168.75.102 (datetime varchar(16), ho...\x0a ^\x0a', query='CREATE TABLE logs_192.168.75.102 (datetime varchar(16), host varchar(32), program varchar(20), pid varchar(10), message varchar(800))' Error creating table, giving up; table='logs_192.168.75.102' The issue seems to be the dot of the IP address. But dots in table names are possible AFAIK. Im able to manually create this table via HeidiSQL with no issues. Does the CREATE TABLE statement need to be escaped in some way? Or is there a way to replace the dots from the $HOST macro with underscores?

2 1

syslog-ng-3.33.1
by László Várady (lvarady) 07 Jul '21

07 Jul '21

3.33.1 Highlights * MQTT destination The new mqtt() destination can be used to publish messages using the MQTT protocol. Currently MQTT 3.1.1 and 3.1 are supported. Supported transports: tcp, ws. Example config: destination { mqtt{ address("tcp://localhost:1883"), topic("syslog/$HOST"), fallback-topic("syslog/fallback")) }; }; Note: MQTT 5.0 and TLS (ssl://, wss://) are currently not supported. (#3703<https://github.com/syslog-ng/syslog-ng/pull/3703>) * discord() destination syslog-ng now has a webhook-based Discord destination. Example usage: destination { discord(url("https://discord.com/api/webhooks/x/y")); }; The following options can be used to customize the destination further: avatar-url(), username("$HOST-bot"), tts(true), template("${MSG:-[empty message]}"). (#3717<https://github.com/syslog-ng/syslog-ng/pull/3717>) Features * kafka-c: batching support in case of sync-send(yes) kafka-c( bootstrap-server("localhost:9092") topic("syslog-ng") sync-send(yes) batch-lines(10) batch-timeout(10000) ); Note1: batch-lines are accepted in case of sync-send(no), but no batching is done. Note2: messages are still sent one at a time to kafka, the batch yields multiple message per transaction. (#3699<https://github.com/syslog-ng/syslog-ng/pull/3699>) * kafka-c: sync-send(yes) enables synchronous message delivery, reducing the possibility of message loss. kafka-c( bootstrap-server("localhost:9092") topic("syslog-ng") sync-send(yes) ); Warning: this option also reduces significantly the performance of kafka-c driver. (#3681<https://github.com/syslog-ng/syslog-ng/pull/3681>) * disk-buffer: Now we optimize the file truncating frequency of disk-buffer. The new behavior saves IO time, but loses some disk space, which is configurable with a new option. The new option in the config is settable at 2 places: * truncate-size-ratio() in the disk-buffer() block, which affects the given disk-buffer. * disk-buffer(truncate-size-ratio()) in the global options block, which affects every disk-buffer which did not set truncate-size-ratio() itself. The default value is 0.01, which operates well with most disk-buffers. If the possible size reduction of the truncation does not reach truncate-size-ratio() x disk-buf-size(), we do not truncate the disk-buffer. To completely turn off truncating (maximal disk space loss, maximal IO time saved) set truncate-size-ratio(1), or to mimic the old behavior (minimal disk space loss, minimal IO time saved) set truncate-size-ratio(0). (#3689<https://github.com/syslog-ng/syslog-ng/pull/3689>) Bugfixes * syslog-format: fixing the check-hostname(yes|no) option The check-hostname(yes|no) option detected every value as invalid, causing a parse error when enabled. (#3690<https://github.com/syslog-ng/syslog-ng/pull/3690>) * disk-buffer(): fix crash when switching between disk-based and memory queues When a disk-buffer was removed from the configuration and the new config was applied by reloading syslog-ng, a crash occurred. (#3700<https://github.com/syslog-ng/syslog-ng/pull/3700>) * logpath: Fixed a message write protection bug, where message modifications (rewrite rules, parsers, etc.) leaked through preceding path elements. This may have resulted not only in unwanted/undefined message modification, but in certain cases crash as well. (#3708<https://github.com/syslog-ng/syslog-ng/pull/3708>) * mongodb(): fix crash with older mongo-c-driver versions syslog-ng crashed (was aborted) when the mongodb() destination was used with older mongo-c-driver versions (< v1.11.0). (#3677<https://github.com/syslog-ng/syslog-ng/pull/3677>) * java(): fix debug logging of Java-based destinations Java debug logging was not enabled previously when syslog-ng was started in debug/trace mode. This has been fixed. (#3679<https://github.com/syslog-ng/syslog-ng/pull/3679>) * kafka-c: fixed a hang during shutdown/reload, when multiple workers is used (workers() option is set to 2 or higher) and the librdkafka internal queue is filled. (error message was kafka: failed to publish message; topic='test-topic', error='Local: Queue full') (#3711<https://github.com/syslog-ng/syslog-ng/pull/3711>) Packaging * kafka: minimum version of librdkafka is changed from 1.0.0 to 1.1.0 (#3706<https://github.com/syslog-ng/syslog-ng/pull/3706>) * configure: now supporting python with two digit minor version (#3713<https://github.com/syslog-ng/syslog-ng/pull/3713>) Other changes * kafka: removed some deprecated options: client-lib-dir(), option(), template(), kafka-bootstrap-servers() (#3698<https://github.com/syslog-ng/syslog-ng/pull/3698>) * kafka: properties-file() option is deprecated. Please list the librdkafka properties in the config() option in syslog-ng's configuration. (#3698<https://github.com/syslog-ng/syslog-ng/pull/3698>) * smtp(): libesmtp is now detected via pkg-config (#3669<https://github.com/syslog-ng/syslog-ng/pull/3669>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler, Balázs Barkó, Benedek Cserhati, Gabor Nagy, L4rS6, Laszlo Budai, Laszlo Szemere, LittleFish33, László Várady, Norbert Takacs, Peter Czanik, Peter Kokai, Todd C. Miller, Tomáš Mózes, Zoltan Pallagi

1 0

query on remote logging
by SIMON BABY 07 Jul '21

07 Jul '21

Hi Team, Can I know if there is a configuration in syslog-ng.conf for logging the messages from a remote client. Can you please send me a sample configuration? Regards Simon

2 1