Hi, My unofficial Fedora / RHEL / CentOS & Co. and openSUSE / SLES packages are now available. As usual, some features are not enabled on all platforms due to problems with dependencies. Read more at https://www.syslog-ng.com/community/b/blog/posts/installing-latest-syslog-ng... FreeBSD ports now includes 3.34, see https://www.freshports.org/sysutils/syslog-ng/ Bye, CzP Peter Czanik (CzP) <peter.czanik@oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik ________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Attila Szakacs (aszakacs) <Attila.Szakacs@oneidentity.com> Sent: Friday, September 10, 2021 14:02 To: syslog-ng@lists.balabit.hu <syslog-ng@lists.balabit.hu> Subject: [syslog-ng] syslog-ng-3.34.1 CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. 3.34.1 Highlights * regexp-parser(): new parser that can parse messages with regular expressions Example: regexp-parser( template("${MESSAGE}") prefix(".regexp.") patterns("(?<DN>foo)", "(?<DN>ball)") ); regexp-parser() can be used as an intuitive replacement for regexp filters that had their store-matches flag set in order to save those matches. (#3702<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsyslog-ng%2Fsyslog-ng%2Fpull%2F3702&data=04%7C01%7Cpeter.czanik%40balabit.com%7C98f1efe02bba4a05af2a08d97452d7b9%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637668721436350342%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=rF4Ncrfnks%2F2Bjs%2BEyXa0p207QDH7Gs0fljVJ15zWp4%3D&reserved=0>) * redis(): workers() and batching support The Redis driver now support the workers() option, which specifies the number of parallel workers, and the batch-lines() option. This could drastically increase the throughput of the Redis destination driver. Example: redis( host("localhost") port(6379) command("HINCRBY", "hosts", "$HOST", "1") workers(8) batch-lines(100) batch-timeout(10000) log-fifo-size(100000) ); (#3732<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsyslog-ng%2Fsyslog-ng%2Fpull%2F3732&data=04%7C01%7Cpeter.czanik%40balabit.com%7C98f1efe02bba4a05af2a08d97452d7b9%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637668721436360292%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=zFn7M8kC%2B3QeSg46JSi8HA4NnxAVCsbixBIKEUqWMKI%3D&reserved=0>, #3745<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsyslog-ng%2Fsyslog-ng%2Fpull%2F3745&data=04%7C01%7Cpeter.czanik%40balabit.com%7C98f1efe02bba4a05af2a08d97452d7b9%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637668721436360292%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=QWzzInjhQ7u%2Fn5YMuNSZTYWzyMTdHEbWWykITMngJl4%3D&reserved=0>) * mqtt(): TLS and WebSocket Secure support The MQTT destination now supports TLS and WSS. Example config: mqtt( address("ssl://localhost:8883") topic("syslog/$HOST") fallback-topic("syslog/fallback") tls( ca-file("/path/to/ca.crt") key-file("/path/to/client.key") cert-file("/path/to/client.crt") peer-verify(yes) ) ); (#3747<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsyslog-ng%2Fsyslog-ng%2Fpull%2F3747&data=04%7C01%7Cpeter.czanik%40balabit.com%7C98f1efe02bba4a05af2a08d97452d7b9%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637668721436360292%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=dhlo1laZBy2IF0DfkziFmM1O6xk%2FqN%2BKCHHjmzO7kyI%3D&reserved=0>) Features * system() source: added support for NetBSD (#3761<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsyslog-ng%2Fsyslog-ng%2Fpull%2F3761&data=04%7C01%7Cpeter.czanik%40balabit.com%7C98f1efe02bba4a05af2a08d97452d7b9%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637668721436370255%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=10aF5aQ%2Bslmlj1tqDMAZGC0LunJptydWgwxfBtrIdxo%3D&reserved=0>) * stats: new statistics counter The following statistics are now available for the HTTP destination, and other file and network based sources/destinations: * msg_size_max/msg_size_avg: Shows the largest/average message size of the given source/destination that has been measured so far. * batch_size_max/batch_size_avg: When batching is enabled, then this shows the largest/average batch size of the given source/destination that has been measured so far. * eps_last_1h, eps_last_24h, eps_since_start: Events per second, measured for the last hour, for the last 24 hours, and since syslog-ng startup, respectively. Notes: * Message sizes are calculated from the incoming raw message length on the source side, and from the outgoing formatted message length on the destination side. * EPS counters are just approximate values, they are updated every minute. (#3753<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsyslog-ng%2Fsyslog-ng%2Fpull%2F3753&data=04%7C01%7Cpeter.czanik%40balabit.com%7C98f1efe02bba4a05af2a08d97452d7b9%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637668721436370255%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=bwBJeMbp90lojs24huK8r5vXf%2FeSMR%2Fb%2Fh3vwvgcndM%3D&reserved=0>) * mqtt(): username/password authentication Example config: mqtt( address("tcp://localhost:1883") topic("syslog/messages") username("user") password("passwd") ); Note: The password is transmitted in cleartext without using ssl:// or wss://. (#3747<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsyslog-ng%2Fsyslog-ng%2Fpull%2F3747&data=04%7C01%7Cpeter.czanik%40balabit.com%7C98f1efe02bba4a05af2a08d97452d7b9%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637668721436380207%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=i1oWgdrrSM5h%2FB5FzqIuCxGXh%2BVOioQKbQ%2Fqt2e6vSg%3D&reserved=0>) * mqtt(): new option http-proxy() for specifying HTTP/HTTPS proxy for WebSocket connections (#3747<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsyslog-ng%2Fsyslog-ng%2Fpull%2F3747&data=04%7C01%7Cpeter.czanik%40balabit.com%7C98f1efe02bba4a05af2a08d97452d7b9%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637668721436380207%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=i1oWgdrrSM5h%2FB5FzqIuCxGXh%2BVOioQKbQ%2Fqt2e6vSg%3D&reserved=0>) * syslog-ng-ctl: new flag for pruning statistics syslog-ng-ctl stats --remove-orphans can be used to remove "orphaned" statistic counters. It is useful when, for example, a templated file destination ($YEAR.$MONTH.$DAY) produces a lot of stats, and one wants to remove those abandoned counters occasionally/conditionally. (#3760<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsyslog-ng%2Fsyslog-ng%2Fpull%2F3760&data=04%7C01%7Cpeter.czanik%40balabit.com%7C98f1efe02bba4a05af2a08d97452d7b9%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637668721436380207%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=pvs7pM%2B2JpR9c1KNrXRQIR7%2FXPRowlANoAbCQR6BbnU%3D&reserved=0>) * disk-buffer(): added a new option to reliable disk-buffer: qout-size(). This option sets the number of messages that are stored in the memory in addition to storing them on disk. The default value is 1000. This serves performance purposes and offers the same no-message-loss guarantees as before. It can be used to maintain a higher throughput when only a small number of messages are waiting in the disk-buffer. (#3754<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsyslog-ng%2Fsyslog-ng%2Fpull%2F3754&data=04%7C01%7Cpeter.czanik%40balabit.com%7C98f1efe02bba4a05af2a08d97452d7b9%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637668721436390161%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=v58JPu60ADw5LP6fPol6rgikUms6owR7ZCbCE%2Fs3OfY%3D&reserved=0>) Bugfixes * network(), syslog(): fixed network sources on NetBSD On NetBSD, TCP-based network sources closed their listeners shortly after startup due to a non-portable TCP keepalive setting. This has been fixed. (#3751<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsyslog-ng%2Fsyslog-ng%2Fpull%2F3751&data=04%7C01%7Cpeter.czanik%40balabit.com%7C98f1efe02bba4a05af2a08d97452d7b9%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637668721436390161%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=3FNpFRLTuWZ0yBmCmJf2MTq6WKl27NNFjVrS9g1xLmo%3D&reserved=0>) * disk-buffer(): fixed a very rare case, where the reliable disk-buffer never resumed after triggering flow-control. (#3752<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsyslog-ng%2Fsyslog-ng%2Fpull%2F3752&data=04%7C01%7Cpeter.czanik%40balabit.com%7C98f1efe02bba4a05af2a08d97452d7b9%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637668721436390161%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=VwkyqGIfJa8UgbJi1Nld%2FZKWYOlR4JsDnhKT2Ob%2BZ%2F8%3D&reserved=0>) * disk-buffer(): fixed a rare memory leak that occurred when mem-buf-length() or mem-buf-size() was configured incorrectly (#3750<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsyslog-ng%2Fsyslog-ng%2Fpull%2F3750&data=04%7C01%7Cpeter.czanik%40balabit.com%7C98f1efe02bba4a05af2a08d97452d7b9%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637668721436400123%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=lvfiEp%2BKsJYNaMtWzFRcNdMmtQInvuWMau15IUtVMaM%3D&reserved=0>) * redis(): fixed command errors that were not detected and marked as successful delivery (#3748<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsyslog-ng%2Fsyslog-ng%2Fpull%2F3748&data=04%7C01%7Cpeter.czanik%40balabit.com%7C98f1efe02bba4a05af2a08d97452d7b9%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637668721436400123%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=6P3h4QSvKf8vbSmMuAt9E02r1iCj6bxNtBnNYurJSms%3D&reserved=0>) Notes to developers * Light framework: new proxy-related options are supported with loggen: --proxy-src-ip, --proxy-dst-ip, --proxy-src-port, --proxy-dst-port (#3766<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsyslog-ng%2Fsyslog-ng%2Fpull%2F3766&data=04%7C01%7Cpeter.czanik%40balabit.com%7C98f1efe02bba4a05af2a08d97452d7b9%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637668721436410077%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=PX6I4uZoMm%2F6aH%2F%2FufBr4pMxNk0T724Sy3WcKdGN49s%3D&reserved=0>) * log-threaded-dest: descendant drivers from LogThreadedDest no longer inherit batch-lines() and batch-timeout() automatically. Each driver have to opt-in for these options with log_threaded_dest_driver_batch_option. log_threaded_dest_driver_option has been renamed to log_threaded_dest_driver_general_option, and log_threaded_dest_driver_workers_option have been added similarly to the batch-related options. (#3741<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsyslog-ng%2Fsyslog-ng%2Fpull%2F3741&data=04%7C01%7Cpeter.czanik%40balabit.com%7C98f1efe02bba4a05af2a08d97452d7b9%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637668721436410077%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=2kT%2FnDQbttsDQeDKO%2BFJfM%2BHxt2%2BSVt%2FxnX%2BoWiOHbc%3D&reserved=0>) Other changes * disk-buffer(): performance improvements Based on our measurements, the following can be expected compared to the previous syslog-ng release (v3.33.1): * non-reliable disk buffer: up to 50% performance gain; * reliable disk buffer: up to 80% increase in performance. (#3743<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsyslog-ng%2Fsyslog-ng%2Fpull%2F3743&data=04%7C01%7Cpeter.czanik%40balabit.com%7C98f1efe02bba4a05af2a08d97452d7b9%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637668721436420032%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=OJO73PFXFVR4vNwglPITx88SwsOVgP0I%2FPZGOAW6ytw%3D&reserved=0>, #3746<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsyslog-ng%2Fsyslog-ng%2Fpull%2F3746&data=04%7C01%7Cpeter.czanik%40balabit.com%7C98f1efe02bba4a05af2a08d97452d7b9%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637668721436420032%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=HFGarMS8TLHYjCeoQMM0frrLLU7haDEXQGg951UjpPg%3D&reserved=0>, #3754<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsyslog-ng%2Fsyslog-ng%2Fpull%2F3754&data=04%7C01%7Cpeter.czanik%40balabit.com%7C98f1efe02bba4a05af2a08d97452d7b9%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637668721436420032%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=iI8q4flwiXVT2TtlG0JA3Ddj51IS1m9KICwGgaKYic4%3D&reserved=0>, #3756<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsyslog-ng%2Fsyslog-ng%2Fpull%2F3756&data=04%7C01%7Cpeter.czanik%40balabit.com%7C98f1efe02bba4a05af2a08d97452d7b9%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637668721436429990%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=yUN5fHNbQkgXc7Dn7ZO%2F3fh5E9Jf%2FigtAiGo5J7kDWc%3D&reserved=0>, #3757<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsyslog-ng%2Fsyslog-ng%2Fpull%2F3757&data=04%7C01%7Cpeter.czanik%40balabit.com%7C98f1efe02bba4a05af2a08d97452d7b9%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637668721436429990%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=aU%2FoPH7YgCFbfVyzsvp9jODUq%2BQlma%2F7ZIHuKWc9oG8%3D&reserved=0>) * disk-buffer(): the default value of the following options has been changed for performance reasons: * truncate-size-ratio(): from 0.01 to 0.1 (from 1% to 10%) * qout-size(): from 64 to 1000 (this affects only the non-reliable disk buffer) (#3757<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsyslog-ng%2Fsyslog-ng%2Fpull%2F3757&data=04%7C01%7Cpeter.czanik%40balabit.com%7C98f1efe02bba4a05af2a08d97452d7b9%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637668721436439943%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=3eoqU3PI3vKMKlyB7aUs6jf3VsxudqkQIwuuZ7azDoA%3D&reserved=0>) * kafka-c(): properties-file() option is removed Please list librdkafka properties in the config() option in syslog-ng's configuration. See librdkafka configuration here<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fedenhill%2Flibrdkafka%2Fblob%2Fmaster%2FCONFIGURATION.md&data=04%7C01%7Cpeter.czanik%40balabit.com%7C98f1efe02bba4a05af2a08d97452d7b9%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637668721436439943%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=h0T%2BoUi%2BXs1%2BIi1k5FUQsa8TwElaplVSzSvxLpZhYdw%3D&reserved=0>. (#3704<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsyslog-ng%2Fsyslog-ng%2Fpull%2F3704&data=04%7C01%7Cpeter.czanik%40balabit.com%7C98f1efe02bba4a05af2a08d97452d7b9%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637668721436439943%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=2ftWLssln3YNg%2BrhSL3hRcAv8ofV3heJroznEPb%2F8ds%3D&reserved=0>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Attila Szakacs, Balazs Scheidler, Balázs Barkó, Benedek Cserhati, Fabrice Fontaine, Gabor Nagy, Laszlo Szemere, LittleFish33, László Várady, Norbert Takacs, Parrag Szilárd, Peter Czanik, Peter Kokai, Zoltan Pallagi