- syslog-ng - lists.balabit.hu

FW: librdkafka config property error on startup
by Sanjay Patel 02 Sep '21

02 Sep '21

I am trying to use Kafka-c with Syslog-ng. I have version Kafka 3.31 of Kafka installed, and I have syslog-ng-mod-rdkafka installed. When I start Syslog-ng, I get the following error. I can't figure out where it is getting this setting from. Syslog starts, but it does not send any logs to Kafka. kafka: error setting librdkafka config property; name='topic.partitioner', value='murmur2_random', error='No such configuration property: My Kafka config looks as follows: @include "scl.conf" destination d_kafka { kafka-c( bootstrap-servers("10.254.164.15:9092") topic("incoming") ); }; log { source(s_network); destination(d_kafka); }; Any help will be deeply appreciated. SKP <https://t.sidekickopen07.com/s3t/o/5/f18dQhb0S7n28cFBYDW5xwfpz2zGCvGW40Fv_6 2SXRqbW4NBnl23Lrg1WW5vbH2n1cd60nf197v5Y04?si=7000000000968799&pi=e03e7148-99 d8-470a-8205-9df1e2d92622>

1 0

No remote logging
by Alexandre Santos 01 Sep '21

01 Sep '21

Hi, I tying to do some remote logging from a Debian10 with syslog-ng 3.19. I have 2 syslog-ng's running, one in the default VRF which receives the logs from the applications, and another syslog-ng in an EXTERNAL VRF which send the logs to log-servers. The default VRF sends the logs to the EXTERNAL VRF via a unix domain. The problem is that sometimes is not streaming anything to remote log servers after configuring it. I have to restart the syslog-ng processes to start the remote logging. Local logging is working fine. Have you come across this issue before? Thanks and Regards, Alex

1 0

Resource temporarily unavailable while reading from system.journal
by Alexandre Santos 17 Aug '21

17 Aug '21

Hi, I am seeing the following strace log when running the syslog-ng 3.32 with the configuration in the attachment: root@myhost:/home/shared# sudo strace -f -yyy -s 256 -e trace=read -p 2539 strace: Process 2539 attached with 4 threads [pid 2539] read(48<anon_inode:[timerfd]>, "\1\0\0\0\0\0\0\0", 8) = 8 [pid 2539] read(11<anon_inode:inotify>, "\2\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 [pid 2539] read(11<anon_inode:inotify>, 0x7ffda019f930, 272) = -1 EAGAIN (Resource temporarily unavailable) [pid 2539] read(11<anon_inode:inotify>, "\2\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 [pid 2539] read(11<anon_inode:inotify>, 0x7ffda019f930, 272) = -1 EAGAIN (Resource temporarily unavailable) [pid 2539] read(11<anon_inode:inotify>, "\2\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 [pid 2539] read(11<anon_inode:inotify>, 0x7ffda019f930, 272) = -1 EAGAIN (Resource temporarily unavailable) [pid 2539] read(11<anon_inode:inotify>, "\2\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 [pid 2539] read(11<anon_inode:inotify>, 0x7ffda019f930, 272) = -1 EAGAIN (Resource temporarily unavailable) [pid 2539] read(11<anon_inode:inotify>, "\2\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 [pid 2539] read(11<anon_inode:inotify>, 0x7ffda019f930, 272) = -1 EAGAIN (Resource temporarily unavailable) [pid 2539] read(11<anon_inode:inotify>, "\2\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 [pid 2539] read(11<anon_inode:inotify>, 0x7ffda019f930, 272) = -1 EAGAIN (Resource temporarily unavailable) [pid 2539] read(11<anon_inode:inotify>, "\2\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 [pid 2539] read(11<anon_inode:inotify>, 0x7ffda019f930, 272) = -1 EAGAIN (Resource temporarily unavailable) [pid 2539] read(11<anon_inode:inotify>, "\2\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 [pid 2539] read(11<anon_inode:inotify>, 0x7ffda019f930, 272) = -1 EAGAIN (Resource temporarily unavailable) [pid 2539] read(11<anon_inode:inotify>, "\2\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 [pid 2539] read(11<anon_inode:inotify>, 0x7ffda019f930, 272) = -1 EAGAIN (Resource temporarily unavailable) ^Cstrace: Process 2539 detached strace: Process 11623 detached strace: Process 20518 detached strace: Process 22517 detached This normal behaviour? Thanks & Regards, Alex

2 1

syslong-ng-ctl stats
by Faine, Mark R. (MSFC-IS40)[NICS] 16 Aug '21

16 Aug '21

Can syslog-ng-ctl stats show me the not just the number of messages processed but size transferred and/or on disk? Thanks, -Mark

2 1

Re: [syslog-ng] syslog-ng Digest, Vol 195, Issue 9
by Linn, William 23 Jul '21

23 Jul '21

Laci, Thanks for your response, I figured out my problem. Thanks again -----Original Message----- From: syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of syslog-ng-request(a)lists.balabit.hu Sent: Friday, July 23, 2021 4:08 AM To: syslog-ng(a)lists.balabit.hu Subject: syslog-ng Digest, Vol 195, Issue 9 Send syslog-ng mailing list submissions to syslog-ng(a)lists.balabit.hu To subscribe or unsubscribe via the World Wide Web, visit https://urldefense.com/v3/__https://lists.balabit.hu/mailman/listinfo/syslo… or, via email, send a message with subject or body 'help' to syslog-ng-request(a)lists.balabit.hu You can reach the person managing the list at syslog-ng-owner(a)lists.balabit.hu When replying, please edit your Subject line so it is more specific than "Re: Contents of syslog-ng digest..." Today's Topics: 1. Re: syslog-ng Digest, Vol 195, Issue 7 (Laszlo Szemere (lszemere)) ---------------------------------------------------------------------- Message: 1 Date: Fri, 23 Jul 2021 08:08:10 +0000 From: "Laszlo Szemere (lszemere)" <Laszlo.Szemere(a)oneidentity.com> To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] syslog-ng Digest, Vol 195, Issue 7 Message-ID: <DM6PR19MB346532CA52F4D51920BFFC529DE59(a)DM6PR19MB3465.namprd19.prod.outlook.com> Content-Type: text/plain; charset="iso-8859-1" Hello Linn, I am sorry, but I haven't noticed your message before. (My email client mixed it because of the subject.) Could you please elaborate on the statement "fails to work"? What are the symptoms? Syslog-ng fails to start, or it is working differently than what you expect? Starting Syslog-ng with debugging enabled, might help to pinpoint the problem. ./syslog-ng -Fdevt -F, --foreground Do not go into the background after initialization -v, --verbose Be a bit more verbose -d, --debug Enable debug messages -t, --trace Enable trace messages -e, --stderr Log messages to stderr Best regards, Laci ________________________________ From: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu> on behalf of Linn, William <wlinn(a)bofa.com> Sent: Tuesday, July 20, 2021 16:08 To: syslog-ng(a)lists.balabit.hu <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] syslog-ng Digest, Vol 195, Issue 7 CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. I am having issues with a previous configuration set up in 2012. How to re-write it to work, here is the configuration that fails to work. destination d_apps { file("/syslog-ng/app/${HOST}/${HOST}-application.log.${DAY}" owner("root") group("root") perm(0644) dir_perm(0755) create_dirs(yes) overwrite_if_older(604800) ); }; destination d_systems {file("/syslog-ng/sys/${HOST}/${HOST}-system.log.${DAY}" owner("root") group("root") perm(0644) dir_perm(0755) create_dirs(yes) overwrite_if_older(604800) ); }; destination d_auth {file("/syslog-ng/auth/auth.log.${MONTH}" owner("root") group("root") perm(0640) dir_perm(0750) create_dirs(yes) overwrite_if_older(604800) ); }; -----Original Message----- From: syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of syslog-ng-request(a)lists.balabit.hu Sent: Tuesday, July 20, 2021 8:00 AM To: syslog-ng(a)lists.balabit.hu Subject: syslog-ng Digest, Vol 195, Issue 7 Send syslog-ng mailing list submissions to syslog-ng(a)lists.balabit.hu To subscribe or unsubscribe via the World Wide Web, visit https://urldefense.com/v3/__https://nam12.safelinks.protection.outlook.com/… or, via email, send a message with subject or body 'help' to syslog-ng-request(a)lists.balabit.hu You can reach the person managing the list at syslog-ng-owner(a)lists.balabit.hu When replying, please edit your Subject line so it is more specific than "Re: Contents of syslog-ng digest..." Today's Topics: 1. syslog-ng-3.33.2 (Laszlo Szemere (lszemere)) ---------------------------------------------------------------------- Message: 1 Date: Mon, 19 Jul 2021 13:38:23 +0000 From: "Laszlo Szemere (lszemere)" <Laszlo.Szemere(a)oneidentity.com> To: "syslog-ng(a)lists.balabit.hu" <syslog-ng(a)lists.balabit.hu> Subject: [syslog-ng] syslog-ng-3.33.2 Message-ID: <DM6PR19MB346594FBE1EDA3F73ACFEB699DE19(a)DM6PR19MB3465.namprd19.prod.outlook.com> Content-Type: text/plain; charset="iso-8859-1" 3.33.2 Bugfixes * disk-buffer: fixed a bug, which was introduced in 3.33.1, where we sometimes corrupted the disk-buffer file when it reached full size. (#3726<https://urldefense.com/v3/__https://nam12.safelinks.protection.outlook.com/… >) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Attila Szakacs, Balázs Barkó, Balazs Scheidler, Benedek Cserhati, Gabor Nagy, Josef Schlehofer, Laszlo Budai, Laszlo Szemere, László Várady, Norbert Takacs, Parrag Szilárd, Peter Kokai, Zoltan Pallagi -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://urldefense.com/v3/__https://nam12.safelinks.protection.outlook.com/… > ------------------------------ Subject: Digest Footer _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://urldefense.com/v3/__https://nam12.safelinks.protection.outlook.com/… ------------------------------ End of syslog-ng Digest, Vol 195, Issue 7 ***************************************** ---------------------------------------------------------------------- This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at https://urldefense.com/v3/__https://nam12.safelinks.protection.outlook.com/… . If you are not the intended recipient, please delete this message. ______________________________________________________________________________ Member info: https://urldefense.com/v3/__https://nam12.safelinks.protection.outlook.com/… Documentation: https://urldefense.com/v3/__https://nam12.safelinks.protection.outlook.com/… FAQ: https://urldefense.com/v3/__https://nam12.safelinks.protection.outlook.com/… -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://urldefense.com/v3/__http://lists.balabit.hu/pipermail/syslog-ng/att… > ------------------------------ Subject: Digest Footer _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://urldefense.com/v3/__https://lists.balabit.hu/mailman/listinfo/syslo… ------------------------------ End of syslog-ng Digest, Vol 195, Issue 9 ***************************************** ---------------------------------------------------------------------- This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.

1 0

Re: [syslog-ng] syslog-ng Digest, Vol 195, Issue 7
by Linn, William 23 Jul '21

23 Jul '21

I am having issues with a previous configuration set up in 2012. How to re-write it to work, here is the configuration that fails to work. destination d_apps { file("/syslog-ng/app/${HOST}/${HOST}-application.log.${DAY}" owner("root") group("root") perm(0644) dir_perm(0755) create_dirs(yes) overwrite_if_older(604800) ); }; destination d_systems {file("/syslog-ng/sys/${HOST}/${HOST}-system.log.${DAY}" owner("root") group("root") perm(0644) dir_perm(0755) create_dirs(yes) overwrite_if_older(604800) ); }; destination d_auth {file("/syslog-ng/auth/auth.log.${MONTH}" owner("root") group("root") perm(0640) dir_perm(0750) create_dirs(yes) overwrite_if_older(604800) ); }; -----Original Message----- From: syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of syslog-ng-request(a)lists.balabit.hu Sent: Tuesday, July 20, 2021 8:00 AM To: syslog-ng(a)lists.balabit.hu Subject: syslog-ng Digest, Vol 195, Issue 7 Send syslog-ng mailing list submissions to syslog-ng(a)lists.balabit.hu To subscribe or unsubscribe via the World Wide Web, visit https://urldefense.com/v3/__https://lists.balabit.hu/mailman/listinfo/syslo… or, via email, send a message with subject or body 'help' to syslog-ng-request(a)lists.balabit.hu You can reach the person managing the list at syslog-ng-owner(a)lists.balabit.hu When replying, please edit your Subject line so it is more specific than "Re: Contents of syslog-ng digest..." Today's Topics: 1. syslog-ng-3.33.2 (Laszlo Szemere (lszemere)) ---------------------------------------------------------------------- Message: 1 Date: Mon, 19 Jul 2021 13:38:23 +0000 From: "Laszlo Szemere (lszemere)" <Laszlo.Szemere(a)oneidentity.com> To: "syslog-ng(a)lists.balabit.hu" <syslog-ng(a)lists.balabit.hu> Subject: [syslog-ng] syslog-ng-3.33.2 Message-ID: <DM6PR19MB346594FBE1EDA3F73ACFEB699DE19(a)DM6PR19MB3465.namprd19.prod.outlook.com> Content-Type: text/plain; charset="iso-8859-1" 3.33.2 Bugfixes * disk-buffer: fixed a bug, which was introduced in 3.33.1, where we sometimes corrupted the disk-buffer file when it reached full size. (#3726<https://urldefense.com/v3/__https://github.com/syslog-ng/syslog-ng/pull/372… >) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Attila Szakacs, Balázs Barkó, Balazs Scheidler, Benedek Cserhati, Gabor Nagy, Josef Schlehofer, Laszlo Budai, Laszlo Szemere, László Várady, Norbert Takacs, Parrag Szilárd, Peter Kokai, Zoltan Pallagi -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://urldefense.com/v3/__http://lists.balabit.hu/pipermail/syslog-ng/att… > ------------------------------ Subject: Digest Footer _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://urldefense.com/v3/__https://lists.balabit.hu/mailman/listinfo/syslo… ------------------------------ End of syslog-ng Digest, Vol 195, Issue 7 ***************************************** ---------------------------------------------------------------------- This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.

2 1

syslog-ng-3.33.2
by Laszlo Szemere (lszemere) 19 Jul '21

19 Jul '21

3.33.2 Bugfixes * disk-buffer: fixed a bug, which was introduced in 3.33.1, where we sometimes corrupted the disk-buffer file when it reached full size. (#3726<https://github.com/syslog-ng/syslog-ng/pull/3726>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Attila Szakacs, Balázs Barkó, Balazs Scheidler, Benedek Cserhati, Gabor Nagy, Josef Schlehofer, Laszlo Budai, Laszlo Szemere, László Várady, Norbert Takacs, Parrag Szilárd, Peter Kokai, Zoltan Pallagi

1 0

Re: [syslog-ng] Logging to postgresql - error creating table with IP
by Herda, Martin 14 Jul '21

14 Jul '21

Thanks for your reply, I did indeed apply the single quotes and also tried with escaped double quotes. In either case the SQL statment string gets malformed to this: CREATE TABLE _logs_192.168.75.130_ Which will result in the same error as before: Error running SQL query; type='pgsql', host='127.0.0.1', port='5432', user='syslog', database='db_syslog', error='6819553: FEHLER: ERROR: Syntax error at >.168<\x0aLINE 1: CREATE TABLE _logs_192.168.75.130_ (datetime varchar(16), ho...\x0a ^\x0a', query='CREATE TABLE _logs_192.168.75.130_ (datetime varchar(16), host varchar(32), program varchar(20), pid varchar(10), message varchar(800))' Error creating table, giving up; table='_logs_192.168.75.130_' Br, MH Von: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu> Im Auftrag von Laszlo Szemere (lszemere) Gesendet: Dienstag, 13. Juli 2021 19:24 An: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Betreff: Re: [syslog-ng] Logging to postgresql - error creating table with IP Hello, as a quick and dirty test I found an online Postgress test env. https://extendsclass.com/postgresql-online.html I made some tests with the following commands: -- create table scientist (id integer, firstname varchar(100), lastname varchar(100)); -- insert into scientist (id, firstname, lastname) values (1, 'albert', 'einstein'); -- insert into scientist (id, firstname, lastname) values (2, 'isaac', 'newton'); -- insert into scientist (id, firstname, lastname) values (3, 'marie', 'curie'); -- select * from scientist; DROP TABLE "logs_192.168.75.102"; CREATE TABLE "logs_192.168.75.102" (datetime varchar(16), host varchar(32), program varchar(20), pid varchar(10), message varchar(800)); insert into "logs_192.168.75.102" (datetime, host, program, pid, message) values ('datetime', 'host', 'program', 'pid', 'message'); insert into "logs_192.168.75.102" (datetime, host, program, pid, message) values ('datetime', 'host', 'program', 'pid', 'message'); insert into "logs_192.168.75.102" (datetime, host, program, pid, message) values ('datetime', 'host', 'program', 'pid', 'message'); select * from "logs_192.168.75.102"; As it turned out, Postgres indeed accepts dots in the table name, however the SQL statement tourns out to be syntactically invalid without quoting it. (Please confirm: Did you apply quotes during the manual test?) Syslog-ng do not apply any (unnecessary?) quotes to the table name: https://github.com/syslog-ng/syslog-ng/blob/master/modules/afsql/afsql.c#L5… It looks like, simply adding it to the template of the table option: table("logs_${HOST}") --> table("'logs_${HOST}'") should do the trick. However, I cannot test it right now. Sorry. (Please inform me if the problem persists, and I will investigate it further when I am back to my machine.) Br, Laci ________________________________ From: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> on behalf of Herda, Martin <Martin.Herda(a)fritz-edv.de<mailto:Martin.Herda@fritz-edv.de>> Sent: Tuesday, July 13, 2021 17:15 To: 'syslog-ng(a)lists.balabit.hu' <syslog-ng(a)lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: [syslog-ng] Logging to postgresql - error creating table with IP CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Greetings to all, Im on Debian 4.19.194-2 using syslog-ng 3.19.1 The .conf is set to send all incoming syslog packages into pg database: destination d_pgsql { sql(type(pgsql) host("127.0.0.1") username("syslog") password("syslog") port("5432") database("db_syslog") table("logs_${HOST}") columns("datetime varchar(16)", "host varchar(32)", "program varchar(20)", "pid varchar(10)", "message varchar(800)") values("$R_DATE", "${HOST}", "$PROGRAM", "$PID", "$MSG") indexes("datetime", "host", "program", "pid", "message")); }; When there is a hostname available, the table is created successfully. When hostname can not be resolved, IP Address is used instead. But in this case syslog-ng is unable to create a new table: syslog-ng[499]: Error running SQL query; type='pgsql', host='127.0.0.1', port='5432', user='syslog', database='db_syslog', error='6819553: ERROR: Syntax error at >.168<\x0aLINE 1: CREATE TABLE logs_192.168.75.102 (datetime varchar(16), ho...\x0a ^\x0a', query='CREATE TABLE logs_192.168.75.102 (datetime varchar(16), host varchar(32), program varchar(20), pid varchar(10), message varchar(800))' Error creating table, giving up; table='logs_192.168.75.102' The issue seems to be the dot of the IP address. But dots in table names are possible AFAIK. Im able to manually create this table via HeidiSQL with no issues. Does the CREATE TABLE statement need to be escaped in some way? Or is there a way to replace the dots from the $HOST macro with underscores?

2 1

Logging to postgresql - error creating table with IP
by Herda, Martin 13 Jul '21

13 Jul '21

Greetings to all, Im on Debian 4.19.194-2 using syslog-ng 3.19.1 The .conf is set to send all incoming syslog packages into pg database: destination d_pgsql { sql(type(pgsql) host("127.0.0.1") username("syslog") password("syslog") port("5432") database("db_syslog") table("logs_${HOST}") columns("datetime varchar(16)", "host varchar(32)", "program varchar(20)", "pid varchar(10)", "message varchar(800)") values("$R_DATE", "${HOST}", "$PROGRAM", "$PID", "$MSG") indexes("datetime", "host", "program", "pid", "message")); }; When there is a hostname available, the table is created successfully. When hostname can not be resolved, IP Address is used instead. But in this case syslog-ng is unable to create a new table: syslog-ng[499]: Error running SQL query; type='pgsql', host='127.0.0.1', port='5432', user='syslog', database='db_syslog', error='6819553: ERROR: Syntax error at >.168<\x0aLINE 1: CREATE TABLE logs_192.168.75.102 (datetime varchar(16), ho...\x0a ^\x0a', query='CREATE TABLE logs_192.168.75.102 (datetime varchar(16), host varchar(32), program varchar(20), pid varchar(10), message varchar(800))' Error creating table, giving up; table='logs_192.168.75.102' The issue seems to be the dot of the IP address. But dots in table names are possible AFAIK. Im able to manually create this table via HeidiSQL with no issues. Does the CREATE TABLE statement need to be escaped in some way? Or is there a way to replace the dots from the $HOST macro with underscores?

2 1

syslog-ng-3.33.1
by László Várady (lvarady) 07 Jul '21

07 Jul '21

3.33.1 Highlights * MQTT destination The new mqtt() destination can be used to publish messages using the MQTT protocol. Currently MQTT 3.1.1 and 3.1 are supported. Supported transports: tcp, ws. Example config: destination { mqtt{ address("tcp://localhost:1883"), topic("syslog/$HOST"), fallback-topic("syslog/fallback")) }; }; Note: MQTT 5.0 and TLS (ssl://, wss://) are currently not supported. (#3703<https://github.com/syslog-ng/syslog-ng/pull/3703>) * discord() destination syslog-ng now has a webhook-based Discord destination. Example usage: destination { discord(url("https://discord.com/api/webhooks/x/y")); }; The following options can be used to customize the destination further: avatar-url(), username("$HOST-bot"), tts(true), template("${MSG:-[empty message]}"). (#3717<https://github.com/syslog-ng/syslog-ng/pull/3717>) Features * kafka-c: batching support in case of sync-send(yes) kafka-c( bootstrap-server("localhost:9092") topic("syslog-ng") sync-send(yes) batch-lines(10) batch-timeout(10000) ); Note1: batch-lines are accepted in case of sync-send(no), but no batching is done. Note2: messages are still sent one at a time to kafka, the batch yields multiple message per transaction. (#3699<https://github.com/syslog-ng/syslog-ng/pull/3699>) * kafka-c: sync-send(yes) enables synchronous message delivery, reducing the possibility of message loss. kafka-c( bootstrap-server("localhost:9092") topic("syslog-ng") sync-send(yes) ); Warning: this option also reduces significantly the performance of kafka-c driver. (#3681<https://github.com/syslog-ng/syslog-ng/pull/3681>) * disk-buffer: Now we optimize the file truncating frequency of disk-buffer. The new behavior saves IO time, but loses some disk space, which is configurable with a new option. The new option in the config is settable at 2 places: * truncate-size-ratio() in the disk-buffer() block, which affects the given disk-buffer. * disk-buffer(truncate-size-ratio()) in the global options block, which affects every disk-buffer which did not set truncate-size-ratio() itself. The default value is 0.01, which operates well with most disk-buffers. If the possible size reduction of the truncation does not reach truncate-size-ratio() x disk-buf-size(), we do not truncate the disk-buffer. To completely turn off truncating (maximal disk space loss, maximal IO time saved) set truncate-size-ratio(1), or to mimic the old behavior (minimal disk space loss, minimal IO time saved) set truncate-size-ratio(0). (#3689<https://github.com/syslog-ng/syslog-ng/pull/3689>) Bugfixes * syslog-format: fixing the check-hostname(yes|no) option The check-hostname(yes|no) option detected every value as invalid, causing a parse error when enabled. (#3690<https://github.com/syslog-ng/syslog-ng/pull/3690>) * disk-buffer(): fix crash when switching between disk-based and memory queues When a disk-buffer was removed from the configuration and the new config was applied by reloading syslog-ng, a crash occurred. (#3700<https://github.com/syslog-ng/syslog-ng/pull/3700>) * logpath: Fixed a message write protection bug, where message modifications (rewrite rules, parsers, etc.) leaked through preceding path elements. This may have resulted not only in unwanted/undefined message modification, but in certain cases crash as well. (#3708<https://github.com/syslog-ng/syslog-ng/pull/3708>) * mongodb(): fix crash with older mongo-c-driver versions syslog-ng crashed (was aborted) when the mongodb() destination was used with older mongo-c-driver versions (< v1.11.0). (#3677<https://github.com/syslog-ng/syslog-ng/pull/3677>) * java(): fix debug logging of Java-based destinations Java debug logging was not enabled previously when syslog-ng was started in debug/trace mode. This has been fixed. (#3679<https://github.com/syslog-ng/syslog-ng/pull/3679>) * kafka-c: fixed a hang during shutdown/reload, when multiple workers is used (workers() option is set to 2 or higher) and the librdkafka internal queue is filled. (error message was kafka: failed to publish message; topic='test-topic', error='Local: Queue full') (#3711<https://github.com/syslog-ng/syslog-ng/pull/3711>) Packaging * kafka: minimum version of librdkafka is changed from 1.0.0 to 1.1.0 (#3706<https://github.com/syslog-ng/syslog-ng/pull/3706>) * configure: now supporting python with two digit minor version (#3713<https://github.com/syslog-ng/syslog-ng/pull/3713>) Other changes * kafka: removed some deprecated options: client-lib-dir(), option(), template(), kafka-bootstrap-servers() (#3698<https://github.com/syslog-ng/syslog-ng/pull/3698>) * kafka: properties-file() option is deprecated. Please list the librdkafka properties in the config() option in syslog-ng's configuration. (#3698<https://github.com/syslog-ng/syslog-ng/pull/3698>) * smtp(): libesmtp is now detected via pkg-config (#3669<https://github.com/syslog-ng/syslog-ng/pull/3669>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler, Balázs Barkó, Benedek Cserhati, Gabor Nagy, L4rS6, Laszlo Budai, Laszlo Szemere, LittleFish33, László Várady, Norbert Takacs, Peter Czanik, Peter Kokai, Todd C. Miller, Tomáš Mózes, Zoltan Pallagi

1 0