July 2021 - syslog-ng - lists.balabit.hu

UDP drops with syslog-ng 3.24.1-1
by brian hoffman 05 Oct '22

05 Oct '22

We've been centrally logging with syslog-ng for about 5 years now. Over that time, the number of sources has grown significantly, and at some point we crossed a line where drops were happening (a quick survey of 3 million syslog packets yielded 420 unique currently sending hosts). After much research and experimentation, we've been able to get to the point where throughout the day there are 0 drops for the most part. This was achieved by installing the latest syslog-ng (not the RedHat packaged one) and creating a source for each CPU. Occasionally, though, we still have periods of drops so I'm trying to eliminate these last few. Here are some relevant configuration items:2 RedHat 7.8 VMs (load balanced via an F5) with 16GB memory and 4 CPUs each running syslog-ng-3.24.1-1.el7.x86_64.net.core.rmem_default = 212992net.core.rmem_max = 268435456 log_fifo_size(268435456); source s_network {network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp1"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp2"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp3"));network(ip("0.0.0.0") port(514) transport("udp") so_rcvbuf(441326592) so-reuseport(1) persist-name("udp4"));network(ip("0.0.0.0") port(514) transport("tcp") max_connections(200) keep_alive(yes) so_rcvbuf(67108864));}; We are limited to UDP, unfortunately, because we do not have control over the devices/networks/etc. that are sending to us, but we have changed as many of the internal senders and destinations to TCP as we can. With a script I created to view the packets, including drops, as well as the individual RECVQs, the issue can be illustrated. Here's what things look like normally:Thu May 7 10:48:15 EDT 2020, 27003 IP pkts rcvd,26980 IP pkts sent,24951 UDP pkts rcvd, 28075 UDP pkts sent,0 UDP pkt rcv err RECVQ-1=2176RECVQ-2=0RECVQ-3=0RECVQ-4=0Thu May 7 10:48:16 EDT 2020, 28453 IP pkts rcvd,28426 IP pkts sent,26185 UDP pkts rcvd, 29180 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=0RECVQ-2=0RECVQ-3=4352RECVQ-4=0Thu May 7 10:48:17 EDT 2020, 28294 IP pkts rcvd,28276 IP pkts sent,26277 UDP pkts rcvd, 28709 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=2176RECVQ-2=0RECVQ-3=0RECVQ-4=0 The RECVQs are sparsely used, and there are no errors. Around 9pm every night, the packet counts go up significantly (probably due to backup related logs):Wed May 6 21:00:08 EDT 2020, 66382 IP pkts rcvd,66366 IP pkts sent,39405 UDP pkts rcvd, 67592 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=1595008RECVQ-2=106217088RECVQ-3=53694976RECVQ-4=31858816Wed May 6 21:00:09 EDT 2020, 69317 IP pkts rcvd,69338 IP pkts sent,44446 UDP pkts rcvd, 75958 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=13056RECVQ-2=126397312RECVQ-3=75568128RECVQ-4=41626880Wed May 6 21:00:10 EDT 2020, 71205 IP pkts rcvd,71227 IP pkts sent,43657 UDP pkts rcvd, 74603 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=920448RECVQ-2=146122752RECVQ-3=100951168RECVQ-4=52622208Wed May 6 21:00:12 EDT 2020, 69578 IP pkts rcvd,69454 IP pkts sent,124465 UDP pkts rcvd, 163367 UDP pkts sent,0 UDP pkt rcv errRECVQ-1=13140864RECVQ-2=44494848RECVQ-3=125579136RECVQ-4=0 Still, though, it's handling it with no errors. But then at some point a threshold is reached and errors start piling up:Wed May 6 21:00:20 EDT 2020, 63177 IP pkts rcvd,63291 IP pkts sent,0 UDP pkts rcvd, 0 UDP pkts sent,38011 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=200357376RECVQ-3=292948352RECVQ-4=28890752Wed May 6 21:00:21 EDT 2020, 69501 IP pkts rcvd,69464 IP pkts sent,0 UDP pkts rcvd, 1 UDP pkts sent,42158 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=223551360RECVQ-3=314995584RECVQ-4=41735680Wed May 6 21:00:23 EDT 2020, 69962 IP pkts rcvd,69978 IP pkts sent,0 UDP pkts rcvd, 2 UDP pkts sent,43775 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=244732544RECVQ-3=338239616RECVQ-4=53858176Wed May 6 21:00:24 EDT 2020, 68266 IP pkts rcvd,68216 IP pkts sent,0 UDP pkts rcvd, 0 UDP pkts sent,43118 UDP pkt rcv errRECVQ-1=536871424RECVQ-2=265258752RECVQ-3=360643712RECVQ-4=65362688 The common denominator I've found is that one of the RECVQs hits 536,871,424. This number seems to be almost exactly double (512 difference) the rmem.max/log_fifo_size (268,435,456). Even though there seems to be capacity in the other RECVQs, just one of those hitting that magic number seems to be enough to throw things out of whack. During this time, the CPU usage for syslog-ng also drops: top - 21:00:11 up 2 days, 8:40, 2 users, load average: 1.09, 1.41, 1.32 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 123.6 10.9 3080:46 syslog-ng top - 21:00:14 up 2 days, 8:40, 2 users, load average: 1.00, 1.39, 1.31 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 24.6 10.9 3080:46 syslog-ng top - 21:00:17 up 2 days, 8:40, 2 users, load average: 1.00, 1.39, 1.31 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7214 root 20 0 2111388 1.7g 5392 S 7.6 10.9 3080:47 syslog-ng As best I can tell, based on the reading I've done, the message counts we're getting should be doable, but it's still not clear exactly how to size some of these options as they relate to UDP (like log_fetch_limit, which we do not have set). Anything else I should try?

2 4

Re: [syslog-ng] syslog-ng Digest, Vol 195, Issue 9
by Linn, William 23 Jul '21

23 Jul '21

Laci, Thanks for your response, I figured out my problem. Thanks again -----Original Message----- From: syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of syslog-ng-request(a)lists.balabit.hu Sent: Friday, July 23, 2021 4:08 AM To: syslog-ng(a)lists.balabit.hu Subject: syslog-ng Digest, Vol 195, Issue 9 Send syslog-ng mailing list submissions to syslog-ng(a)lists.balabit.hu To subscribe or unsubscribe via the World Wide Web, visit https://urldefense.com/v3/__https://lists.balabit.hu/mailman/listinfo/syslo… or, via email, send a message with subject or body 'help' to syslog-ng-request(a)lists.balabit.hu You can reach the person managing the list at syslog-ng-owner(a)lists.balabit.hu When replying, please edit your Subject line so it is more specific than "Re: Contents of syslog-ng digest..." Today's Topics: 1. Re: syslog-ng Digest, Vol 195, Issue 7 (Laszlo Szemere (lszemere)) ---------------------------------------------------------------------- Message: 1 Date: Fri, 23 Jul 2021 08:08:10 +0000 From: "Laszlo Szemere (lszemere)" <Laszlo.Szemere(a)oneidentity.com> To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] syslog-ng Digest, Vol 195, Issue 7 Message-ID: <DM6PR19MB346532CA52F4D51920BFFC529DE59(a)DM6PR19MB3465.namprd19.prod.outlook.com> Content-Type: text/plain; charset="iso-8859-1" Hello Linn, I am sorry, but I haven't noticed your message before. (My email client mixed it because of the subject.) Could you please elaborate on the statement "fails to work"? What are the symptoms? Syslog-ng fails to start, or it is working differently than what you expect? Starting Syslog-ng with debugging enabled, might help to pinpoint the problem. ./syslog-ng -Fdevt -F, --foreground Do not go into the background after initialization -v, --verbose Be a bit more verbose -d, --debug Enable debug messages -t, --trace Enable trace messages -e, --stderr Log messages to stderr Best regards, Laci ________________________________ From: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu> on behalf of Linn, William <wlinn(a)bofa.com> Sent: Tuesday, July 20, 2021 16:08 To: syslog-ng(a)lists.balabit.hu <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] syslog-ng Digest, Vol 195, Issue 7 CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. I am having issues with a previous configuration set up in 2012. How to re-write it to work, here is the configuration that fails to work. destination d_apps { file("/syslog-ng/app/${HOST}/${HOST}-application.log.${DAY}" owner("root") group("root") perm(0644) dir_perm(0755) create_dirs(yes) overwrite_if_older(604800) ); }; destination d_systems {file("/syslog-ng/sys/${HOST}/${HOST}-system.log.${DAY}" owner("root") group("root") perm(0644) dir_perm(0755) create_dirs(yes) overwrite_if_older(604800) ); }; destination d_auth {file("/syslog-ng/auth/auth.log.${MONTH}" owner("root") group("root") perm(0640) dir_perm(0750) create_dirs(yes) overwrite_if_older(604800) ); }; -----Original Message----- From: syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of syslog-ng-request(a)lists.balabit.hu Sent: Tuesday, July 20, 2021 8:00 AM To: syslog-ng(a)lists.balabit.hu Subject: syslog-ng Digest, Vol 195, Issue 7 Send syslog-ng mailing list submissions to syslog-ng(a)lists.balabit.hu To subscribe or unsubscribe via the World Wide Web, visit https://urldefense.com/v3/__https://nam12.safelinks.protection.outlook.com/… or, via email, send a message with subject or body 'help' to syslog-ng-request(a)lists.balabit.hu You can reach the person managing the list at syslog-ng-owner(a)lists.balabit.hu When replying, please edit your Subject line so it is more specific than "Re: Contents of syslog-ng digest..." Today's Topics: 1. syslog-ng-3.33.2 (Laszlo Szemere (lszemere)) ---------------------------------------------------------------------- Message: 1 Date: Mon, 19 Jul 2021 13:38:23 +0000 From: "Laszlo Szemere (lszemere)" <Laszlo.Szemere(a)oneidentity.com> To: "syslog-ng(a)lists.balabit.hu" <syslog-ng(a)lists.balabit.hu> Subject: [syslog-ng] syslog-ng-3.33.2 Message-ID: <DM6PR19MB346594FBE1EDA3F73ACFEB699DE19(a)DM6PR19MB3465.namprd19.prod.outlook.com> Content-Type: text/plain; charset="iso-8859-1" 3.33.2 Bugfixes * disk-buffer: fixed a bug, which was introduced in 3.33.1, where we sometimes corrupted the disk-buffer file when it reached full size. (#3726<https://urldefense.com/v3/__https://nam12.safelinks.protection.outlook.com/… >) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Attila Szakacs, Balázs Barkó, Balazs Scheidler, Benedek Cserhati, Gabor Nagy, Josef Schlehofer, Laszlo Budai, Laszlo Szemere, László Várady, Norbert Takacs, Parrag Szilárd, Peter Kokai, Zoltan Pallagi -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://urldefense.com/v3/__https://nam12.safelinks.protection.outlook.com/… > ------------------------------ Subject: Digest Footer _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://urldefense.com/v3/__https://nam12.safelinks.protection.outlook.com/… ------------------------------ End of syslog-ng Digest, Vol 195, Issue 7 ***************************************** ---------------------------------------------------------------------- This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at https://urldefense.com/v3/__https://nam12.safelinks.protection.outlook.com/… . If you are not the intended recipient, please delete this message. ______________________________________________________________________________ Member info: https://urldefense.com/v3/__https://nam12.safelinks.protection.outlook.com/… Documentation: https://urldefense.com/v3/__https://nam12.safelinks.protection.outlook.com/… FAQ: https://urldefense.com/v3/__https://nam12.safelinks.protection.outlook.com/… -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://urldefense.com/v3/__http://lists.balabit.hu/pipermail/syslog-ng/att… > ------------------------------ Subject: Digest Footer _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://urldefense.com/v3/__https://lists.balabit.hu/mailman/listinfo/syslo… ------------------------------ End of syslog-ng Digest, Vol 195, Issue 9 ***************************************** ---------------------------------------------------------------------- This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.

1 0

Re: [syslog-ng] syslog-ng Digest, Vol 195, Issue 7
by Linn, William 23 Jul '21

23 Jul '21

I am having issues with a previous configuration set up in 2012. How to re-write it to work, here is the configuration that fails to work. destination d_apps { file("/syslog-ng/app/${HOST}/${HOST}-application.log.${DAY}" owner("root") group("root") perm(0644) dir_perm(0755) create_dirs(yes) overwrite_if_older(604800) ); }; destination d_systems {file("/syslog-ng/sys/${HOST}/${HOST}-system.log.${DAY}" owner("root") group("root") perm(0644) dir_perm(0755) create_dirs(yes) overwrite_if_older(604800) ); }; destination d_auth {file("/syslog-ng/auth/auth.log.${MONTH}" owner("root") group("root") perm(0640) dir_perm(0750) create_dirs(yes) overwrite_if_older(604800) ); }; -----Original Message----- From: syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of syslog-ng-request(a)lists.balabit.hu Sent: Tuesday, July 20, 2021 8:00 AM To: syslog-ng(a)lists.balabit.hu Subject: syslog-ng Digest, Vol 195, Issue 7 Send syslog-ng mailing list submissions to syslog-ng(a)lists.balabit.hu To subscribe or unsubscribe via the World Wide Web, visit https://urldefense.com/v3/__https://lists.balabit.hu/mailman/listinfo/syslo… or, via email, send a message with subject or body 'help' to syslog-ng-request(a)lists.balabit.hu You can reach the person managing the list at syslog-ng-owner(a)lists.balabit.hu When replying, please edit your Subject line so it is more specific than "Re: Contents of syslog-ng digest..." Today's Topics: 1. syslog-ng-3.33.2 (Laszlo Szemere (lszemere)) ---------------------------------------------------------------------- Message: 1 Date: Mon, 19 Jul 2021 13:38:23 +0000 From: "Laszlo Szemere (lszemere)" <Laszlo.Szemere(a)oneidentity.com> To: "syslog-ng(a)lists.balabit.hu" <syslog-ng(a)lists.balabit.hu> Subject: [syslog-ng] syslog-ng-3.33.2 Message-ID: <DM6PR19MB346594FBE1EDA3F73ACFEB699DE19(a)DM6PR19MB3465.namprd19.prod.outlook.com> Content-Type: text/plain; charset="iso-8859-1" 3.33.2 Bugfixes * disk-buffer: fixed a bug, which was introduced in 3.33.1, where we sometimes corrupted the disk-buffer file when it reached full size. (#3726<https://urldefense.com/v3/__https://github.com/syslog-ng/syslog-ng/pull/372… >) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Attila Szakacs, Balázs Barkó, Balazs Scheidler, Benedek Cserhati, Gabor Nagy, Josef Schlehofer, Laszlo Budai, Laszlo Szemere, László Várady, Norbert Takacs, Parrag Szilárd, Peter Kokai, Zoltan Pallagi -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://urldefense.com/v3/__http://lists.balabit.hu/pipermail/syslog-ng/att… > ------------------------------ Subject: Digest Footer _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://urldefense.com/v3/__https://lists.balabit.hu/mailman/listinfo/syslo… ------------------------------ End of syslog-ng Digest, Vol 195, Issue 7 ***************************************** ---------------------------------------------------------------------- This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.

2 1

syslog-ng-3.33.2
by Laszlo Szemere (lszemere) 19 Jul '21

19 Jul '21

3.33.2 Bugfixes * disk-buffer: fixed a bug, which was introduced in 3.33.1, where we sometimes corrupted the disk-buffer file when it reached full size. (#3726<https://github.com/syslog-ng/syslog-ng/pull/3726>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Attila Szakacs, Balázs Barkó, Balazs Scheidler, Benedek Cserhati, Gabor Nagy, Josef Schlehofer, Laszlo Budai, Laszlo Szemere, László Várady, Norbert Takacs, Parrag Szilárd, Peter Kokai, Zoltan Pallagi

1 0

Re: [syslog-ng] Logging to postgresql - error creating table with IP
by Herda, Martin 14 Jul '21

14 Jul '21

Thanks for your reply, I did indeed apply the single quotes and also tried with escaped double quotes. In either case the SQL statment string gets malformed to this: CREATE TABLE _logs_192.168.75.130_ Which will result in the same error as before: Error running SQL query; type='pgsql', host='127.0.0.1', port='5432', user='syslog', database='db_syslog', error='6819553: FEHLER: ERROR: Syntax error at >.168<\x0aLINE 1: CREATE TABLE _logs_192.168.75.130_ (datetime varchar(16), ho...\x0a ^\x0a', query='CREATE TABLE _logs_192.168.75.130_ (datetime varchar(16), host varchar(32), program varchar(20), pid varchar(10), message varchar(800))' Error creating table, giving up; table='_logs_192.168.75.130_' Br, MH Von: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu> Im Auftrag von Laszlo Szemere (lszemere) Gesendet: Dienstag, 13. Juli 2021 19:24 An: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Betreff: Re: [syslog-ng] Logging to postgresql - error creating table with IP Hello, as a quick and dirty test I found an online Postgress test env. https://extendsclass.com/postgresql-online.html I made some tests with the following commands: -- create table scientist (id integer, firstname varchar(100), lastname varchar(100)); -- insert into scientist (id, firstname, lastname) values (1, 'albert', 'einstein'); -- insert into scientist (id, firstname, lastname) values (2, 'isaac', 'newton'); -- insert into scientist (id, firstname, lastname) values (3, 'marie', 'curie'); -- select * from scientist; DROP TABLE "logs_192.168.75.102"; CREATE TABLE "logs_192.168.75.102" (datetime varchar(16), host varchar(32), program varchar(20), pid varchar(10), message varchar(800)); insert into "logs_192.168.75.102" (datetime, host, program, pid, message) values ('datetime', 'host', 'program', 'pid', 'message'); insert into "logs_192.168.75.102" (datetime, host, program, pid, message) values ('datetime', 'host', 'program', 'pid', 'message'); insert into "logs_192.168.75.102" (datetime, host, program, pid, message) values ('datetime', 'host', 'program', 'pid', 'message'); select * from "logs_192.168.75.102"; As it turned out, Postgres indeed accepts dots in the table name, however the SQL statement tourns out to be syntactically invalid without quoting it. (Please confirm: Did you apply quotes during the manual test?) Syslog-ng do not apply any (unnecessary?) quotes to the table name: https://github.com/syslog-ng/syslog-ng/blob/master/modules/afsql/afsql.c#L5… It looks like, simply adding it to the template of the table option: table("logs_${HOST}") --> table("'logs_${HOST}'") should do the trick. However, I cannot test it right now. Sorry. (Please inform me if the problem persists, and I will investigate it further when I am back to my machine.) Br, Laci ________________________________ From: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> on behalf of Herda, Martin <Martin.Herda(a)fritz-edv.de<mailto:Martin.Herda@fritz-edv.de>> Sent: Tuesday, July 13, 2021 17:15 To: 'syslog-ng(a)lists.balabit.hu' <syslog-ng(a)lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: [syslog-ng] Logging to postgresql - error creating table with IP CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Greetings to all, Im on Debian 4.19.194-2 using syslog-ng 3.19.1 The .conf is set to send all incoming syslog packages into pg database: destination d_pgsql { sql(type(pgsql) host("127.0.0.1") username("syslog") password("syslog") port("5432") database("db_syslog") table("logs_${HOST}") columns("datetime varchar(16)", "host varchar(32)", "program varchar(20)", "pid varchar(10)", "message varchar(800)") values("$R_DATE", "${HOST}", "$PROGRAM", "$PID", "$MSG") indexes("datetime", "host", "program", "pid", "message")); }; When there is a hostname available, the table is created successfully. When hostname can not be resolved, IP Address is used instead. But in this case syslog-ng is unable to create a new table: syslog-ng[499]: Error running SQL query; type='pgsql', host='127.0.0.1', port='5432', user='syslog', database='db_syslog', error='6819553: ERROR: Syntax error at >.168<\x0aLINE 1: CREATE TABLE logs_192.168.75.102 (datetime varchar(16), ho...\x0a ^\x0a', query='CREATE TABLE logs_192.168.75.102 (datetime varchar(16), host varchar(32), program varchar(20), pid varchar(10), message varchar(800))' Error creating table, giving up; table='logs_192.168.75.102' The issue seems to be the dot of the IP address. But dots in table names are possible AFAIK. Im able to manually create this table via HeidiSQL with no issues. Does the CREATE TABLE statement need to be escaped in some way? Or is there a way to replace the dots from the $HOST macro with underscores?

2 1

Logging to postgresql - error creating table with IP
by Herda, Martin 13 Jul '21

13 Jul '21

Greetings to all, Im on Debian 4.19.194-2 using syslog-ng 3.19.1 The .conf is set to send all incoming syslog packages into pg database: destination d_pgsql { sql(type(pgsql) host("127.0.0.1") username("syslog") password("syslog") port("5432") database("db_syslog") table("logs_${HOST}") columns("datetime varchar(16)", "host varchar(32)", "program varchar(20)", "pid varchar(10)", "message varchar(800)") values("$R_DATE", "${HOST}", "$PROGRAM", "$PID", "$MSG") indexes("datetime", "host", "program", "pid", "message")); }; When there is a hostname available, the table is created successfully. When hostname can not be resolved, IP Address is used instead. But in this case syslog-ng is unable to create a new table: syslog-ng[499]: Error running SQL query; type='pgsql', host='127.0.0.1', port='5432', user='syslog', database='db_syslog', error='6819553: ERROR: Syntax error at >.168<\x0aLINE 1: CREATE TABLE logs_192.168.75.102 (datetime varchar(16), ho...\x0a ^\x0a', query='CREATE TABLE logs_192.168.75.102 (datetime varchar(16), host varchar(32), program varchar(20), pid varchar(10), message varchar(800))' Error creating table, giving up; table='logs_192.168.75.102' The issue seems to be the dot of the IP address. But dots in table names are possible AFAIK. Im able to manually create this table via HeidiSQL with no issues. Does the CREATE TABLE statement need to be escaped in some way? Or is there a way to replace the dots from the $HOST macro with underscores?

2 1

syslog-ng-3.33.1
by László Várady (lvarady) 07 Jul '21

07 Jul '21

3.33.1 Highlights * MQTT destination The new mqtt() destination can be used to publish messages using the MQTT protocol. Currently MQTT 3.1.1 and 3.1 are supported. Supported transports: tcp, ws. Example config: destination { mqtt{ address("tcp://localhost:1883"), topic("syslog/$HOST"), fallback-topic("syslog/fallback")) }; }; Note: MQTT 5.0 and TLS (ssl://, wss://) are currently not supported. (#3703<https://github.com/syslog-ng/syslog-ng/pull/3703>) * discord() destination syslog-ng now has a webhook-based Discord destination. Example usage: destination { discord(url("https://discord.com/api/webhooks/x/y")); }; The following options can be used to customize the destination further: avatar-url(), username("$HOST-bot"), tts(true), template("${MSG:-[empty message]}"). (#3717<https://github.com/syslog-ng/syslog-ng/pull/3717>) Features * kafka-c: batching support in case of sync-send(yes) kafka-c( bootstrap-server("localhost:9092") topic("syslog-ng") sync-send(yes) batch-lines(10) batch-timeout(10000) ); Note1: batch-lines are accepted in case of sync-send(no), but no batching is done. Note2: messages are still sent one at a time to kafka, the batch yields multiple message per transaction. (#3699<https://github.com/syslog-ng/syslog-ng/pull/3699>) * kafka-c: sync-send(yes) enables synchronous message delivery, reducing the possibility of message loss. kafka-c( bootstrap-server("localhost:9092") topic("syslog-ng") sync-send(yes) ); Warning: this option also reduces significantly the performance of kafka-c driver. (#3681<https://github.com/syslog-ng/syslog-ng/pull/3681>) * disk-buffer: Now we optimize the file truncating frequency of disk-buffer. The new behavior saves IO time, but loses some disk space, which is configurable with a new option. The new option in the config is settable at 2 places: * truncate-size-ratio() in the disk-buffer() block, which affects the given disk-buffer. * disk-buffer(truncate-size-ratio()) in the global options block, which affects every disk-buffer which did not set truncate-size-ratio() itself. The default value is 0.01, which operates well with most disk-buffers. If the possible size reduction of the truncation does not reach truncate-size-ratio() x disk-buf-size(), we do not truncate the disk-buffer. To completely turn off truncating (maximal disk space loss, maximal IO time saved) set truncate-size-ratio(1), or to mimic the old behavior (minimal disk space loss, minimal IO time saved) set truncate-size-ratio(0). (#3689<https://github.com/syslog-ng/syslog-ng/pull/3689>) Bugfixes * syslog-format: fixing the check-hostname(yes|no) option The check-hostname(yes|no) option detected every value as invalid, causing a parse error when enabled. (#3690<https://github.com/syslog-ng/syslog-ng/pull/3690>) * disk-buffer(): fix crash when switching between disk-based and memory queues When a disk-buffer was removed from the configuration and the new config was applied by reloading syslog-ng, a crash occurred. (#3700<https://github.com/syslog-ng/syslog-ng/pull/3700>) * logpath: Fixed a message write protection bug, where message modifications (rewrite rules, parsers, etc.) leaked through preceding path elements. This may have resulted not only in unwanted/undefined message modification, but in certain cases crash as well. (#3708<https://github.com/syslog-ng/syslog-ng/pull/3708>) * mongodb(): fix crash with older mongo-c-driver versions syslog-ng crashed (was aborted) when the mongodb() destination was used with older mongo-c-driver versions (< v1.11.0). (#3677<https://github.com/syslog-ng/syslog-ng/pull/3677>) * java(): fix debug logging of Java-based destinations Java debug logging was not enabled previously when syslog-ng was started in debug/trace mode. This has been fixed. (#3679<https://github.com/syslog-ng/syslog-ng/pull/3679>) * kafka-c: fixed a hang during shutdown/reload, when multiple workers is used (workers() option is set to 2 or higher) and the librdkafka internal queue is filled. (error message was kafka: failed to publish message; topic='test-topic', error='Local: Queue full') (#3711<https://github.com/syslog-ng/syslog-ng/pull/3711>) Packaging * kafka: minimum version of librdkafka is changed from 1.0.0 to 1.1.0 (#3706<https://github.com/syslog-ng/syslog-ng/pull/3706>) * configure: now supporting python with two digit minor version (#3713<https://github.com/syslog-ng/syslog-ng/pull/3713>) Other changes * kafka: removed some deprecated options: client-lib-dir(), option(), template(), kafka-bootstrap-servers() (#3698<https://github.com/syslog-ng/syslog-ng/pull/3698>) * kafka: properties-file() option is deprecated. Please list the librdkafka properties in the config() option in syslog-ng's configuration. (#3698<https://github.com/syslog-ng/syslog-ng/pull/3698>) * smtp(): libesmtp is now detected via pkg-config (#3669<https://github.com/syslog-ng/syslog-ng/pull/3669>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler, Balázs Barkó, Benedek Cserhati, Gabor Nagy, L4rS6, Laszlo Budai, Laszlo Szemere, LittleFish33, László Várady, Norbert Takacs, Peter Czanik, Peter Kokai, Todd C. Miller, Tomáš Mózes, Zoltan Pallagi

1 0

query on remote logging
by SIMON BABY 07 Jul '21

07 Jul '21

Hi Team, Can I know if there is a configuration in syslog-ng.conf for logging the messages from a remote client. Can you please send me a sample configuration? Regards Simon

2 1

Dynamic setting value out of message?
by Matthias Gruber 01 Jul '21

01 Jul '21

Hi! I hope it is simple and my thoughts and seeks about it were to complicated :-), simply I didnt know how to do that, perhaps someone has a clue for me I am getting e.g. messages like "MESSAGE": "UIMUC4Maintenance.py: \"== deactivate_uc4_monitoring = ENDE ==\"", (thats out of an JSON-formatted syslog-ng output), What I would like to do is, to extract the 'UIMUC4Maintenance.py:' and put it into a SDATA-Custom-Variable or PROG but based on a regex so some sort of rewrite-rule like (no not a correct syntax, only to describe it) rewrite r_fill_program { set(match("^\w*\.py:" value("MESSAGE")) value("PROG")); }; As far as I understand it, set requires a "string" as first parameter, I could use a lots of rewrites with a condition, but I am in "lack of a static string", this should be some sort of variable :-) or I could do that static with a filter for every "^\w*\.py:"-Text, but I hope I could do that dynamic, every time a match of my regex syslog-ng inserts that part into a variable and so on... Is that possible? cheers Matthias ------------------------------------------------------------------------------------ METZLER Informationstechnologie Matthias Gruber IT-Infrastruktur & -Betrieb B. Metzler seel. Sohn & Co. Aktiengesellschaft Untermainanlage 1 60329 Frankfurt am Main Telefon (0 69) 21 04 - 43 30 Telefax (0 69) 21 04 - 40 40 MGruber(a)metzler.com www.metzler.com Vorstand: Harald Illy, Emmerich Müller, Gerhard Wiesheu Vorsitzender des Aufsichtsrats: Dr. Christoph Schücking Sitz der Gesellschaft: Frankfurt am Main, Handelsregister-Nr. HRB 123 365 Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfänger sein, so bitten wir Sie höflich, dies unverzüglich dem Absender mitzuteilen und die Nachricht zu löschen. Es ist unzulässig, die Nachricht unbefugt weiterzuleiten oder zu kopieren. Da wir nicht die Echtheit oder Vollständigkeit der in dieser Nachricht enthaltenen Informationen garantieren oder zusichern können, sind die vorstehenden Ausführungen rechtlich nicht bindend. Eine Haftung hierfür wird ausgeschlossen. This message is confidential. If you are not the intended recipient, we kindly ask you to inform the sender and delete the information. Any unauthorised dissemination or copying hereof is prohibited. As we cannot guarantee or assure the genuineness or completeness of the information contained in this message, the statements set forth above are not legally binding. Accordingly we cannot accept any liability for their contents.

3 4