- syslog-ng - lists.balabit.hu

syslog-ng-3.33.2
by Laszlo Szemere (lszemere) 19 Jul '21

19 Jul '21

3.33.2 Bugfixes * disk-buffer: fixed a bug, which was introduced in 3.33.1, where we sometimes corrupted the disk-buffer file when it reached full size. (#3726<https://github.com/syslog-ng/syslog-ng/pull/3726>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Attila Szakacs, Balázs Barkó, Balazs Scheidler, Benedek Cserhati, Gabor Nagy, Josef Schlehofer, Laszlo Budai, Laszlo Szemere, László Várady, Norbert Takacs, Parrag Szilárd, Peter Kokai, Zoltan Pallagi

1 0

Re: [syslog-ng] Logging to postgresql - error creating table with IP
by Herda, Martin 14 Jul '21

14 Jul '21

Thanks for your reply, I did indeed apply the single quotes and also tried with escaped double quotes. In either case the SQL statment string gets malformed to this: CREATE TABLE _logs_192.168.75.130_ Which will result in the same error as before: Error running SQL query; type='pgsql', host='127.0.0.1', port='5432', user='syslog', database='db_syslog', error='6819553: FEHLER: ERROR: Syntax error at >.168<\x0aLINE 1: CREATE TABLE _logs_192.168.75.130_ (datetime varchar(16), ho...\x0a ^\x0a', query='CREATE TABLE _logs_192.168.75.130_ (datetime varchar(16), host varchar(32), program varchar(20), pid varchar(10), message varchar(800))' Error creating table, giving up; table='_logs_192.168.75.130_' Br, MH Von: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu> Im Auftrag von Laszlo Szemere (lszemere) Gesendet: Dienstag, 13. Juli 2021 19:24 An: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Betreff: Re: [syslog-ng] Logging to postgresql - error creating table with IP Hello, as a quick and dirty test I found an online Postgress test env. https://extendsclass.com/postgresql-online.html I made some tests with the following commands: -- create table scientist (id integer, firstname varchar(100), lastname varchar(100)); -- insert into scientist (id, firstname, lastname) values (1, 'albert', 'einstein'); -- insert into scientist (id, firstname, lastname) values (2, 'isaac', 'newton'); -- insert into scientist (id, firstname, lastname) values (3, 'marie', 'curie'); -- select * from scientist; DROP TABLE "logs_192.168.75.102"; CREATE TABLE "logs_192.168.75.102" (datetime varchar(16), host varchar(32), program varchar(20), pid varchar(10), message varchar(800)); insert into "logs_192.168.75.102" (datetime, host, program, pid, message) values ('datetime', 'host', 'program', 'pid', 'message'); insert into "logs_192.168.75.102" (datetime, host, program, pid, message) values ('datetime', 'host', 'program', 'pid', 'message'); insert into "logs_192.168.75.102" (datetime, host, program, pid, message) values ('datetime', 'host', 'program', 'pid', 'message'); select * from "logs_192.168.75.102"; As it turned out, Postgres indeed accepts dots in the table name, however the SQL statement tourns out to be syntactically invalid without quoting it. (Please confirm: Did you apply quotes during the manual test?) Syslog-ng do not apply any (unnecessary?) quotes to the table name: https://github.com/syslog-ng/syslog-ng/blob/master/modules/afsql/afsql.c#L5… It looks like, simply adding it to the template of the table option: table("logs_${HOST}") --> table("'logs_${HOST}'") should do the trick. However, I cannot test it right now. Sorry. (Please inform me if the problem persists, and I will investigate it further when I am back to my machine.) Br, Laci ________________________________ From: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> on behalf of Herda, Martin <Martin.Herda(a)fritz-edv.de<mailto:Martin.Herda@fritz-edv.de>> Sent: Tuesday, July 13, 2021 17:15 To: 'syslog-ng(a)lists.balabit.hu' <syslog-ng(a)lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: [syslog-ng] Logging to postgresql - error creating table with IP CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Greetings to all, Im on Debian 4.19.194-2 using syslog-ng 3.19.1 The .conf is set to send all incoming syslog packages into pg database: destination d_pgsql { sql(type(pgsql) host("127.0.0.1") username("syslog") password("syslog") port("5432") database("db_syslog") table("logs_${HOST}") columns("datetime varchar(16)", "host varchar(32)", "program varchar(20)", "pid varchar(10)", "message varchar(800)") values("$R_DATE", "${HOST}", "$PROGRAM", "$PID", "$MSG") indexes("datetime", "host", "program", "pid", "message")); }; When there is a hostname available, the table is created successfully. When hostname can not be resolved, IP Address is used instead. But in this case syslog-ng is unable to create a new table: syslog-ng[499]: Error running SQL query; type='pgsql', host='127.0.0.1', port='5432', user='syslog', database='db_syslog', error='6819553: ERROR: Syntax error at >.168<\x0aLINE 1: CREATE TABLE logs_192.168.75.102 (datetime varchar(16), ho...\x0a ^\x0a', query='CREATE TABLE logs_192.168.75.102 (datetime varchar(16), host varchar(32), program varchar(20), pid varchar(10), message varchar(800))' Error creating table, giving up; table='logs_192.168.75.102' The issue seems to be the dot of the IP address. But dots in table names are possible AFAIK. Im able to manually create this table via HeidiSQL with no issues. Does the CREATE TABLE statement need to be escaped in some way? Or is there a way to replace the dots from the $HOST macro with underscores?

2 1

Logging to postgresql - error creating table with IP
by Herda, Martin 13 Jul '21

13 Jul '21

Greetings to all, Im on Debian 4.19.194-2 using syslog-ng 3.19.1 The .conf is set to send all incoming syslog packages into pg database: destination d_pgsql { sql(type(pgsql) host("127.0.0.1") username("syslog") password("syslog") port("5432") database("db_syslog") table("logs_${HOST}") columns("datetime varchar(16)", "host varchar(32)", "program varchar(20)", "pid varchar(10)", "message varchar(800)") values("$R_DATE", "${HOST}", "$PROGRAM", "$PID", "$MSG") indexes("datetime", "host", "program", "pid", "message")); }; When there is a hostname available, the table is created successfully. When hostname can not be resolved, IP Address is used instead. But in this case syslog-ng is unable to create a new table: syslog-ng[499]: Error running SQL query; type='pgsql', host='127.0.0.1', port='5432', user='syslog', database='db_syslog', error='6819553: ERROR: Syntax error at >.168<\x0aLINE 1: CREATE TABLE logs_192.168.75.102 (datetime varchar(16), ho...\x0a ^\x0a', query='CREATE TABLE logs_192.168.75.102 (datetime varchar(16), host varchar(32), program varchar(20), pid varchar(10), message varchar(800))' Error creating table, giving up; table='logs_192.168.75.102' The issue seems to be the dot of the IP address. But dots in table names are possible AFAIK. Im able to manually create this table via HeidiSQL with no issues. Does the CREATE TABLE statement need to be escaped in some way? Or is there a way to replace the dots from the $HOST macro with underscores?

2 1

syslog-ng-3.33.1
by László Várady (lvarady) 07 Jul '21

07 Jul '21

3.33.1 Highlights * MQTT destination The new mqtt() destination can be used to publish messages using the MQTT protocol. Currently MQTT 3.1.1 and 3.1 are supported. Supported transports: tcp, ws. Example config: destination { mqtt{ address("tcp://localhost:1883"), topic("syslog/$HOST"), fallback-topic("syslog/fallback")) }; }; Note: MQTT 5.0 and TLS (ssl://, wss://) are currently not supported. (#3703<https://github.com/syslog-ng/syslog-ng/pull/3703>) * discord() destination syslog-ng now has a webhook-based Discord destination. Example usage: destination { discord(url("https://discord.com/api/webhooks/x/y")); }; The following options can be used to customize the destination further: avatar-url(), username("$HOST-bot"), tts(true), template("${MSG:-[empty message]}"). (#3717<https://github.com/syslog-ng/syslog-ng/pull/3717>) Features * kafka-c: batching support in case of sync-send(yes) kafka-c( bootstrap-server("localhost:9092") topic("syslog-ng") sync-send(yes) batch-lines(10) batch-timeout(10000) ); Note1: batch-lines are accepted in case of sync-send(no), but no batching is done. Note2: messages are still sent one at a time to kafka, the batch yields multiple message per transaction. (#3699<https://github.com/syslog-ng/syslog-ng/pull/3699>) * kafka-c: sync-send(yes) enables synchronous message delivery, reducing the possibility of message loss. kafka-c( bootstrap-server("localhost:9092") topic("syslog-ng") sync-send(yes) ); Warning: this option also reduces significantly the performance of kafka-c driver. (#3681<https://github.com/syslog-ng/syslog-ng/pull/3681>) * disk-buffer: Now we optimize the file truncating frequency of disk-buffer. The new behavior saves IO time, but loses some disk space, which is configurable with a new option. The new option in the config is settable at 2 places: * truncate-size-ratio() in the disk-buffer() block, which affects the given disk-buffer. * disk-buffer(truncate-size-ratio()) in the global options block, which affects every disk-buffer which did not set truncate-size-ratio() itself. The default value is 0.01, which operates well with most disk-buffers. If the possible size reduction of the truncation does not reach truncate-size-ratio() x disk-buf-size(), we do not truncate the disk-buffer. To completely turn off truncating (maximal disk space loss, maximal IO time saved) set truncate-size-ratio(1), or to mimic the old behavior (minimal disk space loss, minimal IO time saved) set truncate-size-ratio(0). (#3689<https://github.com/syslog-ng/syslog-ng/pull/3689>) Bugfixes * syslog-format: fixing the check-hostname(yes|no) option The check-hostname(yes|no) option detected every value as invalid, causing a parse error when enabled. (#3690<https://github.com/syslog-ng/syslog-ng/pull/3690>) * disk-buffer(): fix crash when switching between disk-based and memory queues When a disk-buffer was removed from the configuration and the new config was applied by reloading syslog-ng, a crash occurred. (#3700<https://github.com/syslog-ng/syslog-ng/pull/3700>) * logpath: Fixed a message write protection bug, where message modifications (rewrite rules, parsers, etc.) leaked through preceding path elements. This may have resulted not only in unwanted/undefined message modification, but in certain cases crash as well. (#3708<https://github.com/syslog-ng/syslog-ng/pull/3708>) * mongodb(): fix crash with older mongo-c-driver versions syslog-ng crashed (was aborted) when the mongodb() destination was used with older mongo-c-driver versions (< v1.11.0). (#3677<https://github.com/syslog-ng/syslog-ng/pull/3677>) * java(): fix debug logging of Java-based destinations Java debug logging was not enabled previously when syslog-ng was started in debug/trace mode. This has been fixed. (#3679<https://github.com/syslog-ng/syslog-ng/pull/3679>) * kafka-c: fixed a hang during shutdown/reload, when multiple workers is used (workers() option is set to 2 or higher) and the librdkafka internal queue is filled. (error message was kafka: failed to publish message; topic='test-topic', error='Local: Queue full') (#3711<https://github.com/syslog-ng/syslog-ng/pull/3711>) Packaging * kafka: minimum version of librdkafka is changed from 1.0.0 to 1.1.0 (#3706<https://github.com/syslog-ng/syslog-ng/pull/3706>) * configure: now supporting python with two digit minor version (#3713<https://github.com/syslog-ng/syslog-ng/pull/3713>) Other changes * kafka: removed some deprecated options: client-lib-dir(), option(), template(), kafka-bootstrap-servers() (#3698<https://github.com/syslog-ng/syslog-ng/pull/3698>) * kafka: properties-file() option is deprecated. Please list the librdkafka properties in the config() option in syslog-ng's configuration. (#3698<https://github.com/syslog-ng/syslog-ng/pull/3698>) * smtp(): libesmtp is now detected via pkg-config (#3669<https://github.com/syslog-ng/syslog-ng/pull/3669>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler, Balázs Barkó, Benedek Cserhati, Gabor Nagy, L4rS6, Laszlo Budai, Laszlo Szemere, LittleFish33, László Várady, Norbert Takacs, Peter Czanik, Peter Kokai, Todd C. Miller, Tomáš Mózes, Zoltan Pallagi

1 0

query on remote logging
by SIMON BABY 07 Jul '21

07 Jul '21

Hi Team, Can I know if there is a configuration in syslog-ng.conf for logging the messages from a remote client. Can you please send me a sample configuration? Regards Simon

2 1

Dynamic setting value out of message?
by Matthias Gruber 01 Jul '21

01 Jul '21

Hi! I hope it is simple and my thoughts and seeks about it were to complicated :-), simply I didnt know how to do that, perhaps someone has a clue for me I am getting e.g. messages like "MESSAGE": "UIMUC4Maintenance.py: \"== deactivate_uc4_monitoring = ENDE ==\"", (thats out of an JSON-formatted syslog-ng output), What I would like to do is, to extract the 'UIMUC4Maintenance.py:' and put it into a SDATA-Custom-Variable or PROG but based on a regex so some sort of rewrite-rule like (no not a correct syntax, only to describe it) rewrite r_fill_program { set(match("^\w*\.py:" value("MESSAGE")) value("PROG")); }; As far as I understand it, set requires a "string" as first parameter, I could use a lots of rewrites with a condition, but I am in "lack of a static string", this should be some sort of variable :-) or I could do that static with a filter for every "^\w*\.py:"-Text, but I hope I could do that dynamic, every time a match of my regex syslog-ng inserts that part into a variable and so on... Is that possible? cheers Matthias ------------------------------------------------------------------------------------ METZLER Informationstechnologie Matthias Gruber IT-Infrastruktur & -Betrieb B. Metzler seel. Sohn & Co. Aktiengesellschaft Untermainanlage 1 60329 Frankfurt am Main Telefon (0 69) 21 04 - 43 30 Telefax (0 69) 21 04 - 40 40 MGruber(a)metzler.com www.metzler.com Vorstand: Harald Illy, Emmerich Müller, Gerhard Wiesheu Vorsitzender des Aufsichtsrats: Dr. Christoph Schücking Sitz der Gesellschaft: Frankfurt am Main, Handelsregister-Nr. HRB 123 365 Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfänger sein, so bitten wir Sie höflich, dies unverzüglich dem Absender mitzuteilen und die Nachricht zu löschen. Es ist unzulässig, die Nachricht unbefugt weiterzuleiten oder zu kopieren. Da wir nicht die Echtheit oder Vollständigkeit der in dieser Nachricht enthaltenen Informationen garantieren oder zusichern können, sind die vorstehenden Ausführungen rechtlich nicht bindend. Eine Haftung hierfür wird ausgeschlossen. This message is confidential. If you are not the intended recipient, we kindly ask you to inform the sender and delete the information. Any unauthorised dissemination or copying hereof is prohibited. As we cannot guarantee or assure the genuineness or completeness of the information contained in this message, the statements set forth above are not legally binding. Accordingly we cannot accept any liability for their contents.

3 4

Is there any compatibility between ver3.2 and ver3.25 about syslog-ng.conf?
by leis 25 Jun '21

25 Jun '21

Hello, I am looking to migrate my environment from CentOS5 syslog-ng ver3.2 to ubnutu20.02 syslog-ng ver3.25. But when I use syslog-ng.conf of ver3.2 to syslog-ng.conf of ver3.25, syslog-ng cannot start. Is there any compatibility between ver3.2's syslog-ng.conf and ver3.25's syslog-ng.conf? Thanks

2 1

Re: [syslog-ng] Tips to diagnose missing syslog messages?
by brian hoffman 15 Jun '21

15 Jun '21

We've spent a lot of time troubleshooting UDP drops, and the so_reuseport option did help quite a bit. There's a max number that the RECVQ can reach (which can be seen with: /usr/bin/netstat -tupln|egrep "(^udp.*syslog-ng)" ) that correlates to the so_rcvbuf. Having more queues won't keep drops from happening, since the hashing in the OSE is related to source IP, so a talkative host can still fill up one of those queues and cause drops. The PE uses eBPF which is apparently better at preventing this, but we found that by using the updated version at "copr:copr.fedorainfracloud.org:czanik:syslog-ng324/x86_64" instead of the stock RH one and bumping up the so_rcvbuf, it did perform much better (and I believe you may need that version for the so_reuseport).-Brian

1 0

Re: [syslog-ng] Request for config help
by Pradeep Gotaparthi 15 Jun '21

15 Jun '21

Hi, Could anybody help us out with the below query? Thanks. On Fri, Mar 26, 2021 at 12:06 AM Pradeep Gotaparthi <pradeep.gp(a)gmail.com> wrote: > Hi, > > We have been using syslog-ng for over a year now and it is working great, > thanks! > > One issue we face is that when we stop the process when there is some > buffer, the threshold is ignored and a bunch of logs get pushed out (as > much as 10 times the threshold value). This is causing problems on the > receiving end as it can't handle the load. We thought the log-fifo-size > parameter can help as syslog uses this parameter to store logs in memory > but it is not helping. > > It would be great if you can guide us. > > > Thanks. >

5 8

Insider 2021-06: Alerting; EoL technologies; Google Summer of Code;
by Peter Czanik (pczanik) 10 Jun '21

10 Jun '21

Dear syslog-ng users, This is the 92nd issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS First steps of sending alerts to Discord and others from syslog-ng: http() and Apprise ------------------------------- A returning question I get is: “I see, that you can send alerts from syslog-ng to Slack and Telegram, but do you happen to support XYZ?” Replace XYZ with Discord and countless others. Up until recently, my regular answer has been: “Take a look at the Slack destination of syslog-ng, and based on that, you can add support for your favorite service”. Then I learned about Apprise, a notification library for Python, supporting dozens of different services. This blog is the first part of a series. It covers how to send log messages to Discord using the http() destination of syslog-ng and an initial try at using Apprise for alerting. https://www.syslog-ng.com/community/b/blog/posts/first-steps-of-sending-ale… Changes in technologies supported by syslog-ng: Python 2, CentOS 6 & Co. ---------------------------------------------------- Technology is continuously evolving. There are regular changes in platforms running syslog-ng: old technologies disappear, and new technologies are introduced. While we try to provide stability and continuity to our users, we also need to adapt. Python 2 reached its end of life a year ago, CentOS 6 in November 2020. Using Java-based drivers has been problematic for many, so they were mostly replaced with native implementations. From this blog you can learn about recent changes affecting syslog-ng development and packaging. https://www.syslog-ng.com/community/b/blog/posts/changes-in-technologies-su… Google Summer of Code 2021 -------------------------- This year, the syslog-ng team participates in Google Summer of Code (GSoC) again as a mentoring organization. Two students paid by GSoC work on syslog-ng under the mentoring of syslog-ng developers. One of the students works on MacOS support, including the new ARM-based systems, while the other one is working developing on a new regular expression parser: https://summerofcode.withgoogle.com/organizations/5548293561516032/ WEBINARS * syslog-ng, Proxy Protocol Support: https://www.syslog-ng.com/event/syslogng-proxy-protocol-support-8148877/ * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0