January 2016 - syslog-ng - lists.balabit.hu

syslog-ng 3.7.2
by Budai, László 22 Mar '16

22 Mar '16

3.7.2 This is the first maintenance release for the 3.7.x series. Changes compared to 3.7.1: Improvements - Added mbox() source. This source can be used to fetch emails from local mbox files: source { mbox("/var/spool/mail/root"); }; This will fetch root emails and parse them into a multiline $MSG. Original implementation by Fabien Wernli, I only converted it into an SCL. - It is possible to append dynamically options into SCL blocks from now. - concurrent_request option added to ElasticSearch with default value 1. - In elasticsearch destinaton, message_template() argument renamed to template(). - SCL added to every Java module (ElasticSearch, Kafka, HDFS). - Linux Audit Parser added for parsing key-value pairs produced by the Linux Audit subsystem. - HTTP destination is now able to receive HTTP method as an option. All the supported methods are available (POST, PUT, HEAD, OPTIONS, DELETE, TRACE, GET). Fixes - In some circumstances syslog-ng mod-journal re-read every already processed messages. - When syslog-ng got a reload and the reload process done within 1 second then mafter the reload, syslog-ng stop generating mark-messages. - When initialization of a network destination in syslog-ng failed (eg. due to DNS resolution failure) we didn't create a queue which caused message loss. - syslog-ng segfaulted on TLS errors when wrong certs was provided (eg.: CA cert with the cert-file directive instead of the server cert). - Fixed a continuous spinning case in the file driver, when the destination file is a device (e.g. /dev/stdout). - A memory leak in around template functions in grammar fixed. - Fixed Python3 support. - Fixed Python GIL issue in python destination. - From now, instead of skipping doc/ alltogether when ENABLE_MANPAGES is not set, only skip the actual man pages, but handle the rest properly. - Allow overriding the python setup.py options. When installing the python modules, allow overriding the options. This is useful for distributions that want to pass extra options. For example, on Debian, we want --install-layout="deb" instead of the --prefix and --root options. With this change, the previous behaviour remains the default, but one can supply PYSETUP_OPTIONS on the make command-line to override it. - The systemd service file read /etc/default/syslog-ng and /etc/sysconfig/syslog-ng, but didn't do anything with their contents. $SYSLOGNG_OPTS added to ExecStart, so that the EnvironmentFiles have an effect (at least on Debian). - Java support checking fixed (not only jdk is required but also gradle). - Memory leak around ping() in Redis fixed. - A crash in pdbtool fixed around r_parser_email(). - Removed cygwin fdlimit statement. Make the default for RLIMIT_NOFILE equal to the current system limits. --fd-limit can still override this, but the default will be configured based on existing system limits. - Fixed BSD year inference. Fixed logic and made clearer the inference of year from bsd-style rfc3164 syslog-messages, which do not include a year. - Handle correctly the epoch 0 timestamp. (Previously, syslog-ng cached the zero timestamp and treated 1970 as it was 1900.) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Adam Arsenault, Adam Istvan Mozes, Andras Mitzki, Avleen Vig, Balazs Scheidler, Fabien Wernli, Gergely Czuczy, Gergely Nagy, Gergo Nagy, Laszlo Budai, Peter Czanik, Robert Fekete, Saurabh Shukla, Tamas Nagy, Tibor Benke, Viktor Juhasz, Vincent Bernat, Wang Long, Zdenek Styblik, Zoltan Pallagi. — View it on GitHub <https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.7.2>.

4 3

Reg; Syslog-ng does not recognize the audit facility
by Justin Kala 22 Feb '16

22 Feb '16

Hi Syslog-ng is unable to recognize the facility audit. When I put filter as audit and restart syslog-ng it errors out. When I put the facility code as 13 ,it does not error on restarting the service but does not capture the syslog message received through this filter code 13 as well. Please advise. -- Kaladhar

3 10

Module development guide
by Marc Falzon 30 Jan '16

30 Jan '16

Hi Is there any documentation on how to develop new syslog-ng module using the native C API, besides reading the incubator modules source code? Cheers, m.

5 17

License check failure
by Juhász, Viktor 25 Jan '16

25 Jan '16

Hi, I really want to keep the license policy, but I don't know, and can't find any description, what if I just moved files to one directory to another directory without any modifications? For example I just moved the logmsg.c from lib to lib/logmsg directory, and the license check fails in travis. What should I do in this case? BR, Viktor Juhász

2 2

Re: [syslog-ng] Dependency on default-jre for Debian package
by Laszlo Budai 22 Jan '16

22 Jan '16

Hi, thank you for sharing this issue(and for suggesting a solution to it). In the next two weeks I'm going to work on the packaging a little bit and I'll fix it. regards, Laszlo Budai Sent from my HTC ----- Reply message ----- From: "Marc Falzon" <m(a)baha.mu> To: <syslog-ng(a)lists.balabit.hu> Subject: [syslog-ng] Dependency on default-jre for Debian package Date: Fri, Jan 22, 2016 10:22 Hi I'm trying to install the unofficial syslog-ng 3.7.2 Debian packages from http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… however I have a problem with the Java-related packages, since they have a package dependency on `default-jre`, and I'm using an Oracle JRE packaged using the `make-jpkg` utility which generates a Debian package providing `java-browser-plugin, java-runtime, java-runtime-headless, java-virtual-machine, java2-runtime, java2-runtime-headless, java6-runtime, java6-runtime-headless, java7-runtime, java7-runtime-headless, java8-runtime, java8-runtime-headless`, but not `default-jre`. Is it possible to modify those packages so that they depend on `java-runtime` instead? The `default-jre` packages also provide `java-runtime`. Thank you, m. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

2 1

Re: [syslog-ng] Module development guide
by Laszlo Budai 22 Jan '16

22 Jan '16

Hi, and we have an unmerged PR that could help implementing a threaded dst. : https://github.com/balabit/syslog-ng-gitbook/pull/52 We should unify these contents... L. Sent from my HTC ----- Reply message ----- From: "Marc Falzon" <m(a)baha.mu> To: "Syslog-ng users' and developers' mailing list" <syslog-ng(a)lists.balabit.hu> Subject: [syslog-ng] Module development guide Date: Fri, Jan 22, 2016 11:42 On Friday 22 January 2016 at 11:35, Tibor Benke wrote: > There is also a shell script which generates a basic destination skeleton. You can find its source on this brach: https://github.com/juhaszviktor/syslog-ng/tree/f/plugin-creator . Check the dev-utils/plugin_skeleton_creator directory and this commit for the usage: https://github.com/juhaszviktor/syslog-ng/commit/89361d4d1817560ce8c9062093… Awesome, thank you! m. > > 2016-01-22 11:33 GMT+01:00 Marc Falzon <m(a)baha.mu (mailto:m@baha.mu)>: > > Hi Bazsi, > > > > > > On Friday 22 January 2016 at 11:28, Scheidler, Balázs wrote: > > > > > not really, but maybe I would read the "official" modules source code instead. and of course you can ask questions on the mailing list. :) > > > > > > Yes that's what I've started to do, but a proper documentation is preferable, saves time and avoid confusion ;) > > > > > if you want to create a destination, the easiest is to create a threaded destination, where the output is an independent thread, that can use a synchronous API. Of course this is going to be slower than using an asynchronous implementation, but that usually is good enough, > > > > Thank you for the hint. Given my use case (HTTP destination) I think I'm safe to go with an asynchronous implementation since each request is stateless, no connections/locks involved. > > > > m. > > > > > > > > On Fri, Jan 22, 2016 at 11:03 AM, Marc Falzon <m(a)baha.mu (mailto:m@baha.mu) (mailto:m@baha.mu)> wrote: > > > > Hi > > > > > > > > Is there any documentation on how to develop new syslog-ng module using the native C API, besides reading the incubator modules source code? > > > > > > > > Cheers, > > > > > > > > m. > > > > > > > > > > > > ______________________________________________________________________________ > > > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > > > > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > > > > > > > > > > ______________________________________________________________________________ > > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > > > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > > > > > > > > ______________________________________________________________________________ > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

Dependency on default-jre for Debian package
by Marc Falzon 22 Jan '16

22 Jan '16

Hi I'm trying to install the unofficial syslog-ng 3.7.2 Debian packages from http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… however I have a problem with the Java-related packages, since they have a package dependency on `default-jre`, and I'm using an Oracle JRE packaged using the `make-jpkg` utility which generates a Debian package providing `java-browser-plugin, java-runtime, java-runtime-headless, java-virtual-machine, java2-runtime, java2-runtime-headless, java6-runtime, java6-runtime-headless, java7-runtime, java7-runtime-headless, java8-runtime, java8-runtime-headless`, but not `default-jre`. Is it possible to modify those packages so that they depend on `java-runtime` instead? The `default-jre` packages also provide `java-runtime`. Thank you, m.

1 0

"Syslog connection closed" but socket not closed
by Patrick Hemmer 21 Jan '16

21 Jan '16

We recently saw an issue where syslog-ng (version 3.6.4 on FreeBSD 10.1) was configured with a `syslog()` source, received an invalid frame, shut down the connection, but the socket remained open. I was looking through the documented fixes in the versions since the one we're using (3.6.4), but nothing looks related. 2016-01-21T07:44:21-05:00 iad1gweb01.ecom.chewy.com ERR syslog-ng[27090]: - Invalid frame header; header='' [meta sequenceId="389"] 2016-01-21T07:44:21-05:00 iad1gweb01.ecom.chewy.com NOTICE syslog-ng[27090]: - Syslog connection closed; fd='19', client='AF_INET(127.0.0.1:59317)', local='AF_INET(127.0.0.1:601)' [meta sequenceId="390"] # netstat -an|grep 59317 tcp4 81660 0 127.0.0.1.601 127.0.0.1.59317 ESTABLISHED tcp4 0 48923 127.0.0.1.59317 127.0.0.1.601 ESTABLISHED From the netstat output (the buffer sizes), it looks like syslog-ng stopped reading from the socket, but didn't close it, and the application on the other end kept trying to write to it. This is just my theory anyway. I can open a bug on github, but I wanted to bring it up here first. Thanks -Patrick

1 0

RFC5425 framing and log_msg_size
by Blake Day 21 Jan '16

21 Jan '16

Hi all, I’m wondering why syslog-ng disconnects clients that try to send log messages exceeding log_msg_size instead of just gracefully accepting and truncating the message? I took a quick glance at the code, and it appears to only affect clients using RFC5425 framing. Here is the relevant code: if (self->frame_len > self->super.options->max_msg_size) { msg_error("Incoming frame larger than log_msg_size()", evt_tag_int("log_msg_size", self->super.options->max_msg_size), evt_tag_int("frame_length", self->frame_len), NULL); return LPS_ERROR; } Here are excerpts from my logs showing the problem: 2016-01-20T11:27:28-05:00 [HOST] ERR syslog-ng[87462]: - Incoming frame larger than log_msg_size(); log_msg_size='16384', frame_length='76218' [meta sequenceId="6560"] 2016-01-20T11:27:28-05:00 [HOST] NOTICE syslog-ng[87462]: - Syslog connection closed; fd='26', client='AF_INET(127.0.0.1:52627)', local='AF_INET(127.0.0.1:601)' [meta sequenceId="6561"]

2 1

Re: [syslog-ng] patterndb and smtp to() issue
by Laszlo Budai 21 Jan '16

21 Jan '16

Hi, templateable afsmtp to() is available from version 3.7.1 . Sent from my HTC ----- Reply message ----- From: "Fabien Wernli" <wernli(a)in2p3.fr> To: "Syslog-ng users' and developers' mailing list" <syslog-ng(a)lists.balabit.hu> Subject: [syslog-ng] patterndb and smtp to() issue Date: Thu, Jan 21, 2016 06:55 Hi Alexandre, Templateable smtp/to() and co. was added only recently. I'll check later in the day on which version it appeared. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

2 1