October 2015 - syslog-ng - lists.balabit.hu

syslog-ng 3.7.2
by Budai, László 22 Mar '16

22 Mar '16

3.7.2 This is the first maintenance release for the 3.7.x series. Changes compared to 3.7.1: Improvements - Added mbox() source. This source can be used to fetch emails from local mbox files: source { mbox("/var/spool/mail/root"); }; This will fetch root emails and parse them into a multiline $MSG. Original implementation by Fabien Wernli, I only converted it into an SCL. - It is possible to append dynamically options into SCL blocks from now. - concurrent_request option added to ElasticSearch with default value 1. - In elasticsearch destinaton, message_template() argument renamed to template(). - SCL added to every Java module (ElasticSearch, Kafka, HDFS). - Linux Audit Parser added for parsing key-value pairs produced by the Linux Audit subsystem. - HTTP destination is now able to receive HTTP method as an option. All the supported methods are available (POST, PUT, HEAD, OPTIONS, DELETE, TRACE, GET). Fixes - In some circumstances syslog-ng mod-journal re-read every already processed messages. - When syslog-ng got a reload and the reload process done within 1 second then mafter the reload, syslog-ng stop generating mark-messages. - When initialization of a network destination in syslog-ng failed (eg. due to DNS resolution failure) we didn't create a queue which caused message loss. - syslog-ng segfaulted on TLS errors when wrong certs was provided (eg.: CA cert with the cert-file directive instead of the server cert). - Fixed a continuous spinning case in the file driver, when the destination file is a device (e.g. /dev/stdout). - A memory leak in around template functions in grammar fixed. - Fixed Python3 support. - Fixed Python GIL issue in python destination. - From now, instead of skipping doc/ alltogether when ENABLE_MANPAGES is not set, only skip the actual man pages, but handle the rest properly. - Allow overriding the python setup.py options. When installing the python modules, allow overriding the options. This is useful for distributions that want to pass extra options. For example, on Debian, we want --install-layout="deb" instead of the --prefix and --root options. With this change, the previous behaviour remains the default, but one can supply PYSETUP_OPTIONS on the make command-line to override it. - The systemd service file read /etc/default/syslog-ng and /etc/sysconfig/syslog-ng, but didn't do anything with their contents. $SYSLOGNG_OPTS added to ExecStart, so that the EnvironmentFiles have an effect (at least on Debian). - Java support checking fixed (not only jdk is required but also gradle). - Memory leak around ping() in Redis fixed. - A crash in pdbtool fixed around r_parser_email(). - Removed cygwin fdlimit statement. Make the default for RLIMIT_NOFILE equal to the current system limits. --fd-limit can still override this, but the default will be configured based on existing system limits. - Fixed BSD year inference. Fixed logic and made clearer the inference of year from bsd-style rfc3164 syslog-messages, which do not include a year. - Handle correctly the epoch 0 timestamp. (Previously, syslog-ng cached the zero timestamp and treated 1970 as it was 1900.) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Adam Arsenault, Adam Istvan Mozes, Andras Mitzki, Avleen Vig, Balazs Scheidler, Fabien Wernli, Gergely Czuczy, Gergely Nagy, Gergo Nagy, Laszlo Budai, Peter Czanik, Robert Fekete, Saurabh Shukla, Tamas Nagy, Tibor Benke, Viktor Juhasz, Vincent Bernat, Wang Long, Zdenek Styblik, Zoltan Pallagi. — View it on GitHub <https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.7.2>.

4 3

Reg; Syslog-ng does not recognize the audit facility
by Justin Kala 22 Feb '16

22 Feb '16

Hi Syslog-ng is unable to recognize the facility audit. When I put filter as audit and restart syslog-ng it errors out. When I put the facility code as 13 ,it does not error on restarting the service but does not capture the syslog message received through this filter code 13 as well. Please advise. -- Kaladhar

3 10

Regex Solaris from Linux hosts in Syslog-ng config file
by vijay amruth 04 Nov '15

04 Nov '15

Hello All, We are drawing logs from several hosts which include solaris(10,11) , linux (centos, ubuntu, rhel) into syslog servers, I want to be able to separate solaris logs, is there any pattern we can match for solaris logs that you may know ? Thanks, Vijay Amrut.

9 20

format-json reverse order?
by Scheidler, Balázs 03 Nov '15

03 Nov '15

Hi, I've encountered a case where format-json orders keys not alphabetically, but rather in the other direction. Can you remember any reason for that? This is the only patch needed to fix the order, but I've figured there may have been some reason behind the ordering. static gint vp_walk_cmp(const gchar *s1, const gchar *s2) { - return strcmp(s2, s1); + return strcmp(s1, s2); } Thanks in advance, -- Bazsi

3 4

unofficial openSUSE/fedora/RHEL syslog-ng 3.7.2 rpm packages
by Czanik, Péter 28 Oct '15

28 Oct '15

Hi, My unofficial syslog-ng 3.7.3 RPM packages for openSUSE/SLES/Fedora/EPEL are now available at the usual locations: Fedora/RHEL7: https://copr.fedoraproject.org/coprs/czanik/syslog-ng37/ RHEL6: https://copr.fedoraproject.org/coprs/czanik/syslog-ng37epel6/ openSUSE/SLES: https://build.opensuse.org/project/show/home:czanik:syslog-ng37 If you want to use the new Java based destination drivers, check my blog at https://czanik.blogs.balabit.com/2015/08/getting-started-with-syslog-ng-3-7… how to get started! Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

syslog-ng OSE: wildcards in source file path status
by Wout Peeters 27 Oct '15

27 Oct '15

Hi list, After some searching, I've noticed this question has been asked a few times before. The most recent answer I saw was from October 2014. Are there any updates regarding the implementation of wildcards in source file paths in syslog-ng OSE? Thanks!

3 2

unofficial syslog-ng 3.7.2 Debian packages
by Budai, László 27 Oct '15

27 Oct '15

Hi, I've just created my unofficial OBS repo[1] for syslog-ng 3.7.2 (latest revision: 3.7.2-2). List of supported OSs (i386/amd64): * Debian 7.0 * Debian 8.0 * Ubuntu 12.04 * Ubuntu 14.04 * Ubuntu 14.10 * Ubuntu 15.04 Install ------- example: Debian 8.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-get install syslog-ng-core=3.7.2-2` You can replace Debian_8.0 to * Debian_7.0 * xUbuntu_12.04, * xUbuntu_14.04 * xUbuntu_14.10 * xUbuntu_15.04 Available packages: * syslog-ng-core * syslog-ng-dbg * syslog-ng-dev * syslog-ng-mod-stomp * syslog-ng-mod-amqp * syslog-ng-mod-elastic * syslog-ng-mod-geoip * syslog-ng-mod-graphite * syslog-ng-mod-hdfs * syslog-ng-mod-http * syslog-ng-mod-java * syslog-ng-mod-java-common-lib * syslog-ng-mod-journal * syslog-ng-mod-json * syslog-ng-mod-kafka * syslog-ng-mod-mongodb * syslog-ng-mod-python * syslog-ng-mod-redis * syslog-ng-mod-riemann * syslog-ng-mod-smtp * syslog-ng-mod-sql * syslog-ng-mod-stomp Notes to the java destinations ------------------------------ About the ``` Error opening plugin module; module='mod-java', error='libjvm.so: cannot open shared object file: No such file or directory' ``` problem. libmod-java.so is linked to the libjvm.so library. If your java installation doesn't create a symlink in /usr/lib directory to libjvm.so then you have to modify your ld.so.conf or expand your LD_LIBRARY_PATH with the containing library before starting syslog-ng. eg.: LD_LIBRARY_PATH=/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server:$LD_LIBRARY_PATH Links -------- [1] https://build.opensuse.org/project/show/home:laszlo_budai:syslog-ng [2] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.7.2 regards, Laszlo Budai

2 1

Re: [syslog-ng] date parser
by Scheidler, Balázs 25 Oct '15

25 Oct '15

Hi, I am basically done with cleaning up the stuff. You can find my latest version at https://github.com/balabit/syslog-ng/tree/f/date-parser I still have to clean up the branch, it is currently just a dump of patches, but the final state should be pretty close that I'd try to push for integration. There's one issue that is still open: strptime() at least as implemented in various libc versions is unusable, so we should probably roll our own implementation. Reasons: 1) tm_gmtoff is not portable 2) %z as a parsing format is not portable 3) %z doesn't accept ISO8601 standard zone offsets when there's a colon between hours and minutes Also, it can be affected by locale settings (which is not good for the syslog-ng case) and is probably very slow. (at one point syslog-ng used strptime for normal log traffic and I remember replacing it with hand-coded routines increased performance a lot). I'll try to get an strptime implementation from somewhere and tailor it to our needs. Any feedback is appreciated. Bazsi -- Bazsi On Sun, Oct 25, 2015 at 4:44 PM, Scheidler, Balázs < balazs.scheidler(a)balabit.com> wrote: > Hi, > > As I see you assume a date is from UTC if there's no timezone information. > I would think that using the local timezone is more intuitive in this case: > > testcase("Tue, 27 Jan 2015 11:48:46", NULL, "%a, %d %b %Y %T", > "2015-01-27T11:48:46+00:00"); > > What do you think? > > > -- > Bazsi > > On Sun, Oct 25, 2015 at 12:36 PM, Scheidler, Balázs < > balazs.scheidler(a)balabit.com> wrote: > >> thanks for the quick reply. >> >> >> -- >> Bazsi >> >> On Sun, Oct 25, 2015 at 12:35 PM, Vincent Bernat <vincent(a)bernat.im> >> wrote: >> >>> ❦ 25 octobre 2015 12:32 +0100, "Scheidler, Balázs" < >>> balazs.scheidler(a)balabit.com> : >>> >>> > I'd have a question though: I can see the date-offset() option for >>> > date parser, which seems to skip the specified number of characters in >>> > the input. >>> > >>> > Can you please describe the usecase behind that? I'd be reluctant to >>> > add such an option to a parser (as none of the other have such an >>> > option), and it should be possible to do the same using template >>> > functions, e.g. something like this: >>> > >>> > date-parser(template("$(substr $MSG 5)")); >>> > >>> > This would similarly skip the first 5 characters. >>> > >>> > If this addresses your original use-case, I'd drop the date-offset() >>> > option. >>> >>> Yes, this would cover my use case. >>> -- >>> Use library functions. >>> - The Elements of Programming Style (Kernighan & Plauger) >>> >> >> >

2 1

date parser
by Scheidler, Balázs 25 Oct '15

25 Oct '15

Hi Vincent, I am working on your date-parser() stuff that was integrated to syslog-ng-incubator recently. It is great stuff, so I'd like to make it part of the main syslog-ng codebase. I am doing some cleanups, which you can follow at the f/date-parser branch in github.com/balabit/syslog-ng I'd have a question though: I can see the date-offset() option for date parser, which seems to skip the specified number of characters in the input. Can you please describe the usecase behind that? I'd be reluctant to add such an option to a parser (as none of the other have such an option), and it should be possible to do the same using template functions, e.g. something like this: date-parser(template("$(substr $MSG 5)")); This would similarly skip the first 5 characters. If this addresses your original use-case, I'd drop the date-offset() option. Thanks in advance, -- Bazsi

1 0

Python external program destination and line buffer
by Giovanni Mancuso 21 Oct '15

21 Oct '15

Hi, i wrote a program destination (in python) that parse a message and insert data in redis. My code is: import re import fileinput import redis r = redis.Redis(host="127.0.0.1", port=6379, db=0) for line in fileinput.input(): tid = re.search('X-TransactionId: (.+?) ', line) #print tid.group(1) regexp_result = re.findall(" Recipient: ([A-z0-9._%+-]+(a)[A-z0-9.-]+)", line) for email in regexp_result: r.sadd(tid.group(1)+"_recs", email) I configure my destination in this way: destination python_to_file { program("/usr/bin/python /etc/syslog-ng/python/Ric_acc_dest.py" flush-lines(1)); }; It works only if i send many message. If i send only one message it doesn't insert nothing in redis. Any ideas? --

2 2