November 2015 - syslog-ng - lists.balabit.hu

syslog-ng 3.7.2
by Budai, László 22 Mar '16

22 Mar '16

3.7.2 This is the first maintenance release for the 3.7.x series. Changes compared to 3.7.1: Improvements - Added mbox() source. This source can be used to fetch emails from local mbox files: source { mbox("/var/spool/mail/root"); }; This will fetch root emails and parse them into a multiline $MSG. Original implementation by Fabien Wernli, I only converted it into an SCL. - It is possible to append dynamically options into SCL blocks from now. - concurrent_request option added to ElasticSearch with default value 1. - In elasticsearch destinaton, message_template() argument renamed to template(). - SCL added to every Java module (ElasticSearch, Kafka, HDFS). - Linux Audit Parser added for parsing key-value pairs produced by the Linux Audit subsystem. - HTTP destination is now able to receive HTTP method as an option. All the supported methods are available (POST, PUT, HEAD, OPTIONS, DELETE, TRACE, GET). Fixes - In some circumstances syslog-ng mod-journal re-read every already processed messages. - When syslog-ng got a reload and the reload process done within 1 second then mafter the reload, syslog-ng stop generating mark-messages. - When initialization of a network destination in syslog-ng failed (eg. due to DNS resolution failure) we didn't create a queue which caused message loss. - syslog-ng segfaulted on TLS errors when wrong certs was provided (eg.: CA cert with the cert-file directive instead of the server cert). - Fixed a continuous spinning case in the file driver, when the destination file is a device (e.g. /dev/stdout). - A memory leak in around template functions in grammar fixed. - Fixed Python3 support. - Fixed Python GIL issue in python destination. - From now, instead of skipping doc/ alltogether when ENABLE_MANPAGES is not set, only skip the actual man pages, but handle the rest properly. - Allow overriding the python setup.py options. When installing the python modules, allow overriding the options. This is useful for distributions that want to pass extra options. For example, on Debian, we want --install-layout="deb" instead of the --prefix and --root options. With this change, the previous behaviour remains the default, but one can supply PYSETUP_OPTIONS on the make command-line to override it. - The systemd service file read /etc/default/syslog-ng and /etc/sysconfig/syslog-ng, but didn't do anything with their contents. $SYSLOGNG_OPTS added to ExecStart, so that the EnvironmentFiles have an effect (at least on Debian). - Java support checking fixed (not only jdk is required but also gradle). - Memory leak around ping() in Redis fixed. - A crash in pdbtool fixed around r_parser_email(). - Removed cygwin fdlimit statement. Make the default for RLIMIT_NOFILE equal to the current system limits. --fd-limit can still override this, but the default will be configured based on existing system limits. - Fixed BSD year inference. Fixed logic and made clearer the inference of year from bsd-style rfc3164 syslog-messages, which do not include a year. - Handle correctly the epoch 0 timestamp. (Previously, syslog-ng cached the zero timestamp and treated 1970 as it was 1900.) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Adam Arsenault, Adam Istvan Mozes, Andras Mitzki, Avleen Vig, Balazs Scheidler, Fabien Wernli, Gergely Czuczy, Gergely Nagy, Gergo Nagy, Laszlo Budai, Peter Czanik, Robert Fekete, Saurabh Shukla, Tamas Nagy, Tibor Benke, Viktor Juhasz, Vincent Bernat, Wang Long, Zdenek Styblik, Zoltan Pallagi. — View it on GitHub <https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.7.2>.

4 3

Reg; Syslog-ng does not recognize the audit facility
by Justin Kala 22 Feb '16

22 Feb '16

Hi Syslog-ng is unable to recognize the facility audit. When I put filter as audit and restart syslog-ng it errors out. When I put the facility code as 13 ,it does not error on restarting the service but does not capture the syslog message received through this filter code 13 as well. Please advise. -- Kaladhar

3 10

rabbitmq errors
by Al Itchon 15 Dec '15

15 Dec '15

I'm seeing an error in my log indicating that messages aren't being sent to a queue in RabbitMQ. This happens a few times a week. Here's the error: Nov 24 13:29:28 syslog09 syslog-ng[7883]: Network error while inserting into AMQP server; time_reopen='10' I have about 1000 servers sending tcp messages through a call to /dev/tcp to port on the syslog-ng server. It seems messages aren't sent to the queue for about 1 or 2 minutes, and then it resumes sending again without error. Could this be a network issue or something within syslog-ng that's limiting the messages?

4 8

unit test valgrind coverage
by Scheidler, Balázs 24 Nov '15

24 Nov '15

Hi, I've wanted to make our unit tests valgrind clean for a long time, here's where I am at right now, the branch fixes a number of production related bugs as well. Feedback, help appreciated. https://github.com/balabit/syslog-ng/tree/f/unit-test-cleanups I am at 57 successful and 18 failing unit tests so far. -- Bazsi

2 1

log_fetch_limit
by Al Itchon 20 Nov '15

20 Nov '15

I have about 1000 servers sending tcp messages through a call to /dev/tcp to a port on my syslog-ng server. I'm assuming each call to /dev/tcp is a single connection and a single message, so am I correct that the log_fetch_limit should be set to 1 since it applies to each connection? Also, should I bother adjusting the so-rcvbuf? I know the manual mentions that if using UDP, the receive buffer should be increased. I'm using TCP and my message rate is about 250 messages/sec.

2 1

rabbitmq source
by Czanik, Péter 18 Nov '15

18 Nov '15

Hi, Someone asked me about the RabitMQ source. As I don't use RabitMQ I don't have much personal experience, so I ask you about it: What are your experiences with the RabitMQ source? What is your message rate? What is your use case? Any problems, missing features? If for any reason, you don't want to share your experiences publicly, please e-mail me directly! Any feedback is very welcome! Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

RFC: groupingby parser
by Scheidler, Balázs 13 Nov '15

13 Nov '15

Hi, I have just created a pull request for the groupingby() parser that I am seeking feedback for: https://github.com/balabit/syslog-ng/pull/785 The commit message is pretty detailed on what it does, but let me reproduce it here for simplicity. Any feedback is appreciated. Thanks. dbparser: add groupingby() parser This patch adds a new parser that can perform simple correllation on log messages, e.g. when multiple input log messages describe the same event. In a way it is similar to the SQL GROUP BY operation, where an aggregate of a set of input records can be calculated. The major difference between SQL GROUP BY and groupingby() is that the first _always_ operates on a enumerable list of records, whereas groupingby() works on a stream of data. groupingby() produces related groups by using a sliding window on time, e.g. it can be specified how much time we need to look back to group related messages together. As a specific use-case, let's see Linux audit logs. Linux audit logs tend to be broken to several lines generated as a list of lines. These tend to be pretty close in time, however there might be multiple events logged at around the same time, which get mixed up in the output. The example below is the audit log for an ntpdate execution: type=SYSCALL msg=audit(1440927434.124:40347): arch=c000003e syscall=59 success=yes exit=0 a0=7f121cef0b88 a1=7f121cef0c00 a2=7f121e690d98 a3=2 items=2 ppid=4312 pid=4347 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ntpdate" exe="/usr/sbin/ntpdate" key=(null) type=EXECVE msg=audit(1440927434.124:40347): argc=3 a0="/usr/sbin/ntpdate" a1="-s" a2="ntp.ubuntu.com" type=CWD msg=audit(1440927434.124:40347): cwd="/" type=PATH msg=audit(1440927434.124:40347): item=0 name="/usr/sbin/ntpdate" inode=2006003 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL type=PATH msg=audit(1440927434.124:40347): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=5243184 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL type=PROCTITLE msg=audit(1440927434.124:40347): proctitle=2F62696E2F7368002F7573722F7362696E2F6E7470646174652D64656269616E002D73 These lines are connected by their 2nd field, msg equals to audit(1440927434.124:40347). This can be processed by the groupingby() parser in a similar way that db-parser() could do correllation. These are the options for groupingby(): * key(): specifies the key for the grouping, e.g. the value that must be the same for all messages in the group * scope(): specifies one of three values: "global", "host", "process", meaning the same as in db-parser; whether to apply grouping for all messages received by syslog-ng (global), only messages coming from the same host (host), or the same process/pid combination. * where(): specifies a filter condition, messages not matching the filter will NOT be added to the group. where() only has access to a single message, the current one being processed. * having(): specifies a filter condition that must match in order for the group to generate an aggregate message. having() has access to the entire group through the "context". * timeout(): specifies the maximum time to wait for all messages in the group to arrive. After this time, the group is assumed to be complete and is aggregation is triggered. * aggregate(): this specifies the aggregate message that's going to be generated when the group is complete. - tags(): - value(): - inherit-mode(): * inject-mode(): how the aggregate message is injected into the syslog-ng message routing, can be one of: "pass-through", "internal". * trigger(): trigger the closure of the group by matching an incoming message. If the filter condition specified here matches the incoming message, it will cause the aggregate message to be calculated, emitted and the group be discarded from the state table. A few use-cases where this can be useful: * Linux audit logs * postfix logs Signed-off-by: Balazs Scheidler <balazs.scheidler(a)balabit.com> -- Bazsi

1 0

OSE 3.7.2 questions
by Fekete, Róbert 13 Nov '15

13 Nov '15

Hi, I've seen that you had a few patches merged to 3.7.2, and I'm not sure what to include in the docs about them: - There were some csv-parser changes, is any of them user-visible? - What does linux-audit-parser do? Does it require any configuration, or it just works? Thanks! Robert

2 1

Re: [syslog-ng] Having trouble with java-modules
by Laszlo Budai 13 Nov '15

13 Nov '15

You shouldn't run autogen.sh for a release tarball. Could you copy-paste the download link for the syslog-ng tarball that doesn't contain the configure script? Sent from my HTC ----- Reply message ----- From: "Rory Toma" <rory(a)ooma.com> To: "Syslog-ng users' and developers' mailing list" <syslog-ng(a)lists.balabit.hu> Subject: [syslog-ng] Having trouble with java-modules Date: Thu, Nov 12, 2015 21:17 This one doesn't have configure, and when I run autogen.sh, it errors with: Running autogen in 'lib/ivykis'... Don't know how to bootstrap submodule 'lib/ivykis' If I remove it, then later I get: configure: error: Internal ivykis sources not found in lib/ivykis. This is a hard dependency, unable to build syslog-ng without them. On 11/10/15 11:45 PM, Budai, László wrote: > You can download releases from here: > https://github.com/balabit/syslog-ng/releases > > L. > > On Wednesday, November 11, 2015, Rory Toma <rory(a)ooma.com > <mailto:rory@ooma.com>> wrote: > > I can't locate the 3.7 source release tarballs, only the git, > which isn't running autogen.sh correctly for me. > > On 11/10/15 9:39 PM, Budai, László wrote: >> >> Hi, >> >> Java is not supported in 3.6.x ( only in syslog-ng incubator ). >> For 3.7 it is ported(reworked) from the incubator, so you should >> give a try to use the java support with syslog-ng-3.7.2. >> >> L. >> >> On Nov 11, 2015 1:44 AM, "Rory Toma" <rory(a)ooma.com >> <javascript:_e(%7B%7D,'cvml','rory(a)ooma.com');>> wrote: >> >> I've downloaded syslog-ng-3.6.4 (I'm not using master because >> I can't >> get the configure script to generate correctly) >> >> I'm following the steps here: >> https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-… >> >> and have even added "--enable-java" >> >> I notice that even with JAVA_HOME set on the command line, >> config.log >> picks up the wrong java, and passing --with-java-home has no >> effect. >> >> In any case, I cannot get java modules installed. Is there an >> additional >> command to run to create/populate the java modules directory? >> >> This is on CentOS-6.7 x86_64 >> ______________________________________________________________________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: >> http://www.balabit.com/support/documentation/?product=syslog-ng >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> >> >> >> ______________________________________________________________________________ >> Member info:https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation:http://www.balabit.com/support/documentation/?product=syslog-… >> FAQ:http://www.balabit.com/wiki/syslog-ng-faq >> > > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

2 1

Having trouble with java-modules
by Rory Toma 12 Nov '15

12 Nov '15

I've downloaded syslog-ng-3.6.4 (I'm not using master because I can't get the configure script to generate correctly) I'm following the steps here: https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-… and have even added "--enable-java" I notice that even with JAVA_HOME set on the command line, config.log picks up the wrong java, and passing --with-java-home has no effect. In any case, I cannot get java modules installed. Is there an additional command to run to create/populate the java modules directory? This is on CentOS-6.7 x86_64

3 5