- syslog-ng - lists.balabit.hu

rabbitmq source
by Czanik, Péter 18 Nov '15

18 Nov '15

Hi, Someone asked me about the RabitMQ source. As I don't use RabitMQ I don't have much personal experience, so I ask you about it: What are your experiences with the RabitMQ source? What is your message rate? What is your use case? Any problems, missing features? If for any reason, you don't want to share your experiences publicly, please e-mail me directly! Any feedback is very welcome! Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

RFC: groupingby parser
by Scheidler, Balázs 13 Nov '15

13 Nov '15

Hi, I have just created a pull request for the groupingby() parser that I am seeking feedback for: https://github.com/balabit/syslog-ng/pull/785 The commit message is pretty detailed on what it does, but let me reproduce it here for simplicity. Any feedback is appreciated. Thanks. dbparser: add groupingby() parser This patch adds a new parser that can perform simple correllation on log messages, e.g. when multiple input log messages describe the same event. In a way it is similar to the SQL GROUP BY operation, where an aggregate of a set of input records can be calculated. The major difference between SQL GROUP BY and groupingby() is that the first _always_ operates on a enumerable list of records, whereas groupingby() works on a stream of data. groupingby() produces related groups by using a sliding window on time, e.g. it can be specified how much time we need to look back to group related messages together. As a specific use-case, let's see Linux audit logs. Linux audit logs tend to be broken to several lines generated as a list of lines. These tend to be pretty close in time, however there might be multiple events logged at around the same time, which get mixed up in the output. The example below is the audit log for an ntpdate execution: type=SYSCALL msg=audit(1440927434.124:40347): arch=c000003e syscall=59 success=yes exit=0 a0=7f121cef0b88 a1=7f121cef0c00 a2=7f121e690d98 a3=2 items=2 ppid=4312 pid=4347 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ntpdate" exe="/usr/sbin/ntpdate" key=(null) type=EXECVE msg=audit(1440927434.124:40347): argc=3 a0="/usr/sbin/ntpdate" a1="-s" a2="ntp.ubuntu.com" type=CWD msg=audit(1440927434.124:40347): cwd="/" type=PATH msg=audit(1440927434.124:40347): item=0 name="/usr/sbin/ntpdate" inode=2006003 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL type=PATH msg=audit(1440927434.124:40347): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=5243184 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL type=PROCTITLE msg=audit(1440927434.124:40347): proctitle=2F62696E2F7368002F7573722F7362696E2F6E7470646174652D64656269616E002D73 These lines are connected by their 2nd field, msg equals to audit(1440927434.124:40347). This can be processed by the groupingby() parser in a similar way that db-parser() could do correllation. These are the options for groupingby(): * key(): specifies the key for the grouping, e.g. the value that must be the same for all messages in the group * scope(): specifies one of three values: "global", "host", "process", meaning the same as in db-parser; whether to apply grouping for all messages received by syslog-ng (global), only messages coming from the same host (host), or the same process/pid combination. * where(): specifies a filter condition, messages not matching the filter will NOT be added to the group. where() only has access to a single message, the current one being processed. * having(): specifies a filter condition that must match in order for the group to generate an aggregate message. having() has access to the entire group through the "context". * timeout(): specifies the maximum time to wait for all messages in the group to arrive. After this time, the group is assumed to be complete and is aggregation is triggered. * aggregate(): this specifies the aggregate message that's going to be generated when the group is complete. - tags(): - value(): - inherit-mode(): * inject-mode(): how the aggregate message is injected into the syslog-ng message routing, can be one of: "pass-through", "internal". * trigger(): trigger the closure of the group by matching an incoming message. If the filter condition specified here matches the incoming message, it will cause the aggregate message to be calculated, emitted and the group be discarded from the state table. A few use-cases where this can be useful: * Linux audit logs * postfix logs Signed-off-by: Balazs Scheidler <balazs.scheidler(a)balabit.com> -- Bazsi

1 0

OSE 3.7.2 questions
by Fekete, Róbert 13 Nov '15

13 Nov '15

Hi, I've seen that you had a few patches merged to 3.7.2, and I'm not sure what to include in the docs about them: - There were some csv-parser changes, is any of them user-visible? - What does linux-audit-parser do? Does it require any configuration, or it just works? Thanks! Robert

2 1

Re: [syslog-ng] Having trouble with java-modules
by Laszlo Budai 13 Nov '15

13 Nov '15

You shouldn't run autogen.sh for a release tarball. Could you copy-paste the download link for the syslog-ng tarball that doesn't contain the configure script? Sent from my HTC ----- Reply message ----- From: "Rory Toma" <rory(a)ooma.com> To: "Syslog-ng users' and developers' mailing list" <syslog-ng(a)lists.balabit.hu> Subject: [syslog-ng] Having trouble with java-modules Date: Thu, Nov 12, 2015 21:17 This one doesn't have configure, and when I run autogen.sh, it errors with: Running autogen in 'lib/ivykis'... Don't know how to bootstrap submodule 'lib/ivykis' If I remove it, then later I get: configure: error: Internal ivykis sources not found in lib/ivykis. This is a hard dependency, unable to build syslog-ng without them. On 11/10/15 11:45 PM, Budai, László wrote: > You can download releases from here: > https://github.com/balabit/syslog-ng/releases > > L. > > On Wednesday, November 11, 2015, Rory Toma <rory(a)ooma.com > <mailto:rory@ooma.com>> wrote: > > I can't locate the 3.7 source release tarballs, only the git, > which isn't running autogen.sh correctly for me. > > On 11/10/15 9:39 PM, Budai, László wrote: >> >> Hi, >> >> Java is not supported in 3.6.x ( only in syslog-ng incubator ). >> For 3.7 it is ported(reworked) from the incubator, so you should >> give a try to use the java support with syslog-ng-3.7.2. >> >> L. >> >> On Nov 11, 2015 1:44 AM, "Rory Toma" <rory(a)ooma.com >> <javascript:_e(%7B%7D,'cvml','rory(a)ooma.com');>> wrote: >> >> I've downloaded syslog-ng-3.6.4 (I'm not using master because >> I can't >> get the configure script to generate correctly) >> >> I'm following the steps here: >> https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-… >> >> and have even added "--enable-java" >> >> I notice that even with JAVA_HOME set on the command line, >> config.log >> picks up the wrong java, and passing --with-java-home has no >> effect. >> >> In any case, I cannot get java modules installed. Is there an >> additional >> command to run to create/populate the java modules directory? >> >> This is on CentOS-6.7 x86_64 >> ______________________________________________________________________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: >> http://www.balabit.com/support/documentation/?product=syslog-ng >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> >> >> >> ______________________________________________________________________________ >> Member info:https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation:http://www.balabit.com/support/documentation/?product=syslog-… >> FAQ:http://www.balabit.com/wiki/syslog-ng-faq >> > > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

2 1

Having trouble with java-modules
by Rory Toma 12 Nov '15

12 Nov '15

I've downloaded syslog-ng-3.6.4 (I'm not using master because I can't get the configure script to generate correctly) I'm following the steps here: https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-… and have even added "--enable-java" I notice that even with JAVA_HOME set on the command line, config.log picks up the wrong java, and passing --with-java-home has no effect. In any case, I cannot get java modules installed. Is there an additional command to run to create/populate the java modules directory? This is on CentOS-6.7 x86_64

3 5

Remote server not keeping message intact
by Gareth Allen 12 Nov '15

12 Nov '15

Hi all I'm sending Apache logs to a remote syslog-ng server, but the remote server isn't keeping the message intact. Source: My Apache log format: LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined What the log looks like: 172.27.15.149 - - [12/Nov/2015:08:30:59 +0000] "GET / HTTP/1.1" 200 3594 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.71 Safari/537.36" My syslog-ng configuration: source s_apache { file("/var/log/apache2/access.log" follow_freq(1) flags(no-parse)); }; destination d_apache_tcp { tcp("x.x.x.x" port(514)); }; log { source(s_apache); destination(d_apache_tcp); }; Log server: source s_net { udp(port(514)); tcp(port(514)); }; template apache { template("${MESSAGE}\n"); template-escape(no); }; destination apache { file("/var/log/apachetest" template(apache)); }; What I see in /var/log/apachetest: - - [12/Nov/2015:08:30:59 +0000] "GET / HTTP/1.1" 200 3594 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.71 Safari/537.36" As you can see the IP at the beginning of the log entry is being removed. I've tried using $MSG and $MSGONLY. Any ideas would be greatly appreciated. Gareth

5 6

Regex Solaris from Linux hosts in Syslog-ng config file
by vijay amruth 04 Nov '15

04 Nov '15

Hello All, We are drawing logs from several hosts which include solaris(10,11) , linux (centos, ubuntu, rhel) into syslog servers, I want to be able to separate solaris logs, is there any pattern we can match for solaris logs that you may know ? Thanks, Vijay Amrut.

9 20

format-json reverse order?
by Scheidler, Balázs 03 Nov '15

03 Nov '15

Hi, I've encountered a case where format-json orders keys not alphabetically, but rather in the other direction. Can you remember any reason for that? This is the only patch needed to fix the order, but I've figured there may have been some reason behind the ordering. static gint vp_walk_cmp(const gchar *s1, const gchar *s2) { - return strcmp(s2, s1); + return strcmp(s1, s2); } Thanks in advance, -- Bazsi

3 4

Syslog-ng 3.5 , "destination plugin python not found ..."
by Alexander Urcioli 02 Nov '15

02 Nov '15

Trying to use a custom python destination I wrote. I wrote the python class, did the necessary PYTHONPATH stuff and wrote a separate .conf file for syslog-ng. When testing this file with the command "syslog-ng -f seclog.conf -Fevd" I get the following error: Error parsing destination, destination plugin python not found in seclog.conf at line 9, column 31: destination d_SECLOG_SERVER { python( ^^^^^^ Is the python destination not supported in 3.5? Alex

2 1

unofficial openSUSE/fedora/RHEL syslog-ng 3.7.2 rpm packages
by Czanik, Péter 28 Oct '15

28 Oct '15

Hi, My unofficial syslog-ng 3.7.3 RPM packages for openSUSE/SLES/Fedora/EPEL are now available at the usual locations: Fedora/RHEL7: https://copr.fedoraproject.org/coprs/czanik/syslog-ng37/ RHEL6: https://copr.fedoraproject.org/coprs/czanik/syslog-ng37epel6/ openSUSE/SLES: https://build.opensuse.org/project/show/home:czanik:syslog-ng37 If you want to use the new Java based destination drivers, check my blog at https://czanik.blogs.balabit.com/2015/08/getting-started-with-syslog-ng-3-7… how to get started! Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0