- syslog-ng - lists.balabit.hu

Remote server not keeping message intact
by Gareth Allen 12 Nov '15

12 Nov '15

Hi all I'm sending Apache logs to a remote syslog-ng server, but the remote server isn't keeping the message intact. Source: My Apache log format: LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined What the log looks like: 172.27.15.149 - - [12/Nov/2015:08:30:59 +0000] "GET / HTTP/1.1" 200 3594 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.71 Safari/537.36" My syslog-ng configuration: source s_apache { file("/var/log/apache2/access.log" follow_freq(1) flags(no-parse)); }; destination d_apache_tcp { tcp("x.x.x.x" port(514)); }; log { source(s_apache); destination(d_apache_tcp); }; Log server: source s_net { udp(port(514)); tcp(port(514)); }; template apache { template("${MESSAGE}\n"); template-escape(no); }; destination apache { file("/var/log/apachetest" template(apache)); }; What I see in /var/log/apachetest: - - [12/Nov/2015:08:30:59 +0000] "GET / HTTP/1.1" 200 3594 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.71 Safari/537.36" As you can see the IP at the beginning of the log entry is being removed. I've tried using $MSG and $MSGONLY. Any ideas would be greatly appreciated. Gareth

5 6

Regex Solaris from Linux hosts in Syslog-ng config file
by vijay amruth 05 Nov '15

05 Nov '15

Hello All, We are drawing logs from several hosts which include solaris(10,11) , linux (centos, ubuntu, rhel) into syslog servers, I want to be able to separate solaris logs, is there any pattern we can match for solaris logs that you may know ? Thanks, Vijay Amrut.

9 20

format-json reverse order?
by Scheidler, Balázs 03 Nov '15

03 Nov '15

Hi, I've encountered a case where format-json orders keys not alphabetically, but rather in the other direction. Can you remember any reason for that? This is the only patch needed to fix the order, but I've figured there may have been some reason behind the ordering. static gint vp_walk_cmp(const gchar *s1, const gchar *s2) { - return strcmp(s2, s1); + return strcmp(s1, s2); } Thanks in advance, -- Bazsi

3 4

Syslog-ng 3.5 , "destination plugin python not found ..."
by Alexander Urcioli 02 Nov '15

02 Nov '15

Trying to use a custom python destination I wrote. I wrote the python class, did the necessary PYTHONPATH stuff and wrote a separate .conf file for syslog-ng. When testing this file with the command "syslog-ng -f seclog.conf -Fevd" I get the following error: Error parsing destination, destination plugin python not found in seclog.conf at line 9, column 31: destination d_SECLOG_SERVER { python( ^^^^^^ Is the python destination not supported in 3.5? Alex

2 1

unofficial openSUSE/fedora/RHEL syslog-ng 3.7.2 rpm packages
by Czanik, Péter 28 Oct '15

28 Oct '15

Hi, My unofficial syslog-ng 3.7.3 RPM packages for openSUSE/SLES/Fedora/EPEL are now available at the usual locations: Fedora/RHEL7: https://copr.fedoraproject.org/coprs/czanik/syslog-ng37/ RHEL6: https://copr.fedoraproject.org/coprs/czanik/syslog-ng37epel6/ openSUSE/SLES: https://build.opensuse.org/project/show/home:czanik:syslog-ng37 If you want to use the new Java based destination drivers, check my blog at https://czanik.blogs.balabit.com/2015/08/getting-started-with-syslog-ng-3-7… how to get started! Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

syslog-ng OSE: wildcards in source file path status
by Wout Peeters 27 Oct '15

27 Oct '15

Hi list, After some searching, I've noticed this question has been asked a few times before. The most recent answer I saw was from October 2014. Are there any updates regarding the implementation of wildcards in source file paths in syslog-ng OSE? Thanks!

3 2

unofficial syslog-ng 3.7.2 Debian packages
by Budai, László 27 Oct '15

27 Oct '15

Hi, I've just created my unofficial OBS repo[1] for syslog-ng 3.7.2 (latest revision: 3.7.2-2). List of supported OSs (i386/amd64): * Debian 7.0 * Debian 8.0 * Ubuntu 12.04 * Ubuntu 14.04 * Ubuntu 14.10 * Ubuntu 15.04 Install ------- example: Debian 8.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-get install syslog-ng-core=3.7.2-2` You can replace Debian_8.0 to * Debian_7.0 * xUbuntu_12.04, * xUbuntu_14.04 * xUbuntu_14.10 * xUbuntu_15.04 Available packages: * syslog-ng-core * syslog-ng-dbg * syslog-ng-dev * syslog-ng-mod-stomp * syslog-ng-mod-amqp * syslog-ng-mod-elastic * syslog-ng-mod-geoip * syslog-ng-mod-graphite * syslog-ng-mod-hdfs * syslog-ng-mod-http * syslog-ng-mod-java * syslog-ng-mod-java-common-lib * syslog-ng-mod-journal * syslog-ng-mod-json * syslog-ng-mod-kafka * syslog-ng-mod-mongodb * syslog-ng-mod-python * syslog-ng-mod-redis * syslog-ng-mod-riemann * syslog-ng-mod-smtp * syslog-ng-mod-sql * syslog-ng-mod-stomp Notes to the java destinations ------------------------------ About the ``` Error opening plugin module; module='mod-java', error='libjvm.so: cannot open shared object file: No such file or directory' ``` problem. libmod-java.so is linked to the libjvm.so library. If your java installation doesn't create a symlink in /usr/lib directory to libjvm.so then you have to modify your ld.so.conf or expand your LD_LIBRARY_PATH with the containing library before starting syslog-ng. eg.: LD_LIBRARY_PATH=/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server:$LD_LIBRARY_PATH Links -------- [1] https://build.opensuse.org/project/show/home:laszlo_budai:syslog-ng [2] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.7.2 regards, Laszlo Budai

2 1

Re: [syslog-ng] date parser
by Scheidler, Balázs 25 Oct '15

25 Oct '15

Hi, I am basically done with cleaning up the stuff. You can find my latest version at https://github.com/balabit/syslog-ng/tree/f/date-parser I still have to clean up the branch, it is currently just a dump of patches, but the final state should be pretty close that I'd try to push for integration. There's one issue that is still open: strptime() at least as implemented in various libc versions is unusable, so we should probably roll our own implementation. Reasons: 1) tm_gmtoff is not portable 2) %z as a parsing format is not portable 3) %z doesn't accept ISO8601 standard zone offsets when there's a colon between hours and minutes Also, it can be affected by locale settings (which is not good for the syslog-ng case) and is probably very slow. (at one point syslog-ng used strptime for normal log traffic and I remember replacing it with hand-coded routines increased performance a lot). I'll try to get an strptime implementation from somewhere and tailor it to our needs. Any feedback is appreciated. Bazsi -- Bazsi On Sun, Oct 25, 2015 at 4:44 PM, Scheidler, Balázs < balazs.scheidler(a)balabit.com> wrote: > Hi, > > As I see you assume a date is from UTC if there's no timezone information. > I would think that using the local timezone is more intuitive in this case: > > testcase("Tue, 27 Jan 2015 11:48:46", NULL, "%a, %d %b %Y %T", > "2015-01-27T11:48:46+00:00"); > > What do you think? > > > -- > Bazsi > > On Sun, Oct 25, 2015 at 12:36 PM, Scheidler, Balázs < > balazs.scheidler(a)balabit.com> wrote: > >> thanks for the quick reply. >> >> >> -- >> Bazsi >> >> On Sun, Oct 25, 2015 at 12:35 PM, Vincent Bernat <vincent(a)bernat.im> >> wrote: >> >>> ❦ 25 octobre 2015 12:32 +0100, "Scheidler, Balázs" < >>> balazs.scheidler(a)balabit.com> : >>> >>> > I'd have a question though: I can see the date-offset() option for >>> > date parser, which seems to skip the specified number of characters in >>> > the input. >>> > >>> > Can you please describe the usecase behind that? I'd be reluctant to >>> > add such an option to a parser (as none of the other have such an >>> > option), and it should be possible to do the same using template >>> > functions, e.g. something like this: >>> > >>> > date-parser(template("$(substr $MSG 5)")); >>> > >>> > This would similarly skip the first 5 characters. >>> > >>> > If this addresses your original use-case, I'd drop the date-offset() >>> > option. >>> >>> Yes, this would cover my use case. >>> -- >>> Use library functions. >>> - The Elements of Programming Style (Kernighan & Plauger) >>> >> >> >

2 1

date parser
by Scheidler, Balázs 25 Oct '15

25 Oct '15

Hi Vincent, I am working on your date-parser() stuff that was integrated to syslog-ng-incubator recently. It is great stuff, so I'd like to make it part of the main syslog-ng codebase. I am doing some cleanups, which you can follow at the f/date-parser branch in github.com/balabit/syslog-ng I'd have a question though: I can see the date-offset() option for date parser, which seems to skip the specified number of characters in the input. Can you please describe the usecase behind that? I'd be reluctant to add such an option to a parser (as none of the other have such an option), and it should be possible to do the same using template functions, e.g. something like this: date-parser(template("$(substr $MSG 5)")); This would similarly skip the first 5 characters. If this addresses your original use-case, I'd drop the date-offset() option. Thanks in advance, -- Bazsi

1 0

Python external program destination and line buffer
by Giovanni Mancuso 22 Oct '15

22 Oct '15

Hi, i wrote a program destination (in python) that parse a message and insert data in redis. My code is: import re import fileinput import redis r = redis.Redis(host="127.0.0.1", port=6379, db=0) for line in fileinput.input(): tid = re.search('X-TransactionId: (.+?) ', line) #print tid.group(1) regexp_result = re.findall(" Recipient: ([A-z0-9._%+-]+(a)[A-z0-9.-]+)", line) for email in regexp_result: r.sadd(tid.group(1)+"_recs", email) I configure my destination in this way: destination python_to_file { program("/usr/bin/python /etc/syslog-ng/python/Ric_acc_dest.py" flush-lines(1)); }; It works only if i send many message. If i send only one message it doesn't insert nothing in redis. Any ideas? --

2 2