- syslog-ng - lists.balabit.hu

file source driver questions
by Michael Hocke 17 Feb '09

17 Feb '09

Hi, I have an application here that does not use syslog but writes all log messages straight into a file. I want to use syslog-ng to pick up the entries of that file and send them over to my log repository via syslog. I am running 3.0.1 and the config looks as follows: * Client side (radserver): source s_accounting { file ("/opt/CSCOar/logs/accounting.log" follow_freq (1) program_override ("radius") flags (no-parse)); }; destination d_archive { udp ("a.b.c.d"); }; log { source (s_accounting); destination (d_archive); }; * Server side (a.b.c.d): source s_udp { udp (); }; destination d_radius { file ("/usr/local/var/log/remote/$HOST_FROM/ accounting.log" template ("$MSGONLY\n")); }; filter f_radius { host ("radserver") and facility (kern); }; log { source (s_udp); filter (f_radius); destination (d_radius); flags (final); }; That works but there are a couple of minor things. - By default all messages coming from the file source driver are tagged with the kern facility. I was not able to find this in the documentation but the source code (cfg-grammar.y) revealed that the file driver accepts the options "facility()" and "level()" to change this behavior. source s_accounting { file ("/opt/CSCOar/logs/accounting.log" follow_freq (1) program_override ("radius") flags (no-parse) facility (local7)); }; fixed that problem. - Empty lines seem to be ignored and not transmitted. The entries: "La la blah blah Hi There good-bye" end up as "La la blah blah Hi There good-bye" in my log file on the server side. Is there a way to enable that? Thanks! - Michael

2 2

[Bug 32] New: syslog-ng stops listening after receiving a zero length udp packet
by bugzilla＠bugzilla.balabit.com 17 Feb '09

17 Feb '09

https://bugzilla.balabit.com/show_bug.cgi?id=32 Summary: syslog-ng stops listening after receiving a zero length udp packet Product: syslog-ng Version: 3.0.x Platform: Other OS/Version: Other Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: ffs(a)ccn.net Type of the Report: bug Estimated Hours: 0.0 Created an attachment (id=12) --> (https://bugzilla.balabit.com/attachment.cgi?id=12) afsocket EOF patch Hi, if syslog-ng has an udp() source configured it stops listening after receiving a zero length udp packet. In debug mode it logs 'EOF occurred while reading'. After this message no udp messages get processed anymore. After a restart syslog-ng is logging again. As this bug is reproducible it could be abused for DOS attacks also. Tested on: Solaris 8/9 Attached is a patch for version 3.0.1. There might be a better way to solve this problem, but it's working for me. Thank you for syslog-ng! Florian -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

3 7

[Bug 33] New: syslog-ng 2.0.10 compile error over solaris
by bugzilla＠bugzilla.balabit.com 17 Feb '09

17 Feb '09

https://bugzilla.balabit.com/show_bug.cgi?id=33 Summary: syslog-ng 2.0.10 compile error over solaris Product: syslog-ng Version: 2.1.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: girl111_2002(a)hotmail.com Type of the Report: bug Estimated Hours: 0.0 gcc -g -O2 -Wall -o loggen loggen.o -lpthread -lresolv -lnsl -lrt -lsocket -ldoor -lfl -lglib-2.0 -L/usr/local/lib -levtlog -lwrap Undefined first referenced symbol in file yylex /usr/sfw/lib/gcc/sparc-sun-solaris2.10/3.4.3/../../../libfl.so deny_severity /usr/sfw/lib/gcc/sparc-sun-solaris2.10/3.4.3/../../../libwrap.so allow_severity /usr/sfw/lib/gcc/sparc-sun-solaris2.10/3.4.3/../../../libwrap.so ld: fatal: Symbol referencing errors. No output written to loggen collect2: ld returned 1 exit status *** Error code 1 make: Fatal error: Command failed for target `loggen' Current working directory /opt/software/syslog-ng/syslog-ng-2.0.10/tests/loggen *** Error code 1 The following command caused the error: set fnord $MAKEFLAGS; amf=$2; \ dot_seen=no; \ target=`echo all-recursive | sed s/-recursive//`; \ list='unit functional loggen'; for subdir in $list; do \ echo "Making $target in $subdir"; \ if test "$subdir" = "."; then \ dot_seen=yes; \ local_target="$target-am"; \ else \ local_target="$target"; \ fi; \ (cd $subdir && make $local_target) \ || case "$amf" in *=*) exit 1;; *k*) fail=yes;; *) exit 1;; esac; \ done; \ if test "$dot_seen" = "no"; then \ make "$target-am" || exit 1; \ fi; test -z "$fail" make: Fatal error: Command failed for target `all-recursive' Current working directory /opt/software/syslog-ng/syslog-ng-2.0.10/tests *** Error code 1 The following command caused the error: set fnord $MAKEFLAGS; amf=$2; \ dot_seen=no; \ target=`echo all-recursive | sed s/-recursive//`; \ list='src tests doc contrib solbuild tgzbuild'; for subdir in $list; do \ echo "Making $target in $subdir"; \ if test "$subdir" = "."; then \ dot_seen=yes; \ local_target="$target-am"; \ else \ local_target="$target"; \ fi; \ (cd $subdir && make $local_target) \ || case "$amf" in *=*) exit 1;; *k*) fail=yes;; *) exit 1;; esac; \ done; \ if test "$dot_seen" = "no"; then \ make "$target-am" || exit 1; \ fi; test -z "$fail" make: Fatal error: Command failed for target `all-recursive' Current working directory /opt/software/syslog-ng/syslog-ng-2.0.10 *** Error code 1 make: Fatal error: Command failed for target `all' -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

grouping of sources
by Joe Shaw 16 Feb '09

16 Feb '09

Hi, Is there a way to create a "group" of sources for log rules? This would be quite helpful when using the new include functionality for configuration files. To give you an idea of my setup: I have two files. One is a site config, which will be the same for every host. It contains global configuration and well-known destinations. The other file differs per-host, with different sources. I would like to be able to set up log rules for routing messages from those disparate sources to the well-known destinations inside the site config, but there doesn't seem to be any way to do it. Here's a pseudo-example. Site config: options { ... }; destination d_whatever { ... }; Host 1 config: source s_one { ... }; log { source(s_one); destination(d_whatever); }; Host 2 config: source s_two_alpha { ... }; source s_two_beta { ... }; log { source(s_two_alpha); source(s_two_beta); destination(d_whatever); }; Ideally, I'd be able to do something like this: Site config: options { ... }; destination d_whatever { ... }; log { source(s_aggregate); destination(d_whatever); }; Host 1 config: source s_one { ... }; source s_aggregate { source(s_one); }; Host 2 config: source s_two_alpha { ... }; source s_two_beta { ... }; source s_aggregate { source(s_two_alpha); source(s_two_beta); }; This setup maintains a nice, clean abstraction of the logging rules away from the source data. Is there any way to do something like this? Thanks, Joe

2 5

File descriptor leaks with 3.0.1
by chris packham 16 Feb '09

16 Feb '09

Hi, We're running a test where syslog-ng gets frequently sighup-ed (about every 5 seconds) and what we are finding is that after a while (sorry don't know specifically how may times) we run into a problem where syslog-ng starts experiencing errors reading its config. We are making use of the new 'include' feature (which is brilliant by the way). Here's a snippet of the internal() log 2009 Feb 17 00:05:01 syslog.notice syslog-ng[838]: Configuration reload request received, reloading configuration; 2009 Feb 17 00:05:04 syslog.notice syslog-ng[838]: Configuration reload request received, reloading configuration; 2009 Feb 17 00:05:37 syslog.notice syslog-ng[838]: Configuration reload request received, reloading configuration; 2009 Feb 17 00:05:43 syslog.notice syslog-ng[838]: Configuration reload request received, reloading configuration; 2009 Feb 17 00:05:46 syslog.err syslog-ng[838]: Error opening include file; filename='/etc/syslog-conf.d/trace-cfg', depth='1' 2009 Feb 17 00:05:46 syslog.notice syslog-ng[838]: Configuration reload request received, reloading configuration; 2009 Feb 17 00:05:47 syslog.err syslog-ng[838]: Error accepting new connection; error='Too many open files (24)' 2009 Feb 17 00:05:47 syslog.err syslog-ng[838]: Error accepting new connection; error='Too many open files (24)' /proc/$(pidof syslog-ng)/fd is also hinting that the included files aren't being closed. lr-x------ 1 root root 64 Feb 17 00:05 4042 -> /tmp/etc/syslog-conf.d/trace-cfg lr-x------ 1 root root 64 Feb 17 00:05 4043 -> /tmp/etc/syslog-conf.d/1_standalone lr-x------ 1 root root 64 Feb 17 00:05 4044 -> /tmp/etc/syslog-conf.d/buffered lr-x------ 1 root root 64 Feb 17 00:05 4045 -> /tmp/etc/syslog-conf.d/console lr-x------ 1 root root 64 Feb 17 00:05 4046 -> /tmp/etc/syslog-conf.d/email lr-x------ 1 root root 64 Feb 17 00:05 4047 -> /tmp/etc/syslog-conf.d/host lr-x------ 1 root root 64 Feb 17 00:05 4048 -> /tmp/etc/syslog-conf.d/permanent lr-x------ 1 root root 64 Feb 17 00:05 4049 -> /tmp/etc/syslog-conf.d/stats lr-x------ 1 root root 64 Feb 17 00:05 4050 -> /tmp/terminal lr-x------ 1 root root 64 Feb 17 00:05 4051 -> /tmp/etc/syslog-conf.d/trace-cfg lr-x------ 1 root root 64 Feb 17 00:05 4052 -> /tmp/etc/syslog-conf.d/1_standalone lr-x------ 1 root root 64 Feb 17 00:05 4053 -> /tmp/etc/syslog-conf.d/buffered lr-x------ 1 root root 64 Feb 17 00:05 4054 -> /tmp/etc/syslog-conf.d/console lr-x------ 1 root root 64 Feb 17 00:05 4055 -> /tmp/etc/syslog-conf.d/email lr-x------ 1 root root 64 Feb 17 00:05 4056 -> /tmp/etc/syslog-conf.d/host lr-x------ 1 root root 64 Feb 17 00:05 4057 -> /tmp/etc/syslog-conf.d/permanent lr-x------ 1 root root 64 Feb 17 00:05 4058 -> /tmp/etc/syslog-conf.d/stats lr-x------ 1 root root 64 Feb 17 00:05 4059 -> /tmp/terminal Also seems to be present in the latest version checked out from the git repo but I'll do some more digging. Thanks, Chris

1 0

Timezone problem?!
by Charles Chan 16 Feb '09

16 Feb '09

Recently I use the syslog-ng for log collection environment encountered a time zone problem,hope somebody can help me.Thanks The environment is like this: The apache server(PDT timezone,UTC-8) forword apache log through syslog to the syslog-ng server（CST timezone，UTC＋8）. The server with same time zones no problem between the forward. Syslog-ng test configuration is as follows: destination int_apache (file ( "/ disk3/cookie_logs/pid1 / $ YEAR-$ MONTH-$ DAY / $ (HOST). apache.log" time_zone (-08:00) template ( "$ DAY-$ HOUR $ S_DAY-$ S_HOUR $ R_DAY-$ R_HOUR $ $ ISODATE $ MSGONLY \ n ") template_escape (no));); The original apache log is: 58.240.69.161 - - - [14/Jan/2009: 22:27:47 -0800] "GET / www.example.com/HTTP/1.1" 200 2028 "-" "Mozilla/4.0 ( compatible; MSIE 7.0; Windows NT 5.1) " The syslog-ng server generated log is: 14-22 14-22 14-22 2009-01-14T22 :27:47-08: 00 58.240.69.161 - - - [14/Jan/2009: 22:27:47 -0800] "GET / www.example. com / HTTP/1.1 "200 2028" - "" Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1) " Originally generated documents should be kept in /disk3/cookie_logs/pid1/2009-01-14 However, the actual file is stored in the /disk3/cookie_logs/pid1/2009-01-15 It can be seen, in fact, syslog-ng configuration item file() use the $DAY and template() use the $DAY not with a identical Macros. Is this the syslog-ng a bug? or I configuration errors. Who can help me answer you?Thanks!

2 8

Snare and MSGONLY macro
by Stewart James 16 Feb '09

16 Feb '09

Hi all, I have had an ongoing problem with snarecore and syslog-ng, this is also happening with loglogics lasso agent too. For some services it makes sense that the system just store the messages without the syslog header. For example IIS logs, apache logs and other misc application logs. Wanting to save having to perform any post processing I figured I would setup a specific listener for those applications, allowing syslog to log to /someplace/with/lots/of/storage/YYYY/MM/DD/raw.program.servername. Using both the windows agents mentioned above, the macros $PROGRAM and $MSGONLY break. $PROGRAM is expanded into some ungodly form (MSWinEventLog\0110\011System\011325\011Mon....) and the MSGONLY macro is garbled. My logic tells me this is due to sysl;og-ng not detecting the syslog header being sent by snare or lasso properly... or it is snare and lasso sending the incorrect header format. I can survive without the $PROGRAM macro working correctly. The MSGONLY macro is a bit more of a pain though, I would rather use the native capabilities of syslog-ng. I have just updated to 2.0.9 (from Debian Lenny) to confirm this is still a problem for me, I see 3.0.x is out but not sure if this is something that would be corrected if I went down that path. Thanks, Stewart

1 0

Parsing messages! help!
by ramesh p 15 Feb '09

15 Feb '09

How to replace the syslog source IP address with the correct loopback IP address before forwarding syslog messages to someother application? PLz give suggestion. How to parse and filter? Thanks in advance! Regards, Rams.

2 2

Compiling syslog-ng 3.01 on CentOS 5.2
by Liam Kirsher 15 Feb '09

15 Feb '09

Hi -- Just thought this might be useful to someone who is compiling on CentOS. There were a few minor issues that came up, but otherwise it went pretty smoothly. Documentation says to use this -- PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:$PKG_CONFIG_PATH However, using that I ended up with a colon at the end, which didn't work, and which I didn't catch immediately... PKG_CONFIG_PATH=/usr/local/lib/pkgconfig: Also, I needed to export -- export PKG_CONFIG_PATH Maybe it could be export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib/pkgconfig I did have to install newer versions of libdbi and libdbi-drivers, from http://libdbi.sourceforge.net/ For libdbi-drivers, remember to specify which ones you want during configuration: ./configure --with-pgqsl After that, this worked: LIBDBI_CFLAGS=-I/usr/local/include LIBDBI_LIBS="-L/usr/local/lib -ldbi" ./configure --sysconfdir=/etc/syslog-ng/ --enable-tcp-wrapper --enable-sql --enable-dynamic-linking (Make sure to link with libs/dirs for my newly installed libdbi which were in /usr/local/include and /usr/local/lib, not the system default ones if they were already installed.) Note that this installs the executables in the default location, which is /usr/local, but looks for the configuration file in the standard place, /etc/syslog-ng/. I adjusted the /etc/init.d/syslog-ng file so it would use the executable in /usr/local/sbin: exec="/usr/local/sbin/syslog-ng" Also, I needed to run ldconfig so that syslog-ng would find libevtlog.so.0 in /usr/local/lib. At that point, I was able to start using the previous (v. 1.6) configuration file, albeit with warnings. This is a Linux system, and initially I added --enable-linux-caps to the configure line, although I wasn't sure what that would do. I got an error and so, removed it. What exactly are linux-caps? Should I go back and try to re-compile with that option? Liam -- Liam Kirsher PGP: http://liam.numenet.com/pgp/

2 1

Missing libraries when trying to compile 3.0
by Tim Boyer 15 Feb '09

15 Feb '09

I'm trying to get 3.0 going - I'm running 2.2.6 currently. configure says I'm missing some libraries: ./configure ... checking for static OpenSSL libraries... no configure: error: static OpenSSL libraries not found (libssl.a and libcrypto.a), either link OpenSSL statically using the --enable-dynamic-linking, or install a static OpenSSL [root@princeton syslog-ng-3.0.1]# ...but the libraries are there: [root@princeton syslog-ng-3.0.1]# locate libssl.a libcrypto.a /usr/lib/libcrypto.a /usr/lib/libssl.a /usr/lib64/libcrypto.a /usr/lib64/libssl.a However, I used the --enable-dynamic-linking for the configure. It configured and compiled just fine. When I went to run, I got this: [root@princeton syslog-ng-3.0.1]# service syslog-ng start Starting system logger: /usr/local/sbin/syslog-ng: error while loading shared libraries: libevtlog.so.0: cannot open shared object file: No such file or directory [FAILED] even though _those_ libraries are there: [root@princeton syslog-ng-3.0.1]# ls -la /usr/local/lib/libevtlog.* -rw-r--r-- 1 root root 65048 Feb 11 11:50 /usr/local/lib/libevtlog.a -rwxr-xr-x 1 root root 822 Feb 11 11:50 /usr/local/lib/libevtlog.la lrwxrwxrwx 1 root root 18 Feb 11 11:50 /usr/local/lib/libevtlog.so -> libevtlog.so.0.0.0 lrwxrwxrwx 1 root root 18 Feb 11 11:50 /usr/local/lib/libevtlog.so.0 -> libevtlog.so.0.0.0 -rwxr-xr-x 1 root root 45369 Feb 11 11:50 /usr/local/lib/libevtlog.so.0.0.0 I'm not sure where to go from here - pointers in the right direction greatly appreciated. -- Tim Boyer Denman Tire Corporation tim(a)denmantire.com

3 3