February 2009 - syslog-ng - lists.balabit.hu

Snare and MSGONLY macro
by Stewart James 16 Feb '09

16 Feb '09

Hi all, I have had an ongoing problem with snarecore and syslog-ng, this is also happening with loglogics lasso agent too. For some services it makes sense that the system just store the messages without the syslog header. For example IIS logs, apache logs and other misc application logs. Wanting to save having to perform any post processing I figured I would setup a specific listener for those applications, allowing syslog to log to /someplace/with/lots/of/storage/YYYY/MM/DD/raw.program.servername. Using both the windows agents mentioned above, the macros $PROGRAM and $MSGONLY break. $PROGRAM is expanded into some ungodly form (MSWinEventLog\0110\011System\011325\011Mon....) and the MSGONLY macro is garbled. My logic tells me this is due to sysl;og-ng not detecting the syslog header being sent by snare or lasso properly... or it is snare and lasso sending the incorrect header format. I can survive without the $PROGRAM macro working correctly. The MSGONLY macro is a bit more of a pain though, I would rather use the native capabilities of syslog-ng. I have just updated to 2.0.9 (from Debian Lenny) to confirm this is still a problem for me, I see 3.0.x is out but not sure if this is something that would be corrected if I went down that path. Thanks, Stewart

1 0

Parsing messages! help!
by ramesh p 15 Feb '09

15 Feb '09

How to replace the syslog source IP address with the correct loopback IP address before forwarding syslog messages to someother application? PLz give suggestion. How to parse and filter? Thanks in advance! Regards, Rams.

2 2

Compiling syslog-ng 3.01 on CentOS 5.2
by Liam Kirsher 15 Feb '09

15 Feb '09

Hi -- Just thought this might be useful to someone who is compiling on CentOS. There were a few minor issues that came up, but otherwise it went pretty smoothly. Documentation says to use this -- PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:$PKG_CONFIG_PATH However, using that I ended up with a colon at the end, which didn't work, and which I didn't catch immediately... PKG_CONFIG_PATH=/usr/local/lib/pkgconfig: Also, I needed to export -- export PKG_CONFIG_PATH Maybe it could be export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib/pkgconfig I did have to install newer versions of libdbi and libdbi-drivers, from http://libdbi.sourceforge.net/ For libdbi-drivers, remember to specify which ones you want during configuration: ./configure --with-pgqsl After that, this worked: LIBDBI_CFLAGS=-I/usr/local/include LIBDBI_LIBS="-L/usr/local/lib -ldbi" ./configure --sysconfdir=/etc/syslog-ng/ --enable-tcp-wrapper --enable-sql --enable-dynamic-linking (Make sure to link with libs/dirs for my newly installed libdbi which were in /usr/local/include and /usr/local/lib, not the system default ones if they were already installed.) Note that this installs the executables in the default location, which is /usr/local, but looks for the configuration file in the standard place, /etc/syslog-ng/. I adjusted the /etc/init.d/syslog-ng file so it would use the executable in /usr/local/sbin: exec="/usr/local/sbin/syslog-ng" Also, I needed to run ldconfig so that syslog-ng would find libevtlog.so.0 in /usr/local/lib. At that point, I was able to start using the previous (v. 1.6) configuration file, albeit with warnings. This is a Linux system, and initially I added --enable-linux-caps to the configure line, although I wasn't sure what that would do. I got an error and so, removed it. What exactly are linux-caps? Should I go back and try to re-compile with that option? Liam -- Liam Kirsher PGP: http://liam.numenet.com/pgp/

2 1

Missing libraries when trying to compile 3.0
by Tim Boyer 15 Feb '09

15 Feb '09

I'm trying to get 3.0 going - I'm running 2.2.6 currently. configure says I'm missing some libraries: ./configure ... checking for static OpenSSL libraries... no configure: error: static OpenSSL libraries not found (libssl.a and libcrypto.a), either link OpenSSL statically using the --enable-dynamic-linking, or install a static OpenSSL [root@princeton syslog-ng-3.0.1]# ...but the libraries are there: [root@princeton syslog-ng-3.0.1]# locate libssl.a libcrypto.a /usr/lib/libcrypto.a /usr/lib/libssl.a /usr/lib64/libcrypto.a /usr/lib64/libssl.a However, I used the --enable-dynamic-linking for the configure. It configured and compiled just fine. When I went to run, I got this: [root@princeton syslog-ng-3.0.1]# service syslog-ng start Starting system logger: /usr/local/sbin/syslog-ng: error while loading shared libraries: libevtlog.so.0: cannot open shared object file: No such file or directory [FAILED] even though _those_ libraries are there: [root@princeton syslog-ng-3.0.1]# ls -la /usr/local/lib/libevtlog.* -rw-r--r-- 1 root root 65048 Feb 11 11:50 /usr/local/lib/libevtlog.a -rwxr-xr-x 1 root root 822 Feb 11 11:50 /usr/local/lib/libevtlog.la lrwxrwxrwx 1 root root 18 Feb 11 11:50 /usr/local/lib/libevtlog.so -> libevtlog.so.0.0.0 lrwxrwxrwx 1 root root 18 Feb 11 11:50 /usr/local/lib/libevtlog.so.0 -> libevtlog.so.0.0.0 -rwxr-xr-x 1 root root 45369 Feb 11 11:50 /usr/local/lib/libevtlog.so.0.0.0 I'm not sure where to go from here - pointers in the right direction greatly appreciated. -- Tim Boyer Denman Tire Corporation tim(a)denmantire.com

3 3

Using tail option
by James B Horwath 15 Feb '09

15 Feb '09

All: This question refers to syslog-ng 2.0.10 running on AIX. If I use a global syslog-ng client file and define 2 text files that do not exist on every system. What happens when syslog-ng is run on a system that does not contain the files? Does it report errors, have performance problems? Example: source s_tail1 { file("/var/apache/log/access_log" follow_freq(1) flags(no-parse)); }; source s_tail2 { file("/var/apache/log/error_log" follow_freq(1) flags(no-parse)); }; Please advise, Jim ----------------------------------------- This message, and any attachments to it, may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are notified that any use, dissemination, distribution, copying, or communication of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately by return e-mail and delete the message and any attachments. Thank you.

2 1

Ip filtering
by Chris Bland 15 Feb '09

15 Feb '09

Hi guys, I am having a problem filtering based on ip address. My predecessor had a config file with over 240 hosts. I just upgraded to 2.0.10-1 and starting getting YACC overflow stack errors. I condensed the config file by using regex. If I stop using DNS everything works fine. What I am trying to do is filter based on ip address and have each log stored in a directory with the servers dns name. My config looks like this options { sync (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); keep_hostname (yes); use_dns (yes); use_fqdn (no); create_dirs (no); }; destination servers { file("/var/log/syslog-ng/hosts/$HOST_FROM/$YEAR/$MONTH/$YEAR-$MONTH-$DAY" owner(root) group(staff) perm(0650) dir_perm(0750) create_dirs(yes)); }; destination teaneck { file("/var/log/syslog-ng/network/TEANECK/$HOST_FROM/$YEAR/$MONTH/$YEAR-$MONTH-$DAY" owner(root) group(staff) perm(0650) dir_perm(0750) create_dirs(yes)); }; destination dslswitches { file("/var/log/syslog-ng/network/TEANECK/Dslswitches/$HOST_FROM/$YEAR/$MONTH/$YEAR-$MONTH-$DAY" owner(root) group(staff) perm(0650) dir_perm(0750) create_dirs(yes)); }; filter f_teaneck { netmask("132.238.11.0/24"); }; filter f_usas { host("132.238.(7|8).*$"); }; filter f_dslswitches { host("132.238.21.40"); }; log { source(s_network); filter(f_usas); destination(servers); }; log { source(s_network); filter(f_teaneck); destination(teaneck); }; log { source(s_network); filter(f_dslswitches); destination(dslswitches); }; The netmask filters work by my host filters don't -Chris

2 1

typo in v.3.0 admin guide
by Philipp 14 Feb '09

14 Feb '09

Hi please note that in the online available syslog-ng-v3.0-guide-admin-en.pdf on page 54 this line seems to have typos in the template() part: destination d_file { file("/var/log/messages" template("$ISODATE $HOST $MSG\n"; template_escape(no))); }; Best regards, Philipp

2 2

question on logging templates: strange log lines
by Philipp 14 Feb '09

14 Feb '09

Dear list users In the near future I would like to migrate from syslog-ng 2 to 3.x. For this reason I've set up a linux testbox. Now, I have a problem I lack to see a solution for. The syslog-ng testing config: @version:3.0 ## GLOBAL options { chain_hostnames(no); keep_hostname(yes); flush_lines(0); stats_freq(3600); use_dns(persist_only); dns_cache_hosts(/etc/hosts); }; ## SOURCES source src { unix-stream("/dev/log"); }; source src-kernel { file("/proc/kmsg" program_override("kernel: ")); }; source src-syslog-ng { internal(); }; ## LOGGING TEMPLATES template t_filetmpl { template("$DATE $FACILITY.$PRIORITY $FULLHOST $MSGHDR$MSG\n"); template_escape(no); }; ## DESTINATIONS destination local4 { file("/var/log/local4.log" template(t_filetmpl)); }; ## FILTERS filter f_local4 { facility(local4); }; ## LOGS log { source(src); filter(f_local4); destination(local4); }; For testing I use the logger command: ` logger -p local4.info -t test testnachricht2`. This configuration causes, and I do not see why, lines like these in the /var/log/local4.log file. [empty line] testnachricht2 test: donald.info local4Feb 11 14:26:01Feb 11 14:26:01 local4.info donald test: testnachricht2 However, if I replace the line below "## DESTINATIONS" with destination local4 { file("/var/log/local4.log" template("$DATE $FACILITY.$PRIORITY $FULLHOST $MSGHDR$MSG\n")); }; all is perfect. The logged line looks like this: Feb 11 14:30:22 local4.info donald test: testnachricht2 Please could you tell me what I've missed? Thanks in advance, Philipp

2 3

Strange template definition problem
by Peter Eckel 14 Feb '09

14 Feb '09

Hi, I am currently using the Open Source edition of syslog-ng version 3.0.1 on RedHat Linux 5.3 and 3.9. So far, there are not too many problems, but one strangeness occurs when defining templates. When I define the template inside the 'destination' configuration, everything works as expected: # + # Definition 1: Works fine. # - source s_local { unix-stream( "/dev/log" ); file( "/proc/kmsg" ); udp( ip( "127.0.0.1" ) port(514) ); tcp( ip( "127.0.0.1" ) port(514) keep-alive(yes) ); }; destination d_test { file( "/opt/sec/var/log/test" template( "$ISODATE -- $FACILITY -- $PRIORITY -- $HOST -- $MESSAGE\n----------\n" ) ); }; log { source(s_local); destination(d_test); }; The log entries look like this: >2009-01-22T19:33:35+01:00 -- authpriv -- notice -- linux4 -- LOG5[17309:3086130064]: syslog_in connected from 192.168.42.129:41105 >---------- >2009-01-22T19:33:35+01:00 -- authpriv -- notice -- linux4 -- LOG5[17309:3086060432]: syslog_in connected from 192.168.42.133:33950 >---------- >2009-01-22T19:33:35+01:00 -- authpriv -- notice -- linux4 -- LOG5[17309:3085990800]: syslog_in connected from 192.168.42.128:43163 >---------- So far, so good. But when I try global templates, I get a rather strange result: # + # Definition 2: Fails miserably. # - source s_local { unix-stream( "/dev/log" ); file( "/proc/kmsg" ); udp( ip( "127.0.0.1" ) port(514) ); tcp( ip( "127.0.0.1" ) port(514) keep-alive(yes) ); }; template t_test { template( "$ISODATE -- $FACILITY -- $PRIORITY -- $HOST -- $MESSAGE\n----------\n" ); }; destination d_test { file( "/opt/sec/var/log/test" template( t_test ) ); }; log { source(s_local); destination(d_test); }; What I'm getting is this (including marked blank lines): > >---------- >-- LOG5[17309:3086130064]: syslog_in connected from 192.168.42.129:41105 -- linux4 -- notice -- authpriv2009-01-22T19:33:35+01:0020 >09-01-22T19:33:35+01:00 -- authpriv -- notice -- linux4 -- LOG5[17309:3086130064]: syslog_in connected from 192.168.42.129:41105 >---------- > >---------- >-- LOG5[17309:3086060432]: syslog_in connected from 192.168.42.133:33950 -- linux4 -- notice -- authpriv2009-01-22T19:33:35+01:0020 >09-01-22T19:33:35+01:00 -- authpriv -- notice -- linux4 -- LOG5[17309:3086060432]: syslog_in connected from 192.168.42.133:33950 >---------- > >---------- >-- LOG5[17309:3085990800]: syslog_in connected from 192.168.42.128:43163 -- linux4 -- notice -- authpriv2009-01-22T19:33:35+01:0020 >09-01-22T19:33:35+01:00 -- authpriv -- notice -- linux4 -- LOG5[17309:3085990800]: syslog_in connected from 192.168.42.128:43163 It looks a bit as if all elements of the filled template are output twice: First in reverse order, then again from start in correct order. This is reproducable on several machines. The only difference is the use of the global template in the second case instead of the inline template in the first. Am I making some stupid mistake, or is this a bug in syslog-ng? Best regards and thanks in advance for any suggestions, Peter. -- ------------------------------------------------------------------------- Peter Eckel System- und Netzwerkmanagement Postfach 1463 D-61284 Bad Homburg v.d.H. Tel.: +49 6172 969495 Mobil: +49 172 6812107 Fax: +49 6172 969845 EMail: Peter.Eckel(a)Eckel-EDV.de -------------------------------------------------------------------------

2 4

patches ported from syslog-ng PE 3.0.2
by Balazs Scheidler 14 Feb '09

14 Feb '09

Hi, Just as I outlined in my previous mail, here goes the list of patches that were ported to syslog-ng OSE from the PE 3.0.2 release. You can find more information in the git tree at http://git.balabit.hu/, but feel free to ask specific questions if you need further information. Balazs Scheidler (30): don't use long double in the parser [tgzbuild] fixed --with-pidfile-dir configure option Fixed: slng_start checks whether syslog-ng already running before starting a new session. (fixes: #16011) Fixed: logic error when checking whether an instance already running. (fixes: #16011) Fixed: On SuSE -n no longer echoed (fixes: #16025) Fixed: obscure missing file error on HP-UX (fixes: #16710) Fixed: Init scripts link /var/run/syslog.pid to syslog-ng's pid. (fixes: #16710) Fixed: Solaris 10 method script links /var/run/syslog.pid to syslog-ng's pid. (fixes: #16710) [config parser] fixed a failed assert in case an include directory does not exist [security] user names were not resolved properly, do it right this time fixed compilation warnings fixed LogMessageMatch structure layout on big-endian 32 bit machines (like PA-RISC) Fixed: init.d script uses different syslogd pid on redhat (fixes: #16710) Fixed: init script don't start syslog-ng when it doesn't run (fixes: #16011) really only move to the newly rotated logfile if the new one already has data [logproto] remember "encoding" value, so that log messages can contain it [logproto] properly handle incomplete encodings (fixes: #15680) [logproto] don't let the conversion buffer increase to unlimited sizes Fixed: init script on solaris 8 links /etc/syslog.pid to pidfile. (fixes: #16710) [security] enable SSL X.509 policy check [tlscontext] reorganize cert validation somewhat, fixed "required-untrusted" (fixes: #16671) [tlscontext] added basicConstraints validation even for trusted CA certificates [afsocket] fixed a possible reference of invalid file descriptors [main] never initialize the configuration in the supervisor process kill the syslog-ng process if reverting to the old config fails fixed compilation error on FreeBSD [templates] use the correct timezone offset when time_zone_info_get_offset() returns -1 [config parser] accept newly introduced keywords as identifiers (fixes: #17653) [config parser] removed unused KW_FRAMED keyword [configure] change default linking mode to "dynamic" Tamas Pal (1): Fixed: on linux, dash no longer prinst -e (fixes: #17559) -- Bazsi

1 0