November 2009 - syslog-ng - lists.balabit.hu

Using pipes for prefiltering?
by Kārlis Repsons 10 Nov '09

10 Nov '09

Hello, Recently I've been trying to squeeze out the most of syslog-ng 3.0 and one of problems pestering me is this: there are some messages, which I would like never to appear in any log file, so I created filters for them, directed real sources through them to pipe destination and created prefiltered sources pointing to those same pipes. Well and what happens is - some logs reach the final destinations, but some not. For example, nothing at all now gets logged form kernel through pipe, even if filters get thrown out! Its on Linux... Also I had to disable syslogng logs, because otherwise CPU was loaded 100% sometimes (one core). After all, how should I do prefiltering? Strange thing is, that even with pipes first of these works, while the other doesn't: log { source(s_compname_usp_f); source(s_compname_kern_f); destination(d_compname_vt12); }; log { source(s_compname_kern_f); filter(f__kern); destination(d_compname_kern); }; filter f__kern { facility(0); }; You can see pipes fragment below. Hope someone has suggestion... Best, Kārlis # prefiltering # destination d_compname_usp_f { pipe("/var/log/filtered-source"); }; destination d_compname_kern_f { pipe("/var/log/filtered-kernel-source"); }; # abusers/spammers dropping here # filter f_compname_antispam_usp { ( not ( program(upsd) and message("some shit there") ) ) and ( not program("syslog") ) and ( not ( program("cron") and ( message(".*rm \-f /var/spool/cron/lastrun/cron\..*") or message(".*test \-x /usr/sbin/run-crons \&\& /usr/sbin/run\-crons.*") ) ) ) and ( not ( program("acpid") and ( message(".*client connected from.*") or message(".*1 client rule loaded") or message(".*client .* has disconnected") ) ) ) ; }; filter f_compname_antispam_kern { ( not message(".*ata1: exception .*") ) and ( not message(".*ata1: .*connection status changed") ) and ( not message(".*Hangcheck\: hangcheck value past margin\!") ) ; }; log { source(s_compname_usp); filter(f_compname_antispam_usp); destination(d_compname_usp_f); }; log { source(s_compname_kern); destination(d_compname_kern_f); }; source s_compname_usp_f { pipe("/var/log/filtered-source"); }; source s_compname_kern_f { pipe("/var/log/filtered-kernel-source"); };

1 1

Re: [syslog-ng] Upper case $HOST
by Pallagi Zoltán 09 Nov '09

09 Nov '09

Tim Boyer írta: > Pallagi Zoltán wrote: >> Hi, >> >> Tim Boyer írta: > >>> Balazs - >>> >>> Looks like I spoke too soon. Something odd is happening. >>> >>> I put the option into the conf file: >>> >>> @version: 3.0 >>> # >>> # global options >>> # >>> >>> options { >>> normalize_hostnames(yes); >>> use_fqdn(no); >>> use_dns(yes); >>> dns_cache(yes); >>> keep_hostname(yes); >>> long_hostnames(off); >>> create_dirs(yes); >>> } >>> >>> > > <snip> > >> You should not use "keep_hostname" in the part of global options >> because this one will block rewriting of the hostname (see syslog-ng >> admin guide: >> http://www.balabit.hu/dl/html/syslog-ng-v3.0-guide-admin-en.html/ch08s09.ht…) >> > > Pallagi - > > Of course - that makes perfect sense; I'm asking it to do two > contradictory actions. Thanks very much... Anyway, I think this is also a problem so I'v just opened a bugzilla ticket: https://bugzilla.balabit.com/show_bug.cgi?id=60 > >

1 0

Upper case $HOST
by Tim Boyer 09 Nov '09

09 Nov '09

I'm running syslog-ng 3.03 on a RHEL5 system, sending logs to various files like so: # global log files destination deservers { file("/var/log/$HOST.log" owner(root) group(hobbit) perm(0640)); }; It's working fine on the Linux boxes. But I'm using Adiscon's EventLog on my Windows machines, and the $HOST name on them are coming up in all caps. [root@buran log]# ls -la /var/log/*.log ... -rw-r----- 1 root hobbit 282 Nov 4 14:37 /var/log/KANTECH.log -rw-r----- 1 root hobbit 535181 Nov 4 14:24 /var/log/PLCDATA.log Where's this $HOST macro get its data from? The DNS entry is lower case; the full computer name on the Windows box is lower case. I don't see where the upper is coming from, unless it's one of those weird Windows-to-Unix translation things. Not a big deal, but a bit of an annoyance. Thanks for any help. -- Tim Boyer Denman Tire Corporation

5 6

Macro question
by Michael J. Bauer 06 Nov '09

06 Nov '09

I am running syslog-ng 3.0.4 on RHEL 5.4. I have a log message that appears in my logfiles as Oct 28 16:41:22 juniper-router {wan-service-set}[FWNAT]: ASP_NAT_RULE_MATCH: proto 6 (TCP) application: any, ge-0/0/3.2:10.3.13.153:49818 -> 66.249.80.148:80, Match NAT rule-set: , rule: nat-outgoing, term: dynamic-nat There are no carriage returns in that entry; any that appear are artifacts of some mail server or client along the way. I want any log entry containing the string FWNAT to go into a separate file. match(FWNAT) on the filter does just that, but syslog-ng complains that I'm not using value() for efficiency. However, it's not at all clear which macro, if any, actually contains the string FWNAT. I can tell you five that don't: APPLICATION EVENT_SOURCE MESSAGE MSG PROGRAM What macro contains FWNAT? And where can I get a definitive list of macros and specifics on how the value in each is set for each line? The documentation is insufficiently detailed on this subject. Thanks, MJB

3 4

$HOST macro and relay host
by Jason Barrett 06 Nov '09

06 Nov '09

Hi all, I'm relaying log messages from one syslog-ng server to another. At the final stop, the only way I can get the $HOST macro to work is if I enable dns resolution on server 1. Is this by design? Here are the relevant configs: ---------------------------------------------------------------------- syslog-ng server 1 (relays to server 2): chain_hostnames(yes); keep_hostname(yes); use_dns(no); source s_udp { udp(port(514)); }; destination df_udpback { udp("192.168.1.157" port(514)); }; log { source(s_udp); destination(df_udpback); }; ---------------------------------------------------------------------- syslog-ng server 2: chain_hostnames(yes); keep_hostname(yes); use_dns(yes); source s_udp { udp(port(514)); }; destination df_udp { file("/var/log/ics/$HOST/$YEAR/$MONTH/$DAY"); }; log { source(s_udp); destination(df_udp); }; ---------------------------------------------------------------------- Sample log message on server 2: Oct 30 09:35:03 10.12.24.46/10.12.24.46 %ASA-5-111005: 10.28.22.55 end configuration: OK 10.12.24.46 is the correct IP address of the originating host, and $HOST resolves to this IP address. I would prefer $HOST to resolve to the hostname as it exists in the /etc/hosts file. Thanks!

3 6

Re: [syslog-ng] Pharmacy Online Sale 84% OFF!
by VIAGRA Inc. 05 Nov '09

05 Nov '09

New from WebMD: Dear syslog-ng(a)lists.balabit.hu!. Sign-up today! You are subscribed as syslog-ng(a)lists.balabit.hu. View and manage your WebMD newsletter preferences. Subscribe to more newsletters. Change/update your email address. WebMD Privacy Policy WebMD Office of Privacy 1175 Peachtree Street, Suite 2400, Atlanta, GA 30361 © 2009 WebMD, LLC. All rights reserved.

1 0

Match/Message/Macros
by R King 03 Nov '09

03 Nov '09

I have updated Syslog-NG to 3 and am trying to figure out: "WARNING: the match() filter without the use of the value() option is deprecated and hinders performance, please update your configuration;" I have one filter that isn't working and I have tried several different ways to fix it. The logs are Windows DHCP logs passed from EPILOG: Nov 3 11:37:55 snsudc02 DHCPLOG[0]: 11,11/03/09,11:37:54,Renew,172.31.0.213,Ashley-PC.nsu.edu,001B9E2A18E9, Nov 3 11:37:55 snsudc02 DHCPLOG[0]: 11,11/03/09,11:37:55,Renew,172.16.0.191,donovan-dcda8cf.,000B7D0993DF, My filter originally was: filter f_dhcp { match("DHCPLOG"); }; I've tried all these without success: filter f_dhcp { message("DHCPLOG"); }; filter f_dhcp { program("DHCPLOG"); }; filter f_dhcp { match("DHCPLOG" flags("ignore-case") value("$PROGRAM")); }; filter f_dhcp { match("0" value("$PID")); }; Any help would be greatly appreciated.

2 2

Re: [syslog-ng] syslog-ng3 with SQL on FreeBSD
by Pal Tamas 03 Nov '09

03 Nov '09

On Wed, Oct 28, 2009 at 12:00:01PM +0100, syslog-ng-request(a)lists.balabit.hu wrote: > Date: Tue, 27 Oct 2009 16:27:31 -0400 > From: "Michael W. Lucas" <mwlucas(a)blackhelicopters.org> > Subject: [syslog-ng] syslog-ng3 with SQL on FreeBSD > To: syslog-ng(a)lists.balabit.hu > Message-ID: > <20091027202731.GA82885(a)bewilderbeast.blackhelicopters.org> > Content-Type: text/plain; charset=us-ascii > > Hi, > > I'm working on getting syslog-ng 3.0.3 to support SQL on FreeBSD, and > hopefully get it added to the FreeBSD package or port. It looks like > something isn't quite right in the configure script, however. > > I have libdbi 0.8.3 installed, but "./configure --enable-sql=yes" > still reports: > > checking for LIBDBI... gnome-config: not found > no Did you installed pkg-config from ports? If not, please install it and re-run configure with the PKG_CONFIG_PATH set to /usr/local/lib/pkgconfig (this is where libdbi installed the dbi.pc config file) > checking for dbi_initialize in -ldbi... no This step will be skipped if library detection using pkg-config detection works. If not, please set include and library detection paths in CFLAGS and LDFLAGS variables before running configure. Ex: CFLAGS="-I/usr/local/include" LDFLAGS="-L/usr/local/lib" ./configure ... Please note this only works in posix shells (bash, /bin/sh), in csh you have to set the variables before running configure. > > I've reviewed /usr/local/include/dbi/dbi.h, and dbi_initialize is in > there. > > Can anyone suggest a fix? If both of these fail, please send in part of config.log from the checking LIBDBI messages to the following text: ## ---------------------- ## ## Running config.status. ## ## ---------------------- ## Regards, -- Pal Tamas/Folti folti(a)balabit.hu

3 5

problem with matching IP address and \d regex operand
by Phil.Newlon＠wendysarbys.com 03 Nov '09

03 Nov '09

I am using this regular expression with Kiwi Syslog to distribute messages to several destinations based on the last number of the third octet (0-4 goes one place, 5-9 goes another). "10\.\d+\.\d*[0-4]\." This doesn't work with syslog-ng, of course, but based on my research of the archives, this should do the same thing because I've escaped the "\d" match("10\.\\d+\.\\d*[0-4]\.") Nope, I get nothing. I've shortened it to just match("10\.\\d+") and still get no matching messages. This sort of works, but gives some unexpected results: match("10\.[0-9]+\.[0-9]*[0-4]\.") The match("10\.[0-9]+\.[0-9]*[0-4]\.") statement resulted in 'true' on this log message. I didn't expect a match on 10.87.48.4 from it because of the '8' as the last number of the third octet not matching '0-4' Oct 29 16:31:20 10.87.48.4 Kiwi_Syslog_Daemon Oct 29 16:31:20 10.87.48.4 MSWinEventLog 0 Security 71000 Thu Oct 29 16:31:17 2009 538 Security pos User Success Audit POS0408748 Logon/Logoff User Logoff: User Name: pos Domain: POS0408748 Logon ID: (0x0,0x4ACB69) Logon Type: 3 42921033 So, I have two questions..... What's wrong with this: match("10\.\\d+\.\\d*[0-4]\.") And why did this match("10\.[0-9]+\.[0-9]*[0-4]\.") match this Oct 29 16:31:20 10.87.48.4 Kiwi_Syslog_Daemon Oct 29 16:31:20 10.87.48.4 MSWinEventLog 0 Security 71000 Thu Oct 29 16:31:17 2009 538 Security pos User Success Audit POS0408748 Logon/Logoff User Logoff: User Name: pos Domain: POS0408748 Logon ID: (0x0,0x4ACB69) Logon Type: 3 42921033 Thanks! Phil Notice: This e-mail message and its attachments are the property of Wendy's/Arby's Group Inc. or one of its subsidiaries and may contain confidential or legally privileged information intended solely for the use of the addressee(s). If you are not an intended recipient, then any use, copying or distribution of this message or its attachments is strictly prohibited. If you received this message in error, please notify the sender and delete this message entirely from your system.

4 6

Seg fault with include file contains an error
by anthony lineham 03 Nov '09

03 Nov '09

Hi Bazsi, I've been investigating a seg fault that occurs when using an include statement in the config file. The error occurs when one of the included files contains an syntax error. The first time syslog-ng is restarted with the bad config, the error is detected and restart is aborted. However, if a second HUP is sent a seg fault occurs. I had a bit of trouble tracking down the cause and I'm still not entirely sure of the mechanism that leads to the crash but I found a couple of apparent problems, which when corrected prevent it. 1. There is a global variable "include_depth" which normally gets decremented back to zero after successful parsing of included config files. However, if an error is detected it stays at its current value and subsequent restarts increment from that point. This doesn't cause the crash, but given enough restarts would lead to overrun or exhaustion of the "include_stack" array. 2. In the case of unsuccessful config parsing certain bits of memory are dealloced but their pointers that not reset. This seems to be what was causing the seg fault but I've found it a bit hard to pin down. 3. There was an off-by-1 error in the included config deinit loop. The following patch fixes the problem, but may not necessarily be the best way to do it - particularly issue 1. Regards, Anthony --- syslog-ng_3.0.1-63-g41f77f5-old/src/cfg-lex.l 2009-09-14 16:02:09.000000000 +1200 +++ syslog-ng_3.0.1-63-g41f77f5/src/cfg-lex.l 2009-09-14 16:15:30.000000000 +1200 @@ -626,17 +626,24 @@ { gint i; - for (i = 0; i < include_depth; i++) + for (i = 0; i <= include_depth; i++) { CfgIncludeLevel *level = &include_stack[i]; if (level->current_file) - g_free(level->current_file); + { + g_free(level->current_file); + level->current_file = NULL; + } g_slist_foreach(level->files, (GFunc) g_free, NULL); g_slist_free(level->files); level->files = NULL; if (level->yybuf) - yy_delete_buffer(level->yybuf); + { + yy_delete_buffer(level->yybuf); + level->yybuf = NULL; + } } + include_depth = 0; }

2 1