September 2010 - syslog-ng - lists.balabit.hu

Pattern Matching Issues
by Peter Mills 16 Sep '10

16 Sep '10

I am new to syslog-ng and just trying to get a basic version 1 XML pattern match file working properly. The output with an associated template is redirected to a file. It currently claims there is no such value. Does this mean that the pattern match has failed? Using syslog-ng (3.0.8) parser p_patterndb { db_parser(file("/opt/syslog-ng/var/db/patterns/v1/test.xml")); }; # Check pattern matching is working log { source(s_ext); parser(p_patterndb); <<<<< destination(df_udp_pattern_output); }; #Check pattern matching destination df_udp_pattern_output { file("/var/log/pattern_output" template("$PROGRAM,${rest},,$MSG\n") template_escape(no)); }; Output of: /opt/syslog-ng/sbin/syslog-ng -e -F -d -v > /tmp/syslog-ng.out 2>&1 Incoming log entry; line='<131>1 - - WXC-192.168.61.202 register 1102 [wx-event(a)juniper.net eventtime="1284626839" metric="Primary Reg Server Unreachable" sev="major" type="sys"] REG: Self registration failed. IP=192.168.61.242.' Filter rule evaluation begins; filter_rule='f_messages' Filter node evaluation result; filter_result='not-match', filter_type='level' Filter node evaluation result; filter_result='not-match', filter_type='AND' Filter rule evaluation result; filter_result='not-match', filter_rule='f_messages' Filter rule evaluation begins; filter_rule='f_udp_nagios_check' Filter node evaluation result; filter_result='not-match' Filter rule evaluation result; filter_result='not-match', filter_rule='f_udp_nagios_check' Filter rule evaluation begins; filter_rule='wxc_all' Filter node evaluation result; filter_result='not-match', filter_type='level' Filter rule evaluation result; filter_result='not-match', filter_rule='wxc_all' Filter rule evaluation begins; filter_rule='wxc_tunnel_info' Filter node evaluation result; filter_result='not-match', filter_type='level' Filter node evaluation result; filter_result='not-match', filter_type='AND' Filter rule evaluation result; filter_result='not-match', filter_rule='wxc_tunnel_info' Filter rule evaluation begins; filter_rule='wxc_tunnel_warn' Filter node evaluation result; filter_result='not-match', filter_type='level' Filter node evaluation result; filter_result='not-match', filter_type='AND' Filter rule evaluation result; filter_result='not-match', filter_rule='wxc_tunnel_warn' Filter rule evaluation begins; filter_rule='wxc_tunnel_crit' Filter node evaluation result; filter_result='not-match', filter_type='level' Filter node evaluation result; filter_result='not-match', filter_type='AND' Filter rule evaluation result; filter_result='not-match', filter_rule='wxc_tunnel_crit' Filter rule evaluation begins; filter_rule='wxc_license_crit' Filter node evaluation result; filter_result='not-match', filter_type='level' Filter node evaluation result; filter_result='not-match', filter_type='AND' Filter rule evaluation result; filter_result='not-match', filter_rule='wxc_license_crit' No such value known; value='rest' <<<< root@nagios-collector:/opt/syslog-ng/var/db/patterns/v1 <mailto:root@nagios-collector:/opt/syslog-ng/var/db/patterns/v1> # cat test1.xml <?xml version="1.0" encoding="utf-8"?> <patterndb version="1" pub_date="2009-04-17"> <program name="1"> <pattern>wxc</pattern> <rule id="1" class="system"> <pattern>@ANYSTRING:rest@</pattern> </rule> </program> </patterndb> root@nagios-collector:/opt/syslog-ng/var/db/patterns/v1 <mailto:root@nagios-collector:/opt/syslog-ng/var/db/patterns/v1> # Thanks Peter Imtech Telecom Global is a limited company registered in England and Wales. Registered number: GB04407184. Registered office: Viables 3, Jays Close, Basingstoke, Hampshire, RG22 4BS. ______________________________________________________________________ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email ______________________________________________________________________

3 5

syslog-ng under HP-UX processing unwanted "padded" data
by Scott Rochford 16 Sep '10

16 Sep '10

Hi, I'm trying to test syslog-ng under HP-UX 11iv2 (both PA-RISC and IA64) to use as an alternative to the supplied syslogd. I have tried using the precompiled 3.0.5 binaries supplied with HP's DSAU (Distributed Systems Administration Utilities) pack, as well as compiling 3.0.5 and also version 3.1.2 from source. In all 3 cases I have the same problem; it seems to be attempting to process the padded data remaining in the 2048-byte chunk that is read from /dev/log, as you can see from the test below. In other words it does not stop processing the 2048-byte chunk of data when it reaches the first null byte. This means that the first log entry after startup is processed correctly, but then all sorts of random things start to happen, including blank (other than date stamp and host name) messages to every connected terminal. I used gcc-3.4.3, glib-2.24.2, popt 1.16, and pkgconfig 0.25 from the HP-UX Porting and Archive Centre to compile it. Have I overlooked something obvious here? I can't see any other similar reports on this mailing list or in the wild. I'd appreciate any suggestions! Regards, Scott. P.S. It's a little confusing that the HP-UX notes in the INSTALL file and the contrib/syslog-ng.conf.HP-UX files disagree with each other about the source syntax; will these be reconciled in a future release? # nohup /usr/local/sbin/syslog-ng --debug --foreground --stderr > syslog-ng.out 2>&1 & # logger -p user.notice "a user.notice test" # grep Incoming syslog-ng.out | head Incoming log entry; line='<13>Sep 6 05:16:52 syslog: a user.notice test' Incoming log entry; line='L\x84yD' Incoming log entry; line='\x01' Incoming log entry; line='\x01' Incoming log entry; line='\x01' Incoming log entry; line='\x09' Incoming log entry; line='\x01\x90T' Incoming log entry; line='\xd1\x9d' Incoming log entry; line='h\x0f\x1c,h\x0f\x1c\x98@' Incoming log entry; line='[Hh\x0dG\xc4h\x0dG\xc4\xc0\x1d?K\xc0\x1d?\x93' # /usr/local/sbin/syslog-ng -V syslog-ng 3.1.2 Installer-Version: 3.1.2 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.1#master#8bf13c304b6ab5fc1a372b49d55c78370efe14ca Compile-Date: Sep 6 2010 01:21:23 Enable-Threads: off Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-Sun-Door: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-SSL: on Enable-SQL: off Enable-Linux-Caps: off Enable-Pcre: off # cat /usr/local/etc/syslog-ng.conf @version: 3.0 # syslog-ng configuration file. # # This should behave pretty much like the original syslog on HP-UX. But # it could be configured a lot smarter. # # See syslog-ng(8) and syslog-ng.conf(8) for more information. # # 20000925 gb(a)sysfive.com options { time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; # default contrib'd version # source s_sys { pipe("/dev/log"); internal(); }; # recommended by INSTALL source s_sys { pipe("/dev/log" pad_size(2048)); }; destination d_cons { file("/dev/console"); }; destination d_mesg { file("/var/adm/syslog/syslog.log"); }; destination d_mail { file("/var/adm/syslog/mail.log"); }; destination d_mlrt { usertty("root"); }; destination d_mlal { usertty("*"); }; filter f_filter1 { facility(mail) and level(debug); }; filter f_filter2 { (facility(mail) and level(debug)) or level(info); }; filter f_filter3 { level(alert); }; filter f_filter4 { level(emerg); }; log { source(s_sys); filter(f_filter1); destination(d_mail); }; log { source(s_sys); filter(f_filter2); destination(d_mesg); }; log { source(s_sys); filter(f_filter3); destination(d_cons); destination(d_mlrt); }; log { source(s_sys); filter(f_filter4); destination(d_mlal); }; #

2 5

rpm GPG keys for syslog-ng RPMs
by Tim Edwards 15 Sep '10

15 Sep '10

Hi, I've downloaded the syslog-ng RPMs from your site but can't find the GPG keys for RPM. I need these to put the RPMs in our local yum repository. Thanks Tim Edwards

2 1

Mass text log files processing over network.
by Tomasz Wrona 13 Sep '10

13 Sep '10

Hello, Could somebody point me if is it possible to configure syslog-ng to automatic, mass, text log files processing over network? Which version eventually support following case? My case: Webservers run virtual servers, which logs their own php error logs [text files]. I want to send this logs to central log server keeping orginal distribution based on virtual servers in the simplest way possible. I would like to have something like this virtual setup: #*** Client config ***# source s_php { # Wildcard match [only for commercial edition?] and TAG it somehow: file("/var/log/php/*-error.log" follow_freq(1) flags(no-parse) TAG("$FILENAME: ")); # eventually manual config for each file if above not possible: file("/var/log/php/service1-error.log" follow_freq(1) flags(no-parse) log_prefix("service1: ")); file("/var/log/php/service2-error.log" follow_freq(1) flags(no-parse) log_prefix("service2: ")); }; destination d_collector_php { udp("collector" port(5501)); }; log { source(s_php); destination(d_collector_php); }; #*** Collector Server config ***# source s_network_php { udp(ip(0.0.0.0) port(5501)); }; # Can I split log stream using MACRO based on custom information send in log [or syslog header]? destination d_php { file("/var/log/hosts/webservers/php/$HOST/$YEAR$MONTH$DAY/$TAG.log"); }; # or maybe: destination d_php { file("/var/log/hosts/webservers/php/$HOST/$YEAR$MONTH$DAY/$log_prefix.log"); }; log { source(s_network_php); destination(d_php); }; Regards Tom

4 3

3.2 alpha 2 build error
by Matthew Hall 13 Sep '10

13 Sep '10

When building using debian/rules binary on 3.2 alpha 2 my build fails here. Any advice on this one? Thanks, Matthew. /bin/bash ../../libtool --tag=CC --mode=compile gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../.. -I../../src -I.. -DENABLE_SQL=1 -pthread -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -I/usr/include/eventlog -D_BSD_SOURCE -D__BSD_SOURCE -D__FAVOR_BSD -DHAVE_NET_ETHERNET_H -I/usr/include -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -g -O2 -Wall -MT libafsql_la-afsql.lo -MD -MP -MF .deps/libafsql_la-afsql.Tpo -c -o libafsql_la-afsql.lo `test -f 'afsql.c' || echo './'`afsql.c libtool: compile: gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../.. -I../../src -I.. -DENABLE_SQL=1 -pthread -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -I/usr/include/eventlog -D_BSD_SOURCE -D__BSD_SOURCE -D__FAVOR_BSD -DHAVE_NET_ETHERNET_H -I/usr/include -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -g -O2 -Wall -MT libafsql_la-afsql.lo -MD -MP -MF .deps/libafsql_la-afsql.Tpo -c afsql.c -fPIC -DPIC -o .libs/libafsql_la-afsql.o afsql.c:120: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘dbi_instance’ afsql.c: In function ‘afsql_dd_insert_db’: afsql.c:583: warning: implicit declaration of function ‘dbi_conn_new_r’ afsql.c:583: error: ‘dbi_instance’ undeclared (first use in this function) afsql.c:583: error: (Each undeclared identifier is reported only once afsql.c:583: error: for each function it appears in.) afsql.c: In function ‘afsql_dd_init’: afsql.c:1007: warning: implicit declaration of function ‘dbi_initialize_r’ afsql.c:1007: error: ‘dbi_instance’ undeclared (first use in this function)

4 14

Re: [syslog-ng] Mass text log files processing over network.
by Tomasz Wrona 09 Sep '10

09 Sep '10

> From: Joe McDonagh <joseph.e.mcdonagh(a)gmail.com> > Probably easier to just pipe the configs to logger, since you can set > the facility and log level with it. This is what I do for my apache > stuff, pretty much a standard practice among the guys I know. > > -- > Joe McDonagh Joe, You are right, piping logs to logger works fine for apache logs, but AFAIK it doesn't work for php error_logs ( php_value error_log "|logger [..]" ). > From: Baron Von Pietri <barnaclebob(a)gmail.com> > If you do use the file() source driver then what i have done in a > similar situation is use the program() flag to set what destination > filename the log data should be in. then in the destination i can > just use file("/var/log/customstuff/$PROGRAM.log"). > > Hope that helps. > -- > Karl Thank You Karl! It works for me. So I use parameter log_prefix("service1: ") as equivalent for program_override() on client side and then $PROGRAM.log on server side. However, it is still needed to configure individually each log file [set "prefix/program"] on client side but server process it smart to dedicated files. Regards Tom

1 0

syslog-ng is not reading level/facility properly
by Charlie Reddington 08 Sep '10

08 Sep '10

Hi List, After my last weeks note, I kept on doing some more digging. Perhaps some of this output will aid in finding a solution. I compiled from source, from the latest stable 2.0 line thinking maybe that was part of my problem since I installed via the epel repositories. Nope, still having the same problem. On a remote host, I have a nginx server, that is sending it's access logs to local5.notice and it's error logs to local5.crit. In the syslog.conf file I have the following entries. local5.notice @syslog-ng server local5.crit @syslog-ng server This data is definitely making it to the systlog server. I have done a tcpdump and it shows this with the following. Also I have it logging to local log files and they are working the way I expect. $ tcpdump port 514 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes 16:16:40.725473 IP CLIENTIP.syslog > SYSLOGNGIP.syslog: SYSLOG local5.notice, length: 304 <-- coming in as local5.notice as I expect. Okay so they are hitting the server how I expect, but they aren't being filtered the way I expect. Below is my syslog-ng config. # Syslog-ng config options { use_dns(no); keep_hostname(yes); long_hostnames(off); chain_hostnames(off); sync(1); log_fifo_size(1024); create_dirs(yes); perm(0644); stats_freq(3600); mark_freq(600); }; # # Data Sources # # External Syslog source s_external { udp(ip("0.0.0.0") port(514) flags(no_parse)); }; # # Where to put Data # # Nginx / Lighttpd / Apache Access Logs destination d_access_log { file("/data01/syslog-ng/$HOST/ access.log"); }; # Nginx / Lighttpd / Apache Error Logs destination d_error_log { file("/data01/syslog-ng/$HOST/error.log"); }; # Everything else destination d_log { file("/data01/syslog-ng/$HOST/$FACILITY.$LEVEL"); }; # # Data Filters # # Facility local5 filter f_local5 { facility(local5); }; # Level info filter f_info { level(info); }; # Level notice filter f_notice { level(notice); }; # Level warn filter f_warn { level(warn); }; # Level crit filter f_crit { level(crit); }; # Level err filter f_err { level(err); }; # # Data Logging Locations # # Access Logs log { source(s_external); filter(f_local5); filter(f_notice); destination(d_access_log); }; # Error Logs log { source(s_external); filter(f_local5); filter(f_crit); destination(d_error_log); }; # Everything Else log { source(s_external); destination(d_log); }; Now the only place that ANY messages seem to go, is into /data01/ syslog-ng/$HOST/kern.emerg. Why is everything being marked as kern.emerg? And lastly here's some debug output that maybe able to shed some light to those who know more than I do. ./sbin/syslog-ng -d -v -e --cfgfile=./etc/syslog-ng.conf Running application hooks; hook='1' syslog-ng starting up; version='2.0.11' Incoming log entry; line='<156>logger: REMOTEIP www.domain.com - [08/ Sep/2010:17:17:44 -0400] "GET /2010/09/08-7912 HTTP/1.0" 200 83134 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) "\x0a' Filter rule evaluation begins; filter_rule='f_local5' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_local5' Filter rule evaluation begins; filter_rule='f_local5' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_local5' For some reason this isn't matching, but I'm officially out of experience on this to know what to do further. I should also note that I'm running splunk on this syslog-ng server and it's able to get my access logs properly. If anyone can throw me a bone here I'd really appreciate it, many thanks in advance. Charlie

3 3

pdbtool parse with success but syslog-ng.conf NOT
by otgovorete＠gmail.com 03 Sep '10

03 Sep '10

Hi guys, I have following situation - I can parse with pdbtool bellow message successfully but when I parse it through syslog-ng.conf and put .classifier.class in DB I got "unknown" in the field where is placed .classifier.class or nothing when put the value FailedLogin_USERNAME in the table. Please see my syslog-ng.conf file below. Please help me to find the mistake. Thank you. Kosta Values: MESSAGE=Sep 13 17:34:00 server1 sshd[20981]: Failed keyboard-interactive/pam for invalid user dfgdf from xxxx port 3602 ssh2 PROGRAM=ssh .classifier.class=violation .classifier.rule_id=ssh-failed FailedLogin_MONTH=Sep FailedLogin_DATE=13 FailedLogin_TIME=17:34:00 FailedLogin_SERVER=server1 FailedLogin_SERVICE.ID2=sshd[20981]: FailedLogin_USERNAME=dfgdf FailedLogin_SOURCE_IP=xxxx FailedLogin_SOURCE.PORT=3602 kosta@Kostadin:~$ /opt/syslog-ng/bin/pdbtool match -D -c -p /opt/syslog-ng/var/login.parser.new.xml -P "ssh" -M "Sep 13 17:34:00 server1 sshd[20981]: Failed keyboard-interactive/pam for invalid user dfgdf from xxxx port 3602 ssh2" <rule provider='balabit' id='ssh-failed' class='violation'> <patterns> <pattern>@ESTRING:FailedLogin_MONTH: @@ESTRING:FailedLogin_DATE: @@ESTRING:FailedLogin_TIME: @@ESTRING:FailedLogin_SERVER: @@ESTRING:FailedL$ </patterns> </rule> ################ # DESTINATIONS # ################ destination d_mssql_unix { sql(type(mssql) host("medea.mobiltel.bg") port("1433") username("syslog_ng") password("nglogp@ss") database("SysLog") table("SYSLOG_unix1")columns("Date varchar(40)", "Time varchar(16)", "SourceServer varchar(8)", "SendingProgram varchar(5)", "PID varchar(6)$ values("$DATE_", "$TIME_", "$HOST", "$PROGRAM", "$PID", "${.classifier.class}")); }; ############### # PARSER # ############### parser pattern_db { db_parser(file("/opt/syslog-ng/var/login.parser.new.xml")); }; parser DateTime { csv-parser(columns("DATE_", "TIME_") delimiters("T") flags(escape-none) template("${ISODATE}")); }; ############### # LOG # ############### log { source(s_net); parser (DateTime); parser(pattern_db); destination(d_mssql_unix); destination(d_messages_successful); };

4 3

Re: [syslog-ng] [PATCH] remove auto-generated files
by chris packham 03 Sep '10

03 Sep '10

On Sat, 2010-09-04 at 04:23 +1200, chris packham wrote: > On Fri, 2010-09-03 at 13:24 +0200, Balazs Scheidler wrote: > > On Fri, 2010-09-03 at 06:09 +1200, chris packham wrote: > > > INSTALL is generated by the build process. If the build host has a different > > > version of autotools then the contents of INSTALL can change which in turn > > > causes problems for developers working from the git repository. > > > > > > This patch removes the INSTALL file and adds it to .gitignore. > > > > So, in your case autotools generates an INSTALL file? That's funny, this > > is not the same as the autotools generated one (in fact it is a bit > > outdated, but I wrote that to document how to install syslog-ng). > > > > For me automake doesn't generate this file. Why does it for you? > > > > autogen.sh contains: > > > > > > aclocal > > sed -i -e 's/PKG_PROG_PKG_CONFIG(\[0\.16\])/PKG_PROG_PKG_CONFIG([0.14])/g' aclocal.m4 > > > > autoheader > > automake --foreign --add-missing > > autoconf > > > > > > as I understand automake will only be adding the INSTALL file if it doesn't exist. > > Are you using -f for automake? > > > > Actually our problem might be this > > $(call MESSAGE,"Running autogen.sh") > $(Q)cd $(SYSLOG_NG_GIT_DIR) && $(AUTORECONF) 2> /dev/null > $(Q)touch $@ > > So we're lying to our developers for a start. If I do run autogen.sh > instead I get this error > > configure.in:513: warning: macro `AM_PATH_GLIB_2_0' not found in library > configure.in:513: error: possibly undefined macro: AM_PATH_GLIB_2_0 > > which is probably the reason we didn't just use autogen.sh in the first > place. > > Looking at the expanded $(AUTORECONF) it expands to "/usr/bin/autoreconf > -f -i" (plus a lot of environment vars for our cross-compile > environment). I'll try fiddling with these options to see if some of > them changes the behaviour. Sure enough, removing the -f option stops it from overwriting INSTALL.

1 0

Solving parsing problem
by otgovorete＠gmail.com 03 Sep '10

03 Sep '10

Hi guys, I have done parser for the authentication logs of HPUX. Here is the situation: A)Sep 22 13:14:24 serverone sshd[12934]: Accepted keyboard-interactive/pam for username from xxxx port 1691 ssh2 B)Sep 2 13:14:24 serverone sshd[12934]: Accepted keyboard-interactive/pam for username from xxxx port 1691 ssh2 The parser i've made parses successfully A but not B. The problem is extra whitespace when the date is one digit. I am testing parsing with pdbtool of syslog-ng. The outcome of parsing of A is: PROGRAM=ssh .classifier.class=legitimate .classifier.rule_id=ssh-succeed SucceedLogin_MONTH=Sep SucceedLogin_DATE=22 SucceedLogin_TIME=13:14:24 SucceedLogin_SERVER=serverone SucceedLogin_SERVICE.ID=sshd[12934]: SucceedLogin_USER_NAME=username SucceedLogin_DESTINATION=xxxx SucceedLogin_SOURCE.PORT=1691 The outcome of parsing if B is: Matching part: Sep 2 13:14:24 serverone Values: MESSAGE=Sep 2 13:14:24 serverone sshd[12934]: Accepted keyboard-interactive/pam for username from xxxx port 1691 ssh2 PROGRAM=ssh .classifier.class=unknown Here is the parser from xml file: <patterns> <pattern>@ESTRING:SucceedLogin_MONTH: @@ESTRING:SucceedLogin_DATE: @@ESTRING:SucceedLogin_TIME: @@ESTRING:SucceedLogin_SERVER: @@ESTRING:SucceedLogin_SERVICE.ID: @Accepted keyboard-interactive/pam for @ESTRING:SucceedLogin_USER_NAME: @from @ESTRING:SucceedLogin_DESTINATION: @port @ESTRING:SucceedLogin_SOURCE.PORT: @ssh2</pattern> </patterns> and here is the command pdbtool: /opt/syslog-ng/bin/pdbtool match -D -c -p login.parser.new.xml -P "ssh" -M "Sep 22 13:14:24 serverone sshd[12934]: Accepted keyboard-interactive/pam for username from xxxx port 1691 ssh2" Please, advice me how to proceed. Thank you. Kosta

5 5