- syslog-ng - lists.balabit.hu

Re: [syslog-ng] Mass text log files processing over network.
by Tomasz Wrona 09 Sep '10

09 Sep '10

> From: Joe McDonagh <joseph.e.mcdonagh(a)gmail.com> > Probably easier to just pipe the configs to logger, since you can set > the facility and log level with it. This is what I do for my apache > stuff, pretty much a standard practice among the guys I know. > > -- > Joe McDonagh Joe, You are right, piping logs to logger works fine for apache logs, but AFAIK it doesn't work for php error_logs ( php_value error_log "|logger [..]" ). > From: Baron Von Pietri <barnaclebob(a)gmail.com> > If you do use the file() source driver then what i have done in a > similar situation is use the program() flag to set what destination > filename the log data should be in. then in the destination i can > just use file("/var/log/customstuff/$PROGRAM.log"). > > Hope that helps. > -- > Karl Thank You Karl! It works for me. So I use parameter log_prefix("service1: ") as equivalent for program_override() on client side and then $PROGRAM.log on server side. However, it is still needed to configure individually each log file [set "prefix/program"] on client side but server process it smart to dedicated files. Regards Tom

1 0

syslog-ng is not reading level/facility properly
by Charlie Reddington 09 Sep '10

09 Sep '10

Hi List, After my last weeks note, I kept on doing some more digging. Perhaps some of this output will aid in finding a solution. I compiled from source, from the latest stable 2.0 line thinking maybe that was part of my problem since I installed via the epel repositories. Nope, still having the same problem. On a remote host, I have a nginx server, that is sending it's access logs to local5.notice and it's error logs to local5.crit. In the syslog.conf file I have the following entries. local5.notice @syslog-ng server local5.crit @syslog-ng server This data is definitely making it to the systlog server. I have done a tcpdump and it shows this with the following. Also I have it logging to local log files and they are working the way I expect. $ tcpdump port 514 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes 16:16:40.725473 IP CLIENTIP.syslog > SYSLOGNGIP.syslog: SYSLOG local5.notice, length: 304 <-- coming in as local5.notice as I expect. Okay so they are hitting the server how I expect, but they aren't being filtered the way I expect. Below is my syslog-ng config. # Syslog-ng config options { use_dns(no); keep_hostname(yes); long_hostnames(off); chain_hostnames(off); sync(1); log_fifo_size(1024); create_dirs(yes); perm(0644); stats_freq(3600); mark_freq(600); }; # # Data Sources # # External Syslog source s_external { udp(ip("0.0.0.0") port(514) flags(no_parse)); }; # # Where to put Data # # Nginx / Lighttpd / Apache Access Logs destination d_access_log { file("/data01/syslog-ng/$HOST/ access.log"); }; # Nginx / Lighttpd / Apache Error Logs destination d_error_log { file("/data01/syslog-ng/$HOST/error.log"); }; # Everything else destination d_log { file("/data01/syslog-ng/$HOST/$FACILITY.$LEVEL"); }; # # Data Filters # # Facility local5 filter f_local5 { facility(local5); }; # Level info filter f_info { level(info); }; # Level notice filter f_notice { level(notice); }; # Level warn filter f_warn { level(warn); }; # Level crit filter f_crit { level(crit); }; # Level err filter f_err { level(err); }; # # Data Logging Locations # # Access Logs log { source(s_external); filter(f_local5); filter(f_notice); destination(d_access_log); }; # Error Logs log { source(s_external); filter(f_local5); filter(f_crit); destination(d_error_log); }; # Everything Else log { source(s_external); destination(d_log); }; Now the only place that ANY messages seem to go, is into /data01/ syslog-ng/$HOST/kern.emerg. Why is everything being marked as kern.emerg? And lastly here's some debug output that maybe able to shed some light to those who know more than I do. ./sbin/syslog-ng -d -v -e --cfgfile=./etc/syslog-ng.conf Running application hooks; hook='1' syslog-ng starting up; version='2.0.11' Incoming log entry; line='<156>logger: REMOTEIP www.domain.com - [08/ Sep/2010:17:17:44 -0400] "GET /2010/09/08-7912 HTTP/1.0" 200 83134 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) "\x0a' Filter rule evaluation begins; filter_rule='f_local5' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_local5' Filter rule evaluation begins; filter_rule='f_local5' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_local5' For some reason this isn't matching, but I'm officially out of experience on this to know what to do further. I should also note that I'm running splunk on this syslog-ng server and it's able to get my access logs properly. If anyone can throw me a bone here I'd really appreciate it, many thanks in advance. Charlie

3 3

pdbtool parse with success but syslog-ng.conf NOT
by otgovorete＠gmail.com 03 Sep '10

03 Sep '10

Hi guys, I have following situation - I can parse with pdbtool bellow message successfully but when I parse it through syslog-ng.conf and put .classifier.class in DB I got "unknown" in the field where is placed .classifier.class or nothing when put the value FailedLogin_USERNAME in the table. Please see my syslog-ng.conf file below. Please help me to find the mistake. Thank you. Kosta Values: MESSAGE=Sep 13 17:34:00 server1 sshd[20981]: Failed keyboard-interactive/pam for invalid user dfgdf from xxxx port 3602 ssh2 PROGRAM=ssh .classifier.class=violation .classifier.rule_id=ssh-failed FailedLogin_MONTH=Sep FailedLogin_DATE=13 FailedLogin_TIME=17:34:00 FailedLogin_SERVER=server1 FailedLogin_SERVICE.ID2=sshd[20981]: FailedLogin_USERNAME=dfgdf FailedLogin_SOURCE_IP=xxxx FailedLogin_SOURCE.PORT=3602 kosta@Kostadin:~$ /opt/syslog-ng/bin/pdbtool match -D -c -p /opt/syslog-ng/var/login.parser.new.xml -P "ssh" -M "Sep 13 17:34:00 server1 sshd[20981]: Failed keyboard-interactive/pam for invalid user dfgdf from xxxx port 3602 ssh2" <rule provider='balabit' id='ssh-failed' class='violation'> <patterns> <pattern>@ESTRING:FailedLogin_MONTH: @@ESTRING:FailedLogin_DATE: @@ESTRING:FailedLogin_TIME: @@ESTRING:FailedLogin_SERVER: @@ESTRING:FailedL$ </patterns> </rule> ################ # DESTINATIONS # ################ destination d_mssql_unix { sql(type(mssql) host("medea.mobiltel.bg") port("1433") username("syslog_ng") password("nglogp@ss") database("SysLog") table("SYSLOG_unix1")columns("Date varchar(40)", "Time varchar(16)", "SourceServer varchar(8)", "SendingProgram varchar(5)", "PID varchar(6)$ values("$DATE_", "$TIME_", "$HOST", "$PROGRAM", "$PID", "${.classifier.class}")); }; ############### # PARSER # ############### parser pattern_db { db_parser(file("/opt/syslog-ng/var/login.parser.new.xml")); }; parser DateTime { csv-parser(columns("DATE_", "TIME_") delimiters("T") flags(escape-none) template("${ISODATE}")); }; ############### # LOG # ############### log { source(s_net); parser (DateTime); parser(pattern_db); destination(d_mssql_unix); destination(d_messages_successful); };

4 3

Re: [syslog-ng] [PATCH] remove auto-generated files
by chris packham 03 Sep '10

03 Sep '10

On Sat, 2010-09-04 at 04:23 +1200, chris packham wrote: > On Fri, 2010-09-03 at 13:24 +0200, Balazs Scheidler wrote: > > On Fri, 2010-09-03 at 06:09 +1200, chris packham wrote: > > > INSTALL is generated by the build process. If the build host has a different > > > version of autotools then the contents of INSTALL can change which in turn > > > causes problems for developers working from the git repository. > > > > > > This patch removes the INSTALL file and adds it to .gitignore. > > > > So, in your case autotools generates an INSTALL file? That's funny, this > > is not the same as the autotools generated one (in fact it is a bit > > outdated, but I wrote that to document how to install syslog-ng). > > > > For me automake doesn't generate this file. Why does it for you? > > > > autogen.sh contains: > > > > > > aclocal > > sed -i -e 's/PKG_PROG_PKG_CONFIG(\[0\.16\])/PKG_PROG_PKG_CONFIG([0.14])/g' aclocal.m4 > > > > autoheader > > automake --foreign --add-missing > > autoconf > > > > > > as I understand automake will only be adding the INSTALL file if it doesn't exist. > > Are you using -f for automake? > > > > Actually our problem might be this > > $(call MESSAGE,"Running autogen.sh") > $(Q)cd $(SYSLOG_NG_GIT_DIR) && $(AUTORECONF) 2> /dev/null > $(Q)touch $@ > > So we're lying to our developers for a start. If I do run autogen.sh > instead I get this error > > configure.in:513: warning: macro `AM_PATH_GLIB_2_0' not found in library > configure.in:513: error: possibly undefined macro: AM_PATH_GLIB_2_0 > > which is probably the reason we didn't just use autogen.sh in the first > place. > > Looking at the expanded $(AUTORECONF) it expands to "/usr/bin/autoreconf > -f -i" (plus a lot of environment vars for our cross-compile > environment). I'll try fiddling with these options to see if some of > them changes the behaviour. Sure enough, removing the -f option stops it from overwriting INSTALL.

1 0

Solving parsing problem
by otgovorete＠gmail.com 03 Sep '10

03 Sep '10

Hi guys, I have done parser for the authentication logs of HPUX. Here is the situation: A)Sep 22 13:14:24 serverone sshd[12934]: Accepted keyboard-interactive/pam for username from xxxx port 1691 ssh2 B)Sep 2 13:14:24 serverone sshd[12934]: Accepted keyboard-interactive/pam for username from xxxx port 1691 ssh2 The parser i've made parses successfully A but not B. The problem is extra whitespace when the date is one digit. I am testing parsing with pdbtool of syslog-ng. The outcome of parsing of A is: PROGRAM=ssh .classifier.class=legitimate .classifier.rule_id=ssh-succeed SucceedLogin_MONTH=Sep SucceedLogin_DATE=22 SucceedLogin_TIME=13:14:24 SucceedLogin_SERVER=serverone SucceedLogin_SERVICE.ID=sshd[12934]: SucceedLogin_USER_NAME=username SucceedLogin_DESTINATION=xxxx SucceedLogin_SOURCE.PORT=1691 The outcome of parsing if B is: Matching part: Sep 2 13:14:24 serverone Values: MESSAGE=Sep 2 13:14:24 serverone sshd[12934]: Accepted keyboard-interactive/pam for username from xxxx port 1691 ssh2 PROGRAM=ssh .classifier.class=unknown Here is the parser from xml file: <patterns> <pattern>@ESTRING:SucceedLogin_MONTH: @@ESTRING:SucceedLogin_DATE: @@ESTRING:SucceedLogin_TIME: @@ESTRING:SucceedLogin_SERVER: @@ESTRING:SucceedLogin_SERVICE.ID: @Accepted keyboard-interactive/pam for @ESTRING:SucceedLogin_USER_NAME: @from @ESTRING:SucceedLogin_DESTINATION: @port @ESTRING:SucceedLogin_SOURCE.PORT: @ssh2</pattern> </patterns> and here is the command pdbtool: /opt/syslog-ng/bin/pdbtool match -D -c -p login.parser.new.xml -P "ssh" -M "Sep 22 13:14:24 serverone sshd[12934]: Accepted keyboard-interactive/pam for username from xxxx port 1691 ssh2" Please, advice me how to proceed. Thank you. Kosta

5 5

Re: [syslog-ng] [PATCH] remove auto-generated files
by chris packham 03 Sep '10

03 Sep '10

On Fri, 2010-09-03 at 13:24 +0200, Balazs Scheidler wrote: > On Fri, 2010-09-03 at 06:09 +1200, chris packham wrote: > > INSTALL is generated by the build process. If the build host has a different > > version of autotools then the contents of INSTALL can change which in turn > > causes problems for developers working from the git repository. > > > > This patch removes the INSTALL file and adds it to .gitignore. > > So, in your case autotools generates an INSTALL file? That's funny, this > is not the same as the autotools generated one (in fact it is a bit > outdated, but I wrote that to document how to install syslog-ng). > > For me automake doesn't generate this file. Why does it for you? > > autogen.sh contains: > > > aclocal > sed -i -e 's/PKG_PROG_PKG_CONFIG(\[0\.16\])/PKG_PROG_PKG_CONFIG([0.14])/g' aclocal.m4 > > autoheader > automake --foreign --add-missing > autoconf > > > as I understand automake will only be adding the INSTALL file if it doesn't exist. > Are you using -f for automake? > Actually our problem might be this $(call MESSAGE,"Running autogen.sh") $(Q)cd $(SYSLOG_NG_GIT_DIR) && $(AUTORECONF) 2> /dev/null $(Q)touch $@ So we're lying to our developers for a start. If I do run autogen.sh instead I get this error configure.in:513: warning: macro `AM_PATH_GLIB_2_0' not found in library configure.in:513: error: possibly undefined macro: AM_PATH_GLIB_2_0 which is probably the reason we didn't just use autogen.sh in the first place. Looking at the expanded $(AUTORECONF) it expands to "/usr/bin/autoreconf -f -i" (plus a lot of environment vars for our cross-compile environment). I'll try fiddling with these options to see if some of them changes the behaviour.

1 0

syslog-ng on Solaris 10
by Riaz, Javed (WW Global Network Services) 03 Sep '10

03 Sep '10

This the first time I am running syslog-ng 3.0.4 on Solaris 10. Following are the issues I am seeing: 1. Two processes of syslog-ng are running. Is this normal: root 15423 1 0 Aug 31 ? 0:00 /usr/local/sbin/syslog-ng root 15424 15423 0 Aug 31 ? 0:00 /usr/local/sbin/syslog-ng 2. I need to disable/enable syslog-ng multiple times to get syslog-ng online in Solaris SVCADM after updating conf file. 3. Also if I disable syslog-ng in Solaris SVCADM should the tw0 processes stop? Please help! # /usr/local/sbin/syslog-ng -V syslog-ng 3.0.4 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.0#master#1b5d618e301ad94aa20e692ffba16469dece8d10 Compile-Date: Sep 2 2009 06:15:53 Enable-Threads: off Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: on Enable-Sun-Door: on Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: off Enable-SSL: on Enable-SQL: off Enable-Linux-Caps: off Enable-Pcre: on # uname -a SunOS usplsbavisp007 5.10 Generic_118833-33 sun4u sparc SUNW,Sun-Fire-V445 Thanks,

2 1

feature request (parallel processing)
by syslogng＠feystorm.net 03 Sep '10

03 Sep '10

So, after months of work, we finally turned on our production environment for syslog collection. However, we hit one immediate snag. Currently were writing to the database, and the way the database works is that it collects enough data to fill a single block, and then it flushes out that block. Well every time it goes to flush the block out, the insert takes an extra couple milliseconds. Now when I'm doing about 220000 inserts a second, that millisecond delay is significant. So basically syslog has to pause on that log statement while it waits for the database to flush. (1 out of 10 messages was getting dropped) Now I tried to solve this by writing multiple destination drivers so that a second database thread could be processing while the first was flushing, but that didnt work as it appears syslog waits for the destination driver to complete before it hands data off to the second driver. Instead I managed to solve the problem by creating yet more syslog processes. So basically the master process listens for data from all the hosts. It then runs a match on the $PID and sends all even numbered PIDs to one syslog process, and all odd numbered PIDs to a second syslog process. This way both processes can be inserting to the database at the same time. It effectively cuts the amount of work each database thread does in half, so that when it has to pause to flush, it doesnt cause the syslog buffer to fill up. Ultimately my request is this, allow multiple destination drivers to work at the same time. I realize this is probably not a simple change, but seems like it would be a significant speed enhancement.

4 7

logic and duplicate suppression
by John Kristoff 03 Sep '10

03 Sep '10

I have a couple of scenarios where I'm looking to enhance how I handle and process some logs. I'm looking for suggestions on what my options are, but maybe these are potential feature requests? 1. In using a parser (cvs or the patterndb), I'd like to use some conditionals based on a resultant macro value. So for example, if I have an sshd authentication log message with a source address in a macro and that address is contained w/in a specific prefix, I'd like to handle that message differently. Perhaps not log it all or set another MACRO to a certain value. 2. I'd like to be able to suppress duplicate messages even if they are not necessarily contiguous at the destination. So for example, if I have a SSH client that generates a log of its SSH client protocol and software, I don't need to see that over and over again (e.g. as you might commonly see today in SSH brute force attacks). John

4 10

Insert hostname instead of overwrite ?
by stucky 03 Sep '10

03 Sep '10

Guys I'm trying to log to a loadbalanced VIP. It seems to work ok except that the loadbalancer uses SNAT so I loose my source IP. This means I cannot use dns or even the source ip to get the source host as all logs appear to come from the same source (the loadbalancer). This means I have no choice but to rely on the hostname field which works about 98% of the time but some stuff like Dell OpenManage skips the hostname field. So I'd get logs like this on host "cage" f.e. Aug 16 21:47:22 Server Administrator: Storage Service EventID: 2242 The Patrol Read has started.: Controller 0 (PERC 5/i Integrated) I fixed that by telling the syslog-ng client to force itself to figure out a proper hostname and now the log looks like this Aug 17 13:51:10 cage Administrator[]: Instrumentation Service EventID: 1000 Server Administrator starting I thought syslog-ng inserts the hostname but by the looks of it it simply replaces whatever is in the expected field with the hostname it has just figured out. As you can see it overwrote the entry "Server". No biggie in the above case but what if this field contained valuable information ? I'd loose that. Any way to squeeze in the hostname so to speak ? -- stucky

5 12