- syslog-ng - lists.balabit.hu

Debugging Pattern Match Failures
by Matthew Hall 04 Aug '10

04 Aug '10

Hello list, Recently I created a series of blasphemous scripts which convert some large collections of recorded log messages in my environment into pattern DB XML files. At first there were some syntax errors but I fixed all of these and the XML files are loading successfully. However I am running into some problems with the next step: getting the patterns to match against the incoming log messages. I suspect I am not properly stripping the headers off of the disk files of recorded messages I am using to generate the pattern DB XML files. I am wondering how I can enable the right debugging capabilities to get more detailed debug output from the pattern DB parser where I can see what strings are being processed so that I can fix this right instead of guessing repeatedly and incorrectly. Thanks, Matthew Hall.

3 12

syslog-ng 3.1.2 has been released
by devel＠balabit.hu 04 Aug '10

04 Aug '10

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.1.2 SUMMARY : new stable release DATE : Aug 4, 2010 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (3.1.2) has been released. For latest fixes in the 3.1.x feature branch you are recommended to upgrade to this version. CHANGES: 3.1.2 Mon, 02 Aug 2010 17:06:28 +0200 INCOMPATIBLE CHANGES: * This release changes the default for 'store-legacy-msghdr' flag for log sources. In previous releases this had to be enabled explicitly, because of the performance penalties, which do not apply to 3.1. The original behaviour can be restored by explicitly specifying 'dont-store-legacy-msghdr'. See this blog post for more details: http://bazsi.blogs.balabit.com/2010/05/small-incompatible-change-for-31.html Bugfixes: * When syslog-ng is reloaded, the local hostname value was not refreshed, causing syslog-ng to remember the hostname until the next restart. This may not play nice with DHCP configured hostnames, which may change dynamically. * When several SIGHUP signals are received in quick succession, the last one may have been dropped. This problem is fixed. * Make sure that numbers never get resolved using getpwnam/getgrnam (e.g. libnss functions), because this may cause deadlocks if the NSS provider is LDAP and the LDAP server is trying to log a message to syslog about invalid usernames. * Fixed flush_timeout() handling in for non-file destinations. Enabling flush_lines()/flush_timeout() for a non-file destination, either using a global option, or a per-destination option could cause excess CPU usage as long as the destination wasn't writable. * In case patterndb has a syntax error when reloading the file, syslog-ng automatically reverts to the old version instead of dropping pattern matching altogether. * Fixed pdbtool to properly handle accented characters in the command line. * Fixed the use of the greedy and drop-invalid flags for csv-parser. * Fixed a possible but rare memory leak in the handling of message suppression. Other changes: * Added "update-patterndb" command that merges patterndb files from ${sysconfdir}/patterndb.d and produces ${localstatedir}/patterndb.xml, the default file of the db-parser(). * Make it possible to specify the path to the syslog-ng control socket using a command line option in order to make it easier to launch multiple syslog-ng instances. * Introduced a new, cleaner syntax to specify that the permissions of a file are not to be changed. Use owner/group/perm options of a file without any parameters instead of "-1". The old mechanism had the problem that both per-file and global settings had to use "-1" to effectively stop syslog-ng from changing the uid/gid/perm values. The new syntax will make it possible to specify defaults in the global options section and customize those on a per-file basis. * syslog-ng will not go into the background if either the --syntax-only or --debug options are specified. Loggen: * Added SSL support. * Added the ability to read from a sample file Credits: * We would like to send our thank for all contributors sending us bugreports, documentation issues or patches. They are: * Marius Tomaschewski * Corey Hickey * Evan Rempel * Andy Kitchingman * andrewn at locus.net Hopefully noone was missed from this list. DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

1 0

Strange behavior with the option "mark"
by Ilas, Yann 03 Aug '10

03 Aug '10

Hello everybody, I'm currently using the application syslog-ng version 3.1.1 and there is an odd behavior with the "mark" option. Here is the configuration file used for the test : @version: 3.0 options { mark(7); }; source s_local { internal(); unix-stream( "/dev/log" ); }; destination d_f_msg_unknown { file ( /var/log/messages__unknown ); }; log { source (s_local); destination (d_f_msg_unknown); }; ...and I start the syslog-ng application like this : # cd /opt/syslog-ng/sbin/ # ./syslog-ng -Fevd --foreground --no-caps --cfgfile=/opt/syslog-ng/etc/syslog-ng.conf.MARK --pidfile=/tmp/syslog-ng.conf.pid The output of the log file : Aug 2 17:44:22 serveur01 -- MARK -- Aug 2 17:44:29 serveur01 -- MARK -- Aug 2 17:44:36 serveur01 -- MARK -- (...) Then, I change the configuration file from "mark(7)" to "mark(0)" and send a HUP signal between "17:44:36" and "17:44:43" # kill -HUP $(cat /tmp/syslog-ng.conf.pid) Here is the output (tail -f /var/log/messages__unknown) : Aug 2 17:44:36 serveur01 -- MARK -- <HUP signal> Aug 2 17:44:43 serveur01 -- MARK -- Aug 2 17:44:50 serveur01 -- MARK -- Aug 2 17:44:50 serveur01 -- MARK -- Aug 2 17:44:50 serveur01 -- MARK -- Aug 2 17:44:50 serveur01 -- MARK -- Aug 2 17:44:50 serveur01 -- MARK -- Aug 2 17:44:50 serveur01 -- MARK -- Aug 2 17:44:50 serveur01 -- MARK -- (...) ...then the server syslog-ng sends a lot of "MARK" messages. I have to stop the process with a "kill" or "ctrl+c". I have the same behaviour if I use "mark_freq" instead of "mark". What's wrong with that option ? Did I miss something ? Regards, Yann I.

2 1

[Bug 84] New: Program destination blocks if program stops reading input
by bugzilla＠bugzilla.balabit.com 02 Aug '10

02 Aug '10

https://bugzilla.balabit.com/show_bug.cgi?id=84 Summary: Program destination blocks if program stops reading input Product: syslog-ng Version: 3.0.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 Syslog-ng OSE 3.0.5 and 3.0.6 - binary download from balabit and self compiled all behave the same. If the program in a destination stops reading its input, syslog-ng buffers a small number of messages and then stops reading the source (in my case a pipe). I would expect syslog-ng to read from its input until the log_fifo_size limit was reached and then drop messages. an strace is attached. The configuration file is -------------- options { log_fifo_size(1000000); use_fqdn(yes); keep_hostname(yes); chain_hostnames(yes); time_reap(60); time_reopen(5); flush_lines(1000); flush_timeout(1000); }; template t_application { template("$MESSAGE\n"); template_escape(no); }; template t_localsyslog { template("<$PRI>$S_ISODATE $HOST erempel: $MSGONLY\n"); template_escape(no); }; source local { pipe("/var/log/syslog.pipes/erempel" flags(no-parse) ); }; source int { internal(); }; destination localsyslog { tcp("localhost" port(1514) localip(localhost) template(t_localsyslog) log_fifo_size(5000) ); }; destination erempel { program("/home1l/erempel/read-noread.pl" template(t_application) log_fifo_size(8000000) ); }; log { source(local); destination(erempel); }; log { source(int); destination(localsyslog); }; ---------------------- -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 8

Pattern for Eventlog-to-Syslog
by Martin Holste 31 Jul '10

31 Jul '10

I've recently started using the great util Eventlog-to-Syslog (http://code.google.com/p/eventlog-to-syslog/) and needed a parser. This pattern should do the job. A word of warning, I opted to go with no program match because I didn't want to write a pattern per event log source, so it's possible this would hit on other log sources. However, in testing, the pattern seems to work well and I haven't gotten any "false" matches. I also gave up trying to parse out the user name as the second param because it isn't always included and the semicolon used as a delimiter regularly shows up naturally in messages. One could write per-event-id message parsers which grabbed all kinds of fields, but this should get the ball rolling to at least grab the event id. <ruleset name="Windows" id='4'>  <rules> <rule provider="local" class='Windows' id='4'> <patterns> <pattern>@NUMBER:event_id:@: @ANYSTRING:msg:@</pattern> </patterns> <examples> <example> <test_message program="Service_Control_Manager">7035: NT AUTHORITY\SYSTEM: The COH_Mon service was successfully sent a start control.</test_message> <test_values> <test_value name="event_id">7035</test_value> <test_value name="msg">NT AUTHORITY\SYSTEM: The COH_Mon service was successfully sent a start control.</test_value> </test_values> </example> <example> <test_message program="SceCli">1202: Security policies were propagated with warning. 0x4b8 : An extended error has occurred. For best results in resolving this event, log on with a non-administrative account and search http://support.microsoft.com for "Troubleshooting Event 1202's".</test_message> <test_value name="event_id">1202</test_value> <test_value name="msg">Security policies were propagated with warning. 0x4b8 : An extended error has occurred. For best results in resolving this event, log on with a non-administrative account and search http://support.microsoft.com for "Troubleshooting Event 1202's".</test_value> </example> </examples> <tags> <tag>Windows</tag> </tags> </rule> </rules> </ruleset>

2 2

Syntax in 3.0 Admin Guide not working for MySQL
by Chuck 31 Jul '10

31 Jul '10

I am trying to log to mysql. I am running syslog-ng 3.0.4 on Solaris 10 Update 8. I have the following stanza in syslog-ng.conf: destination r_mysql { sql(type(mysql) host("localhost") username("syslog") password("xxxxxxx") database("syslog") table("messages_${HOST}_${R_YEAR}${R_MONTH}${R_DAY}") columns("datetime", "host", "program", "pid", "message") values("$R_DATE", "$HOST", "$PROGRAM", "$PID", "$MSGONLY") indexes("datetime", "host", "program", "pid", "message")); }; It is complaining about the syntax.. Anyone have any ideas? Thx

4 4

Syslog IO Error
by Pedro Rafael Alves Simoes 30 Jul '10

30 Jul '10

Hello, My Syslog-ng shows this error message: I/O error occurred while reading; fd='183', error='Connection reset by peer (104)' Anybody have a clue in how to resolve this? I've tried a google search about this but didn't get much information... Using RedHat Linux 2.6.18-164.el5 syslog-ng-2.1.4-1.el5 Thanks.

2 1

Solaris 10 Compilation Woes
by Chuck 30 Jul '10

30 Jul '10

Trying to build 3.1.1 on Solaris 10 (update 8) so I can enable SQL support. I've installed the required version of gcc, bison, flex, and etc.. per the INSTALL doc. Here is the error I get: (I've searched google all morning and find tons of others with this error but no solution) ./configure --prefix=/usr/local --enable-sql --enable-pcre checking for pkg-config... /bin/pkg-config checking pkg-config is at least version 0.9.0... yes checking for a BSD-compatible install... /usr/local/bin/install -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... /usr/local/bin/mkdir -p checking for gawk... no checking for mawk... no checking for nawk... nawk checking whether make sets $(MAKE)... yes checking for gcc... gcc checking for C compiler default output file name... a.out checking whether the C compiler works... yes checking whether we are cross compiling... no checking for suffix of executables... checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether gcc accepts -g... yes checking for gcc option to accept ISO C89... none needed checking for style of include used by make... GNU checking dependency style of gcc... gcc3 checking for gcc... (cached) gcc checking whether we are using the GNU C compiler... (cached) yes checking whether gcc accepts -g... (cached) yes checking for gcc option to accept ISO C89... (cached) none needed checking dependency style of gcc... (cached) gcc3 checking for gcc option to accept ISO C99... -std=gnu99 checking for bison... bison -y checking for flex... flex checking lex output file root... configure: error: cannot find output from flex; giving up I am using the following environment vars: CXX=g++ CFLAGS=-I/usr/local/include CPPFLAGS=-I/usr/local/include CC=gcc All my open source tools are under /usr/local. I have tried have /usr/ccs/bin in my PATH before /yusr/local/bin, tried having it after, and tried not having it in my PATH at all. Anyone have any ideas? Thx, CC

2 3

Re: [syslog-ng] Small problem with pdbtool and the parser ?
by Ilas, Yann 29 Jul '10

29 Jul '10

Hello, Thanks ! It works... Regards, Yann I. Hi, Please use this patch (fix by Viktor Tusa): diff --git a/src/dbparser/pdbtool.c b/src/dbparser/pdbtool.c index 5ce3e79..cde859a 100644 --- a/src/dbparser/pdbtool.c +++ b/src/dbparser/pdbtool.c @@ -11,6 +11,7 @@ #include <stdio.h> #include <string.h> #include <stdlib.h> +#include <locale.h> #if HAVE_GETOPT_H #include <getopt.h> @@ -618,6 +620,7 @@ main(int argc, char *argv[]) usage(); } + setlocale(LC_ALL,""); if (!g_option_context_parse(ctx, &argc, &argv, &error)) { fprintf(stderr, "Error parsing command line arguments: %s\n", error ? error->message : "Invalid arguments"); M On Wed, 2010-07-28 at 15:39 +0200, Ilas, Yann wrote: > Hello, > > I have a small problem with pdbtool. When I try to match this message "> Message message n?2 message 255", I obtain this error : "Error parsing command line arguments: Invalid byte sequence in conversion input". > The problem seems to be the "?" which is bad interpreted by pdbtool. But the syslog-ng application has not that problem. > > Here is the xml file I used : > <?xml version='1.0' encoding='UTF-8'?> > <patterndb version='3' pub_date='2010-07-28'> > <ruleset name='test' id='ab-cd-123456789'> > <pattern>test</pattern> > <rules> > <rule provider='yann' id='xxxxx:syslog:abcdef:1234567890:id001' class='system'> > <patterns> > <pattern>> Message message n?@NUMBER@ message @NUMBER@</pattern> > <values> > <value name=".classifier.facility">local5</value> > <value name=".classifier.severity">notice</value> > <value name=".classifier.priority">173</value> > </values> > </patterns> > </rule> > </rules> > </ruleset> > </patterndb> > > If I use the pattern "<pattern>> Message message n?@NUMBER@ message @NUMBER@</pattern>", I obtain : > $ /opt/syslog-ng/bin/pdbtool match -p patterndb.xml -P test -M '> Message message n?2 message 255' > Error parsing command line arguments: Invalid byte sequence in conversion input > => It's not OK > > If I use the pattern "<pattern>> Message message nx@NUMBER@ message @NUMBER@</pattern>", I obtain : > (the diffence is the charactere "x" after the " n") > $ /opt/syslog-ng/bin/pdbtool match -p patterndb.xml -P test -M '> Message message nx2 message 255' > MESSAGE=> Message message nx2 message 255 > PROGRAM=test > .classifier.class=system > .classifier.rule_id=xxxxx:syslog:abcdef:1234567890:id001 > .classifier.facility=local5 > .classifier.severity=notice > .classifier.priority=173 > => It's OK > > With syslog-ng and the pattern "<pattern>> Message message n?@NUMBER@ message @NUMBER@</pattern>", I tried to send the same message (with the "?") by using "nc" and it works, ie the message is well interpreted by the application : > $ LANG=C date +"<70>%b %d %R:%S server.domain test[1234]: > Message message n?123 message 456" | nc -4 127.0.0.1 514 > > Why pdbtool can't parse that string ? > > Regards, > > Yann I. >

1 0

[patterndb] more login/logout samples
by Balazs Scheidler 28 Jul '10

28 Jul '10

Hi, Just as I wrote in the other thread, I'm almost finished with marking up the log samples that you posted here on this list. This means that we can keep the ball rolling. I guess there are other services running on your computers doing authentication, so please go ahead and email bomb me with further samples. :) Also, I'd appreciate some review on the markup, and some testing with real life data. See my blog post about "grep on steroids" to see how you can immediately use the stuff finished so far. -- Bazsi

1 0