- syslog-ng - lists.balabit.hu

Pattern for Eventlog-to-Syslog
by Martin Holste 31 Jul '10

31 Jul '10

I've recently started using the great util Eventlog-to-Syslog (http://code.google.com/p/eventlog-to-syslog/) and needed a parser. This pattern should do the job. A word of warning, I opted to go with no program match because I didn't want to write a pattern per event log source, so it's possible this would hit on other log sources. However, in testing, the pattern seems to work well and I haven't gotten any "false" matches. I also gave up trying to parse out the user name as the second param because it isn't always included and the semicolon used as a delimiter regularly shows up naturally in messages. One could write per-event-id message parsers which grabbed all kinds of fields, but this should get the ball rolling to at least grab the event id. <ruleset name="Windows" id='4'>  <rules> <rule provider="local" class='Windows' id='4'> <patterns> <pattern>@NUMBER:event_id:@: @ANYSTRING:msg:@</pattern> </patterns> <examples> <example> <test_message program="Service_Control_Manager">7035: NT AUTHORITY\SYSTEM: The COH_Mon service was successfully sent a start control.</test_message> <test_values> <test_value name="event_id">7035</test_value> <test_value name="msg">NT AUTHORITY\SYSTEM: The COH_Mon service was successfully sent a start control.</test_value> </test_values> </example> <example> <test_message program="SceCli">1202: Security policies were propagated with warning. 0x4b8 : An extended error has occurred. For best results in resolving this event, log on with a non-administrative account and search http://support.microsoft.com for "Troubleshooting Event 1202's".</test_message> <test_value name="event_id">1202</test_value> <test_value name="msg">Security policies were propagated with warning. 0x4b8 : An extended error has occurred. For best results in resolving this event, log on with a non-administrative account and search http://support.microsoft.com for "Troubleshooting Event 1202's".</test_value> </example> </examples> <tags> <tag>Windows</tag> </tags> </rule> </rules> </ruleset>

2 2

Syntax in 3.0 Admin Guide not working for MySQL
by Chuck 31 Jul '10

31 Jul '10

I am trying to log to mysql. I am running syslog-ng 3.0.4 on Solaris 10 Update 8. I have the following stanza in syslog-ng.conf: destination r_mysql { sql(type(mysql) host("localhost") username("syslog") password("xxxxxxx") database("syslog") table("messages_${HOST}_${R_YEAR}${R_MONTH}${R_DAY}") columns("datetime", "host", "program", "pid", "message") values("$R_DATE", "$HOST", "$PROGRAM", "$PID", "$MSGONLY") indexes("datetime", "host", "program", "pid", "message")); }; It is complaining about the syntax.. Anyone have any ideas? Thx

4 4

Syslog IO Error
by Pedro Rafael Alves Simoes 30 Jul '10

30 Jul '10

Hello, My Syslog-ng shows this error message: I/O error occurred while reading; fd='183', error='Connection reset by peer (104)' Anybody have a clue in how to resolve this? I've tried a google search about this but didn't get much information... Using RedHat Linux 2.6.18-164.el5 syslog-ng-2.1.4-1.el5 Thanks.

2 1

Solaris 10 Compilation Woes
by Chuck 30 Jul '10

30 Jul '10

Trying to build 3.1.1 on Solaris 10 (update 8) so I can enable SQL support. I've installed the required version of gcc, bison, flex, and etc.. per the INSTALL doc. Here is the error I get: (I've searched google all morning and find tons of others with this error but no solution) ./configure --prefix=/usr/local --enable-sql --enable-pcre checking for pkg-config... /bin/pkg-config checking pkg-config is at least version 0.9.0... yes checking for a BSD-compatible install... /usr/local/bin/install -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... /usr/local/bin/mkdir -p checking for gawk... no checking for mawk... no checking for nawk... nawk checking whether make sets $(MAKE)... yes checking for gcc... gcc checking for C compiler default output file name... a.out checking whether the C compiler works... yes checking whether we are cross compiling... no checking for suffix of executables... checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether gcc accepts -g... yes checking for gcc option to accept ISO C89... none needed checking for style of include used by make... GNU checking dependency style of gcc... gcc3 checking for gcc... (cached) gcc checking whether we are using the GNU C compiler... (cached) yes checking whether gcc accepts -g... (cached) yes checking for gcc option to accept ISO C89... (cached) none needed checking dependency style of gcc... (cached) gcc3 checking for gcc option to accept ISO C99... -std=gnu99 checking for bison... bison -y checking for flex... flex checking lex output file root... configure: error: cannot find output from flex; giving up I am using the following environment vars: CXX=g++ CFLAGS=-I/usr/local/include CPPFLAGS=-I/usr/local/include CC=gcc All my open source tools are under /usr/local. I have tried have /usr/ccs/bin in my PATH before /yusr/local/bin, tried having it after, and tried not having it in my PATH at all. Anyone have any ideas? Thx, CC

2 3

Re: [syslog-ng] Small problem with pdbtool and the parser ?
by Ilas, Yann 29 Jul '10

29 Jul '10

Hello, Thanks ! It works... Regards, Yann I. Hi, Please use this patch (fix by Viktor Tusa): diff --git a/src/dbparser/pdbtool.c b/src/dbparser/pdbtool.c index 5ce3e79..cde859a 100644 --- a/src/dbparser/pdbtool.c +++ b/src/dbparser/pdbtool.c @@ -11,6 +11,7 @@ #include <stdio.h> #include <string.h> #include <stdlib.h> +#include <locale.h> #if HAVE_GETOPT_H #include <getopt.h> @@ -618,6 +620,7 @@ main(int argc, char *argv[]) usage(); } + setlocale(LC_ALL,""); if (!g_option_context_parse(ctx, &argc, &argv, &error)) { fprintf(stderr, "Error parsing command line arguments: %s\n", error ? error->message : "Invalid arguments"); M On Wed, 2010-07-28 at 15:39 +0200, Ilas, Yann wrote: > Hello, > > I have a small problem with pdbtool. When I try to match this message "> Message message n?2 message 255", I obtain this error : "Error parsing command line arguments: Invalid byte sequence in conversion input". > The problem seems to be the "?" which is bad interpreted by pdbtool. But the syslog-ng application has not that problem. > > Here is the xml file I used : > <?xml version='1.0' encoding='UTF-8'?> > <patterndb version='3' pub_date='2010-07-28'> > <ruleset name='test' id='ab-cd-123456789'> > <pattern>test</pattern> > <rules> > <rule provider='yann' id='xxxxx:syslog:abcdef:1234567890:id001' class='system'> > <patterns> > <pattern>> Message message n?@NUMBER@ message @NUMBER@</pattern> > <values> > <value name=".classifier.facility">local5</value> > <value name=".classifier.severity">notice</value> > <value name=".classifier.priority">173</value> > </values> > </patterns> > </rule> > </rules> > </ruleset> > </patterndb> > > If I use the pattern "<pattern>> Message message n?@NUMBER@ message @NUMBER@</pattern>", I obtain : > $ /opt/syslog-ng/bin/pdbtool match -p patterndb.xml -P test -M '> Message message n?2 message 255' > Error parsing command line arguments: Invalid byte sequence in conversion input > => It's not OK > > If I use the pattern "<pattern>> Message message nx@NUMBER@ message @NUMBER@</pattern>", I obtain : > (the diffence is the charactere "x" after the " n") > $ /opt/syslog-ng/bin/pdbtool match -p patterndb.xml -P test -M '> Message message nx2 message 255' > MESSAGE=> Message message nx2 message 255 > PROGRAM=test > .classifier.class=system > .classifier.rule_id=xxxxx:syslog:abcdef:1234567890:id001 > .classifier.facility=local5 > .classifier.severity=notice > .classifier.priority=173 > => It's OK > > With syslog-ng and the pattern "<pattern>> Message message n?@NUMBER@ message @NUMBER@</pattern>", I tried to send the same message (with the "?") by using "nc" and it works, ie the message is well interpreted by the application : > $ LANG=C date +"<70>%b %d %R:%S server.domain test[1234]: > Message message n?123 message 456" | nc -4 127.0.0.1 514 > > Why pdbtool can't parse that string ? > > Regards, > > Yann I. >

1 0

[patterndb] more login/logout samples
by Balazs Scheidler 28 Jul '10

28 Jul '10

Hi, Just as I wrote in the other thread, I'm almost finished with marking up the log samples that you posted here on this list. This means that we can keep the ball rolling. I guess there are other services running on your computers doing authentication, so please go ahead and email bomb me with further samples. :) Also, I'd appreciate some review on the markup, and some testing with real life data. See my blog post about "grep on steroids" to see how you can immediately use the stuff finished so far. -- Bazsi

1 0

patterndb: collect login/logout samples
by Balazs Scheidler 28 Jul '10

28 Jul '10

Hi, After getting the generic patterndb policy into shape, I'd like to start collecting log samples, preferably in a domain that is useful for everyone. My target is at first is login/logout/login failure events. I'd start with a generic Linux installation and try to cover all applications that perform authentication. As a starter, I've commited access/sshd.pdb, containing three rules for OpenSSH login/logout/login failure events. I'd head towards standard services, ftp, pop3 and imap authentication, using their "default" implementation in Ubuntu/Debian. (if there's no default, I'll just pick one at random). If any of you can collect these 3 samples of any of the applications that you run daily on your system and submit them here, it'd be tremendous use and would be appreciated. The format of the submission would be preferred in patterndb format (see the ssh sample I've just pushed), but if you are afraid of that, even simple samples would be useful, I'll do the markup myself. -- Bazsi

9 27

Small problem with pdbtool and the parser ?
by Ilas, Yann 28 Jul '10

28 Jul '10

Hello, I have a small problem with pdbtool. When I try to match this message "> Message message n°2 message 255", I obtain this error : "Error parsing command line arguments: Invalid byte sequence in conversion input". The problem seems to be the "°" which is bad interpreted by pdbtool. But the syslog-ng application has not that problem. Here is the xml file I used : <?xml version='1.0' encoding='UTF-8'?> <patterndb version='3' pub_date='2010-07-28'> <ruleset name='test' id='ab-cd-123456789'> <pattern>test</pattern> <rules> <rule provider='yann' id='xxxxx:syslog:abcdef:1234567890:id001' class='system'> <patterns> <pattern>> Message message n°@NUMBER@ message @NUMBER@</pattern> <values> <value name=".classifier.facility">local5</value> <value name=".classifier.severity">notice</value> <value name=".classifier.priority">173</value> </values> </patterns> </rule> </rules> </ruleset> </patterndb> If I use the pattern "<pattern>> Message message n°@NUMBER@ message @NUMBER@</pattern>", I obtain : $ /opt/syslog-ng/bin/pdbtool match -p patterndb.xml -P test -M '> Message message n°2 message 255' Error parsing command line arguments: Invalid byte sequence in conversion input => It's not OK If I use the pattern "<pattern>> Message message nx@NUMBER@ message @NUMBER@</pattern>", I obtain : (the diffence is the charactere "x" after the " n") $ /opt/syslog-ng/bin/pdbtool match -p patterndb.xml -P test -M '> Message message nx2 message 255' MESSAGE=> Message message nx2 message 255 PROGRAM=test .classifier.class=system .classifier.rule_id=xxxxx:syslog:abcdef:1234567890:id001 .classifier.facility=local5 .classifier.severity=notice .classifier.priority=173 => It's OK With syslog-ng and the pattern "<pattern>> Message message n°@NUMBER@ message @NUMBER@</pattern>", I tried to send the same message (with the "°") by using "nc" and it works, ie the message is well interpreted by the application : $ LANG=C date +"<70>%b %d %R:%S server.domain test[1234]: > Message message n°123 message 456" | nc -4 127.0.0.1 514 Why pdbtool can't parse that string ? Regards, Yann I.

2 1

sql() and pgsql
by John Kristoff 28 Jul '10

28 Jul '10

Hello, First thanks to Bazsi and co. for years of great work on syslog-ng. I've used it off and on for some time and have always found it to be well implemented and supported. I've experimented a bit with rsyslog since some distros have been making it their new default and found syslog-ng to be easier to work with and more reliable for my tasks. I'm been playing around with v3.0.8 and the sql() / pgsql output driver to insert log data into a specific table on a specific schema. I couldn't figure out how to specify the schema and maybe its just not supoprted. For instance, if the following is defined: database: foo schema: bar table: baz but the PostgreSQL search path for the user doesn't include bar, then syslog-ng will attempt to create a table in the default schema (usually 'public'). I tried setting the table name to "bar.baz", but that just created a table named bar_baz in the default schema. I did a quick check in the source code and didn't see where this was occurring, maybe its a function of a third party library? If schema support was available that would be nice. For posterity sake and maybe a FAQ, the work around is to alter the search_path for the user (e.g. ALTER ROLE user SET search_path = bar,public;). On a related note, it might be nice to have an option to avoid creating a table if it does not already exist. If nothing else, this helps limit the permissions granted to the syslog-ng db user, which would be appreciated. Likewise, it might also be nice to have an option that can disable checking if the table exists (the "SELECT * FROM %s WHERE 0=1" query) to further lock down user permissions. John

3 3

config file parsing bugs in syslog-ng 3.0.7
by Matthew Hall 28 Jul '10

28 Jul '10

Hello list, In the course of my work I discovered two small defects in syslog-ng syslog-ng 3.0.7. Since I am not familiar enough with the code to patch these I am trying to do the best I can by reporting them to the experts. There are workarounds for these. I am in the process of doing some other work on the product which I would like to contribute back in the near future after I figure out how the open-source contribution process works at this new company I joined. Thank you for providing this product. I am impressed with the documentation and all of the capabilities and I look forward to becoming more expert with it over the coming few months. Best Regards, Matthew Hall. 1) specifying the flags directive in a log stanza other than in the last position causes a syntax error This specific format works: log { filter(filter_local1); destination(dst_local1); parser(parse_switch); flags(final); }; Any format like this raises an error: log { filter(filter_local1); destination(dst_local1); flags(final); parser(parse_switch); }; Is there some reason why or just a parser glitch? 2) using a version directive newer than the daemon version has unexpected behavior If I feed 3.0 a file marked with @version: 3.1 tag, I get this error: Configuration file has no version number, assuming syslog-ng 2.1 format. Please add @version: maj.min to the beginning of the file; I think it does not make sense to fall back to the oldest format if the file has a newer version tag. Would it be better to try to process the file as the newest version with a warning, or print an error that the file is too new for 3.0 to read?

2 1