[syslog-ng] Logging with db-parser issue

[Balazs Scheidler]

On Thu, 2009-05-07 at 16:26 -0400, Nate Hausrath wrote:
> Hey everyone,
> 
> Sorry to ask what is probably an easy question.
> 
> I'm attempting to use the new db-parser feature in syslog-ng to parse
> Windows events generated by Snare.  What I've attempted to do is write
> a simple XML file that will match one simple rule.  This way I can
> test that everything is functioning properly and I can move on from
> there.  Unfortunately, it's not working, so I'm unsure as to whether
> it is my rule or my installation.  So I have a few basic questions
> that will hopefully clear up my confusion!
> 
> - I did not install syslog-ng from source.  I downloaded the .deb for
> version 3.01 and used that.  Does this even include db-parser? (I'm
> assuming yes because it does not cause an error from the config file
> alone.)
> 
> - My installation is in /opt/syslog-ng, so I've put the example XML
> files (http://www.balabit.com/downloads/files/patterndb/) in
> /opt/syslog-ng/var.  Is this the right location?

the default location for the patterndb file is 
${localstatedir}/patterndb.xml, but you can override that using the file
option, e.g.

db-parser(file(/path/to/patterndb));

the reason it is in var that in the future we assume that this file is
going to be generated from several source files automatically.

> 
> Other than that, here is my syslog-ng.conf file:
> 
> @version: 3.0
> #Default configuration file for syslog-ng.
> #
> # For a description of syslog-ng configuration file directives, please read
> # the syslog-ng Administrator's guide at:
> #
> # http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html
> #
> 
> options {
> };
> 
> ######
> # sources
> source s_local {
>         # message generated by Syslog-NG
>         internal();
>         # standard Linux log source (this is the default place for the syslog()
>         # function to send logs to)
>         unix-stream("/dev/log");
>         # messages from the kernel
>         file("/proc/kmsg" program_override("kernel: "));
> };
> 
> source s_net {
>         udp();
>         tcp();
> };
> 
> parser p_db { db-parser(); };
> 
> ######
> # destinations
> destination d_messages { file("/var/log/messages"); };
> destination d_parsed { file("/var/log/remote/parsed.log"); };
> 
> log {
>         source(s_local);
>         destination(d_messages);
> };
> 
> log {
>         source(s_net);
>         destination(d_parsed);
>         parser(p_db);
> };

the log statements define a 'pipeline', thus your messages will reach
the db-parser() only after having been written to the d_parsed
destination.


> 
> And here is my windows.xml file:
> 
> <?xml version='1.0' encoding='UTF-8'?>
> <patterndb version='2' pub_date='2009-05-07'>
>   <ruleset name='windows'>
>     <pattern>MSWinEventLog</pattern>
>     <rules>
>       <rule provider='nate' id='1' class='system'>
>         <patterns>
>           <pattern>540</pattern>
>         </patterns>
>         <description>This is a terrible terrible message to receive.
> Game over man!  Game over!</description>
>       </rule>
>     </rules>
>   </ruleset>
> </patterndb>
> 
> 
> As you see, to test I just want to match on the number '540' and put
> that log message in /var/log/remote/parsed.log.  I have verified that
> messages that should match this are arriving at the machine when I
> send them.  If i turn off the db-parser, the messages appear in the
> parsed.log file.

All messages should end up in your parsed.log file anyway. In order to
only match those which matched any of the patterns, you need to put
filters into your log statement.

> 
> If there is any documentation that explains the basics of this, sorry
> that I missed it.  I've only been able to find some of the other
> messages on this mailing list and the blog entries on balabit.

Just feel free to ask about it here, I'm trying to give you timely
answers, as other source of information about db-parser() is scarse. In
the documentation you will find information about parsers in general,
and db-parser() is one such parser, basically behaving quite similar to
csv-parser(), except it uses a database to extract information from log
messages.

Also, you can find a working example in my presentation at OSDC:

http://people.balabit.hu/bazsi/slides/osdc-2009-syslog-ng-3.0.odp

-- 
Bazsi