Thanks for your response. I must be doing something else wrong, because after changing everything to .classifier.rule_id, all my logs are placed in the other.log file. Is there anything else that jumps out at you? Thanks, Nate On Jan 7, 2010, at 1:27 PM, Martin Holste wrote:

It should be ".classifier.rule_id" not ".classifier_rule_id."

By the way, I find I need to use a rewrite template for Snare to normalize the program. I use:

rewrite r_snare { subst("MSWinEventLog.+(Security|Application|System).+", "$1", value("PROGRAM") flags(global)); };

--Martin

On Thu, Jan 7, 2010 at 11:56 AM, Nate Hausrath <hausrath.mailing.list@gmail.com> wrote:

...

Hi everyone,

I've set up a central log server using syslog-ng that receives logs from many different locations on the network. I want to break these logs up into different files so I can using another program to parse and interpret each one individually. So for example, I want my Windows logs to go to /var/log/remote/windows.log, which my ASA logs go to /var/log/remote/asa.log. Anything that doesn't fit into a group should go to /var/log/remote/other.log.

I'm trying to figure out the best way to do this. I was going to create a custom xml file for the db-parser, but I can't seem to get it to work the way I want. Here is my db-parser XML file so far:

<?xml version='1.0' encoding='UTF-8'?> <patterndb version='2' pub_date='2009-12-07'> <ruleset name='capcxml' id='1923-ab2b'> <pattern/> <rules> <rule provider='capc' id='1' class='system'> <description>Detects ASA logs</description> <patterns> <pattern>%ASA</pattern> </patterns> </rule> <rule provider='capc' id='2' class='system'> <description>Detects Windows logs from Snare</description> <patterns> <pattern>MSWinEventLog</pattern> </patterns> </rule> </rules> </ruleset> </patterndb>

So my question is, how can I specify a filter in my syslog-ng.conf file that will allow me to separate logs from these two different rules? Along with that, how can I separate everything that doesn't match?

For instance, I could have something like this:

filter f_class_asa { match("1" value(".classifier_rule_id") type("string") ); };

filter f_class_windows { match("2" value(".classifier_rule_id") type("string") ); };

filter f_class_other { not match("1" value(".classifier_rule_id") type("string")) and not match("2" value(".classifier_rule_id") type("string")); };

log { source(s_remote); parser(p_capc); filter(f_class_asa); destination(df_asa); };

log { source(s_remote); parser(p_capc); filter(f_class_windows); destination(df_windows); };

log { source(s_remote); parser(p_capc); filter(f_class_other); destination(df_other); };

But this doesn't seem to work. Everything gets placed in the df_asa file.

Am I approaching this the wrong way? Should I not use db-parser for this task?

Thanks for any help! Nate ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html

______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html

Thanks again for your response Martin. Unfortunately, it is still not parsing correctly and I am not sure where to go from here. I can try switching to the pattern db 3.0, but where do I find information about the differences? Right now, the ASA logs are being placed in the other.log file, and no other logs are being placed anywhere (even though I have verified they are being received). Just to reiterate, I'm trying to place the Windows logs in a windows.log file, ASA logs in an asa.log file, and everything else in the other.log file. Also, just in case, here is my latest XML file: <?xml version='1.0' encoding='UTF-8'?> <patterndb version='2' pub_date='2009-12-07'> <ruleset name='asa' id='1'> <pattern>%ASA</pattern> <rules> <rule provider='capc' id='1' class='system'> <description>Detects ASA logs</description> <patterns> <pattern></pattern> </patterns> </rule> </rules> </ruleset> <ruleset name='win' id='2'> <pattern>MSWinEventLog</pattern> <rules> <rule provider='capc' id='2' class='system'> <description>Detects Windows logs from Snare</description> <patterns> <pattern></pattern> </patterns> </rule> </rules> </ruleset> </patterndb> And here are the relevant parts of my syslog-ng.conf file: destination df_asa { file("/var/log/remote/asa.log"); }; destination df_windows { file("/var/log/remote/windows.log"); }; destination df_other { file("/var/log/remote/other.log"); }; filter f_class_asa { match("1" value(".classifier.rule_id") type("string") ); }; filter f_class_windows { match("2" value(".classifier.rule_id") type("string") ); }; filter f_class_other { not match("1" value(".classifier.rule_id") type("string")) and not match("2" value(".classifier.rule_id") type("string")); }; parser p_capc{ db-parser( file("/opt/syslog-ng/var/capcdb2.xml") ); }; log { source(s_remote); parser(p_capc); filter(f_class_asa); destination(df_asa); }; log { source(s_remote); parser(p_capc); filter(f_class_windows); destination(df_windows); }; log { source(s_remote); parser(p_capc); filter(f_class_other); destination(df_other); }; It should be relatively straightforward, so I must be overlooking something. I've tested things with pdbtool as well, but it's not giving me the responses I would expect. # pdbtool dump -p /opt/syslog-ng/var/capcdb2.xml -P '%ASA' '' '' rule_id='1' # pdbtool match -p /opt/syslog-ng/var/capcdb2.xml -P "%ASA" -M "Jan 6 13:49:22 10.X.X.X %ASA-5-304001: 10.Y.Y.Y Accessed URL " MESSAGE=Jan 6 13:49:22 10.48.8.111 %ASA-5-304001: 10.48.10.82 Accessed URL PROGRAM=%ASA .classifier.class=unknown Thanks again for any suggestions. -Nate On Fri, Jan 8, 2010 at 3:01 PM, Martin Holste <mcholste@gmail.com> wrote:

The pattern is a bit misleading in the 2.0 pattern db schema, as it means the pattern of the $PROGRAM macro sometimes, and the $MSG macro within a rule element.  I think what you want is this:

<?xml version='1.0' encoding='UTF-8'?> <patterndb version='2' pub_date='2009-12-07'>       <ruleset name='capcxml' id='1923-ab2b'>               <pattern>%ASA</pattern>               <rules>                       <rule provider='capc' id='1' class='system'>                               <description>Detects ASA logs</description>                               <patterns>                                       <pattern></pattern>                               </patterns>                       </rule>               </rules>       </ruleset>       <ruleset>          <pattern>MSWinEventLog</pattern>         <rules>         <rule provider='capc' id='2' class='system'>             <description>Detects Windows logs from Snare</description>              <patterns>               <pattern></pattern>             </patterns>         </rule>       </rules> </patterndb>

You want to consider switching over to the 3.0 patterndb version as it is a bitmore user-friendly in my opinion.  There are 2.0 examples here: http://www.balabit.com/downloads/files/patterndb/1.0-20081117/patterndb/patt... .

--Martin

On Fri, Jan 8, 2010 at 10:45 AM, Nate Hausrath <hausrath.mailing.list@gmail.com> wrote:

...

Thanks for your response.

I must be doing something else wrong, because after changing everything to .classifier.rule_id, all my logs are placed in the other.log file.  Is there anything else that jumps out at you?

Thanks, Nate

On Jan 7, 2010, at 1:27 PM, Martin Holste wrote:

...

It should be ".classifier.rule_id" not ".classifier_rule_id."

By the way, I find I need to use a rewrite template for Snare to normalize the program.  I use:

rewrite r_snare {  subst("MSWinEventLog.+(Security|Application|System).+", "$1", value("PROGRAM") flags(global)); };

--Martin

On Thu, Jan 7, 2010 at 11:56 AM, Nate Hausrath <hausrath.mailing.list@gmail.com> wrote:

...

Hi everyone,

I've set up a central log server using syslog-ng that receives logs from many different locations on the network.  I want to break these logs up into different files so I can using another program to parse and interpret each one individually.  So for example, I want my Windows logs to go to /var/log/remote/windows.log, which my ASA logs go to /var/log/remote/asa.log.  Anything that doesn't fit into a group should go to /var/log/remote/other.log.

I'm trying to figure out the best way to do this.  I was going to create a custom xml file for the db-parser, but I can't seem to get it to work the way I want.  Here is my db-parser XML file so far:

<?xml version='1.0' encoding='UTF-8'?> <patterndb version='2' pub_date='2009-12-07'>        <ruleset name='capcxml' id='1923-ab2b'>                <pattern/>                <rules>                        <rule provider='capc' id='1' class='system'>                                <description>Detects ASA logs</description>                                <patterns>                                        <pattern>%ASA</pattern>                                </patterns>                        </rule>                        <rule provider='capc' id='2' class='system'>                                <description>Detects Windows logs from Snare</description>                                <patterns>                                        <pattern>MSWinEventLog</pattern>                                </patterns>                        </rule>                </rules>        </ruleset> </patterndb>

So my question is, how can I specify a filter in my syslog-ng.conf file that will allow me to separate logs from these two different rules?  Along with that, how can I separate everything that doesn't match?

For instance, I could have something like this:

filter f_class_asa {        match("1"        value(".classifier_rule_id")        type("string")        ); };

filter f_class_windows {        match("2"        value(".classifier_rule_id")        type("string")        ); };

filter f_class_other {        not match("1" value(".classifier_rule_id") type("string"))        and not match("2" value(".classifier_rule_id") type("string")); };

log {        source(s_remote);        parser(p_capc);        filter(f_class_asa);        destination(df_asa); };

log {        source(s_remote);        parser(p_capc);        filter(f_class_windows);        destination(df_windows); };

log {        source(s_remote);        parser(p_capc);        filter(f_class_other);        destination(df_other); };

But this doesn't seem to work.  Everything gets placed in the df_asa file.

Am I approaching this the wrong way?  Should I not use db-parser for this task?

Thanks for any help! Nate ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html

______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html

______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html

______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html

Everything appears to work properly with the patterndb.xml file. Dumping worked fine, and here is what happened when I matched: # ./pdbtool match -p /opt/syslog-ng/var/patterndb.xml -P su -M "+ pts/2 root:nateh" MESSAGE=+ pts/2 root:nateh PROGRAM=su .classifier.class=system .classifier.rule_id=04ba999a-75fe-11dd-9bba-001e6806451b However, when I use my custom XML file with a message that should match, it doesn't work: # ./pdbtool match -p /opt/syslog-ng/var/capcdb2.xml -P "MSWinEventLog" -M "This is the message" MESSAGE=This is the message PROGRAM=MSWinEventLog .classifier.class=unknown Here is the relevant part in the XML: <ruleset name='win' id='2'> <pattern>MSWinEventLog</pattern> <rules> <rule provider='capc' id='2' class='system'> <description>Detects Windows logs from Snare</description> <patterns> <pattern></pattern> </patterns> </rule> </rules> </ruleset> I'm assuming that leaving the <pattern> part blank should cause it to match on anything with "MSWinEventLog", right? Thanks! -Nate On Mon, Jan 11, 2010 at 10:33 AM, SZALAY Attila <sasa@balabit.hu> wrote:

Hi!

On Mon, 2010-01-11 at 09:55 -0500, Nate Hausrath wrote:

...

Right now, the ASA logs are being placed in the other.log file, and no other logs are being placed anywhere (even though I have verified they are being received).  Just to reiterate, I'm trying to place the Windows logs in a windows.log file, ASA logs in an asa.log file, and everything else in the other.log file.

You can try to match a log message with the given pattern ruleset with the pdbtool command.

First try to dump the patterndb with the dump command pdbtool dump -p /opt/ssb/var/db/patterndb.xml -T

Then check the programs:

pdbtool dump -p /opt/ssb/var/db/patterndb.xml -P zcv

After that (if everything is good) try to match a log message:

pdbtool match -p /opt/ssb/var/db/patterndb.xml -P zcv -M "Iam the message part."

Do not forget to set the program with the -P option.

Is the pdbtool found the correct rule?

______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html