Re: [syslog-ng] Separating Remote Logs

11 Jan 2010

      Thanks again for your response Martin.  Unfortunately, it is still not
parsing correctly and I am not sure where to go from here.  I can try
switching to the pattern db 3.0, but where do I find information about
the differences?

Right now, the ASA logs are being placed in the other.log file, and no
other logs are being placed anywhere (even though I have verified they
are being received).  Just to reiterate, I'm trying to place the
Windows logs in a windows.log file, ASA logs in an asa.log file, and
everything else in the other.log file.

Also, just in case, here is my latest XML file:

<?xml version='1.0' encoding='UTF-8'?>
<patterndb version='2' pub_date='2009-12-07'>
    <ruleset name='asa' id='1'>
        <pattern>%ASA</pattern>
        <rules>
            <rule provider='capc' id='1' class='system'>
                <description>Detects ASA logs</description>
                <patterns>
                    <pattern></pattern>
                </patterns>
            </rule>
        </rules>
    </ruleset>
    <ruleset name='win' id='2'>
        <pattern>MSWinEventLog</pattern>
        <rules>
            <rule provider='capc' id='2' class='system'>
                <description>Detects Windows logs from Snare</description>
                <patterns>
                    <pattern></pattern>
                </patterns>
            </rule>
        </rules>
    </ruleset>
</patterndb>

And here are the relevant parts of my syslog-ng.conf file:

destination df_asa { file("/var/log/remote/asa.log"); };
destination df_windows { file("/var/log/remote/windows.log"); };
destination df_other { file("/var/log/remote/other.log"); };

filter f_class_asa {
        match("1"
        value(".classifier.rule_id")
        type("string")
        );
};

filter f_class_windows {
        match("2"
        value(".classifier.rule_id")
        type("string")
        );
};

filter f_class_other {
        not match("1" value(".classifier.rule_id") type("string"))
        and not match("2" value(".classifier.rule_id") type("string"));
};

parser p_capc{
        db-parser( file("/opt/syslog-ng/var/capcdb2.xml") );
};

log {
        source(s_remote);
        parser(p_capc);
        filter(f_class_asa);
        destination(df_asa);
};

log {
        source(s_remote);
        parser(p_capc);
        filter(f_class_windows);
        destination(df_windows);
};

log {
        source(s_remote);
        parser(p_capc);
        filter(f_class_other);
        destination(df_other);
};

It should be relatively straightforward, so I must be overlooking something.

I've tested things with pdbtool as well, but it's not giving me the
responses I would expect.

# pdbtool dump -p /opt/syslog-ng/var/capcdb2.xml -P '%ASA'
''
 '' rule_id='1'

# pdbtool match -p /opt/syslog-ng/var/capcdb2.xml -P "%ASA" -M "Jan  6
13:49:22 10.X.X.X %ASA-5-304001: 10.Y.Y.Y Accessed URL "
MESSAGE=Jan  6 13:49:22 10.48.8.111 %ASA-5-304001: 10.48.10.82 Accessed URL
PROGRAM=%ASA
.classifier.class=unknown

Thanks again for any suggestions.

-Nate

On Fri, Jan 8, 2010 at 3:01 PM, Martin Holste <mcholste@gmail.com> wrote:
...
The pattern is a bit misleading in the 2.0 pattern db schema, as it
means the pattern of the $PROGRAM macro sometimes, and the $MSG macro
within a rule element.  I think what you want is this:
<?xml version='1.0' encoding='UTF-8'?>
<patterndb version='2' pub_date='2009-12-07'>
      <ruleset name='capcxml' id='1923-ab2b'>
              <pattern>%ASA</pattern>
              <rules>
                      <rule provider='capc' id='1' class='system'>
                              <description>Detects ASA logs</description>
                              <patterns>
                                      <pattern></pattern>
                              </patterns>
                      </rule>
              </rules>
      </ruleset>
      <ruleset>
         <pattern>MSWinEventLog</pattern>
        <rules>
        <rule provider='capc' id='2' class='system'>
            <description>Detects Windows logs from Snare</description>
             <patterns>
              <pattern></pattern>
            </patterns>
        </rule>
      </rules>
</patterndb>
You want to consider switching over to the 3.0 patterndb version as it
is a bitmore user-friendly in my opinion.  There are 2.0 examples
here: http://www.balabit.com/downloads/files/patterndb/1.0-20081117/patterndb/patt...
.
--Martin
On Fri, Jan 8, 2010 at 10:45 AM, Nate Hausrath
<hausrath.mailing.list@gmail.com> wrote:
...
Thanks for your response.
I must be doing something else wrong, because after changing everything to .classifier.rule_id, all my logs are placed in the other.log file.  Is there anything else that jumps out at you?
Thanks,
Nate
On Jan 7, 2010, at 1:27 PM, Martin Holste wrote:
...
It should be ".classifier.rule_id" not ".classifier_rule_id."
By the way, I find I need to use a rewrite template for Snare to
normalize the program.  I use:
rewrite r_snare {
 subst("MSWinEventLog.+(Security|Application|System).+", "$1",
value("PROGRAM") flags(global));
};
--Martin
On Thu, Jan 7, 2010 at 11:56 AM, Nate Hausrath
<hausrath.mailing.list@gmail.com> wrote:
...
Hi everyone,
I've set up a central log server using syslog-ng that receives logs from many different locations on the network.  I want to break these logs up into different files so I can using another program to parse and interpret each one individually.  So for example, I want my Windows logs to go to /var/log/remote/windows.log, which my ASA logs go to /var/log/remote/asa.log.  Anything that doesn't fit into a group should go to /var/log/remote/other.log.
I'm trying to figure out the best way to do this.  I was going to create a custom xml file for the db-parser, but I can't seem to get it to work the way I want.  Here is my db-parser XML file so far:
<?xml version='1.0' encoding='UTF-8'?>
<patterndb version='2' pub_date='2009-12-07'>
       <ruleset name='capcxml' id='1923-ab2b'>
               <pattern/>
               <rules>
                       <rule provider='capc' id='1' class='system'>
                               <description>Detects ASA logs</description>
                               <patterns>
                                       <pattern>%ASA</pattern>
                               </patterns>
                       </rule>
                       <rule provider='capc' id='2' class='system'>
                               <description>Detects Windows logs from Snare</description>
                               <patterns>
                                       <pattern>MSWinEventLog</pattern>
                               </patterns>
                       </rule>
               </rules>
       </ruleset>
</patterndb>
So my question is, how can I specify a filter in my syslog-ng.conf file that will allow me to separate logs from these two different rules?  Along with that, how can I separate everything that doesn't match?
For instance, I could have something like this:
filter f_class_asa {
       match("1"
       value(".classifier_rule_id")
       type("string")
       );
};
filter f_class_windows {
       match("2"
       value(".classifier_rule_id")
       type("string")
       );
};
filter f_class_other {
       not match("1" value(".classifier_rule_id") type("string"))
       and not match("2" value(".classifier_rule_id") type("string"));
};
log {
       source(s_remote);
       parser(p_capc);
       filter(f_class_asa);
       destination(df_asa);
};
log {
       source(s_remote);
       parser(p_capc);
       filter(f_class_windows);
       destination(df_windows);
};
log {
       source(s_remote);
       parser(p_capc);
       filter(f_class_other);
       destination(df_other);
};
But this doesn't seem to work.  Everything gets placed in the df_asa file.
Am I approaching this the wrong way?  Should I not use db-parser for this task?
Thanks for any help!
Nate
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html