[syslog-ng] logic and duplicate suppression

[John Kristoff]

On Thu, 29 Jul 2010 10:03:37 +0200
Balazs Scheidler <bazsi at balabit.hu> wrote:

> I had a similar idea for a while and as an incentive for you to try
> the latest-greatest stuff, I've implemented it in OSE 3.2:

Thank you, thought I can't promise I'll try it in the immediate
future.  By way of another example for the sort of thing I was thinking
of here are a couple of patterndb rules for BIND 9.x messages:

 <ruleset id='c7a73d172adcd7c5' name='named'>
    <pattern>named</pattern>
    <rules>
      <rule id='3409bd184b4dd59c' provider='drg.lamer' class='info'>
        <description>BIND 9.x lame server - see lib/dns/resolver.c</description>
        <patterns>
          <pattern>lame-servers: info: @ESTRING:LAMER.REASON: resolving@ '@ESTRING:LAMER.QNAME:/@@STRING:LAMER.TYPE@/@STRING:LAMER.CLASS@': @IPvANY:LAMER.SADDR@#@NUMBER@</pattern>
          <pattern>lame-servers: info: lame server resolving @QSTRING:LAMER.QNAME:'@ (in @QSTRING:LAMER.ZONE:'@?): @IPvANY:LAMER.SADDR@#@NUMBER@</pattern>
        </patterns>
        <examples>
          <example>
            <test_message>lame-servers: info: unexpected RCODE (REFUSED) resolving 'ns1.example.org/AAAA/IN': 192.0.2.1#53</test_message>
            <test_message>lame-servers: info: lame server resolving '1.2.0.192.in-addr.arpa' (in '2.0.192.in-addr.arpa'?): 192.2.0.1#53</test_message>
          </example>
        </examples>
      </rule>

      <rule id='2531e2f1a9304259' provider='drg.query' class='info'>
        <description>BIND 9.x query</description>
        <patterns>
          <pattern>queries: info: client @IPvANY:QUERY.SADDR@#@NUMBER:QUERY.SPORT@: view @STRING:QUERY.VIEW@: query: @ESTRING:QUERY.QNAME: @@STRING:QUERY.CLASS@ @STRING:QUERY.TYPE@ @ANYSTRING:QUERY.FLAGS@</pattern>
        </patterns>
        <examples>
          <example>
            <test_message>queries: info: client 192.0.2.1#1024: view external: query: . IN NS -</test_message>
            <test_message>queries: info: client 127.0.0.1#1024: view loopback: query: 1.0.0.127.in-addr.arpa IN PTR +</test_message>
          </example>
        </examples>
      </rule>

    </rules>
  </ruleset>

I'll happily take improvements to the above patterns.  Do note, the
prefix strings (e.g. 'queries: info: ' and 'lame-servers: info: ') will
only be present if BIND print-severity and print-category options are
set.  A better set of named rules to account for various combinations of
those options may be desirable.  Likewise, to account for the existence
or lack of using views for queries.

In the case of the query pattern, being able to set a MACRO based on
the presence of a flag (e.g. if FLAGS =~ /\+/ then RD=1 else RD=0).

> This is more difficult. The sane way of doing this is to keep state
> on a per-host basis, which is the area of correllation. Of course
> this is on the radar for syslog-ng, but we're not there yet.

I'm already doing as Martin suggested (thanks for your reply Martin)
and will just log everything and optimize queries and reporting.  If
really needed, I could post process entries in the database to keep
only the first and last seen log of interest.  Though 

Thanks again,

John