[syslog-ng] logic and duplicate suppression

[Balazs Scheidler]

Hi,

Sorry for taking such an awful long time to respond, but better late
than never, so here it comes.

On Thu, 2010-08-05 at 16:44 -0500, John Kristoff wrote:
> On Mon, 02 Aug 2010 16:29:39 +0200
> Balazs Scheidler <bazsi at balabit.hu> wrote:
> 
> > can you describe what 'generic' application level events these do
> > describe? For example, user login/logout are described using the
> > "usracct" schema, which defines which name-value pairs need to be
> > marked in the incoming log message. Does this idea apply to here as
> > well?
> 
> Without knowing what the choices are and what the goals are, maybe they
> are both under a DNS or more generic netinfo schema?  The drg.lamer
> pattern identifies a lame delegation.  They are both informational as
> the prefix tags suggest.  In the generic sense, maybe renaming the
> LAMER part of the name to a generic DNS tag would be appropriate?

Now that I think of it, the DNS query portion is quite simple: it logs
the contents of the DNS query and probably the same parameters would
probably be present in all DNS server logs, thus I just have to decide
the naming policy to be used on "transaction logs in general".

If we had a mail server and we'd want to normalize SMTP transactions,
how would they be called?

I guess "smtptxn" for SMTP transaction would be a good name, right? In
that way your DNS transactions (= query logs) would need to be called
"dnstxn", how does that sound to you?

Also, lame delegation is not a query, right? (I'd really need to refresh
my memories about lame delegation, it's been a while since I last ran my
own DNS server) If I understand the pattern/log message correctly, lame
delegation would happen if my bind instance would try to resolve because
recursion was requested, and the respond received from the upstream DNS
server was bogus. Is this right?

If I'm right, it should go under "dnslame".

> 
> > > In the case of the query pattern, being able to set a MACRO based on
> > > the presence of a flag (e.g. if FLAGS =~ /\+/ then RD=1 else RD=0).
> > 
> > I don't understand this, can you elaborate please?
> 
> A ISC BIND query log message may contain the following flags appended
> onto the log message:
> 
>   flag | description
>   --------------------------
>     +  |  recursion desired
>     -  |  recursion not requested
>     S  |  signed query
>     E  |  EDNS options in use
>     T  |  TCP in use
>     D  |  DNSSEC OK set
>     C  |  checking disabled
> 
> I was thinking of a way to set a macro based on the presence of a
> particular flag.  For for instance, if the following logs appear:
> 
>   client 127.0.0.1#49152: query: www.example.org IN A +
>   client 192.0.2.1#49152: query: www.example.org IN A +E
>   client 2001:DB8::1#49152: query: www.example.org IN A +SE
> 
> In any case, I was thinking if I could set ${DNS.RECURSION} = 1 that
> would be nice unless there is a better, more efficient way within the
> existing capabilities.

Hmm.. flag like stuff is not easy to parse using patterndb, it is not
very convinient, but creating all the possible patterns will not cause
it to be parsed too much slower, however if all flags are independent
that'd mean 2^7 = 128 rules.

One way to do that, I'd suggest to parse the whole field as text, put it
in a database and filter on that at query time.

An alternative is to use rewrite() rules with conditions:

rewrite r_dns_flags { set("dnstxn.recursion", "1" condition(match("+"
type(string) value("dnstxn.flags"))); };

Another alterantive would be in the future is to use the
"template-functions" idea and extend syslog-ng with a specific function.
However I didn't finish with that yet:

in a template you'll be able to call functions, like this:

template("$(function $value)");

For example:

<value name="dnstxn.recursion">$(substring-present-p $value +))</value>

where "substring-present-p" would be a function that returns either "0"
or "1" depending on whether the $value contains a substring or not.

-- 
Bazsi