[syslog-ng] using correlation to filter out some messages ?

[Balazs Scheidler]

Hi,

Sorry for not answering any sooner, but as this post shows I usually
process my backlog every now-and-then. Matthew and Martin usually helps
a lot of people encountering this stuff, but certainly they may not be
able to help everyone :)

The question itself is great, this is how we can cover with
newer-and-newer problems with syslog-ng functionality. Your use-case is
not simply as db-parser() was never meant to be used to drop messages.

So, here are some ideas how to cope with your problem.

1) you should create a very specific rule for the first message (e.g. it
should contain the IP address as literal string, instead of a parser).

  This will make the db-parser() engine prefer this rule over the one
with a parser reference in it (assuming of course that there's another
similar rule)

  In this first rule, use the following correllation options:

   context-scope="program"
   context-id="lbprobe_${slapd.connection_id}"
   context-timeout="5"   # or the time you'd assume you'll see the second message

  Attach a tag to the message, which you can use to drop the message 
  later on using a filter (e.g. tag "dropthis").

  Then, create a timeout action (e.g. <action trigger="timeout">, which should 
  generate a message identical to the matched one, but this time 
  without the tag which indicates that the message should be dropped.

2) create a rule for the 2nd message

the correllation options of both rules should be the same as the 1st
rule, except that the timeout can be 0.

  Attach a tag to the message, which you can use to drop the message 
  later on using a filter (e.g. tag "dropthis").

  Then, create a matched action (e.g. <action trigger="match">), with 
  a filter condition attached (e.g. condition="filter-expr"). This condition 
  should check that there was is no other message in the correllation state (for 
  example: ${MESSAGE}@1 == ""). The action will be fired, when the closed 
  message is coming from a non-probe client. In this case the content of the action
  should just repeat the original message. (e.g. it should be emitted) without 
  the use of the dropthis tag.

I know this is complicated. It could be made simpler a number of ways.

1) possibly an explicit "drop" action for the main message (makes the explicit 
tag, plus filtering in the configuration unnecessary). This should be 
trivial to do.

2) possibly build some stack based mini-language into the db-parser() 
engine, with stuff like push/pop/apply a surprising number of 
operations can be implemented easily. I'll give a thought to this.

On Wed, 2010-12-29 at 15:58 +0100, Guillaume Rousse wrote:
> Hello.
> 
> I've had a look at message correlation in the 3.2 manual, but it's still
> a bit obscure for me to understand if it could allow me to achieve
> filtering of pairs of log messages, as described here:
> https://lists.balabit.hu/pipermail/syslog-ng/2010-January/013857.html
> 

-- 
Bazsi