[syslog-ng] using correlation to filter out some messages ?

[Martin Holste]

Ok, so the first time the raw message is seen, it is not forwarded
through normally then and has effectively been put in limbo, not
duplicated, right?

On Sat, Jan 15, 2011 at 5:41 AM, Balazs Scheidler <bazsi at balabit.hu> wrote:
> On Fri, 2011-01-14 at 13:18 -0600, Martin Holste wrote:
>> Ok, I'm with you so far on this.  So when this is run, there will be
>> two messages, one which has a tag of "dropthis" and gets filtered out
>> by the f_drop, and the other proceeds normally?  If that's correct,
>> then I don't understand the strategy but I do understand the
>> configuration.
>>
>
> The aim was to drop the lines unless proved to be part of a real
> session. It is part of a real session, if:
>
>  1) the closure message comes 5 seconds after the first one (that's the
> correllation timeout)
>  2) or if the connection request came from a different host from the
> load balancer.
>
> In effect, it'll hold the connection open/close messages until either of
> the above conditions will prove that the message can be written out. But
> in order to do this, we need to drop the 'raw' log message first and
> reproduce it once we know we have to.
>
>> On Fri, Jan 14, 2011 at 5:44 AM, Balazs Scheidler <bazsi at balabit.hu> wrote:
>> > On Thu, 2011-01-13 at 19:44 -0600, Martin Holste wrote:
>> >> > Sorry for not answering any sooner, but as this post shows I usually
>> >> > process my backlog every now-and-then. Matthew and Martin usually helps
>> >> > a lot of people encountering this stuff, but certainly they may not be
>> >> > able to help everyone :)
>> >> >
>> >>
>> >> Ha, this one is well beyond me--I'm also struggling with the correlation stuff.
>> >>
>> >> >  Then, create a timeout action (e.g. <action trigger="timeout">, which should
>> >> >  generate a message identical to the matched one, but this time
>> >> >  without the tag which indicates that the message should be dropped.
>> >> >
>> >>
>> >> This is where you're losing me.  Can you provide a working example of
>> >> the whole thing?
>> >
>> > something like this:
>> >
>> > <action trigger="timeout">
>> >  <message>
>> >    <values>
>> >      <!-- HOST, PROGRAM, PID are filled automatically based on the current message -->
>> >      <value name="MESSAGE">${MSG}@1</value>
>> >    </values>
>> >  </message>
>> > </action>
>> >
>> > while the body of the rule contains this:
>> >
>> > <tags>
>> >  <tag>dropthis</tag>
>> > <tags>
>> >
>> > This tag can then be used to drop messages by a syslog-ng
>> > filter, something like this:
>> >
>> > filter f_drop { tags("dropthis"); };
>> >
>> > parser p_db { db-parser(); };
>> >
>> > log { parser(p_db);
>> >        log { filter(f_drop); flags(final); };
>> >        log { destination(normal-destination); };
>> > };
>> >
>> > Or an equivalent.
>> >
>> > When db-parser() matches a rule, it does the following:
>> >  1) process the current message (matching the rule) by adding name-value pairs and tags as described by the rule
>> >  2) possibly create new messages based on actions associated with the rule.
>> >
>> > The body of the rule (e.g. the contents inside the main <rule></rule> tag)
>> > are applied to the current message. Actions are processed when their
>> > specified trigger is hit, e.g. either on timeout (trigger="timeout") or
>> > at the same time the rule is matched (trigger="match")
>> >
>> > This way a single rule match can in theory result in several messages
>> > to be emitted.
>> >  1) the message coming in and triggering the rule
>> >  2) all the messages generated by associated actions.
>> >
>> >
>> >
>> > --
>> > Bazsi
>> >
>> >
>> > ______________________________________________________________________________
>> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>> > FAQ: http://www.campin.net/syslog-ng/faq.html
>> >
>> >
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.campin.net/syslog-ng/faq.html
>>
>>
>
> --
> Bazsi
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>