[syslog-ng] Bazsi's blog: Syslog-ng correllation

[Martin Holste]

This is powerful stuff!  I'm really looking forward to exploring all
of the ways it can be used.  The Postfix usage example is a great one.
 One clarification: is $sessionid autogenerated by Syslog-NG, or do we
have to create it ourselves using tuples from the messages?  I'm
assuming it'll use something akin to the $SEQNUM macro.

One other question, can you join an entire preceding message by using
something like ${MSG}@0?  If so, it would be great if there were a
built-in to say all preceding messages, like ${MSG}@-1 or @ALL or
something, but I guess that's getting a little more app-level than I'd
prefer.

On Wed, Sep 29, 2010 at 3:26 AM, Balazs Scheidler <bazsi at balabit.hu> wrote:
>
> Syslog-ng correllation
>
> I think we’ve reached an important milestone with syslog-ng: log message
> correllation was added to db-parser(). As you probably know dbparser and its
> sister project patterndb is able to transform unstructured syslog messages
> into a normalized format: the human readable string content becomes a set of
> name-value pairs. The problem is that in a lot of cases messages miss one or
> two details that would really be needed to understand them and this
> information usually comes in a followup message.
>
> For example: one message in postfix logs contain the sender address and
> while the recipient information comes in the next message. It is trivial to
> understand that in reality most cases you want the information in
> sender,recipient pairs. Another example is sshd, where the authentication
> failure comes in one and the exact reason for the failure comes in the next.
>
> Currently what you can do with syslog-ng is to put the separate messages
> into two SQL tables and join them at query time. This gets ugly quite fast:
> increased storage needs, the hassle with managing two tables instead of one
> and not to mention the increase of the time needed to query the database.
> Sometimes the sole reason for creating SQL tables in this case is to perform
> the correllation, otherwise you’d be happier with a CSV file.
>
> And that’s what became possible now with the latest git commit of syslog-ng
> 3.2. The idea is simple: when a patterndb rule matches, you can tell
> syslog-ng to remember that message by adding it to a correllation state.
> This state is identified with information extracted from the message making
> it a unique session identifier. When the next line comes in you can
> reference the information stored earlier.
>
> Basically the correllation state is a list of log messages associated with a
> session id. To add a new message to this state, you need a store rule:
>
> <rule id=”…”>
> <patterns>
> <pattern>foo session: @STRING:sessionid@, param: @STRING:param@</pattern>
> </patterns>
> <store id=”$sessionid” timeout=”60″/>
> </rule>
>
> The id attribute of the store element specifies a template containing any
> syslog-ng name-value pairs, probably extracted from the current message
> itself.
>
> When the final information comes in you can use the join attribute of the
> values tag:
>
> <rule id=”…”>
> <patterns>
> <pattern>bar session: @STRING:sessionid@</pattern>
> </patterns>
> <values join=”$sessionid”>
> <value name=”param”>${param}@1</value>
> </values>
> </rule>
>
> here the join attribute specifies the session to look up (which must match
> in the two messages), and if there’s a match all messages stored in the
> correllation state becomes available when evaluating the name-value pairs
> associated with the current message.
>
> The key here is the new syntax in the template string “@1″ appended to a
> name-value pair reference. After the “@” character, you can reference a
> message in the correllation state by specifying the index backward from the
> current message. This way @0 is the current message, @1 is the one prior to
> the current one, @2 is before that and so on.
>
> There are more complex ways to use/query the contents of the correllation
> state, but those will appear in a followup post. Stay tuned!
>
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>
>