[syslog-ng] [patterndb] classification

[Balazs Scheidler]

Hi,

As you probably know one goal for patterndb is to implement message
classification.

E.g. in addition to extracting information from log messages, it also
associates a "class", later available in the "${.classifier.class}"
value.

Right now, syslog-ng doesn't really care what this string is. But the
XML schema validating patterndb file lists the following four classes
(taken from the logcheck project)

violation     - security violation
security      - other security events
system        - system information
unknown       - no rule matches

One one hand, the tagging functionality (e.g. the ability to also
associate tags with each message) is superior to classes. 

On the other hand, all tags are equivalent, thus if a message has 5
tags, then currently syslog-ng only provides functions to _filter_ based
on tags, but not use it as a macro.

So for example it is possible to do:

file d_class_files { file("/var/log/messages.${.classifier.class}.log"); };

But it is difficult to do with tags (except for using filters and 
different destinations), as there's no such functionality. Another problem 
is that tags/classes are completely independent, in order to filter on the
class of the message, one would have to use a match() filter like this:

filter f_class { match("violation" value(".classifier.class")); };

My conclusion is that classes are better when used in templates, 
tags are better when filtering. The two should be merged somehow.

So I'm thinking on how to move forward. Here are the alternatives 
I'm considering:

1) the class of the message is always a tag in some generated format (e.g.
if a message has class XXX, then a tag named ".class.XXX" would 
be automatically associated with the message.

This is somewhat cumbersome.

2) the class of the message is created as a tag as well, with
the same name as the class.

e.g. we'd have a tag named "violation", but that'd preclude the use of the 
"violation" name as a tag.

3) drop the class stuff and implement a macro trick that 
makes it possible to use tags in macro context

One way to do this:

file d_class_files { file("/var/log/messages.$(expand-tag-name violation security system unknown).log"); };

The "expand-tag-name" macro function would try to look for the tags 
listed as parameters, and if the message matches it'd expand to the
tagname.

This is not intuitive and if someone wants to use such an expansion 
in a lot of templates, it is also irritating and difficult to 
get right.

On an independent matter, the set of classes may need some thought. As 
I said the original list is borrowed from logcheck, but I think it 
probably needs to be expanded. Last time I got patterns for 
DNS queries, and although I could shove them into "system", right now I 
feel that the point of classification is to categorize events by 
"importance", in a similar spirit to syslog severity, but one that works
even if the application developer uses a bogus severity when sending 
syslog messages.

So one email, two questions, feedback appreciated.
Thanks.

-- 
Bazsi