[syslog-ng] patterndb repositoy and best practices

[Fekete Róbert]

Hi Evan, 

First of all, thank you very much for your lots of useful feedback and comments on patterndb, and our apologies for the slow reply (year-end rush hit us early this year). I try to answer your questions where I can, and hope that Bazsi'll find some time to correct me and answer the rest. See my answers inline.

Regards, 

Robert
 
On Friday, November 25, 2011 19:55 CET, Evan Rempel <erempel at uvic.ca> wrote: 
 
> We are preparing to leverage the patterndb functionality in a very big way.
> A few questions before I jump.
> 
> 1. Is there a public repository of pattern databases?
> 
Yes, though the number of patterns is rather limited. The current repository is available here: http://git.balabit.hu/?p=bazsi/syslog-ng-patterndb.git

Some Windows patterns that are not yet in this repository are available here: http://people.balabit.hu/czanik/patterndb-win2k8.xml

A bunch of logcheck rules have been also converted to patterndbs, but from what I have heard the results were not very good, but may or may not be useful for a start (http://www.balabit.com/downloads/files/patterndb-snapshot/).

> 2. Is there some registry for creating the uniq IDs for rules and rule sets?
>     For example, is there anything that prevents me from creating a rule with
>     a duplicate rule ID that would result in an ID collision when merged
>     with some other patterndb author?
No, but if you use some kind of hash, the chance for collision should be pretty low. For example, if you store your patterns under version control (git, whatever), the hash of the parent commit will probably do just fine.

> 
> 3. Is there any registry for tag names, or key value pair names so that
>     tags that I place on a message will be usable by others syslog-ng
>     configuration filters?
> 
Originally we started to develop a schema for the tags, you can still find it in the SCHEMAS.txt of the repository. Then we heard that MITRE is working (among others) on a standard description for log messages (called Common Event Expression, CEE (http://cee.mitre.org)), and hoped that we could use it instead of reinventing the wheel. Unfortunately, the development of CEE is going slow, so we are stuck in the middle: having stopped to create an own schema so we can use a standard one that'll become more widespread, but that is not here yet. Once a working CEE specification is out, we will convert the tagging of our currently published patterns (which use our own tagging schema) to the CEE schema, so (strictly IMHO) if you use our schema, we will be able to give you at least a mapping on how to convert the tags to the CEE standard.

> 4. Are there any best practices for tag names or key names to provide
>     any kind of grouping? For example;
> 
>     user.name
>     user.uid
>     user.gid
>     host.address.ipv4
>     host.address.ipv6
>     host.name
> 
> 
See the above comment.
> 
> Any pointers or discussion will be helpful because we are looking at producing
> a complete "artificial ignorance" infrastructure for our entire organization,
> from linux, research compute clusters, network gear, Windows hosts,
> web hosting, database services etc.
> 
That sounds really exciting (and a whole lot of work). Please keep us posted about our experiences, both good and bad.

Robert

> Thanks for your time.
> 
> Evan
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> 
>