[syslog-ng] few questions about patterndb

[Martin Holste]

Currently, there's not a community repository for pattern-db, but I
believe Balabit is hoping to have one.  That said, it does not sound
like you need to use it for what you're trying to do.  The value of
pattern-db is in fine-tuned classification and also parsing fields out
of the bodies of messages.  For instance, here's a pattern I use to
parse the basic fields in a Snort IDS log message:

<ruleset name="snort" id='8'>
  <pattern>snort</pattern>
  <rules>
   <rule provider="local" class='8' id='8'>
     <patterns>
       <pattern>@QSTRING:sig_sid:[]@
@ESTRING:sig_msg:[@Classification:@QSTRING:sig_classification: ]@
[Priority: @NUMBER:sig_priority:@]: @QSTRING:proto:{}@
@IPv4:srcip:@:@NUMBER:srcport:@ ->
@IPv4:dstip:@:@NUMBER:dstport:@</pattern>
      </patterns>
    </rule>
  </rules>
</ruleset>

This will take effect on messages with a program matching "snort",
assign some meta data (class=8 and id=8) and then extract fields from
the message text.  I can then put this directly into a database with
each field as a column for reporting, etc.

This is a pretty drastic rule.  Most of the example rules provided by
Balabit are demonstrating that messages can be classified based on the
program name and assigned tags, etc. which is basically providing
another way of attaching unlimited meta data as the logs come in.  I'm
using them for parsing because pattern-db is much more efficient than
using regular expressions, so it is necessary for high-volume parsing.

--Martin

On Wed, Jun 23, 2010 at 1:46 AM, Hendrik Pahl <pahl at team-datentechnik.de> wrote:
> Hi folks,
>
> i'm currently testing the patterndb with syslog-ng. After a few test
> runs,there are some open questions.
>
> Will there be some future releases of the patterndb by balabit? Last
> Snapshot has a timestamp from sep. 2009. Or is there a community
> somewhere out there, that i haven't seen yet?:-)
>
> I tested the classification into different logfiles
> (system,violation,unknown) with 1.5M lines of logfiles, and there was
> just one rule matching 'em all:
> 734adfc8-8a4d-40ad-923d-9d3fbabda34b, which is something about a newly
> connected usb mass storage device. This behaviour can't be correct?
>
> The patternfile "system-apache2.xml" still containts just plain
> messages, no XML anywhere - i thougt this should have been fixed yet?
>
>
> After some testing with logging messages to file like
> "messages-system/messages-violation/messages-unknown" i'm just asking
> myself about the profit i get when using patterndb. With a standard
> syslog-ng i usually can log things into files divided by the priority
> of the messages  - that's all i want it to do, just have a logfile
> where all errors und failures are logged, just to get the pure number
> of lines down. When controlling a logfile i usually don't really care
> about a cronjob to be successful. What is the profit of using
> patterndb right there?
>
> Thanks in advance,
>
>
>
> i.A. Hendrik Pahl
> System Engineering
>
> team! datentechnik GmbH & Co.KG
> Werner von Siemens Straße 12a
> 49124 Georgsmarienhuette
> Tel.: +49 (0)5401-8226-50
> Fax : +49 (0)5401-8226-55
>
> E-Mail: pahl at team-datentechnik.de
> Internet: www.team-datentechnik.de
> HRA 110397, Amtsgericht Osnabrück
> Geschäftsführung: Reemt Lükenga
>
> Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen.
> Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich
> erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie
> diese E-Mail. Vielen Dank.
>
> Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail ist nicht
> gestattet.
>
> This e-mail contains confidential and/or privileged information. If
> you are not
> the intended recipient (or have received this e-mail in error) please notify
> the sender and delete this message.
> Thank you.
>
> Any unauthorized copying, disclosure or distribution of the material in this
> e-mail is strictly forbidden.
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>