[syslog-ng] new db-parser() utility: pdbtool

[ILLES, Marton]

Hi,

As I wrote in my previous post I kept myself busy with
patterndb/db-parser enhancements. Recently I added a small utility
called pdbtool to help working with pattern databases. Creating,
maintaining and especially troubleshooting patterns are probably not the
most easy task if you try it with the first time. pdbtool is a
collection of small utilities bounded into one binary similar to samba's
net utility. You can find pdbtool in my 3.1 syslog-ng git repo
(http://git.balabit.hu/?p=marci/syslog-ng-3.1.git;a=summary), hopefully
Bazsi is gonna merge it soon mainline.

MATCH
The first command included in pdbtool is called "match" which can be
used to test match rules in a pattern database. Debuging or
troubleshooting patterns was somewhat complicated as you had to feed the
messages into syslog-ng to see if a given pattern matches a message. The
match command does the same. It tries to match a given message against
patterndb rules and also evaluates parser while extracting parsed part
of the message.

To test rules simple rune pdbtool with the message and optionally with
the program name to test:

$ pdbtool match -p patterndb.xml -P sshd -M "Accepted publickey for marci from 127.0.0.1 port 59357 ssh2"


match returns with 0 or 1 return code and prints the assigned values as
well:

.classifier.rule_id=fc44a9fe-75fd-11dd-9bba-001e6806451b
.classifier.class=system


I also plan to add some more verbose debug output if you specify debug
option, so you can see how the given message is matched against the
tree. Till than you can still test and debug your ruleset.

DUMP
The dump command can be used to dump the radix tree built from the
patterns. This way you can see how your patterns are represented in
syslog-ng and it might also help you tracking down problems with
patterns. The dump utility can dump the tree used for matching the
PROGRAM or the MSG parts.

To see the patterns for the 'sshd' PROGRAM one should run the following
command:

$ pdbtool dump -p patterndb.xml  -P 'sshd'


The output looks like this (just part of the whole output):

   'p'
    'assword for'
     @QSTRING:@
      'from'
       @QSTRING:@
        'port '
         @NUMBER:@ rule_id='fc49054e-75fd-11dd-9bba-001e6806451b'
          ' ssh' rule_id='fc55cf86-75fd-11dd-9bba-001e6806451b'
           '2' rule_id='fc4b7982-75fd-11dd-9bba-001e6806451b'
    'ublickey for'
     @QSTRING:@
      'from'
       @QSTRING:@
        'port '
         @NUMBER:@ rule_id='fc4d377c-75fd-11dd-9bba-001e6806451b'
          ' ssh' rule_id='fc5441ac-75fd-11dd-9bba-001e6806451b'
           '2' rule_id='fc44a9fe-75fd-11dd-9bba-001e6806451b'


MERGE
The third command "merge" can be used to convert and merge pattern
databases into one big file which can be loaded into syslog-ng. It makes
sense to store the patterns in separate files per applications or on
other grouping bases, however they need to be merged as syslog-ng can
utilize only one file per db-parser() instance.

My idea is to store patterns in different files along with the syslog-ng
configuration and merge them automatically (or manually) into one big
file which is referenced from syslog-ng configuration. By default
syslog-ng reads patterndb from the /var/lib/syslog-ng/patterndb.xml
(depending on the compile time parameters) file which can be the output
of a merge of the separate patterndb files.

Having separate files help maintaining and troubleshooting patterns and
also helps distributing or moving patterns from one node to an other.

Besides merging the patterndb files the "merge" command takes care of
converting/upgrading of older patterndb to the latest version (as of
this writing version 3). This way you can easily use your old patterns
with newer syslog-ng versions without any problem. Of course it might
make sense to convert them with the tool as some features (like values,
tags etc.) are only provided by newer versions. (The tool can be used to
merge only 1 file, which case it upgrades it only if necessary.)


I still need to create a man page for the utility as probably that is
the first source of information most people is looking at. I hope you
find the tool useful and as always any feedback, comment is very
welcomed.

Marton
-- 
Key fingerprint = F78C 25CA 5F88 6FAF EA21 779D 3279 9F9E 1155 670D