[syslog-ng] destination based on custom field question

[Balazs Scheidler]

On Thu, 2011-01-06 at 16:52 -0800, Matthew Hall wrote:
> On Thu, Jan 06, 2011 at 06:01:13PM -0500, Christopher Barry wrote:
> > Greetings,
> > 
> > I'm new to syslog-ng, and I'm trying to key off of a custom field in a
> > log entry, and put the message in a particular directory named for the
> > field.
> > 
> > a sample log entry would look like this:
> > Dec 16 14:08:51 u910-05 testapp: \
> > 00000000000000a7:00007f62d170a910:DEBUG :part.cpp     :  183: \ |
> > PartitionInfo [0x275f720]
> > 
> > '\' denotes line continuation.
> > 
> > The field I want to key off in this line is:
> > 00000000000000a7
> > 
> > I created a filter to only get stuff from 'testapp', but now I want to
> > make the destination be tied to the field. The field is positional, but
> > can have any value, and cannot be known a priori.
> > 
> > Not sure how to go about it. Any links to examples or where to start
> > would be very much appreciated.
> > 
> > --
> > Thanks,
> > -Christopher
> 
> This is certainly doable.
> 
> What you need is the patterndb feature described here:
> 
> http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.2-guide-admin-en.html/index.html-single.html#chapter-patterndb
> 
> You will create a series of rules which match your messages to the level 
> of granularity desired, then you can capture variables from the message 
> with names you select. Then you can use the variables anywhere 
> 'downstream' from where you applied the patterndb matching, such as the 
> output file template or rewrite rules, etc.
> 
> You could also do this with PCRE capturing, but it would be less 
> efficient. How many messages/sec. are you seeing and planning to 
> support?

If this stream can easily separated from the rest of your logs, then
csv-parser with a custom separator might be somewhat easier to use, and
still you can apply db-parser() filtering after the csv-parser has
exploded the fields into name-value pairs.

-- 
Bazsi