[syslog-ng] Buffering AF_UNIX Destination, Batch Post Processing Messages

[Balazs Scheidler]

On Wed, 2010-09-15 at 09:51 -0500, Martin Holste wrote:
> > I guess syslog-ng could also write tab separated data into files and can
> > also do per-minute batches (by using the $MIN macro). Are there any
> > other things the perl stuff does?
> 
> Yes, you certainly could get Syslog-NG to write TSV in minute batches
> without any other program's intervention.  I pipe to Perl because I do
> some minor data alterations before writing to file.  Specifically, I
> CRC the program name to generate a program ID so that I can store the
> program in a programs table and the program_id in the main logs table
> to conserve space and keep the database in as close to 3rd normal form
> as I can.  

Perfect example for a template-function.

> I also do basic conversions like INET_ATON all IP addresses
> to store them in integer columns.  

Again, should be a great example for a template-function. It was such a
good idea, that I've actually coded it here:

commit 70e91556b6af8724334443347fd6488745405344
Author: Balazs Scheidler <bazsi at balabit.hu>
Date:   Mon Sep 20 17:12:27 2010 +0200

    convertfuncs: new plugin to contain conversion template functions
    
    The plugin now only contains ipv4-to-int which converts an IPv4 address
    to a long integer.

Usage:

$(ipv4-to-int $SOURCEIP)

> In the future, I would put advanced
> correlation capabilities there (probably nothing like what Matthew is
> cooking up, though!) as well as any real-time cluster messaging I need
> to do.

Expect a  blog post on this topic, a simple correllation engine is now
built into patterndb.

I'm afraid the information about this topic is not very much, but
anyway, here's the patch that implements it:

commit 9d07e274bdf2ba00b0e697a13299140f4bf04ed3
Author: Balazs Scheidler <bazsi at balabit.hu>
Date:   Mon Sep 20 15:54:37 2010 +0200

    db-parser: initial support for simple message correllation
    
    This feature is not yet complete, will probably leak memory, but if not leak
    will probably use a _lot_ of memory, but still makes it possible to use
    simple log event correllation for those who want it.



-- 
Bazsi