[syslog-ng] Buffering AF_UNIX Destination, Batch Post Processing Messages

[Balazs Scheidler]

On Mon, 2010-09-20 at 14:35 -0500, Martin Holste wrote:
> > commit 70e91556b6af8724334443347fd6488745405344
> > Author: Balazs Scheidler <bazsi at balabit.hu>
> > Date:   Mon Sep 20 17:12:27 2010 +0200
> >
> >    convertfuncs: new plugin to contain conversion template functions
> >
> >    The plugin now only contains ipv4-to-int which converts an IPv4 address
> >    to a long integer.
> >
> > Usage:
> >
> > $(ipv4-to-int $SOURCEIP)
> >
> 
> Very cool stuff!
> 
> > Expect a  blog post on this topic, a simple correllation engine is now
> > built into patterndb.
> 
> Hm, very interesting, I'll be taking a look.
> 
> Regarding detecting the batches being complete:  It seems a little
> inelegant to have a baby-sitter script that looks for an appropriately
> named file in a given directory and hoping it's the right buffer.  It
> would be really nice if Syslog-NG could execute program() on a file
> that has just been written to for the last time.

I was thinking about adding "events" to sources/destinations which could
invoke 3rd party tools/scripts when something happens.

Events could be time based, but other setup/teardown style stuff can
come in handy.

e.g.

destination d_file { file("/var/log/messages.$HOUR" 
			events(cron(min(5) hour(*) exec("/usr/local/bin/messages-file-finished"));
};

Not sure about the syntax though. Also I want it to be able to run processes like tail -f:

source s_follow { pipe("/var/run/syslog-ng/tail-pipe" 
			events(startup(supervise("/usr/bin/tail -f /var/log/apache.log > /var/run/syslog-ng/tail-pipe"))));
};

I know that syslog-ng is capable for tailing files, but the point is that there
are sometimes complex log systems of various applications, and the only sane interface
to them to run a process to tail its otherwise binary logfile. I want syslog-ng to
manage these processes.


-- 
Bazsi