[syslog-ng] Buffering AF_UNIX Destination, Batch Post Processing Messages

[Balazs Scheidler]

On Wed, 2010-09-08 at 13:48 -0500, Martin Holste wrote:
> Those are good suggestions.  However, we have had some luck with a
> different method.  I will share my experience thus far with the exact
> problem you're tackling and what's been working for us:
> 
> Use the program() destination and open(FH, "-|") in Perl to read it.
> This saves the UDP packet creation overhead as well as ensures that
> there are no lost logs.  I have experimented with having N number of
> preforked Perl child workers which all listen on "sub" pipes in a
> round-robin (modulo on Perl's $. variable), but I quickly found what
> you've already pointed out, that this is a sync pipe, so there's no
> sense in round-robin-ing since the parent can't move on to the next
> child pipe until the first child is done reading anyway.  That's fine,
> since I have never found the Syslog-NG -> Perl end of things to be a
> bottleneck.  In our setup, I have Perl do some simple massaging of the
> logs and then write out to a tab-separated file in one minute batches.

I guess syslog-ng could also write tab separated data into files and can
also do per-minute batches (by using the $MIN macro). Are there any
other things the perl stuff does?

-- 
Bazsi