[syslog-ng] Buffering AF_UNIX Destination, Batch Post Processing Messages

[Martin Holste]

Those are good suggestions.  However, we have had some luck with a
different method.  I will share my experience thus far with the exact
problem you're tackling and what's been working for us:

Use the program() destination and open(FH, "-|") in Perl to read it.
This saves the UDP packet creation overhead as well as ensures that
there are no lost logs.  I have experimented with having N number of
preforked Perl child workers which all listen on "sub" pipes in a
round-robin (modulo on Perl's $. variable), but I quickly found what
you've already pointed out, that this is a sync pipe, so there's no
sense in round-robin-ing since the parent can't move on to the next
child pipe until the first child is done reading anyway.  That's fine,
since I have never found the Syslog-NG -> Perl end of things to be a
bottleneck.  In our setup, I have Perl do some simple massaging of the
logs and then write out to a tab-separated file in one minute batches.
 I then load the file in using MySQL LOAD DATA INFILE, and this can
get you 100k mps sustained into a database if you're light on the
indexing.  There's also no reason you couldn't simply write the logs
from Perl to flat file in sqlite format, which would allow you to skip
the MySQL step entirely.  It really depends what you want the final
format of the logs to be in.

In any case, I would discourage you from trying the async framework
route as it adds way too much overhead.

If you do in fact find a bottleneck with pipes, I would think that a
solution involving UDP sent to a local port could work with some fancy
iptables load balancing.  You would be limited to netstat counters to
detect losses, but it would probably work.  But unless you hit a pipe
bottleneck, I think all of that is more trouble than it is worth.

--Martin

On Wed, Sep 8, 2010 at 12:02 AM,  <syslogng at feystorm.net> wrote:
>
>
> Sent: Martedì 7 Settembre 2010 19.42.52
> From: Matthew Hall <mhall at mhcomputing.net>
> To: Syslog-ng users' and developers' mailing list
> <syslog-ng at lists.balabit.hu>
> Subject: Re: [syslog-ng] Buffering AF_UNIX Destination, Batch Post
> Processing Messages
>
> Syslog-ng will queue all the destination messages until the oldest
> message is 60 seconds old, and then flushes them all out at once.
>
>
> This part is tricky. How do I tell if I have received all the messages?
> How do I know when I have hit the end of the batch? Is it possible to
> have the daemon insert a marker message, or is there some other way I
> can check for this?
>
>
> I do not believe there is an elegant way. Best idea I can come up with is to
> put a timeout on the receiving end so that when it goes quiet for more than
> X seconds or whatnot, it sees that as end of batch.
> You might be able to request that the mark option be allowed for non-local
> destinations. Basically that would allow you to set a mark of 1 second, and
> when you receive 2 mark messages back-to-back, that would be end-of-batch
> (would basically mean there was no data in between).
>
> Thanks,
> Matthew.
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>
>