Those are good suggestions. However, we have had some luck with a different method. I will share my experience thus far with the exact problem you're tackling and what's been working for us: Use the program() destination and open(FH, "-|") in Perl to read it. This saves the UDP packet creation overhead as well as ensures that there are no lost logs. I have experimented with having N number of preforked Perl child workers which all listen on "sub" pipes in a round-robin (modulo on Perl's $. variable), but I quickly found what you've already pointed out, that this is a sync pipe, so there's no sense in round-robin-ing since the parent can't move on to the next child pipe until the first child is done reading anyway. That's fine, since I have never found the Syslog-NG -> Perl end of things to be a bottleneck. In our setup, I have Perl do some simple massaging of the logs and then write out to a tab-separated file in one minute batches. I then load the file in using MySQL LOAD DATA INFILE, and this can get you 100k mps sustained into a database if you're light on the indexing. There's also no reason you couldn't simply write the logs from Perl to flat file in sqlite format, which would allow you to skip the MySQL step entirely. It really depends what you want the final format of the logs to be in. In any case, I would discourage you from trying the async framework route as it adds way too much overhead. If you do in fact find a bottleneck with pipes, I would think that a solution involving UDP sent to a local port could work with some fancy iptables load balancing. You would be limited to netstat counters to detect losses, but it would probably work. But unless you hit a pipe bottleneck, I think all of that is more trouble than it is worth. --Martin On Wed, Sep 8, 2010 at 12:02 AM, <syslogng@feystorm.net> wrote:
Sent: Martedì 7 Settembre 2010 19.42.52 From: Matthew Hall <mhall@mhcomputing.net> To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Buffering AF_UNIX Destination, Batch Post Processing Messages
Syslog-ng will queue all the destination messages until the oldest message is 60 seconds old, and then flushes them all out at once.
This part is tricky. How do I tell if I have received all the messages? How do I know when I have hit the end of the batch? Is it possible to have the daemon insert a marker message, or is there some other way I can check for this?
I do not believe there is an elegant way. Best idea I can come up with is to put a timeout on the receiving end so that when it goes quiet for more than X seconds or whatnot, it sees that as end of batch. You might be able to request that the mark option be allowed for non-local destinations. Basically that would allow you to set a mark of 1 second, and when you receive 2 mark messages back-to-back, that would be end-of-batch (would basically mean there was no data in between).
Thanks, Matthew. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html