On Wed, 2010-09-08 at 13:48 -0500, Martin Holste wrote:
Those are good suggestions. However, we have had some luck with a different method. I will share my experience thus far with the exact problem you're tackling and what's been working for us:
Use the program() destination and open(FH, "-|") in Perl to read it. This saves the UDP packet creation overhead as well as ensures that there are no lost logs. I have experimented with having N number of preforked Perl child workers which all listen on "sub" pipes in a round-robin (modulo on Perl's $. variable), but I quickly found what you've already pointed out, that this is a sync pipe, so there's no sense in round-robin-ing since the parent can't move on to the next child pipe until the first child is done reading anyway. That's fine, since I have never found the Syslog-NG -> Perl end of things to be a bottleneck. In our setup, I have Perl do some simple massaging of the logs and then write out to a tab-separated file in one minute batches.
I guess syslog-ng could also write tab separated data into files and can also do per-minute batches (by using the $MIN macro). Are there any other things the perl stuff does? -- Bazsi