[syslog-ng] Syslog-ng 3.0.2 statistics

[Martin Holste]

In my setup, I'm planning on using Cisco gear to load balance a massive
amount of unique log senders on a round-robin to several syslog-ng nodes.
You could do the same with your Xen VM's so that each one is only processing
1/Nth of the logs.  I'm sure you could work out some ACL which would send
certain log generator IP's to certain VM's.  It might even be a good idea to
have a different destination port for each VM IP address so that the UDP
buffer on the host OS of the VM's doesn't get overwhelmed (as in,
10.0.0.1:514, 10.0.0.2:515, etc.).

Alternatively, I'm sure that with some IPTables entries on the Xen host box
using the REDIRECT target, you could multiplex the logs out to the correct
Xen VM's.  I've done that before and it works, but I've had occasional
issues with IPTables not resetting correctly when the configuration is
altered, so YMMV.

On Fri, Jun 19, 2009 at 10:14 AM, Aaron Robel <megawott at gmail.com> wrote:

> Thanks for the reply Bazsi.
>
> I think we are running into multiple issues.  Here are my findings from
> yesterday with regard to the config file on the archive box and the UDP
> receive errors on the kernel.
>
> Archive host:
> TCP seemed to resolve the issue.  This was a por test because once TCP was
> configured we lost our source_spoof function and everything dumped to a
> single file.
>
> I then tried dumping everything to a single file and used UDP, this
> resolved the MPS descrepancy as well. Are there limitations to how many
> filters and destinations you can have.  We have about 140 filters and same
> for destinations.  We are breaking everything out by device, it's pretty
> simplistic but there are a lot.
>
> UDP buffer problems:
> I read about this yesterday where heavy UDP syslog traffic the UDP receive
> buffer in the kernel can have problems.  I checked the relay's and archive
> boxes for udp receive errors and sure enough we're seeing 20% udp receive
> errors on the relay and 60% on the archive.  I adjusted the relay box to an
> 8meg udp receive buffer in the kernel and this has dropped the udp receive
> errors to 8-10%.
>
> There was still a question of how Xen in the virtualized environment might
> play a role in these issues and after a bit of research it looks like there
> are logged bugs with regard to domU's and UDP.
>
> My thought now is to go with the suggestion Martin had, scrap the virtual
> environment and utilize the box in whole.  Just seems like such a waste,
> it's a 16proc, 32gig, 13TB box leftover from a project gone awry.  Lot's of
> horsepower for a single syslog server that can't thread. Is there a way to
> utilize this horsepower for syslog-ng?
>
> And the hits just keep on coming...
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20090619/82a53123/attachment.htm