In my setup, I'm planning on using Cisco gear to load balance a massive
amount of unique log senders on a round-robin to several syslog-ng
nodes. You could do the same with your Xen VM's so that each one is
only processing 1/Nth of the logs. I'm sure you could work out some
ACL which would send certain log generator IP's to certain VM's. It
might even be a good idea to have a different destination port for each
VM IP address so that the UDP buffer on the host OS of the VM's doesn't
get overwhelmed (as in, <a href="http://10.0.0.1:514/" target="_blank">10.0.0.1:514</a>, <a href="http://10.0.0.2:515/" target="_blank">10.0.0.2:515</a>, etc.).<br>
<br>Alternatively, I'm sure that with some IPTables entries on the Xen
host box using the REDIRECT target, you could multiplex the logs out to
the correct Xen VM's. I've done that before and it works, but I've had
occasional issues with IPTables not resetting correctly when the
configuration is altered, so YMMV.<br><br><div class="gmail_quote">On Fri, Jun 19, 2009 at 10:14 AM, Aaron Robel <span dir="ltr"><<a href="mailto:megawott@gmail.com">megawott@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div>Thanks for the reply Bazsi.</div>
<div> </div>
<div>I think we are running into multiple issues. Here are my findings from yesterday with regard to the config file on the archive box and the UDP receive errors on the kernel.</div>
<div> </div>
<div>Archive host:</div>
<div>TCP seemed to resolve the issue. This was a por test because once TCP was configured we lost our source_spoof function and everything dumped to a single file.</div>
<div> </div>
<div>I then tried dumping everything to a single file and used UDP, this resolved the MPS descrepancy as well. Are there limitations to how many filters and destinations you can have. We have about 140 filters and same for destinations. We are breaking everything out by device, it's pretty simplistic but there are a lot.</div>
<div> </div>
<div>UDP buffer problems:</div>
<div>I read about this yesterday where heavy UDP syslog traffic the UDP receive buffer in the kernel can have problems. I checked the relay's and archive boxes for udp receive errors and sure enough we're seeing 20% udp receive errors on the relay and 60% on the archive. I adjusted the relay box to an 8meg udp receive buffer in the kernel and this has dropped the udp receive errors to 8-10%. </div>
<div> </div>
<div>There was still a question of how Xen in the virtualized environment might play a role in these issues and after a bit of research it looks like there are logged bugs with regard to domU's and UDP. </div>
<div> </div>
<div>My thought now is to go with the suggestion Martin had, scrap the virtual environment and utilize the box in whole. Just seems like such a waste, it's a 16proc, 32gig, 13TB box leftover from a project gone awry. Lot's of horsepower for a single syslog server that can't thread. Is there a way to utilize this horsepower for syslog-ng?</div>
<div> </div>
<div>And the hits just keep on coming... </div></blockquote></div><br>