<div>Wow, I'm trying to create as simple of a solution as possible for our ops folks to manage. It's also being done with no budget. I don't know that I want to dive into the load balancing arena with this environment yet, although the topic certainly has raised it's head.</div>
<div> </div>
<div>On the positive side it looks like I've got things working, for now. I was reading what Bazsi said about the udp buffer sizing and realized I had only played with the kernel buffer. So, I adjusted kernel and syslog config to match and, BAM, no more udp receive errors. We're now able to push logs onto their file destinations without dropping Our archive is now getting about 4500 mps which equals the relay's added together.</div>
<div> </div>
<div>So, for now we aren't being impacted by the Xen bug that I mentioned earlier. The only outstanding issue is that now that the archive box is processing correctly our CPU is hovering around 85 to 90%. I tried playing with time_sleep, fifo, fetch, and flush but only created udp receive errors and log dropping when I did. I'm running the udp receive buffers at 4 meg.</div>
<div> </div>
<div>I'm going to let it burn in over the weekend and make sure that things are still working then tackle CPU next week. </div>
<div> </div>
<div>Just a buffer issue, it's always something simple. It's even in the admin guide under "Possible causes of losing log messages."</div>
<div> </div>
<div>Thanks for helping the linx/xen/syslog rookie out. Sorry for the ramblings.</div>
<div> </div>
<div> </div>
<div class="gmail_quote">On Fri, Jun 19, 2009 at 10:24 AM, Martin Holste <span dir="ltr"><<a href="mailto:mcholste@gmail.com">mcholste@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">In my setup, I'm planning on using Cisco gear to load balance a massive amount of unique log senders on a round-robin to several syslog-ng nodes. You could do the same with your Xen VM's so that each one is only processing 1/Nth of the logs. I'm sure you could work out some ACL which would send certain log generator IP's to certain VM's. It might even be a good idea to have a different destination port for each VM IP address so that the UDP buffer on the host OS of the VM's doesn't get overwhelmed (as in, <a href="http://10.0.0.1:514/" target="_blank">10.0.0.1:514</a>, <a href="http://10.0.0.2:515/" target="_blank">10.0.0.2:515</a>, etc.).<br>
<br>Alternatively, I'm sure that with some IPTables entries on the Xen host box using the REDIRECT target, you could multiplex the logs out to the correct Xen VM's. I've done that before and it works, but I've had occasional issues with IPTables not resetting correctly when the configuration is altered, so YMMV.
<div>
<div></div>
<div class="h5"><br><br>
<div class="gmail_quote">On Fri, Jun 19, 2009 at 10:14 AM, Aaron Robel <span dir="ltr"><<a href="mailto:megawott@gmail.com" target="_blank">megawott@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0pt 0pt 0pt 0.8ex; BORDER-LEFT: rgb(204,204,204) 1px solid">
<div>Thanks for the reply Bazsi.</div>
<div> </div>
<div>I think we are running into multiple issues. Here are my findings from yesterday with regard to the config file on the archive box and the UDP receive errors on the kernel.</div>
<div> </div>
<div>Archive host:</div>
<div>TCP seemed to resolve the issue. This was a por test because once TCP was configured we lost our source_spoof function and everything dumped to a single file.</div>
<div> </div>
<div>I then tried dumping everything to a single file and used UDP, this resolved the MPS descrepancy as well. Are there limitations to how many filters and destinations you can have. We have about 140 filters and same for destinations. We are breaking everything out by device, it's pretty simplistic but there are a lot.</div>
<div> </div>
<div>UDP buffer problems:</div>
<div>I read about this yesterday where heavy UDP syslog traffic the UDP receive buffer in the kernel can have problems. I checked the relay's and archive boxes for udp receive errors and sure enough we're seeing 20% udp receive errors on the relay and 60% on the archive. I adjusted the relay box to an 8meg udp receive buffer in the kernel and this has dropped the udp receive errors to 8-10%. </div>
<div> </div>
<div>There was still a question of how Xen in the virtualized environment might play a role in these issues and after a bit of research it looks like there are logged bugs with regard to domU's and UDP. </div>
<div> </div>
<div>My thought now is to go with the suggestion Martin had, scrap the virtual environment and utilize the box in whole. Just seems like such a waste, it's a 16proc, 32gig, 13TB box leftover from a project gone awry. Lot's of horsepower for a single syslog server that can't thread. Is there a way to utilize this horsepower for syslog-ng?</div>
<div> </div>
<div>And the hits just keep on coming... </div></blockquote></div><br></div></div><br>______________________________________________________________________________<br>Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>FAQ: <a href="http://www.campin.net/syslog-ng/faq.html" target="_blank">http://www.campin.net/syslog-ng/faq.html</a><br>
<br><br></blockquote></div><br><br clear="all">
<div></div><br>-- <br>Aaron Robel<br>