[syslog-ng] Syslog-ng 3.0.2 statistics

[Aaron Robel]

Wow, I'm trying to create as simple of a solution as possible for our ops
folks to manage.  It's also being done with no budget. I don't know that I
want to dive into the load balancing arena with this environment yet,
although the topic certainly has raised it's head.

On the positive side it looks like I've got things working, for now.  I was
reading what Bazsi said about the udp buffer sizing and realized I had only
played with the kernel buffer.  So, I adjusted kernel and syslog config to
match and, BAM, no more udp receive errors. We're now able to push logs onto
their file destinations without dropping  Our archive is now getting about
4500 mps which equals the relay's added together.

So, for now we aren't being impacted by the Xen bug that I mentioned
earlier.  The only outstanding issue is that now that the archive box is
processing correctly our CPU is hovering around 85 to 90%.  I tried playing
with time_sleep, fifo, fetch, and flush but only created udp receive errors
and log dropping when I did.  I'm running the udp receive buffers at 4 meg.

I'm going to let it burn in over the weekend and make sure that things are
still working then tackle CPU next week.

Just a buffer issue, it's always something simple. It's even in the admin
guide under "Possible causes of losing log messages."

Thanks for helping the linx/xen/syslog rookie out. Sorry for the ramblings.


On Fri, Jun 19, 2009 at 10:24 AM, Martin Holste <mcholste at gmail.com> wrote:

> In my setup, I'm planning on using Cisco gear to load balance a massive
> amount of unique log senders on a round-robin to several syslog-ng nodes.
> You could do the same with your Xen VM's so that each one is only processing
> 1/Nth of the logs.  I'm sure you could work out some ACL which would send
> certain log generator IP's to certain VM's.  It might even be a good idea to
> have a different destination port for each VM IP address so that the UDP
> buffer on the host OS of the VM's doesn't get overwhelmed (as in,
> 10.0.0.1:514, 10.0.0.2:515, etc.).
>
> Alternatively, I'm sure that with some IPTables entries on the Xen host box
> using the REDIRECT target, you could multiplex the logs out to the correct
> Xen VM's.  I've done that before and it works, but I've had occasional
> issues with IPTables not resetting correctly when the configuration is
> altered, so YMMV.
>
>
> On Fri, Jun 19, 2009 at 10:14 AM, Aaron Robel <megawott at gmail.com> wrote:
>
>> Thanks for the reply Bazsi.
>>
>> I think we are running into multiple issues.  Here are my findings from
>> yesterday with regard to the config file on the archive box and the UDP
>> receive errors on the kernel.
>>
>> Archive host:
>> TCP seemed to resolve the issue.  This was a por test because once TCP was
>> configured we lost our source_spoof function and everything dumped to a
>> single file.
>>
>> I then tried dumping everything to a single file and used UDP, this
>> resolved the MPS descrepancy as well. Are there limitations to how many
>> filters and destinations you can have.  We have about 140 filters and same
>> for destinations.  We are breaking everything out by device, it's pretty
>> simplistic but there are a lot.
>>
>> UDP buffer problems:
>> I read about this yesterday where heavy UDP syslog traffic the UDP receive
>> buffer in the kernel can have problems.  I checked the relay's and archive
>> boxes for udp receive errors and sure enough we're seeing 20% udp receive
>> errors on the relay and 60% on the archive.  I adjusted the relay box to an
>> 8meg udp receive buffer in the kernel and this has dropped the udp receive
>> errors to 8-10%.
>>
>> There was still a question of how Xen in the virtualized environment might
>> play a role in these issues and after a bit of research it looks like there
>> are logged bugs with regard to domU's and UDP.
>>
>> My thought now is to go with the suggestion Martin had, scrap the virtual
>> environment and utilize the box in whole.  Just seems like such a waste,
>> it's a 16proc, 32gig, 13TB box leftover from a project gone awry.  Lot's of
>> horsepower for a single syslog server that can't thread. Is there a way to
>> utilize this horsepower for syslog-ng?
>>
>> And the hits just keep on coming...
>>
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>
>


-- 
Aaron Robel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20090619/81a6089b/attachment.htm