[syslog-ng] log failback groups

[Balazs Scheidler]

On Fri, 2010-03-19 at 00:56 -0400, Jan Schaumann wrote:
> Martin Holste <mcholste at gmail.com> wrote:
> > How many messages per second is the system attempting to handle?  I'm
> > very surprised that you're seeing that level of utilization.  In our
> > setup we've never had a problem pushing up through 30,000 messages per
> > second written to disk with Syslog-NG in production, and I've pushed
> > more than 70,000 per second in development.
> 
> Could you provide your configuration for these systems (including
> sysctls or kernel tunables etc.)?  I've so far not been able to get my
> systems to accept and process (without any regex matching) more than
> approximagely 25K - 30K UDP messages/s.
>  

udp() may be the problem here, in fact since syslog-ng is not thread
based, its latency to poll for the udp socket may be a bit too long.

increasing the udp socket buffer to insane values (like 256MB) could
help alleviate the problem somewhat, but the latency is the root cause.

adding time_sleep() to the mix increases latency even further. It only
helps if you have a lot of connections (and udp is only one even if you
have thousands of clients, while tcp keeps a separate connection for
each client).

After I get to the end of integrating the plugins branch, I intend to
work on enabling multiple threads, thus decreasing the latency.


-- 
Bazsi