On Fri, 2010-03-19 at 00:56 -0400, Jan Schaumann wrote:
Martin Holste <mcholste@gmail.com> wrote:
How many messages per second is the system attempting to handle? I'm very surprised that you're seeing that level of utilization. In our setup we've never had a problem pushing up through 30,000 messages per second written to disk with Syslog-NG in production, and I've pushed more than 70,000 per second in development.
Could you provide your configuration for these systems (including sysctls or kernel tunables etc.)? I've so far not been able to get my systems to accept and process (without any regex matching) more than approximagely 25K - 30K UDP messages/s.
udp() may be the problem here, in fact since syslog-ng is not thread based, its latency to poll for the udp socket may be a bit too long. increasing the udp socket buffer to insane values (like 256MB) could help alleviate the problem somewhat, but the latency is the root cause. adding time_sleep() to the mix increases latency even further. It only helps if you have a lot of connections (and udp is only one even if you have thousands of clients, while tcp keeps a separate connection for each client). After I get to the end of integrating the plugins branch, I intend to work on enabling multiple threads, thus decreasing the latency. -- Bazsi