You didn't mention the version of syslog-ng. My 2.0.4 system uses a similar amount of CPU, which means that a single CPU (syslog-ng is single threaded) can only handle a sustained rate about 10,000 per second. A lot of that time (as much as 30%) will be the IP stack doing packet processing. You can use a TCP/IP offload network interface card to basically eliminate this. The remaining time will have to be addressed by tuning syslog-ng. The authors have produced a nice section in the syslog-ng manual http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/ch07.html Evan. Rory wrote:
I'm using syslog-ng as a receiver to relay messages to local processes and other remote syslog listeners. I'm not currently using any local file destinations. The issue I'm having is that as soon as I start receiving more than a few hundred messages per second (udp) syslog-ng uses significant CPU resources, even on an intel core-duo.
At 1000 messages per second with an average message size < 300 bytes syslog-ng consumes 9-12% CPU. With an event rate closer to 4000 messages per second the CPU use is 35-30%. This seems excessive given that I'm doing no filtering at all. In a test configuration, I am only receiving (no log{} clauses at all) and this behaviour is still demonstrated. It seems that all the CPU use is related to incoming packet processing. The test system is Linux 2.6.20 (Ubuntu) running on a Core2 dual 2.16GHz.
Once I have completed some profiling I will share the findings but in the meantime: am I missing something that should be obvious? Some searching through the mailing list archives hasn't helped me much. My test configuration file:
options { time_reopen(10); chain_hostnames(yes); keep_hostname(no); log_fifo_size(65535); gc_idle_threshold(100); gc_busy_threshold(65535); use_dns(no); };
source remote_sources { udp(ip(0.0.0.0) port(514)); };
destination event_collector { tcp("127.0.0.1" port(1028)); };
# log { # source(remote_sources); # destination(event_collector); # };
Thanks, Rory
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
-- Evan Rempel erempel@uvic.ca Senior Programmer Analyst 250.721.7691 Computing Services University of Victoria