I'm using syslog-ng as a receiver to relay messages to local processes and other remote syslog listeners. I'm not currently using any local file destinations. The issue I'm having is that as soon as I start receiving more than a few hundred messages per second (udp) syslog-ng uses significant CPU resources, even on an intel core-duo. At 1000 messages per second with an average message size < 300 bytes syslog-ng consumes 9-12% CPU. With an event rate closer to 4000 messages per second the CPU use is 35-30%. This seems excessive given that I'm doing no filtering at all. In a test configuration, I am only receiving (no log{} clauses at all) and this behaviour is still demonstrated. It seems that all the CPU use is related to incoming packet processing. The test system is Linux 2.6.20 (Ubuntu) running on a Core2 dual 2.16GHz. Once I have completed some profiling I will share the findings but in the meantime: am I missing something that should be obvious? Some searching through the mailing list archives hasn't helped me much. My test configuration file: options { time_reopen(10); chain_hostnames(yes); keep_hostname(no); log_fifo_size(65535); gc_idle_threshold(100); gc_busy_threshold(65535); use_dns(no); }; source remote_sources { udp(ip(0.0.0.0) port(514)); }; destination event_collector { tcp("127.0.0.1" port(1028)); }; # log { # source(remote_sources); # destination(event_collector); # }; Thanks, Rory
You didn't mention the version of syslog-ng. My 2.0.4 system uses a similar amount of CPU, which means that a single CPU (syslog-ng is single threaded) can only handle a sustained rate about 10,000 per second. A lot of that time (as much as 30%) will be the IP stack doing packet processing. You can use a TCP/IP offload network interface card to basically eliminate this. The remaining time will have to be addressed by tuning syslog-ng. The authors have produced a nice section in the syslog-ng manual http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/ch07.html Evan. Rory wrote:
I'm using syslog-ng as a receiver to relay messages to local processes and other remote syslog listeners. I'm not currently using any local file destinations. The issue I'm having is that as soon as I start receiving more than a few hundred messages per second (udp) syslog-ng uses significant CPU resources, even on an intel core-duo.
At 1000 messages per second with an average message size < 300 bytes syslog-ng consumes 9-12% CPU. With an event rate closer to 4000 messages per second the CPU use is 35-30%. This seems excessive given that I'm doing no filtering at all. In a test configuration, I am only receiving (no log{} clauses at all) and this behaviour is still demonstrated. It seems that all the CPU use is related to incoming packet processing. The test system is Linux 2.6.20 (Ubuntu) running on a Core2 dual 2.16GHz.
Once I have completed some profiling I will share the findings but in the meantime: am I missing something that should be obvious? Some searching through the mailing list archives hasn't helped me much. My test configuration file:
options { time_reopen(10); chain_hostnames(yes); keep_hostname(no); log_fifo_size(65535); gc_idle_threshold(100); gc_busy_threshold(65535); use_dns(no); };
source remote_sources { udp(ip(0.0.0.0) port(514)); };
destination event_collector { tcp("127.0.0.1" port(1028)); };
# log { # source(remote_sources); # destination(event_collector); # };
Thanks, Rory
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
-- Evan Rempel erempel@uvic.ca Senior Programmer Analyst 250.721.7691 Computing Services University of Victoria
Rory wrote:
options { time_reopen(10); chain_hostnames(yes); keep_hostname(no); gc_idle_threshold(100); gc_busy_threshold(65535); use_dns(no); log_fetch_limit(10); log_iw_size(10000); log_fifo_size(100000); };
source remote_sources { udp(ip(0.0.0.0) port(514)); };
If you don't need to receive data from ALL local IP addresses, you can save some CPU by only binding and thus polling the specific IP addresses that you need.
destination event_collector { tcp("127.0.0.1" port(1028)); };
If you event collector does parsing the the syslog message, then you could stop syslog-ng from parsing the incomming message, and using a $MSGONLY template on the destination.
log { source(remote_sources); destination(event_collector); flags(flow-control); };
I'm not sure what platform you are on, but you could compile a later binary, like i686 rather than i386. It can make a large difference. Evan.
My syslog-ng server /var/log/syslog-ng contains IPs not hostnames How to correct that?
Have you set the option use_dns(yes) in syslog-ng.conf? Is DNS working fine on your server otherwise? -----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Tamer Tayea Sent: Wednesday, July 25, 2007 11:34 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] /var/log/syslog-ng contains IPs not hostnames My syslog-ng server /var/log/syslog-ng contains IPs not hostnames How to correct that? _______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
participants (4)
-
Darwin, Samuel
-
Evan Rempel
-
Rory
-
Tamer Tayea