[syslog-ng] Syslog-ng performance (CPU)

[Evan Rempel]

You didn't mention the version of syslog-ng.

My 2.0.4 system uses a similar amount of CPU, which means that a single CPU (syslog-ng is single threaded) can only 
handle a sustained rate about 10,000 per second. A lot of that time (as much as 30%) will be the IP stack
doing packet processing. You can use a TCP/IP offload network interface card to basically eliminate this.

The remaining time will have to be addressed by tuning syslog-ng.

The authors have produced a nice section in the syslog-ng manual

http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/ch07.html

Evan.

Rory wrote:
> I'm using syslog-ng as a receiver to relay messages to local processes
> and other remote syslog listeners.  I'm not currently using any local
> file destinations.  The issue I'm having is that as soon as I start
> receiving more than a few hundred messages per second (udp) syslog-ng
> uses significant CPU resources, even on an intel core-duo.
> 
> At 1000 messages per second with an average message size < 300 bytes
> syslog-ng consumes 9-12% CPU.  With an event rate closer to 4000
> messages per second the CPU use is 35-30%.  This seems excessive given
> that I'm doing no filtering at all.  In a test configuration, I am only
> receiving (no log{} clauses at all) and this behaviour is still
> demonstrated.  It seems that all the CPU use is related to incoming
> packet processing.  The test system is Linux 2.6.20 (Ubuntu) running on
> a Core2 dual 2.16GHz.
> 
> Once I have completed some profiling I will share the findings but in
> the meantime: am I missing something that should be obvious?  Some
> searching through the mailing list archives hasn't helped me much.  My
> test configuration file:
> 
> options {
>     time_reopen(10);
>     chain_hostnames(yes);
>     keep_hostname(no);
>     log_fifo_size(65535);
>     gc_idle_threshold(100);
>     gc_busy_threshold(65535);
>     use_dns(no);
>  };
> 
>  source remote_sources {
>     udp(ip(0.0.0.0) port(514));
>  };
> 
>  destination event_collector {
>     tcp("127.0.0.1" port(1028));
>  };
> 
> # log {
> #    source(remote_sources);
> #    destination(event_collector);
> # };
> 
> Thanks,
> Rory
> 
> _______________________________________________
> syslog-ng maillist  -  syslog-ng at lists.balabit.hu
> https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
> 


-- 
Evan Rempel                erempel at uvic.ca
Senior Programmer Analyst        250.721.7691
Computing Services
University of Victoria