Sorry I should have mentioned that I'm using 2.0.5.  Thanks for the pointers so far.  Unfortunately, in the intended application cpu is at a premium so I may not be able to tune my way out of this.  In a standalone log collector this would be more than adequate performance (in case anyone thought I was being overly critical :) )

However, I have found a couple of things that helped.  I wasn't paying attention to the compiler output the first time and didn't realize that I had built a debug binary.   Changing that was worth a few % in cpu use.

Second, enabling flow-control helped considerably.  At this point I am able to handle about 8000 messages per second while using an average of about 45% cpu with some periods closer to 60%.  The tcp listener I am using is simply netcat.  Some output from top plus my current test configuration:


top - 16:44:26 up  7:48,  5 users,  load average: 1.03, 1.02, 0.94
Tasks: 166 total,   3 running, 163 sleeping,   0 stopped,   0 zombie
Cpu0  : 14.0%us, 18.0%sy,  0.0%ni, 48.5%id,  4.0%wa,  0.0%hi, 15.5%si,  0.0%st
Cpu1  :  9.0%us, 24.5%sy,  0.0%ni, 58.5%id,  0.0%wa,  0.5%hi,  7.5%si,  0.0%st
Mem:   2074688k total,  2020636k used,    54052k free,     1584k buffers
Swap:  4096564k total,   242692k used,  3853872k free,   746772k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
30458 root      15   0  8548 7460  804 R   46  0.4  12:09.62 ./syslog-ng-2.0.5/src/syslog-ng -F -v -f ./syslog-ng.conf
30462 rory      15   0  1688  608  512 S   14  0.0   3:02.96 nc -l -p 1028



 options {
    time_reopen(10);
    chain_hostnames(yes);
    keep_hostname(no);
    gc_idle_threshold(100);
    gc_busy_threshold(65535);
    use_dns(no);
    log_fetch_limit(10);
    log_iw_size(10000);
    log_fifo_size(100000);
 };

 source remote_sources {
    udp(ip(0.0.0.0) port(514));
 };

 destination event_collector {
    tcp("127.0.0.1" port(1028));
 };

 log {
    source(remote_sources);
    destination(event_collector);
    flags(flow-control);
 };


Evan Rempel wrote:
You didn't mention the version of syslog-ng.

My 2.0.4 system uses a similar amount of CPU, which means that a single CPU (syslog-ng is single threaded) can only 
handle a sustained rate about 10,000 per second. A lot of that time (as much as 30%) will be the IP stack
doing packet processing. You can use a TCP/IP offload network interface card to basically eliminate this.

The remaining time will have to be addressed by tuning syslog-ng.

The authors have produced a nice section in the syslog-ng manual

http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/ch07.html

Evan.

Rory wrote:
  
I'm using syslog-ng as a receiver to relay messages to local processes
and other remote syslog listeners.  I'm not currently using any local
file destinations.  The issue I'm having is that as soon as I start
receiving more than a few hundred messages per second (udp) syslog-ng
uses significant CPU resources, even on an intel core-duo.

At 1000 messages per second with an average message size < 300 bytes
syslog-ng consumes 9-12% CPU.  With an event rate closer to 4000
messages per second the CPU use is 35-30%.  This seems excessive given
that I'm doing no filtering at all.  In a test configuration, I am only
receiving (no log{} clauses at all) and this behaviour is still
demonstrated.  It seems that all the CPU use is related to incoming
packet processing.  The test system is Linux 2.6.20 (Ubuntu) running on
a Core2 dual 2.16GHz.

Once I have completed some profiling I will share the findings but in
the meantime: am I missing something that should be obvious?  Some
searching through the mailing list archives hasn't helped me much.  My
test configuration file:

options {
    time_reopen(10);
    chain_hostnames(yes);
    keep_hostname(no);
    log_fifo_size(65535);
    gc_idle_threshold(100);
    gc_busy_threshold(65535);
    use_dns(no);
 };

 source remote_sources {
    udp(ip(0.0.0.0) port(514));
 };

 destination event_collector {
    tcp("127.0.0.1" port(1028));
 };

# log {
#    source(remote_sources);
#    destination(event_collector);
# };

Thanks,
Rory

_______________________________________________
syslog-ng maillist  -  syslog-ng@lists.balabit.hu
https://lists.balabit.hu/mailman/listinfo/syslog-ng
Frequently asked questions at http://www.campin.net/syslog-ng/faq.html