Ok, so finally got time to look at this again.
I tried the time_sleep option, and it was a horrible failure. With it enabled syslog-ng started losing about 98% of all incoming lines (and no, thats not an exaggeration). I'm guessing that time_sleep does not play well with udp as thats what incoming data is being sent over.
However I do have the master-slave multi-process thing going and its working really good. I was able to put time_sleep on the child processes (the one doing regex matches), and it dropped their cpu utilization from around 40% to about 20% (master process uses tcp to talk to slave processes, so no drops). Another thing is that when I tried using the syslog protocol to talk to the child processes, the slaves were terminating the connection within seconds of being established. I poked and prodded and could not get this to work without constantly dropping the connection, so I had to switch back to plain tcp.

Anyway, the attached config is what it looked like when I had all regexes run within a single process (the config that was utilizing over 90% cpu).



Sent: Thursday, March 18, 2010 10:56:16 PM
From: Jan Schaumann <jschauma@netmeister.org>
To: syslog-ng@lists.balabit.hu
Subject: Re: [syslog-ng] log failback groups 
Martin Holste <mcholste@gmail.com> wrote:
  
How many messages per second is the system attempting to handle?  I'm
very surprised that you're seeing that level of utilization.  In our
setup we've never had a problem pushing up through 30,000 messages per
second written to disk with Syslog-NG in production, and I've pushed
more than 70,000 per second in development.
    

Could you provide your configuration for these systems (including
sysctls or kernel tunables etc.)?  I've so far not been able to get my
systems to accept and process (without any regex matching) more than
approximagely 25K - 30K UDP messages/s.
 
-Jan
  

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html