[syslog-ng] Profiling syslog-ng

[John Morrissey]

On Fri, Feb 10, 2006 at 09:14:44PM -0600, Kevin wrote:
> On 2/10/06, John Morrissey <jwm at horde.net> wrote:
> > I'm trying to profile syslog-ng on a set of servers that generate high
> > levels of logging traffic (a few GB/day). Currently, syslog-ng is
> > consuming anywhere from 40-80% of a CPU on a dual Pentium IV Xeon
> > machine.
> 
> Even for the slowest possible P4, that seems immensely high.
> Can you give more information about what on these servers generates
> the log events, the average log message length, and the average and
> peak events/second rate?

They're our inbound mail cluster (running Postfix), handling a few million
mail messages a day. They usually have several hundred processes running
simultaneously, almost all of which log.

As far as log messages go, 13-19,000 messages/minute is a general average,
with frequent peaks of 23-28,000 messages/minute, and occasional heavy peaks
of up to 35,000 messages/minute for a couple minutes at a time. Yesterday,
one machine syslogged about 23 million messages (~16,000/minute average for
the day).

Here's a breakdown of log lengths:
< 100 bytes - 34%
  100       - 45%
  200       - 7%
  300       - 12%
  400       - 2%

> >  can post my configuration if it'd be helpful,
> 
> Please do.

It's up at http://horde.net/~jwm/syslog-ng.conf.

john
-- 
John Morrissey          _o            /\         ----  __o
jwm at horde.net        _-< \_          /  \       ----  <  \,
www.horde.net/    __(_)/_(_)________/    \_______(_) /_(_)__