[syslog-ng] Re: syslog-ng performance question

[Balazs Scheidler]

> About how many messages per second do you think syslog-ng can process?
> 
> I know it depends on a lot of factors, but have you done any testing?

Sending messages in an infinite loop from shell script causes no problem.
Even if several scripts are running at the same time. 

However if this is done from within a compiled C program, it may lose
messages if non-connection-oriented messages are used (udp, and unix-dgram).
Sending messages from C meant about 3000 accepted messages per second. (e.g.
messages which were actually written to the logfile), while the C program
sent about 25000 messages per second.

Lost messages are caused by garbage collection. While it runs, syslog-ng
doesn't fetch messages from the kernel queue. 

You can set the number of allocated objects when the gc runs with the
gc_threshold option. The larger this value is, the longer the gc phase will
run (of course if it's larger it runs less times).

So syslog-ng can work with at most 3000 messages per second if used with
connectionless protocols. Connection oriented protocols will block the
sending process while the message is sent. Messages are not lost, but
overall system performance decreases because of logging.

I measured this on my Celeron 300A running at 450Mhz (100Mhz FSB, 128k
full-speed cache, 64MB 100Mhz SDRAM) with a threshold value of 1000.

I think threshold should be set to largest burst you are counting for. 3000
messages / second means about 300000 bytes log info per second, 18megs in 
a minute, 1 gigs in an hour, and 24 gigs in a day. (using an estimate of 100
bytes per message)

I may try to add more options to finetune gc phase in the future.

-- 
Bazsi
PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1
     url: http://www.balabit.hu/pgpkey.txt