[syslog-ng]Calibration of the syslog-ng server

[Nate Campi]

On Thu, Jun 12, 2003 at 05:33:24PM +0200, Loubet Jean-Michel wrote:
> 
> I want to collect about 1500 hosts.
> I've estimated that it represents about  8 000 000 messages (1 Gb) daily.
> 
> My syslog-ng server, which will be dedicated to this, will run on solaris 8.
> 
> Have you got any idea about needed cpu number and memory ?

The biggest issues with large log volumes have two broad issues:

1) network throughput
2) disk subsystem throughput

Bursts of messages will probably be the biggest enemy of getting logs
committed to disk. People regularly report issues with UDP receive
buffers filling and the OS dropping packets silently. Using TCP
everywhere possible and/or increase your UDP receive buffer size is a
good start.

As for hardware, lots of memory is good for filesystem caching, on
Solaris all disk I/O goes through memory. I'd say get some nice 10k RPM
lvd scsi disks, balanced with raid across several spindles (with
parity/mirroring of some sort since it's logs, maybe raid 5 or raid
1+0). Two cpus might be good, you'd hope one CPU handles most system
stuff and the second does syslog-ng. If Solaris does one thing well it's
scaling almost linearly with additional processors.

Lots of people put a couple syslog servers behind a load balancer, which
makes configuration simple all around (no syslog configs for clients or
servers have to know anything about the load balancer). If your peak
periods are too bursty that might end up a requirement.

See:
 http://www.campin.net/syslog-ng/faq.html#how_fast
-- 
Nate Campi    http://www.campin.net