[syslog-ng] About Syslog-NG performance

[Balazs Scheidler]

On Tue, 2008-10-14 at 19:28 +0900, Yu Watanabe wrote:
> Hello all,
> 
> I am planning to build a syslog server using syslog-ng v 2.0.5.
> 
> However, I am very concerned with the how much performace could 
> syslog-ng can handle.
> 
> Following is the system specification I am using.
> 
> OS : RHEL 4 or RHEL 5
> CPU : Xeon(R) E5205 (1.86GHz)×2
> Memory : 4 GB
> 
> I would like get an advice with the performace of the syslog-ng.
> Does any one has a testimonial with this?

Performance depends greatly on the exact configuration you are planning
to deploy.

There's a small utility in the syslog-ng distribution, called loggen
(available starting at 2.0.8), it is a simple log traffic generator,
that you can use to measure performance.

I have some raw numbers though:
 * syslog-ng OSE 2.0 was the baseline, it did not receive the
performance improvements that I made during the 2.1 PE development
cycle,
 * syslog-ng PE 2.1 performed about 70000 messages/sec at 150
bytes/message, writing everything in a single plain text file
 * syslog-ng OSE 2.1 received _some_ of the performance updates, so it
should be better than OSE 2.0, but not up to PE 2.1
 * syslog-ng OSE 3.0 (currently released as a git tree and nightly
snapshots) should be on par with syslog-ng PE 3.0 and 2.1; some
performance related work went into PE 3.0 as well, so it should be
somewhat better than PE 2.1, especially if the new protocol is in use; I
measured about 75000msg/sec into plain text files and about 50000msg/sec
when writing into logstore.

The PE 2.1 performance work was carried out on an 2.2GHz Opteron box in
64 bit mode. 64 bit mode mattered, so that should be preferred, but I
can't remember the exact numbers.

I only used "tcp" as transport, udp becomes very lossy at large pkt/sec.

I only used a single tcp connection on the input side.

syslog-ng uses threads only in limited ways (when using SQL for
instance), so it'll not scale to multiple CPUs.

Please note that 70k msg/sec at 150 bytes/msg each, you get 10MB/sec, or
about 36GB/hour.

-- 
Bazsi