[syslog-ng] About Syslog-NG performance

[Yu Watanabe]

Hello Bazsi,

Thank you for the reply.

It is very helpful !

May I confirm you two things ?

1. Whenever Syslog-NG receives large amount of messages from the servers, 
does syslog-ng queues the messages when it can't procede the message at once?

I just want make sure that there aren't any messages that are lost.

2. Is the following the average performance data among Syslog OSE 2.0, 2.1 and 3.0, when
the specification which I have posted is used?

>Please note that 70k msg/sec at 150 bytes/msg each, you get 10MB/sec, or
>about 36GB/hour.

Best Regards,
Yu Watanabe


Balazs Scheidler さんは書きました:
>On Tue, 2008-10-14 at 19:28 +0900, Yu Watanabe wrote:
>> Hello all,
>> 
>> I am planning to build a syslog server using syslog-ng v 2.0.5.
>> 
>> However, I am very concerned with the how much performace could 
>> syslog-ng can handle.
>> 
>> Following is the system specification I am using.
>> 
>> OS : RHEL 4 or RHEL 5
>> CPU : Xeon(R) E5205 (1.86GHz)×2
>> Memory : 4 GB
>> 
>> I would like get an advice with the performace of the syslog-ng.
>> Does any one has a testimonial with this?
>
>Performance depends greatly on the exact configuration you are planning
>to deploy.
>
>There's a small utility in the syslog-ng distribution, called loggen
>(available starting at 2.0.8), it is a simple log traffic generator,
>that you can use to measure performance.
>
>I have some raw numbers though:
> * syslog-ng OSE 2.0 was the baseline, it did not receive the
>performance improvements that I made during the 2.1 PE development
>cycle,
> * syslog-ng PE 2.1 performed about 70000 messages/sec at 150
>bytes/message, writing everything in a single plain text file
> * syslog-ng OSE 2.1 received _some_ of the performance updates, so it
>should be better than OSE 2.0, but not up to PE 2.1
> * syslog-ng OSE 3.0 (currently released as a git tree and nightly
>snapshots) should be on par with syslog-ng PE 3.0 and 2.1; some
>performance related work went into PE 3.0 as well, so it should be
>somewhat better than PE 2.1, especially if the new protocol is in use; I
>measured about 75000msg/sec into plain text files and about 50000msg/sec
>when writing into logstore.
>
>The PE 2.1 performance work was carried out on an 2.2GHz Opteron box in
>64 bit mode. 64 bit mode mattered, so that should be preferred, but I
>can't remember the exact numbers.
>
>I only used "tcp" as transport, udp becomes very lossy at large pkt/sec.
>
>I only used a single tcp connection on the input side.
>
>syslog-ng uses threads only in limited ways (when using SQL for
>instance), so it'll not scale to multiple CPUs.
>
>Please note that 70k msg/sec at 150 bytes/msg each, you get 10MB/sec, or
>about 36GB/hour.
>
>-- 
>Bazsi
>
>______________________________________________________________________________
>Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>FAQ: http://www.campin.net/syslog-ng/faq.html
>