[syslog-ng]performance test questions

[Roberto Nibali]

Hi,

> no, your feedback is appreciated.

Ok, meanwhile I've conducted some tests with syslog-ng-1.5.26 and libol-0.3.9 
plus a hand-applied templates patch. I've written a little test tool to send log 
messages to /dev/log where they will be fetched by syslog[d|-ng] and transported 
via UDP or in case of syslog-ng over UDP/TCP to a central syslog-ng server. The 
tests were all done over a 10Mbit/s switched network with no other traffic. The 
hosts are all running a 2.2.x kernel and tools have been compiled with glibc-2.1.3.

I was interested in how many log messages (burst mode, meaning that there is no 
usleep between sending messages off via syslog()) one can reliably send over the 
net and store on the disk depending on the level of enabled MACRO expansion. The 
results during the tests were quite instable so I had to do a lot of repetitions 
to get a useful mean per test case.

I do not claim full correctness of these results but they sure give you an 
impression on the impact of using UDP vs. TCP or enabling MACRO expansion on 
different levels.

Using 1 client (5000 messages total):
--------------
syslogd as a client (UDP transfer)
    no   MACRO         : ~5000 messages (100%) #loss encountered but minimal
    file MACRO         : ~4250 messages (83%)
    file+template MACRO: ~3200 messages (64%)

syslog-ng as a client (UDP transfer)
    no   MACRO         : ~3500 messages (100% / 70%)
    file MACRO         : ~2800 messages (80%  / 66%)
    file+template MACRO: ~2250 messages (64%  / 70%)

syslog-ng as a client (TCP transfer)
    no   MACRO         : ~5000 messages (100%)
    file MACRO         : ~4300 messages (83%)  #between 3400 and 5000 :(
    file+template MACRO: ~3600 messages (72%)


Using 3 clients (5000 messages each, 15000 messages total):
---------------
syslogd as a client (UDP transfer)
    no   MACRO         : ~5800 messages (39%)
    file MACRO         : ~4000 messages (27%)
    file+template MACRO: ~3200 messages (21%)

syslog-ng as a client (UDP transfer)
    no   MACRO         : ~7200 messages (48% / 124%)
    file MACRO         : ~5200 messages (35% / 130%)
    file+template MACRO: ~3700 messages (25% / 116%)

syslog-ng as a client (TCP transfer)
    no   MACRO         : ~15000 messages (100%) #always 15000!
    file MACRO         : ~11000 messages (73%)
    file+template MACRO: ~8800  messages (59%)

I can provide you with more information but I do not want to unneccessary cludge 
this list with useless or even offtopic information. The options I've used on 
the server side were (actually I tried a lot of different option but all in all 
the global picture of message loss didn't vary a lot):

options {
         keep_hostname(no);
         long_hostnames(off);
         #check_hostname(yes);
         use_time_recvd(yes);
         use_dns(no);
	dns_cache_expire(600);
	dns_cache_size(10000);
	dns_cache(yes);
	use_fqdn(no);
	create_dirs(yes);
	sync(0);
         stats(10);
         time_reopen (5);
         log_fifo_size (1000);
};

There is a lot to say about those numbers but I'd rather not interprete too much 
at this point. It seems though that increasing the amount of clients and using 
UDP as a transfer mechanism definitely worsens the performance, which to a 
certain degree is logical.

Another thing is the rather high performance loss with enabled templates. I 
understand that in case of templates, more of a message has to be inspected but 
I'm asking myself if there would be a possibility to add some kind of a caching 
mechanism?

I'm very open to give you more details on the test or to discuss the test 
conduct or even to repeat it with different settings (provided time permits it).

> If you are using syslog-ng under Linux, try sending messages to a terminal,
> /dev/tty12 for example, and press Ctrl+S on that terminal.
> 
> This will block writes and should make syslog-ng drop messages.

Hmm, so only outgoing buffers on blocking destinations get accredited. Seems 
kind  of logic. So for incoming buffers one would need to implement sequence 
numbers. Have you thought about implementing that?

> I've also added this to my CVS tree since then. I have created scripts to
> generate automatic daily snapshots. They are available at:
> 
> http://www.balabit.com/downloads/syslog-ng/1.5/src-snapshot/

Nice, except the hindering shockwave and html stuff around that on the page. 
Would it be possible to export it to a txt-only directory (for the links/lynx 
users)? ;)

> Can you check if it works for you?

I'll compile and beat it tonight.

> I am trying to push 1.6.0 out but I don't want to release a broken version.

As an OSS developer myself I completely understand that.

Best regards,
Roberto Nibali, ratz
-- 
echo '[q]sa[ln0=aln256%Pln256/snlbx]sb3135071790101768542287578439snlbxq' | dc