UDP source performance improvement sugestions
Hello, I'm doing some performance tests between syslog-ng and rs####g (yes I guess I'm not the first to think/do something like this). I've noticed that syslog-ng is much slower in the case of a single UDP source and destination to file due to the nature of the message reception code. I then decided to play a bit with the buffering parameters that syslog-ng allows me to set to try improve its performance in this specific test. I expect that when I set time_sleep to 0, it even avoids calling nanosleep from what I saw, so it should continuously read from the UDP socket. I then set a big log_fifo_size 10000 (without knowing exactly if its useful or not in this case, destination is file) and on the source udp I set log_fetch_limit also something like 5000, log_iw_size to 10000. These steps didn't improved the performance that much, maybe I could be doing something wrong..
From what I understood, this will make syslog-ng read 5000 messages from the socket (if they are available) and only after that it queues them to the fifo? fifo waits for flush_lines number of lines or flush_timeout to write the data to the file?
Is there anything else that you know I could do to improve? flush_lines > 0? .. or maybe some resource/doc I could/should read. Thanks for your time. Jean Mousinho
On Fri, 2010-02-12 at 12:55 +0000, Jean F. Mousinho wrote:
Hello,
I'm doing some performance tests between syslog-ng and rs####g (yes I guess I'm not the first to think/do something like this).
I've noticed that syslog-ng is much slower in the case of a single UDP source and destination to file due to the nature of the message reception code.
I then decided to play a bit with the buffering parameters that syslog-ng allows me to set to try improve its performance in this specific test.
I expect that when I set time_sleep to 0, it even avoids calling nanosleep from what I saw, so it should continuously read from the UDP socket. I then set a big log_fifo_size 10000 (without knowing exactly if its useful or not in this case, destination is file) and on the source udp I set log_fetch_limit also something like 5000, log_iw_size to 10000.
These steps didn't improved the performance that much, maybe I could be doing something wrong..
From what I understood, this will make syslog-ng read 5000 messages from the socket (if they are available) and only after that it queues them to the fifo? fifo waits for flush_lines number of lines or flush_timeout to write the data to the file?
Is there anything else that you know I could do to improve? flush_lines > 0? .. or maybe some resource/doc I could/should read.
By performance you mean that incoming udp messages are dropped? Does "netstat -ns" show the udp receive error count increasing? if that's what you mean under performance problems, then the root cause might be the latency it takes syslog-ng to go back to the udp socket after writing a chunk of data to a destination file. apart from increasing the udp receive buffer, there's not much you can do. refactoring syslog-ng to use a dedicated receiver thread for incoming UDP traffic could help the latency issue, but it's quite complex change. I plan to do that in the nearfuture though. -- Bazsi
On Mon, 2010-02-15 at 17:12 +0100, Balazs Scheidler wrote:
On Fri, 2010-02-12 at 12:55 +0000, Jean F. Mousinho wrote:
Hello,
I'm doing some performance tests between syslog-ng and rs####g (yes I guess I'm not the first to think/do something like this).
I've noticed that syslog-ng is much slower in the case of a single UDP source and destination to file due to the nature of the message reception code.
By performance you mean that incoming udp messages are dropped? Does "netstat -ns" show the udp receive error count increasing?
Yes, I'm actually using that parameter to compare the syslog daemons. So for the same amount of send syslog packets with aproximate idle interval between them, syslog-ng takes longer to receive and process them so packets are lost. Although I belive that in the case of multisource reception syslog-ng might be faster again, due to the nature of the reception code..
if that's what you mean under performance problems, then the root cause might be the latency it takes syslog-ng to go back to the udp socket after writing a chunk of data to a destination file.
OK. The project I'm working its specially important to have good 514 UDP and TCP reception performance, so thats why I focus the performance tests on those.
I plan to do that in the nearfuture though.
Its good to know that in advance :-) I personally prefer the syslog-ng configuration structure, and the code looks more organized (its just my opinion). But I need to get some conclusions (with numbers) on the best solution for the probject.. lets see how it goes, going to test the UDP+TCP performance next... Thanks for your time. Jean Mousinho
participants (2)
-
Balazs Scheidler
-
Jean F. Mousinho