Loss of messages: what is the limits ?
Dear all, I realized some bench tests in order to determine the limits supported in tcp connection with a syslog-ng (client/server mode) and what is the optimal configuration (buffer size,etc..). My setup is as follows: 2 syslog-ng (client A - server B) on dedicated servers with 8Go RAM and quadripro CPU. Client A is configured as follow: destination to_serverB { tcp("x.x.x.x" port(1999)); }; destination to_clientA { file("/var/log/syslog-ng.log"); }; Server B: destination all { file("/var/log/syslog-ng.log"); }; On the client A i have a a C program (named stressyslog) who send send 10.000 to 500.000 messages on 10 to 100 threads and the delay between sending is 1ms to 10 ms to Client A with : syslog(LOG_NOTICE, "go %d pThread for %d messages and %d ms", iThreads, iMessages, iDelay); I make a diff between clientA and serverB and i lose some lines on ServerB (delai less than 5ms and thread greater than 50) example: delay: 2ms ; messages: 50000; thread 100; Server B has 4947888 lines in syslog-ng.log Client A has 5000087 lines in syslog-ng.log Another problem is syslog-ng never alert if some line are lost beteween Client A and Server B. I have tried to increase the buffer size (10.000 and 100.000) but the same problem. If someone can help me i would be graceful. arnaud _____________________________________________________________________________ Ne gardez plus qu'une seule adresse mail ! Copiez vos mails vers Yahoo! Mail http://mail.yahoo.fr
On Wed, 2007-12-12 at 13:40 +0100, FILEUX arnaud wrote:
Dear all,
I realized some bench tests in order to determine the limits supported in tcp connection with a syslog-ng (client/server mode) and what is the optimal configuration (buffer size,etc..).
My setup is as follows:
2 syslog-ng (client A - server B) on dedicated servers with 8Go RAM and quadripro CPU.
Client A is configured as follow:
destination to_serverB { tcp("x.x.x.x" port(1999)); }; destination to_clientA { file("/var/log/syslog-ng.log"); };
Server B: destination all { file("/var/log/syslog-ng.log"); };
On the client A i have a a C program (named stressyslog) who send send 10.000 to 500.000 messages on 10 to 100 threads and the delay between sending is 1ms to 10 ms to Client A with : syslog(LOG_NOTICE, "go %d pThread for %d messages and %d ms", iThreads, iMessages, iDelay);
I make a diff between clientA and serverB and i lose some lines on ServerB (delai less than 5ms and thread greater than 50)
example: delay: 2ms ; messages: 50000; thread 100;
Server B has 4947888 lines in syslog-ng.log Client A has 5000087 lines in syslog-ng.log
Another problem is syslog-ng never alert if some line are lost beteween Client A and Server B.
I have tried to increase the buffer size (10.000 and 100.000) but the same problem.
It'd be useful to know where the message is lost. There are multiple possibilities: 1) on the local log transport between your application and syslog-ng (e.g. /dev/log) are you using unix-dgram or unix-stream ? 2) by syslog-ng itself, if the message does not fit into the output buffer it'll be dropped, but these drops are counted, check the "log statistics" log line, it includes the dropped counter 3) if you are using tcp() then the TCP connection will not lose messages 4) the receiving syslog-ng may drop a message if the fifo of the destination file gets full, but this also shows up in the log statistics message of the server syslog-ng If you have identified where message gets lost, we might be able to help you. -- Bazsi
Balazs Scheidler wrote:
On Wed, 2007-12-12 at 13:40 +0100, FILEUX arnaud wrote:
Dear all,
I realized some bench tests in order to determine the limits supported in tcp connection with a syslog-ng (client/server mode) and what is the optimal configuration (buffer size,etc..).
My setup is as follows:
2 syslog-ng (client A - server B) on dedicated servers with 8Go RAM and quadripro CPU.
Client A is configured as follow:
destination to_serverB { tcp("x.x.x.x" port(1999)); }; destination to_clientA { file("/var/log/syslog-ng.log"); };
Server B: destination all { file("/var/log/syslog-ng.log"); };
On the client A i have a a C program (named stressyslog) who send send 10.000 to 500.000 messages on 10 to 100 threads and the delay between sending is 1ms to 10 ms to Client A with : syslog(LOG_NOTICE, "go %d pThread for %d messages and %d ms", iThreads, iMessages, iDelay);
I make a diff between clientA and serverB and i lose some lines on ServerB (delai less than 5ms and thread greater than 50)
example: delay: 2ms ; messages: 50000; thread 100;
Server B has 4947888 lines in syslog-ng.log Client A has 5000087 lines in syslog-ng.log
Another problem is syslog-ng never alert if some line are lost beteween Client A and Server B.
I have tried to increase the buffer size (10.000 and 100.000) but the same problem.
It'd be useful to know where the message is lost. There are multiple possibilities:
1) on the local log transport between your application and syslog-ng (e.g. /dev/log) are you using unix-dgram or unix-stream ? 2) by syslog-ng itself, if the message does not fit into the output buffer it'll be dropped, but these drops are counted, check the "log statistics" log line, it includes the dropped counter 3) if you are using tcp() then the TCP connection will not lose messages 4) the receiving syslog-ng may drop a message if the fifo of the destination file gets full, but this also shows up in the log statistics message of the server syslog-ng
My experience tells me that these drops that are destined to a file are NOT logged in the statistics message that syslog-ng reports, in fact there are no stats on any file destinations at all. -- Evan Rempel erempel@uvic.ca Senior Programmer Analyst 250.721.7691 Computing Services University of Victoria
participants (3)
-
Balazs Scheidler
-
Evan Rempel
-
FILEUX arnaud