[syslog-ng] Loss of messages: what is the limits ?

[Evan Rempel]

Balazs Scheidler wrote:
> On Wed, 2007-12-12 at 13:40 +0100, FILEUX arnaud wrote:
>> Dear all,
>>
>> I realized some bench tests in order to determine the
>> limits supported in tcp connection with a syslog-ng
>> (client/server mode) and what is the optimal
>> configuration (buffer size,etc..).
>>
>> My setup is as follows:
>>
>> 2 syslog-ng (client A  - server B) on dedicated
>> servers with 8Go RAM and quadripro CPU.
>>
>>
>> Client A is configured as follow:
>>
>> destination  to_serverB { tcp("x.x.x.x" port(1999));
>> };
>> destination  to_clientA {
>> file("/var/log/syslog-ng.log"); };
>>
>> Server B:
>> destination all { file("/var/log/syslog-ng.log"); };
>>
>>
>> On the client A i have a a C program (named
>> stressyslog) who send send 10.000 to 500.000 messages
>> on 10 to 100 threads and the delay between sending is
>> 1ms to 10 ms to Client A with :
>> syslog(LOG_NOTICE, "go %d pThread for %d messages and
>> %d ms", iThreads, iMessages, iDelay);
>>
>> I make a diff between clientA and serverB and i lose
>> some lines on ServerB (delai less than 5ms and thread
>> greater than 50)
>>
>> example:
>> delay: 2ms ;
>> messages: 50000;
>> thread 100;
>>
>> Server B has 4947888 lines in syslog-ng.log
>> Client A has 5000087 lines in syslog-ng.log
>>
>> Another problem is syslog-ng never alert if some line
>> are lost beteween Client A and Server  B.
>>
>> I have tried to increase the buffer size (10.000 and
>> 100.000) but the same problem.
> 
> It'd be useful to know where the message is lost. There are multiple
> possibilities:
> 
> 1) on the local log transport between your application and syslog-ng
> (e.g. /dev/log) are you using unix-dgram or unix-stream ?
> 2) by syslog-ng itself, if the message does not fit into the output
> buffer it'll be dropped, but these drops are counted, check the "log
> statistics" log line, it includes the dropped counter
> 3) if you are using tcp() then the TCP connection will not lose messages
> 4) the receiving syslog-ng may drop a message if the fifo of the
> destination file gets full, but this also shows up in the log statistics
> message of the server syslog-ng

My experience tells me that these drops that are destined to a file are NOT logged in
the statistics message that syslog-ng reports, in fact there are no stats on any file
destinations at all.

-- 
Evan Rempel                erempel at uvic.ca
Senior Programmer Analyst        250.721.7691
Computing Services
University of Victoria