[syslog-ng] Odd behavior with very small log_fifo_size()

[Balazs Scheidler]

On Tue, 2008-07-01 at 13:52 -0400, Joe Shaw wrote:
> Hi,
> 
> I was doing a little bit of testing with syslog-ng today, and I think
> that with a very small log_fifo_size() it was dropping messages but
> not telling me.
> 
> If I set my log_fifo_size to 1, here are the results:
> 
> [jshaw at ars1dev7 ~/syslog-ng-2.0.9/tests/loggen]$ ./loggen -r 1000000
> -s 1024000 -I 60 localhost 2000
> average rate = 96.48 msg/sec, count=5789
> [jshaw at ars1dev7 ~/syslog-ng-2.0.9/tests/loggen]$ ls -lh /ita/jshaw/messages
> -rw-------  1 jshaw ita 101M Jul  1 13:30 /ita/jshaw/messages
> 
> However, 1024000 * 5789 = 5.9 GB
> 
> Grepping for drop messages in the output file:
> 
> [jshaw at ars1dev7 ~/syslog-ng-2.0.9/tests/loggen]$ grep -i drop
> /ita/jshaw/messages
> 
> returns nothing.  I have stats_freq(5) in my config file, and that
> messages file is the only output.  I do have other statistical lines,
> which imply to me that no messages are dropped:
> 
> Jul  1 13:49:52 s_local at ars1dev7 syslog-ng[6344]: Log statistics;
> processed='center(queued)=6045', processed='center(received)=6045',
> processed='destination(d_file)=6045', processed='source(s_local)=6045'
> 
> But it seems clear to me that they are -- or loggen is lying.
> 
> With more normal log_fifo_sizes (100, 2000) the throughput rates from
> loggen are lower and the file sizes match.

syslog-ng does not log message loss on file destinations, because that'd
clutter the log statistics line, on the asssumption that no messages are
can be lost there.

However this is not always the case, if you log_fifo_size() is smaller
than the window size of incoming sources, messages can be lost.

Here is a description of how flow-control and window sizes work in
syslog-ng:

http://www.balabit.hu/dl/html/syslog-ng-admin-guide_en.html/ch08s03.html

Please let me (and the doc team) know if the description could be made
clearer somehow.


-- 
Bazsi