On Tue, 2008-07-01 at 13:52 -0400, Joe Shaw wrote:
Hi,
I was doing a little bit of testing with syslog-ng today, and I think that with a very small log_fifo_size() it was dropping messages but not telling me.
If I set my log_fifo_size to 1, here are the results:
[jshaw@ars1dev7 ~/syslog-ng-2.0.9/tests/loggen]$ ./loggen -r 1000000 -s 1024000 -I 60 localhost 2000 average rate = 96.48 msg/sec, count=5789 [jshaw@ars1dev7 ~/syslog-ng-2.0.9/tests/loggen]$ ls -lh /ita/jshaw/messages -rw------- 1 jshaw ita 101M Jul 1 13:30 /ita/jshaw/messages
However, 1024000 * 5789 = 5.9 GB
Grepping for drop messages in the output file:
[jshaw@ars1dev7 ~/syslog-ng-2.0.9/tests/loggen]$ grep -i drop /ita/jshaw/messages
returns nothing. I have stats_freq(5) in my config file, and that messages file is the only output. I do have other statistical lines, which imply to me that no messages are dropped:
Jul 1 13:49:52 s_local@ars1dev7 syslog-ng[6344]: Log statistics; processed='center(queued)=6045', processed='center(received)=6045', processed='destination(d_file)=6045', processed='source(s_local)=6045'
But it seems clear to me that they are -- or loggen is lying.
With more normal log_fifo_sizes (100, 2000) the throughput rates from loggen are lower and the file sizes match.
syslog-ng does not log message loss on file destinations, because that'd clutter the log statistics line, on the asssumption that no messages are can be lost there. However this is not always the case, if you log_fifo_size() is smaller than the window size of incoming sources, messages can be lost. Here is a description of how flow-control and window sizes work in syslog-ng: http://www.balabit.hu/dl/html/syslog-ng-admin-guide_en.html/ch08s03.html Please let me (and the doc team) know if the description could be made clearer somehow. -- Bazsi