[syslog-ng] Stats dropped

[Balazs Scheidler]

On Thu, 2007-02-01 at 18:24 +0000, Hari Sekhon wrote:
> yes this is what I was thinking but the server also dropped some. I will 
> try to increase the fifo by the amount of dropped stats on the server, 
> make it 40000 instead of 20000 and then on the agents give them a fifo 
> of 30000 just to be safe. From what I've measure it doesn't seem to take 
> much ram so this should be ok. Although this wasn't done at the times of 
> heavier load when the machine's fifo is filling up.
> 
> If anybody knows more on this or has some info on the mem consumption 
> related to doing this then please let me know

We have investigated a similar issue in-house, and as it seems the
performance improvements in 1.6.10 and .11 and 2.0.0 increase the load
on the destination buffers significantly, which in turn can cause
message loss in cases where it did not occur before.

For example, previously file destinations could only drop messages, if
there were at least 100 log connections, producing a log message in a
single poll iteration. (assuming log_fifo_size(100), the default).

Later versions have a much higher probability of internal message drops.
This can be solved with increasing log_fifo_size(), the out-of-the-box
defaults seem to be unadequate right now.

The problem is:
  * message sources read maximum 30 messages at a single poll iteration 
  * each socket connection on /dev/log is a message source in this sense
(e.g. if there are 10 programs connecting to /dev/log, then a single
input iteration can produce as much as 10*30 = 300 messages)
  * these messages are put into the destination's fifo, with a default
size of 100 messages.

Previously each source generated at most 1 message per poll cycle, now
it is increased to 30. It is trivial to see that the default
log_fifo_size() value is not adequate.

I'm thinking about increasing the default log_fifo_size to 1000 in 1.6.x

syslog-ng 2.0 is also affected, albeit a bit differently:
  * the number of messages fetched in a single go can be controlled by a
configuration option (per source fetch_limit() option)
  * its default value is 10 instead of 30

I too think, that fifo size would be forced to a minimum value of 1000.

-- 
Bazsi