[syslog-ng]Answer to the reliability problem

[Jeffrey W. Baker]

On Sat, 30 Sep 2000, Balazs Scheidler wrote:

> > So you can see that when the queue is full, the messages simply get
> > dropped.  That's bad!
> > 
> > The solution is to flush the queue when it is full, and then continue
> > looping over the available input.  The question is, what code should be
> > responsible for doing this flushing?  Shouldn't some piece of code be
> > checking for the ST_FAIL return code?
> 
> Could you explain why it is better to flush already buffered data?

Yes.

Assume that you have a queue which can hold N, and initially holds n == 0
entries.  You have a source which can produce entries at the rate p', and
a destination that can accept entries at the rate of r'.  r' < p'.  
Therefore, the queue will become full at the rate n' = p' - r'.  
Eventually n will reach N, and the queue must be flushed immediately,
regardless of the fact that more entries will be read from the source.  
The only other option is to simply throw away the extra entries from the
source, which is what is currently happening.

Note that using the log_fifo_size() option to increase N is not
acceptable, since at steady-state that means that syslog-ng's memory
consumption will grow without bounds.

-jwb