[syslog-ng]TCP transport reliability

[Scott McDermott]

Balazs Scheidler on Thu  4/01 19:20 +0100:
> > > I'm testing syslog-ng with TCP transport, and I'm a little
> > > surprised- perhaps I'm doing something wrong? I'm finding severe
> > > message loss, which I had imagined TCP would eliminate.
> > 
> > I've reported the same thing on this list in the past.  Indeed, my
> > machines can even lose messages when the input is a pipe and the
> > output is a local file!
> 
> And I replied that you are _not_ correct. Several times. Please stop
> spreading misinformation.

Actually, he is correct, from a simple point of view.  Message *are*
lost if the queue is not large enough.  This is quite clear below; the
string is freed and the loop continues, without the log ever being
written to backing store anywhere.

> > The culprit is this bit of code at line 71 in libol:
> > 
> >         if (self->queue_size == self->queue_max) {
> >                 /* fifo full */
> >                 ol_string_free(string);
> >                 return ST_FAIL | ST_OK;
> >         }
> > 
> > If the queue is full, your message is dismissed and cannot be
> > recovered.  If you put a printf() statement in that code block, I
> > bet you will see that you are hitting this misfeature.  I tried
> > fixing this problem, but there is not a simple way.  It requires
> > significant changes to syslog-ng's message handling loop.
> 
> You have to raise the log_fifo_size variable.

This shouldn't require configuration.  Ideally, log_fifo_size should
grow and shrink dynamically.