[syslog-ng]TCP transport reliability

[Balazs Scheidler]

Hello,

> > I'm testing syslog-ng with TCP transport, and I'm a little surprised-
> > perhaps I'm doing something wrong? I'm finding severe message loss,
> > which I had imagined TCP would eliminate.
> 
> I've reported the same thing on this list in the past.  Indeed, my
> machines can even lose messages when the input is a pipe and the output is
> a local file!

And I replied that you are _not_ correct. Several times. Please stop
spreading misinformation.

> The culprit is this bit of code at line 71 in libol:
> 
>         if (self->queue_size == self->queue_max) {
>                 /* fifo full */
>                 ol_string_free(string);
>                 return ST_FAIL | ST_OK;
>         }
> 
> If the queue is full, your message is dismissed and cannot be recovered.
> If you put a printf() statement in that code block, I bet you will see
> that you are hitting this misfeature.  I tried fixing this problem, but
> there is not a simple way.  It requires significant changes to syslog-ng's
> message handling loop.

You have to raise the log_fifo_size variable. Try to estimate the number of
possible received messages in a single poll run. Each message source
receives maximum 1024 bytes of data. A single log message is about 20 to 30
bytes. So on a single run each log source may emit 1024/20 = 50 messages.

Your log fifo must be able to hold this number of messages for each source.
So for two sources, you'll need at least 100 slots in your fifo:

destination d_tcp { tcp(x.x.x.x log_fifo_size(100)); };

-- 
Bazsi
PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1
     url: http://www.balabit.hu/pgpkey.txt