Balazs Scheidler on Thu 4/01 19:20 +0100:
I'm testing syslog-ng with TCP transport, and I'm a little surprised- perhaps I'm doing something wrong? I'm finding severe message loss, which I had imagined TCP would eliminate.
I've reported the same thing on this list in the past. Indeed, my machines can even lose messages when the input is a pipe and the output is a local file!
And I replied that you are _not_ correct. Several times. Please stop spreading misinformation.
Actually, he is correct, from a simple point of view. Message *are* lost if the queue is not large enough. This is quite clear below; the string is freed and the loop continues, without the log ever being written to backing store anywhere.
The culprit is this bit of code at line 71 in libol:
if (self->queue_size == self->queue_max) { /* fifo full */ ol_string_free(string); return ST_FAIL | ST_OK; }
If the queue is full, your message is dismissed and cannot be recovered. If you put a printf() statement in that code block, I bet you will see that you are hitting this misfeature. I tried fixing this problem, but there is not a simple way. It requires significant changes to syslog-ng's message handling loop.
You have to raise the log_fifo_size variable.
This shouldn't require configuration. Ideally, log_fifo_size should grow and shrink dynamically.