[syslog-ng]Urgent: Mangled messages in pipe() destinations in versions >1.5.8

[Andreas Schulze]

Balazs Scheidler wrote:
> On Sat, Dec 28, 2002 at 10:01:07AM +0300, Borzenkov Andrey wrote:
> 
>>Another possibility is to push buffer back into output queue when write
>>returns EAGAIN/EWOULDBLOCK. It has some potential of blocking output to a
>>single destination, but then if destination cannot accept input anymore, it
>>is probably broken anyway. Messages can be dropped on input side then (i.e.
>>if queue is full, simply no new message is queued for this destination).
> 
> This was the indented behaviour and the patch I sent in my previous mail
> does this.

Sounds good. This is the other solution.
I'll try the patch out. Thanks a lot for it.

But here are two questions for this:

- file() destinations have a feature (log_fifo_size()), that allows
configuration of output queue.
Seems, this's a good feature for pipe() destinations, too?

- But, the more important...
As you say above, if the output queue is full, syslog-ng drops on the
input side.
But we must ensure, that _exactly_ whole messages are dropped or queued,
regardless of the available queue size in bytes.
This is what I meant with line based queuing.
We shouldn't forget over all the I/O buffering, that our basic data
structure is a newline terminated syslog message.

>>BTW it looks like syslog-ng makes writes in fixed 4096 buffer length. For
>>pipes it is wrong - system guarantees atomic delivery of a single write up
>>to PIPE_BUF; system is free to reorder parts of any single write with larger
>>size. Portable way to query for PIPE_BUF on modern systems is to use
>>(f)pathconf utility.

Yepp. PIPE_BUF is 4KB on linux, but 5KB on Solaris (See limits(4)).
So, to ensure an atomic write(2) on Solaris, you can safely
use a 5k buffer.

> BTW: syslog-ng puts the pipe fd in nonblocking mode, so if the chunk is too
> large, write() should simply indicate that fewer bytes was written than
> originally asked to.

Yes. Its the usual way, use only atomic write(2) with <=PIPE_BUF on
nonblocking pipes, to avoid all the stuff that can happen, if only
parts of the buffer are written to the fd.

-- 
Best regards --Andreas Schulze
                [phone: +49.5246.80.1275, fax: +49.5246.80.2275]

| I believe, it was Dennis Ritchie who said something like:
|   "C is rarely the best language for a given task,
|    but it's often the second-best".
| The implication being that: "[...]"
|
| sh# cat>$$.c<<EOT
| main(l,a,n,d)char**a;{for(d=atoi(a[1])/10*80-atoi(a[2])/5-596;n="@NK\
| ACLCCGZAAQBEAADAFaISADJABBA^SNLGAQABDAXIMBAACTBATAHDBANZcEMMCCCCAAhE\
| IJFAEAAABAfHJETBdFLDAANEfDNBPHdBcBBBEA_AL H E L L O,    W O R L D! "
| [l++-3];)for(;n-->64;)putchar(!d+++33^l&1);}
| EOT
| gcc -o$$ $$.c;clear;./$$ 52 8;rm -f $$*