[syslog-ng]Urgent: Mangled messages in pipe() destinations in versions >1.5.8

[Balazs Scheidler]

On Mon, Dec 30, 2002 at 06:22:36PM +0100, Andreas Schulze wrote:
> Balazs Scheidler wrote:
> >On Sat, Dec 28, 2002 at 10:01:07AM +0300, Borzenkov Andrey wrote:
> 
> - file() destinations have a feature (log_fifo_size()), that allows
> configuration of output queue.
> Seems, this's a good feature for pipe() destinations, too?

This problem is solved for syslog-ng 2 where all the common parameters are
parsed once. It is of course possible to add parameters like this to all
drivers but is a bit more difficult in 1.5.x.

> 
> - But, the more important...
> As you say above, if the output queue is full, syslog-ng drops on the
> input side.
> But we must ensure, that _exactly_ whole messages are dropped or queued,
> regardless of the available queue size in bytes.
> This is what I meant with line based queuing.
> We shouldn't forget over all the I/O buffering, that our basic data
> structure is a newline terminated syslog message.

the buffering itself is message based (log_fifo_size specifies the buffer
measured in messages not in bytes). The final I/O buffer is used to prevent
writing messages in multiple write() calls. Instead a buffer is filled (4kB)
and is written as a single chunk.

> 
> >>BTW it looks like syslog-ng makes writes in fixed 4096 buffer length. For
> >>pipes it is wrong - system guarantees atomic delivery of a single write up
> >>to PIPE_BUF; system is free to reorder parts of any single write with 
> >>larger
> >>size. Portable way to query for PIPE_BUF on modern systems is to use
> >>(f)pathconf utility.
> 
> Yepp. PIPE_BUF is 4KB on linux, but 5KB on Solaris (See limits(4)).
> So, to ensure an atomic write(2) on Solaris, you can safely
> use a 5k buffer.
> 
> >BTW: syslog-ng puts the pipe fd in nonblocking mode, so if the chunk is too
> >large, write() should simply indicate that fewer bytes was written than
> >originally asked to.
> 
> Yes. Its the usual way, use only atomic write(2) with <=PIPE_BUF on
> nonblocking pipes, to avoid all the stuff that can happen, if only
> parts of the buffer are written to the fd.

syslog-ng handles the cases when write() is not atomic. So this is a
non-issue. The value of 4096 bytes might not be the best but syslog-ng
should work regardless the actual value of PIPE_BUF.

-- 
Bazsi
PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1