[syslog-ng]Urgent: Mangled messages in pipe() destinations in versions >1.5.8

[Borzenkov Andrey]

[...]
> What happens:
>=20
> write(48, " < @ - - I L M - - @ >  ".., 4096)	=3D 4096
> [-> Try to write 4096 bytes. Success. No problem.
>      (There are more than one message inside this 4KB)]
> write(48, " c v / % l o s s   =3D   5".., 4096)	=3D 4096
> [-> Try to write the next 4 KBytes. Success. No problem?
>      (WARNING:
>       Because we use a byte buffer instead a line buffer,
>       there is NO tailing '\n' in the pipe now!!)]
> write(48, " c m p d :   |   i c m p".., 508)	Err#11 EAGAIN
> [-> Try to write the last bytes. Pipe is bussy. Drop.

[...]
>=20
> I think, you changed the internal buffering from a line based buffer
> to byte based buffer between 1.5.8 and 1.5.9.
> In 1.5.7 there isn't that problem.
> Maybe, line based buffering seems a better idea in this context.
>=20

Another possibility is to push buffer back into output queue when write
returns EAGAIN/EWOULDBLOCK. It has some potential of blocking output to =
a
single destination, but then if destination cannot accept input =
anymore, it
is probably broken anyway. Messages can be dropped on input side then =
(i.e.
if queue is full, simply no new message is queued for this =
destination).


BTW it looks like syslog-ng makes writes in fixed 4096 buffer length. =
For
pipes it is wrong - system guarantees atomic delivery of a single write =
up
to PIPE_BUF; system is free to reorder parts of any single write with =
larger
size. Portable way to query for PIPE_BUF on modern systems is to use
(f)pathconf utility.

Regards

-andrey