On Mon, Dec 30, 2002 at 06:22:36PM +0100, Andreas Schulze wrote:
Balazs Scheidler wrote:
On Sat, Dec 28, 2002 at 10:01:07AM +0300, Borzenkov Andrey wrote:
- file() destinations have a feature (log_fifo_size()), that allows configuration of output queue. Seems, this's a good feature for pipe() destinations, too?
This problem is solved for syslog-ng 2 where all the common parameters are parsed once. It is of course possible to add parameters like this to all drivers but is a bit more difficult in 1.5.x.
- But, the more important... As you say above, if the output queue is full, syslog-ng drops on the input side. But we must ensure, that _exactly_ whole messages are dropped or queued, regardless of the available queue size in bytes. This is what I meant with line based queuing. We shouldn't forget over all the I/O buffering, that our basic data structure is a newline terminated syslog message.
the buffering itself is message based (log_fifo_size specifies the buffer measured in messages not in bytes). The final I/O buffer is used to prevent writing messages in multiple write() calls. Instead a buffer is filled (4kB) and is written as a single chunk.
BTW it looks like syslog-ng makes writes in fixed 4096 buffer length. For pipes it is wrong - system guarantees atomic delivery of a single write up to PIPE_BUF; system is free to reorder parts of any single write with larger size. Portable way to query for PIPE_BUF on modern systems is to use (f)pathconf utility.
Yepp. PIPE_BUF is 4KB on linux, but 5KB on Solaris (See limits(4)). So, to ensure an atomic write(2) on Solaris, you can safely use a 5k buffer.
BTW: syslog-ng puts the pipe fd in nonblocking mode, so if the chunk is too large, write() should simply indicate that fewer bytes was written than originally asked to.
Yes. Its the usual way, use only atomic write(2) with <=PIPE_BUF on nonblocking pipes, to avoid all the stuff that can happen, if only parts of the buffer are written to the fd.
syslog-ng handles the cases when write() is not atomic. So this is a non-issue. The value of 4096 bytes might not be the best but syslog-ng should work regardless the actual value of PIPE_BUF. -- Bazsi PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1