[syslog-ng] Syslog-ng to Pipe question

[Valdis.Kletnieks at vt.edu]

On Thu, 20 Jul 2006 09:57:44 CDT, "SOLIS, ALEX" said:
> I guess I forgot that a pipe is not a file.  I thought it would just
> store the logs until the reader woke up again.  Is the pipe is limited
> by system resources or something else?

The basic problem is that all the buffered data ends up sitting in memory,
and usually in locked, unswappable pages.  As a result, if you let a pipe
have a 128K buffer, then each pipe that gets stuck can potentially nail
down 128K of RAM, causing extreme memory pressure.  (As an aside, making
these pages swappable is often harder than you think, which is why they
end up being nailed-down pages).

In the Linux 2.16.18-rc1-mm1 kernel I'm running now, a writer to a pipe is
allowed up to 16 4K pages (or a total of 64K) of writes before it blocks
waiting for a existing reader to actually accept data - and if the reader goes
away entirely, it generates an immediate SIGPIPE.

> This explains the dropped stat messages but the fifo is only one
> destination.  Why doesn't syslog-ng drop the messages destined for
> destination(pipe /foo/fifo/} and continue to send messages to all the
> other destinations successfully?  I have evidence to support that some
> messages to all destinations are dropped.  Why?

It *should* send to other destinations OK.  I'm pretty sure that "dropped"
gets incremented if *any* destination fails (so if 5 success, one fails, that's
still a dropped++).  What evidence do you have that some are being dropped
entirely?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 226 bytes
Desc: not available
Url : http://lists.balabit.hu/pipermail/syslog-ng/attachments/20060720/313a4339/attachment.pgp