On Thu, 20 Jul 2006 09:57:44 CDT, "SOLIS, ALEX" said:
I guess I forgot that a pipe is not a file. I thought it would just store the logs until the reader woke up again. Is the pipe is limited by system resources or something else?
The basic problem is that all the buffered data ends up sitting in memory, and usually in locked, unswappable pages. As a result, if you let a pipe have a 128K buffer, then each pipe that gets stuck can potentially nail down 128K of RAM, causing extreme memory pressure. (As an aside, making these pages swappable is often harder than you think, which is why they end up being nailed-down pages). In the Linux 2.16.18-rc1-mm1 kernel I'm running now, a writer to a pipe is allowed up to 16 4K pages (or a total of 64K) of writes before it blocks waiting for a existing reader to actually accept data - and if the reader goes away entirely, it generates an immediate SIGPIPE.
This explains the dropped stat messages but the fifo is only one destination. Why doesn't syslog-ng drop the messages destined for destination(pipe /foo/fifo/} and continue to send messages to all the other destinations successfully? I have evidence to support that some messages to all destinations are dropped. Why?
It *should* send to other destinations OK. I'm pretty sure that "dropped" gets incremented if *any* destination fails (so if 5 success, one fails, that's still a dropped++). What evidence do you have that some are being dropped entirely?