[syslog-ng] Program destination seems to block

[Evan Rempel]

A little background.

There is a "server" syslog-ng process that accepts messages from the network and
sends the messages to a variety of destinations. For this report, I am only interested
in one destination that happens to be a pipe.

There is a "slave" syslog-ng process that reads from the pipe that the "server" writes to,
and writes to a program destination.

The program reads the standard in, and does "something".
All works well.

At some point our application (we know why and don't want to discuss it) application stops
reading standard in for a while (1,000,000 lines over an hour). We expect that the memory
footprint of syslog-ng "slave" to grow during this time but it does not. Instead, the
memory footprint of the syslog-ng "server" grows. When our application starts reading
its standard in again, the memory footprint of the syslog "slave" grows very quickly,
and all messages reach the destination.

I think that the syslog-ng "slave" get blocked on the program destination in a way that
prevents it from reading its source, resulting in the upstream syslog-ng "server" having
to buffer all of the messages.

There is no flow control anywhere, and both syslog-ng instances have log_fifo_size(8000000)
for all of the destinations.

Anyone have any suggestions?

Evan.