[syslog-ng] syslog-ng dropping log messages forwarded w/TCP

[Jeffrey W. Baker]

You wrote:
>Probably the speed of your network is less than required. How fast are
>you
>sending those messages? syslog-ng can do with quite high loads (2G logs
>a
>day), but it still cannot widen your network bandwidth. Note that log
>connections in syslog-ng (e.g. a log statement) is not flowcontrolled.
>This
>means that messages are continously read even if they have not yet been
>flushed to the destination. The reason behind this is to prevent
>syslog-ng
>to become the bottleneck. 
>

I don't think we have been quite clear on the nature of the problem.  We
are sending messages at a very high rate over a named pipe to the local
syslog-ng, and these messages are being forwarded via tcp to a remote
machine, and also are logged in a local file.  Our network is switched 100
Mbps, and the machines at each end are 450 MHz and 600 MHz Intel Pentium
III with 3Com 3c905B NICs.  It seems unlikely that our network is the
problem.

The exact sequence of events is that our program is started which makes
100000 log entries in a tight loop.  After the program exits, the local
log file has all 100000 entries, but the remote log file has only about
25000 entries.  No amount of waiting increases the number of entries in
the remote log.  The entries in the remote log are not sequentially
numbered.  For example, they skip from 95432 to 96109.  The intervening
log entries are simply dropped.  This is why we believe the problem is in
syslog-ng.

At any rate it is possible that James has found the problem.  Apparently
the return value of write is not being corretly interpreted when zero
bytes were written.  Perhaps James will have a patch soonish.

Regards and thanks for your software,
Jeffrey Baker

>I may add an option to log statements to stop reading from the source if
>the
>destination fifo is full. This is however not trivial, since several
>sources
>can log to the same destination. I plan something like this:
>
>log { source(src); options(flow_control(yes)); destination(dst); };