[syslog-ng] syslog-ng dropping log messages forwarded w/TCP

[Jeffrey W. Baker]

Sorry about that last message.  My real email is of course
jwbaker@acm.org, not jwb@localhost.

Cheers,
-jwb


On Tue, 7 Mar 2000, Jeffrey W. Baker wrote:

> 
> 
> You wrote:
> >Probably the speed of your network is less than required. How fast are
> >you
> >sending those messages? syslog-ng can do with quite high loads (2G logs
> >a
> >day), but it still cannot widen your network bandwidth. Note that log
> >connections in syslog-ng (e.g. a log statement) is not flowcontrolled.
> >This
> >means that messages are continously read even if they have not yet been
> >flushed to the destination. The reason behind this is to prevent
> >syslog-ng
> >to become the bottleneck. 
> >
> 
> I don't think we have been quite clear on the nature of the problem.  We
> are sending messages at a very high rate over a named pipe to the local
> syslog-ng, and these messages are being forwarded via tcp to a remote
> machine, and also are logged in a local file.  Our network is switched 100
> Mbps, and the machines at each end are 450 MHz and 600 MHz Intel Pentium
> III with 3Com 3c905B NICs.  It seems unlikely that our network is the
> problem.
> 
> The exact sequence of events is that our program is started which makes
> 100000 log entries in a tight loop.  After the program exits, the local
> log file has all 100000 entries, but the remote log file has only about
> 25000 entries.  No amount of waiting increases the number of entries in
> the remote log.  The entries in the remote log are not sequentially
> numbered.  For example, they skip from 95432 to 96109.  The intervening
> log entries are simply dropped.  This is why we believe the problem is in
> syslog-ng.
> 
> At any rate it is possible that James has found the problem.  Apparently
> the return value of write is not being corretly interpreted when zero
> bytes were written.  Perhaps James will have a patch soonish.
> 
> Regards and thanks for your software,
> Jeffrey Baker
> 
> >I may add an option to log statements to stop reading from the source if
> >the
> >destination fifo is full. This is however not trivial, since several
> >sources
> >can log to the same destination. I plan something like this:
> >
> >log { source(src); options(flow_control(yes)); destination(dst); };
> 
> 
> 
>