[syslog-ng] syslog-ng dropping log messages forwarded w/TCP

[Balazs Scheidler]

> I have two Linux machines (slackware 7 dist.) running syslog-ng 1.4.0.
> The first machine is set to forward all of its log entries, and write
> them to a file:
> 
> source src { pipe("/home/james/log_foo/logfifo"); internal(); };
> filter all { match("*"); };
> destination dest {
>     file("/home/james/dev/syslog-ng/log/foo.log");
>     tcp("192.168.1.201" port(8081));
> };
> log { source(src); filter(all); destination(dest); };
> 
> The second machine is set up to collect these and log them to a file:
> 
> source src { tcp(ip(192.168.1.201) port(8081)); internal(); };
> filter all { match("*"); };
> destination dest { file("/home/james/dev/syslog-ng/log/foo.log"); };
> log { source(src); filter(all); destination(dest); };
> 
> When I send a batch of 100k log messages to the first machine, they all
> end up in its file, but about 25% of them never make it through to the
> second machine's log.
> 
> Any ideas?

Probably the speed of your network is less than required. How fast are you
sending those messages? syslog-ng can do with quite high loads (2G logs a
day), but it still cannot widen your network bandwidth. Note that log
connections in syslog-ng (e.g. a log statement) is not flowcontrolled. This
means that messages are continously read even if they have not yet been
flushed to the destination. The reason behind this is to prevent syslog-ng
to become the bottleneck. 

I may add an option to log statements to stop reading from the source if the
destination fifo is full. This is however not trivial, since several sources
can log to the same destination. I plan something like this:

log { source(src); options(flow_control(yes)); destination(dst); };

-- 
Bazsi
PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1
     url: http://www.balabit.hu/pgpkey.txt