On Fri, 6 Oct 2000, Scott McDermott wrote:
Jeffrey W. Baker on Thu 5/10 15:07 -0700:
You had better make sure that the disk on the destination is faster than the sum of the logging rates of all the other hosts, or the syslog-ng on the destination machine will start throwing entries away, and *then* you'll really be embarrassed :)
Why does syslog-ng "throw messages away?" Shouldn't they be buffered instead of discarded? Surely memory can keep up. It is unacceptable for messages to be thrown away. You might as well just use UDP and `hope' all messages arrive.
I don't know, and yes. The current syslog-ng design works like this. There is an iteration loop that syslog-ng goes through. At the beginning of the loop, it reads log entries from the source. Then it adds these log entries to a destination queue. Then it flushes the destination queue. Repeat. The problem is that if you read N entries from the input, and your queue size is only n, where N > n, N - n entries will be discarded. Unfortunately, raising the queue size (via log_fifo_size) is not the solution. If you do that, then syslog-ng's memory consumption will increase at the rate of N' - n', and eventually will either overflow the queue size anyway, or exhaust your system of memory.
Syslog-ng could be more efficient still by allocating large chunks of memory (maybe using obstacks) for each destination and then batch-writing them (say, when an alarm expires). I imagine that syslog-ng spends a lot of time in system calls because it writes each message individually.