[syslog-ng]syslog-ng vs (of all things) Win2k + IIS

[Jeffrey W. Baker]

On Fri, 6 Oct 2000, Scott McDermott wrote:

> Jeffrey W. Baker on Thu  5/10 15:07 -0700:
> > You had better make sure that the disk on the destination is faster
> > than the sum of the logging rates of all the other hosts, or the
> > syslog-ng on the destination machine will start throwing entries away,
> > and *then* you'll really be embarrassed :)
> 
> Why does syslog-ng "throw messages away?" Shouldn't they be buffered
> instead of discarded? Surely memory can keep up.  It is unacceptable
> for messages to be thrown away.  You might as well just use UDP and
> `hope' all messages arrive.

I don't know, and yes.

The current syslog-ng design works like this.  There is an iteration loop
that syslog-ng goes through.  At the beginning of the loop, it reads log
entries from the source.  Then it adds these log entries to a destination
queue.  Then it flushes the destination queue.  Repeat.

The problem is that if you read N entries from the input, and your queue
size is only n, where N > n, N - n entries will be discarded.

Unfortunately, raising the queue size (via log_fifo_size) is not the
solution.  If you do that, then syslog-ng's memory consumption will
increase at the rate of N' - n', and eventually will either overflow the
queue size anyway, or exhaust your system of memory.

> Syslog-ng could be more efficient still by allocating large chunks of
> memory (maybe using obstacks) for each destination and then
> batch-writing them (say, when an alarm expires).  I imagine that
> syslog-ng spends a lot of time in system calls because it writes each
> message individually.