[syslog-ng]syslog-ng vs (of all things) Win2k + IIS

[Balazs Scheidler]

> > > You had better make sure that the disk on the destination is faster
> > > than the sum of the logging rates of all the other hosts, or the
> > > syslog-ng on the destination machine will start throwing entries away,
> > > and *then* you'll really be embarrassed :)
> > 
> > Why does syslog-ng "throw messages away?" Shouldn't they be buffered
> > instead of discarded? Surely memory can keep up.  It is unacceptable
> > for messages to be thrown away.  You might as well just use UDP and
> > `hope' all messages arrive.
> 
> I don't know, and yes.
> 
> The current syslog-ng design works like this.  There is an iteration loop
> that syslog-ng goes through.  At the beginning of the loop, it reads log
> entries from the source.  Then it adds these log entries to a destination
> queue.  Then it flushes the destination queue.  Repeat.
> 
> The problem is that if you read N entries from the input, and your queue
> size is only n, where N > n, N - n entries will be discarded.

This is not true.

Syslog-ng uses a fully nonblocking I/O backend based on poll() and
callbacks. If poll detects that an fd is readable, a callback is called
which reads all available data from the fd (but maximum MAX_LINE bytes,
which is currently 2048) and  each line in this chunk is sent towards the
destination. This procedure is repeated for each fd (either to be read or
written).

The worst case is the following scenario:
1) the destination fd is the first to be polled, so there's nothing waiting
to be written.
2) each source fd is after the destination fd, and each has some data
available
3) the destination will not be flushed until the next iteration

You only have to make sure that the destination fifo doesn't overflow in a
single poll loop. If you assume that a 2048 byte block may contain 20
messages, and you have 10 sources, you'll have to increase the fifo size to
at least 10*20=200 entries.

> Unfortunately, raising the queue size (via log_fifo_size) is not the
> solution.  If you do that, then syslog-ng's memory consumption will
> increase at the rate of N' - n', and eventually will either overflow the
> queue size anyway, or exhaust your system of memory.

see above.

-- 
Bazsi
PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1
     url: http://www.balabit.hu/pgpkey.txt