[syslog-ng]syslog-ng vs (of all things) Win2k + IIS

[Balazs Scheidler]

> > > You had better make sure that the disk on the destination is faster
> > > than the sum of the logging rates of all the other hosts, or the
> > > syslog-ng on the destination machine will start throwing entries away,
> > > and *then* you'll really be embarrassed :)
> > 
> > Why does syslog-ng "throw messages away?" Shouldn't they be buffered
> > instead of discarded? Surely memory can keep up.  It is unacceptable for
> > messages to be thrown away.  You might as well just use UDP and `hope'
> > all messages arrive.
> 
> I would imagine that syslog-ng would be able to keep up fine with not many
> filters and a fast machine.  I have about 120 Unix/NT systems logging to syslog-ng, on the 
> most part it does well keeping up with it but I do have a few filters which
> I'm sure are quite intensive considering each line logged is sent through
> these filters.  What does it do if it can't deal with the log straight away?
> Puts it in the buffer for when it can deal with it?  What if the buffer gets
> full?
> 
> I would imagine that's why there is the garbage collection stuff, cause it does
> buffer.  I could be very wrong ?!  Where is Balazs. :-)

syslog-ng reads messages, and processes them, if the load is high processing
them takes longer, so it has no time reading its input file descriptors. If
it doesn't read those input fds, the kernel begins to drop messages (in case
of UDP, in case of TCP, windows are not acknowledged)

> > Syslog-ng could be more efficient still by allocating large chunks of
> > memory (maybe using obstacks) for each destination and then
> > batch-writing them (say, when an alarm expires).  I imagine that
> > syslog-ng spends a lot of time in system calls because it writes each
> > message individually.
> 
> I didn't realise that.  From what the documentation says sync() does something
> that might help you.  

sync doesn't batch writes, it simply controls the number of lines collected
in the buffer before _anything_ is written. So if you have sync(10) 10 lines
is collected, and then each of them is written individually.

So this could be improved.

-- 
Bazsi
PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1
     url: http://www.balabit.hu/pgpkey.txt