flushing / dropping messages (was Re: [syslog-ng]syslog-ng vs (of all things) Win2k + IIS)

[Scott McDermott]

John on Fri  6/10 16:21 +0100:
> > Why does syslog-ng "throw messages away?" Shouldn't they be buffered
> > instead of discarded? Surely memory can keep up.  It is unacceptable
> > for messages to be thrown away.  You might as well just use UDP and
> > `hope' all messages arrive.
> 
> I would imagine that syslog-ng would be able to keep up fine with not
> many filters and a fast machine.  I have about 120 Unix/NT systems
> logging to syslog-ng, on the most part it does well keeping up with it
> but I do have a few filters which I'm sure are quite intensive
> considering each line logged is sent through these filters.  What does
> it do if it can't deal with the log straight away?  Puts it in the
> buffer for when it can deal with it?  What if the buffer gets full?

Then it reallocs more space :) The only thing that should cause it to
lose messages is an OOM event.  You just keep writing to a big buffer
for the destination, then have it flushed periodically.  If the disk
can't keep up, I guess you are screwed, but it should at least try, and
increase the buffer size if necessary.  Then it should be able to write
out the buffer better during periods when it isn't as swamped with
incoming messages.  Currently there seems to be a hard limit on the
buffer size and this surely will lead to tragedy.

> I would imagine that's why there is the garbage collection stuff,
> cause it does buffer.  I could be very wrong ?!  Where is Balazs. :-)
> 
> > Syslog-ng could be more efficient still by allocating large chunks
> > of memory (maybe using obstacks) for each destination and then
> > batch-writing them (say, when an alarm expires).  I imagine that
> > syslog-ng spends a lot of time in system calls because it writes
> > each message individually.
> 
> I didn't realise that.

Well, it seems I was only partially right.  The logging subsystem
actually does do it using a FIFO queue, as Jeff Baker points out.  I am
not sure how often the queue is flushed (every loop? what's the point of
that?), but it does seem that when it *is* flushed to the IO subsystem,
each message is written out individually (strace/truss reveals this) to
the descriptor.  These should be batched, IMO, so perhaps dozens or
hundreds of messages are written out in one write(2); this would allow
syslog-ng to keep up with a much larger message stream.  A variable
buffer for each FD, and possibly an alarm per destination, or even a
separate metathread for flushing might be one way to go.

> From what the documentation says sync() does something that might help
> you.  

No, this just determines how often sync() is called, which flushes all
file descriptors before it returns (or at least schedules them for
flushing...the semantics depend on OS flavor as to the result of the
flush when sync(2) returns).  What I want to do is be able to schedule
the writes themselves, not how often we force the writes to be
committed by the kernel.