[syslog-ng] Odd behavior with very small log_fifo_size()

[Balazs Scheidler]

On Wed, 2008-07-02 at 13:21 -0400, Joe Shaw wrote:
> Hi,
> 
> On Wed, Jul 2, 2008 at 5:13 AM, Balazs Scheidler <bazsi at balabit.hu> wrote:
> > syslog-ng does not log message loss on file destinations, because that'd
> > clutter the log statistics line, on the asssumption that no messages are
> > can be lost there.
> >
> > However this is not always the case, if you log_fifo_size() is smaller
> > than the window size of incoming sources, messages can be lost.
> 
> Could something be added if -v were passed to syslog-ng then?  It's
> (very) useful diagnostic information when you're tuning an
> installation and I probably wouldn't have noticed if not for the data
> that loggen outputs and the fact that the log file seemed small.
> 
> > Here is a description of how flow-control and window sizes work in
> > syslog-ng:
> >
> > http://www.balabit.hu/dl/html/syslog-ng-admin-guide_en.html/ch08s03.html
> >
> > Please let me (and the doc team) know if the description could be made
> > clearer somehow.
> 
> I hadn't seen this before -- I was only using the docs shipped with
> the tarball -- and it's good information

I know I should've removed that old documentation from the source
tarball. It is unmaintained there, and false information is worse than
no information at all. So that's what I'm going to do, remove the doc
and add a pointer to the website. (or maybe add a txt version if I can
figure out how to do that).

> .
> 
> My setting of the log_fifo_size() in testing was for two reasons: (1)
> to see how setting it affected memory usage when being hit with many
> large messages and (2) to demonstrably see what its behavior was if it
> was overloaded and had to drop messages.  I would have liked a little
> more info in the latter.
> 
> As for the docs, it does mention that without the flow control
> messages may be lost, but doesn't indicate how this might be detected
> or what happens if they are.

Without flow control messages are lost when the output FIFO size is
full. That's basically the only possibility. It even displays a debug
message in this case:

  if (!log_queue_push_tail(self->queue, lm, path_flags))
    {
      /* drop incoming message, we must ack here, otherwise the sender might
       * block forever, however this should not happen unless the sum of
       * window_sizes of sources feeding this writer exceeds log_fifo_size
       * or if flow control is not turned on.
       */
      
      /* we don't send a message here since the system is draining anyway */
      
      if (self->dropped_messages)
        (*self->dropped_messages)++;
      msg_debug("Destination queue full, dropping message",
                evt_tag_int("queue_len", log_queue_get_length(self->queue)),
                evt_tag_int("mem_fifo_size", self->options->mem_fifo_size),
                NULL);
      log_msg_drop(lm, path_flags);
      return;
    }



-- 
Bazsi