[syslog-ng] Odd behavior with very small log_fifo_size()

[Balazs Scheidler]

On Sun, 2008-07-20 at 12:59 -0300, Carlos Carvalho wrote:
> Balazs Scheidler (bazsi at balabit.hu) wrote on 2 July 2008 11:13:
>  >syslog-ng does not log message loss on file destinations, because that'd
>  >clutter the log statistics line, on the asssumption that no messages are
>  >can be lost there.
>  >
>  >However this is not always the case, if you log_fifo_size() is smaller
>  >than the window size of incoming sources, messages can be lost.
>  >
>  >Here is a description of how flow-control and window sizes work in
>  >syslog-ng:
>  >
>  >http://www.balabit.hu/dl/html/syslog-ng-admin-guide_en.html/ch08s03.html
>  >
>  >Please let me (and the doc team) know if the description could be made
>  >clearer somehow.
> 
> I found this part confusing:
> 
>    The syslog-ng application monitors (polls) the sources defined in
>    its configuration file, periodically checking each source for
>    messages. When a log message is found in one of the sources,
>    syslog-ng polls every source and reads the available messages.
> 
> This means it breaks the periodicity of the checks, because when it
> finds something in one source it checks the others immediately. Is
> this true? If it is it doesn't look very useful because each source
> should be polled frequently enough independently of the others.

You are right, this is a bit confusing. 

syslog-ng polls its inputs independently from each other, using the
poll() system call. poll() blocks until no input is available, and wakes
up when something becomes available _OR_ a timer elapses.

Normal sources (network connections or pipes) are polled using the first
method, but others like file sources, are polled using timers: syslog-ng
periodically checks whether the files in question have grown in size or
not.

Now, poll() can return that multiple sources have inputs available. All
of these are processed in the same poll loop iteration. File sources on
the other hand are checked whenever poll wakes up: either because the
follow-freq() timer of the file source elapsed or because another source
indicated that data is available. This is because checking a file
whether it has grown is simple and less CPU intensive, than waiting for
another poll loop. Therefore file-size checking is always performed
regardless how much time elapsed.

This way normal sources are treated equally. File sources are checked
every time when poll wakes up, or the follow-freq timer elapses.

-- 
Bazsi