[syslog-ng]Over zealous syslog-ng problem

[Andreas Schulze]

Balazs Scheidler wrote:
> On Thu, Jan 02, 2003 at 06:36:32PM +0100, Andreas Schulze wrote:
> 
> syslog-ng polls, which means that it waits until the pipe is writable, which
> is not when the buffer is full.

Yepp. Imagine the following worst case scenario:
There isn't any process that reads from the pipe.
In this case you need in theory a unlimited queue size
to buffer all the informations, that would never read...

> and I don't think pipes would be lossy. what about:
> gunzip -dc somethingbig.tar.gz | tar xvf -
> (ok, I know this is not a named pipe, but they are internally the same)

IMHO shell pipes working BLOCKING.
In this case, you are right, of course.

>>The problem is, that the process (your script), that read(2)'s
>>from the pipe, MUST do this at least as fast as the process (syslogd)
>>that write(2)'s to the pipe.
>>Otherwise the pipes 'buffer' overruns. And the writing process
>>can't write more than PIPE_BUF bytes to the pipe and drops the msgs.
>>If this happens, there are two solutions:
>>Make the reading process faster or implement output queueing
>>for the writing process.

Sorry. I'm afraid, that I mispronounce some of the above stuff.
My english isn't the best, I know. :(

We have to worry about information loss in pipes, only if the writing
process write()'s NONBLOCKING to this pipe.
In this case the reader must at least as fast as the writer.
You implement queueing, of course. But queueing means only a strategy
to catch short periods of bursty message flow in this manner.
If, seen over a longer period, your reader is to slow, your writer
must drop messages or your writers queue size blows up.

On the other hand, in theory, more than one process can write
nonblocking to the same pipe. So its better to use atomic writes with
buffers <= PIPE_BUF, to avoid, that the processes interleaves.

> Hmm... could you send me a reference where you read this? I doubt it is
> true.

My favorite references are:
APUE and UNP books by R.W.Stevens (the godfather himself :)
and many of the linux/solaris man(2,3*,4,5,7) manpages.

-- 
Best regards --Andreas Schulze
                [phone: +49.5246.80.1275, fax: +49.5246.80.2275]

| I believe, it was Dennis Ritchie who said something like:
|   "C is rarely the best language for a given task,
|    but it's often the second-best".
| The implication being that: "[...]"
|
| sh# cat>$$.c<<EOT
| main(l,a,n,d)char**a;{for(d=atoi(a[1])/10*80-atoi(a[2])/5-596;n="@NK\
| ACLCCGZAAQBEAADAFaISADJABBA^SNLGAQABDAXIMBAACTBATAHDBANZcEMMCCCCAAhE\
| IJFAEAAABAfHJETBdFLDAANEfDNBPHdBcBBBEA_AL H E L L O,    W O R L D! "
| [l++-3];)for(;n-->64;)putchar(!d+++33^l&1);}
| EOT
| gcc -o$$ $$.c;clear;./$$ 52 8;rm -f $$*