[syslog-ng]Over zealous syslog-ng problem

[Balazs Scheidler]

On Thu, Jan 02, 2003 at 06:36:32PM +0100, Andreas Schulze wrote:
> Hi,
> 
> >My perl script does the same thing.  It basically sits asleep and checks 
> >every few seconds to see if something has been written to the pipe.  If 
> >so, it reads 1 line at a time until nothing else is there and then goes 
> >back to sleep.  The pipe entries are also preformatted sql statements.  
> >What is strange to me is that the sending machine seems to have no 
> >problems writting 16 million entries to disk and the receiving machine 
> >has the same syslog-ng binary and, for the most part, the same 
> >syslog-ng.conf file.  So either the messages are getting lost in 
> >transport, or the perl sql inserts are not blocking and they happen too 
> >fast for mysql to deal with.  To me, the latter is more troubling.  
> >Either way, my setup needs to be refined.
> >
> >>>My Questions:
> >>>Is there any way to throttle syslog-ng, or make syslog-ng not accept 
> >>>all  log messages when an app goes crazy?  I want to log some of 
> >>>these messages, so I know when to restart the service, but I don't 
> >>>want all 16 million.  Also, it seems that around 15 million log 
> >>>messages didn't make it to my central server.  Where were they lost?  
> >>>Is this a problem with the UDP transport?
> 
> If you are in a LAN environment UDP is in all most cases _not_
> the problem.
> 
> You're losing messages in pipe() destinations? Right?
> 
> Your problem is unix pipe related.
> A pipe has a limited 'buffer' size (4k on Linux, 5k on Solaris).
> It seems, that your syslogd tries to write more messages to the
> pipe than your script can read.
> If your script polls e.g. all 1 sec., its absolutly possible that
> syslogd tries to write more than PIPE_BUF bytes to the pipe during
> this time slot of 1 sec.

syslog-ng polls, which means that it waits until the pipe is writable, which
is not when the buffer is full.

and I don't think pipes would be lossy. what about:

gunzip -dc somethingbig.tar.gz | tar xvf -

(ok, I know this is not a named pipe, but they are internally the same)

> 
> The problem is, that the process (your script), that read(2)'s
> from the pipe, MUST do this at least as fast as the process (syslogd)
> that write(2)'s to the pipe.
> Otherwise the pipes 'buffer' overruns. And the writing process
> can't write more than PIPE_BUF bytes to the pipe and drops the msgs.
> If this happens, there are two solutions:
> Make the reading process faster or implement output queueing
> for the writing process.

Hmm... could you send me a reference where you read this? I doubt it is
true.

> 
> Balazs announced a patch, that fix output queueing for pipes()
> in libol.
> 
> But this patch doesn't fix the mangled messages problem.
> I'll explain this in the related list thread.

I would be more than interested.

-- 
Bazsi
PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1