[syslog-ng]Over zealous syslog-ng problem

[Andreas Schulze]

Hi,

> My perl script does the same thing.  It basically sits asleep and checks 
> every few seconds to see if something has been written to the pipe.  If 
> so, it reads 1 line at a time until nothing else is there and then goes 
> back to sleep.  The pipe entries are also preformatted sql statements.  
> What is strange to me is that the sending machine seems to have no 
> problems writting 16 million entries to disk and the receiving machine 
> has the same syslog-ng binary and, for the most part, the same 
> syslog-ng.conf file.  So either the messages are getting lost in 
> transport, or the perl sql inserts are not blocking and they happen too 
> fast for mysql to deal with.  To me, the latter is more troubling.  
> Either way, my setup needs to be refined.
> 
>>> My Questions:
>>> Is there any way to throttle syslog-ng, or make syslog-ng not accept 
>>> all  log messages when an app goes crazy?  I want to log some of 
>>> these messages, so I know when to restart the service, but I don't 
>>> want all 16 million.  Also, it seems that around 15 million log 
>>> messages didn't make it to my central server.  Where were they lost?  
>>> Is this a problem with the UDP transport?

If you are in a LAN environment UDP is in all most cases _not_
the problem.

You're losing messages in pipe() destinations? Right?

Your problem is unix pipe related.
A pipe has a limited 'buffer' size (4k on Linux, 5k on Solaris).
It seems, that your syslogd tries to write more messages to the
pipe than your script can read.
If your script polls e.g. all 1 sec., its absolutly possible that
syslogd tries to write more than PIPE_BUF bytes to the pipe during
this time slot of 1 sec.

The problem is, that the process (your script), that read(2)'s
from the pipe, MUST do this at least as fast as the process (syslogd)
that write(2)'s to the pipe.
Otherwise the pipes 'buffer' overruns. And the writing process
can't write more than PIPE_BUF bytes to the pipe and drops the msgs.
If this happens, there are two solutions:
Make the reading process faster or implement output queueing
for the writing process.

Balazs announced a patch, that fix output queueing for pipes()
in libol.

But this patch doesn't fix the mangled messages problem.
I'll explain this in the related list thread.
-- 
Best regards --Andreas Schulze
                [phone: +49.5246.80.1275, fax: +49.5246.80.2275]

| I believe, it was Dennis Ritchie who said something like:
|   "C is rarely the best language for a given task,
|    but it's often the second-best".
| The implication being that: "[...]"
|
| sh# cat>$$.c<<EOT
| main(l,a,n,d)char**a;{for(d=atoi(a[1])/10*80-atoi(a[2])/5-596;n="@NK\
| ACLCCGZAAQBEAADAFaISADJABBA^SNLGAQABDAXIMBAACTBATAHDBANZcEMMCCCCAAhE\
| IJFAEAAABAfHJETBdFLDAANEfDNBPHdBcBBBEA_AL H E L L O,    W O R L D! "
| [l++-3];)for(;n-->64;)putchar(!d+++33^l&1);}
| EOT
| gcc -o$$ $$.c;clear;./$$ 52 8;rm -f $$*