On Thu, Jan 02, 2003 at 06:36:32PM +0100, Andreas Schulze wrote:
Hi,
My perl script does the same thing. It basically sits asleep and checks every few seconds to see if something has been written to the pipe. If so, it reads 1 line at a time until nothing else is there and then goes back to sleep. The pipe entries are also preformatted sql statements. What is strange to me is that the sending machine seems to have no problems writting 16 million entries to disk and the receiving machine has the same syslog-ng binary and, for the most part, the same syslog-ng.conf file. So either the messages are getting lost in transport, or the perl sql inserts are not blocking and they happen too fast for mysql to deal with. To me, the latter is more troubling. Either way, my setup needs to be refined.
My Questions: Is there any way to throttle syslog-ng, or make syslog-ng not accept all log messages when an app goes crazy? I want to log some of these messages, so I know when to restart the service, but I don't want all 16 million. Also, it seems that around 15 million log messages didn't make it to my central server. Where were they lost? Is this a problem with the UDP transport?
If you are in a LAN environment UDP is in all most cases _not_ the problem.
You're losing messages in pipe() destinations? Right?
Your problem is unix pipe related. A pipe has a limited 'buffer' size (4k on Linux, 5k on Solaris). It seems, that your syslogd tries to write more messages to the pipe than your script can read. If your script polls e.g. all 1 sec., its absolutly possible that syslogd tries to write more than PIPE_BUF bytes to the pipe during this time slot of 1 sec.
syslog-ng polls, which means that it waits until the pipe is writable, which is not when the buffer is full. and I don't think pipes would be lossy. what about: gunzip -dc somethingbig.tar.gz | tar xvf - (ok, I know this is not a named pipe, but they are internally the same)
The problem is, that the process (your script), that read(2)'s from the pipe, MUST do this at least as fast as the process (syslogd) that write(2)'s to the pipe. Otherwise the pipes 'buffer' overruns. And the writing process can't write more than PIPE_BUF bytes to the pipe and drops the msgs. If this happens, there are two solutions: Make the reading process faster or implement output queueing for the writing process.
Hmm... could you send me a reference where you read this? I doubt it is true.
Balazs announced a patch, that fix output queueing for pipes() in libol.
But this patch doesn't fix the mangled messages problem. I'll explain this in the related list thread.
I would be more than interested. -- Bazsi PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1