Hi,
My perl script does the same thing. It basically sits asleep and checks every few seconds to see if something has been written to the pipe. If so, it reads 1 line at a time until nothing else is there and then goes back to sleep. The pipe entries are also preformatted sql statements. What is strange to me is that the sending machine seems to have no problems writting 16 million entries to disk and the receiving machine has the same syslog-ng binary and, for the most part, the same syslog-ng.conf file. So either the messages are getting lost in transport, or the perl sql inserts are not blocking and they happen too fast for mysql to deal with. To me, the latter is more troubling. Either way, my setup needs to be refined.
My Questions: Is there any way to throttle syslog-ng, or make syslog-ng not accept all log messages when an app goes crazy? I want to log some of these messages, so I know when to restart the service, but I don't want all 16 million. Also, it seems that around 15 million log messages didn't make it to my central server. Where were they lost? Is this a problem with the UDP transport?
If you are in a LAN environment UDP is in all most cases _not_ the problem. You're losing messages in pipe() destinations? Right? Your problem is unix pipe related. A pipe has a limited 'buffer' size (4k on Linux, 5k on Solaris). It seems, that your syslogd tries to write more messages to the pipe than your script can read. If your script polls e.g. all 1 sec., its absolutly possible that syslogd tries to write more than PIPE_BUF bytes to the pipe during this time slot of 1 sec. The problem is, that the process (your script), that read(2)'s from the pipe, MUST do this at least as fast as the process (syslogd) that write(2)'s to the pipe. Otherwise the pipes 'buffer' overruns. And the writing process can't write more than PIPE_BUF bytes to the pipe and drops the msgs. If this happens, there are two solutions: Make the reading process faster or implement output queueing for the writing process. Balazs announced a patch, that fix output queueing for pipes() in libol. But this patch doesn't fix the mangled messages problem. I'll explain this in the related list thread. -- Best regards --Andreas Schulze [phone: +49.5246.80.1275, fax: +49.5246.80.2275] | I believe, it was Dennis Ritchie who said something like: | "C is rarely the best language for a given task, | but it's often the second-best". | The implication being that: "[...]" | | sh# cat>$$.c<<EOT | main(l,a,n,d)char**a;{for(d=atoi(a[1])/10*80-atoi(a[2])/5-596;n="@NK\ | ACLCCGZAAQBEAADAFaISADJABBA^SNLGAQABDAXIMBAACTBATAHDBANZcEMMCCCCAAhE\ | IJFAEAAABAfHJETBdFLDAANEfDNBPHdBcBBBEA_AL H E L L O, W O R L D! " | [l++-3];)for(;n-->64;)putchar(!d+++33^l&1);} | EOT | gcc -o$$ $$.c;clear;./$$ 52 8;rm -f $$*