My apologies if this has been discussed, as I haven't been able to find anything useful via google or the FM... Syoposis: Syslog-ng drops something on the order of 90% of the logs remotely flung at it. Detail: I'm using syslog-ng 1.5.15 from the Debian stable package archive. I've been tasked with setting up two remote log servers for my employer; both log servers have fairly beefy IDE raids (IOZone gives me an unbuffered write speed of about 40M/sec), and as far as I can tell with vmstat(8) and Our Friend top(1), syslog-ng isn't running into any I/O bottlenecks. The systems have insane CPUs (Athlon XP 2000) and 512M RAM a pop, which considering their intended tasks (syslog and serial console server), should be more than adequate. I've set up several systems to push their logs onto the log server; a Solaris 8 (with native syslog) box, a Debian Linux box (native syslog again), and a Debian Linux box using syslog-ng. Yet syslog-ng seems to dump between 75% and 90% of all the logs handed to it on the floor. Several of the systems we wish to have logging to syslog will throw out about a thousand lines within a second or two, and this is where most of our problems come in. I've tried both TCP and UDP; TCP barely seems to work at all -- I can throw a thousand lines in on one end, see them in the logfile, and see one line of output out on the other side. I've futzed a bit with the FIFO size and the gc_idle/gc_busy numbers, but the latter two are more-or-less undocumented in the manual, so I have no idea what they really do (no units or anything), and the code regarding them is a bit too complex for me to read. I've been googling and chomping upon the syslog-ng documentation for about the past two weeks, and have yet to find a solution. Any suggestions? An FM for me to read? A USENET article I missed? Thanks-in-advance! -- Don Werve <donw@examen.com> (Unix System Administrator) Yorn desh born, der ritt de gitt der gue, Orn desh, dee born desh, de umn bork! bork! bork!
2004-03-18, cs keltezéssel 20:35-kor donw@examen.com ezt írta:
My apologies if this has been discussed, as I haven't been able to find anything useful via google or the FM...
Syoposis:
Syslog-ng drops something on the order of 90% of the logs remotely flung at it.
you should first try to identify where the message is dropped. it is possible that the packet is not received by syslog-ng at all, because the local IP stack drops it (for instance because the local receive buffer is full) (the UDP case) You could check this by comparing a tcpdump and an strace dump of the same time interval and check whether each incoming packet has its associated recvfrom() system call. TCP should not drop messages on the network but messages might still be lost on the sending side, syslog-ng itself is not flow controlled, once the output pipe send buffer is full, it will start dropping messages. You could check the "STATS" message syslog-ng is producing, it reports all messages dropped within syslog-ng. You could also update your syslog-ng, the version you are using is quite outdated, you can find binaries for woody in our ZorpOS repository. (check: http://www.balabit.com/downloads/zorp/zorp-os/pool/s/syslog-ng/)
I've set up several systems to push their logs onto the log server; a Solaris 8 (with native syslog) box, a Debian Linux box (native syslog again), and a Debian Linux box using syslog-ng.
Yet syslog-ng seems to dump between 75% and 90% of all the logs handed to it on the floor. Several of the systems we wish to have logging to syslog will throw out about a thousand lines within a second or two, and this is where most of our problems come in.
I've tried both TCP and UDP; TCP barely seems to work at all -- I can throw a thousand lines in on one end, see them in the logfile, and see one line of output out on the other side. I've futzed a bit with the FIFO size and the gc_idle/gc_busy numbers, but the latter two are more-or-less undocumented in the manual, so I have no idea what they really do (no units or anything), and the code regarding them is a bit too complex for me to read.
I've been googling and chomping upon the syslog-ng documentation for about the past two weeks, and have yet to find a solution.
-- Bazsi PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1
participants (2)
-
Balazs Scheidler
-
donw＠examen.com