Syslog-ng 1.6.9 just stops...
I'm using 1.6.9 (upgraded from 1.6.6 because I was seeing the same problem). I have it running on 8 different servers at different locations, some are SMP, some aren't. On these hosts, we have anywhere from 2 to 14 devices logging to the servers, some via 514/tcp, others via 514/udp. All of the loggers typically sit with a load average < 1 (usually not even registering), and a cpu idle of 99%. 7 of these remote loggers also log to our local machine, but only 5 lines every 2 minutes (for stats). What I've noticed (and I've seen this on all of them at one time or another), is that syslog-ng just stops. ps shows it running, but the log file (/logs/messages) never changes. If I tcpdump on the interface that it's listening on, I see traffic, and it seems that the act of tcpdumping causes the log file to start to grow again, then a little while later, it may stop again. It's sporadic though, on one of my systems, it hasn't done it in over 2 months, on another, it's done it 3 times today. I've pulled out my last hair and still haven't come any closer to a solution. I've recompiled the source, loaded 3 different versions, etc. The only thing common is that all of these systems are running RHEL3. Any help would be greatly appreciated. Thanks, Tony
You haven't got Logrotate running, have you? I found that unless I restarted syslog-ng in the rotate scripts, then nothing further was logged.
You might want to try copytruncate in your logrotate configuration, that will leave the file there for syslog-ng to keep writing to instead of rotating it off. if the files you are rotating are large, then I would suggest to leave things the way they are to save the disk I/O. Mike
--
Best regards,
David mailto:dma@pern.co.uk
On Wed, 2006-03-01 at 10:01 -0500, Andreoli, Tony A. USNUNK NAVAIR B1490, R215 wrote:
I'm using 1.6.9 (upgraded from 1.6.6 because I was seeing the same problem). I have it running on 8 different servers at different locations, some are SMP, some aren't. On these hosts, we have anywhere from 2 to 14 devices logging to the servers, some via 514/tcp, others via 514/udp. All of the loggers typically sit with a load average < 1 (usually not even registering), and a cpu idle of 99%. 7 of these remote loggers also log to our local machine, but only 5 lines every 2 minutes (for stats).
What I've noticed (and I've seen this on all of them at one time or another), is that syslog-ng just stops. ps shows it running, but the log file (/logs/messages) never changes. If I tcpdump on the interface that it's listening on, I see traffic, and it seems that the act of tcpdumping causes the log file to start to grow again, then a little while later, it may stop again. It's sporadic though, on one of my systems, it hasn't done it in over 2 months, on another, it's done it 3 times today.
I've pulled out my last hair and still haven't come any closer to a solution. I've recompiled the source, loaded 3 different versions, etc. The only thing common is that all of these systems are running RHEL3.
Don't you happen to read /proc/kmsg by both syslog-ng and klogd ? That is a known bad situation and the symptoms are exactly what you describe. (poll indicates readability but by the time syslog-ng gets to read the file the data has already been read) This is documented in the FAQ as well. -- Bazsi
participants (4)
-
Andreoli, Tony A. USNUNK NAVAIR B1490, R215
-
Balazs Scheidler
-
David Anderson
-
Mike