Hi there I just reinstalled my Fedora workstation recently and hadn't moved over to syslog-ng like I normally do - so this is a question about syslog really. In the past two days I've come in to find my workstation fairly screwed up by the syslog daemon being hung/broken. It would be running, but not working any more. Anyway, as the socket file /dev/log exists, all apps that write to syslog were also hung! So I had my mail server screwed, cronjobs unable to finish, etc, etc. Simply stopping the syslog daemon caused my load average to jump to 40+ whilst all the pent-up processes started working again :-) I guess once the socket file disappeared under them, they just carried on nicely... So - am I right in saying /dev/log blocks, and is there some way at an application layer to limit how long a process attempts to write to syslog before giving up? I know an alarm around the syslog call should work - but is that the only way? Thanks! -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +64 3 9635 377 Fax: +64 3 9635 417 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
On Wed, 2005-06-15 at 11:35 +1200, Jason Haar wrote:
Hi there
I just reinstalled my Fedora workstation recently and hadn't moved over to syslog-ng like I normally do - so this is a question about syslog really.
In the past two days I've come in to find my workstation fairly screwed up by the syslog daemon being hung/broken. It would be running, but not working any more.
Anyway, as the socket file /dev/log exists, all apps that write to syslog were also hung! So I had my mail server screwed, cronjobs unable to finish, etc, etc. Simply stopping the syslog daemon caused my load average to jump to 40+ whilst all the pent-up processes started working again :-) I guess once the socket file disappeared under them, they just carried on nicely...
So - am I right in saying /dev/log blocks, and is there some way at an application layer to limit how long a process attempts to write to syslog before giving up? I know an alarm around the syslog call should work - but is that the only way?
Basically, this is how it was designed to work. The syslog() call in libc uses a blocking call to send messages to syslogd, if the socket is opened (e.g. syslogd is running) then there's no timeout involved. -- Bazsi
Balazs Scheidler wrote:
Basically, this is how it was designed to work. The syslog() call in libc uses a blocking call to send messages to syslogd, if the socket is opened (e.g. syslogd is running) then there's no timeout involved.
So am alarm around the call should stop an app from hanging forever (i.e. set an alarm of 10 sec, call syslog and if it hasn't returned after 10 secs, the alarm will kill that exec)? Is there any way I can emulate this condition? I tried sending a STOP signal to syslog-ng - it stopped outputting to the syslog file all right - but didn't cause any apps to hang - so it's not quite what I'm after... [I have an app I wrote that writes to syslog that I'm now concerned for it's reliability now that I've experienced this] Thanks! Now that I've installed syslog-ng - I'm *not* expecting to come into work to see the same condition tomorrow ;-) -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +64 3 9635 377 Fax: +64 3 9635 417 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
On Wed, 2005-06-15 at 20:08 +1200, Jason Haar wrote:
Balazs Scheidler wrote:
Basically, this is how it was designed to work. The syslog() call in libc uses a blocking call to send messages to syslogd, if the socket is opened (e.g. syslogd is running) then there's no timeout involved.
So am alarm around the call should stop an app from hanging forever (i.e. set an alarm of 10 sec, call syslog and if it hasn't returned after 10 secs, the alarm will kill that exec)? Is there any way I can emulate this condition? I tried sending a STOP signal to syslog-ng - it stopped outputting to the syslog file all right - but didn't cause any apps to hang - so it's not quite what I'm after...
[I have an app I wrote that writes to syslog that I'm now concerned for it's reliability now that I've experienced this]
Thanks! Now that I've installed syslog-ng - I'm *not* expecting to come into work to see the same condition tomorrow ;-)
syslog-ng is not different in that regard from stock syslogd, if syslog-ng stops, and /dev/log is unix-stream then the application is going to block. if /dev/log is unix-dgram then it will probably not (I have not tested this myself), though in this case the log message will be dropped. I don't know whether alarm() is a good choice, I prefer not to rely on it too much as it might interfere with low-level libraries (libc might retry sending the message if it receives an EINTR from send() that alarm would cause) -- Bazsi
participants (2)
-
Balazs Scheidler
-
Jason Haar