Critical hang while opening tty
Hi there, I'm using syslog-ng 2.0.9-4.1 on Ubuntu 9.04. In the configuration file, emergency messages are configured to be logged to all ttys. Sometimes syslog-ng blocks the whole system while opening /dev/ttyS0: # pgrep -f syslog-ng # strace -f -p 3172 Process 3172 attached - interrupt to quit open("/dev/ttyS0", O_WRONLY|O_NOCTTY|O_APPEND I'm surprised because I saw the same bug report, and this was supposed to be fixed in Ubuntu's 2.0.9-1 version as stated in the changelog at See http://changelogs.ubuntu.com/changelogs/pool/universe/s/syslog-ng/syslog-ng_... The changelog says: syslog-ng (2.0.9-1) unstable; urgency=low * New upstream version. (Closes: #397650, #447105) This corresponds to this bug: syslog-ng hangs writing to /dev/ttyN http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=397650 I notice the bug is fixed in syslog-ng: http://git.balabit.hu/?p=bazsi/syslog-ng-2.0.git;a=commit;h=c5cc92d54d2e33b9... Is there maybe another call to tty open() that was not covered by alarm()? The issue happens on several machines, but I don't know what triggers this. Reloading syslog-ng fixes the problem. Thanks in advance, -- Jean-Baptiste Quenot http://jbq.caraldi.com/
On Fri, 2009-06-05 at 13:22 +0200, Jean-Baptiste Quenot wrote:
Hi there,
I'm using syslog-ng 2.0.9-4.1 on Ubuntu 9.04. In the configuration file, emergency messages are configured to be logged to all ttys. Sometimes syslog-ng blocks the whole system while opening /dev/ttyS0:
# pgrep -f syslog-ng # strace -f -p 3172 Process 3172 attached - interrupt to quit open("/dev/ttyS0", O_WRONLY|O_NOCTTY|O_APPEND
I'm surprised because I saw the same bug report, and this was supposed to be fixed in Ubuntu's 2.0.9-1 version as stated in the changelog at See http://changelogs.ubuntu.com/changelogs/pool/universe/s/syslog-ng/syslog-ng_...
The changelog says:
syslog-ng (2.0.9-1) unstable; urgency=low
* New upstream version. (Closes: #397650, #447105)
This corresponds to this bug:
syslog-ng hangs writing to /dev/ttyN http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=397650
I notice the bug is fixed in syslog-ng: http://git.balabit.hu/?p=bazsi/syslog-ng-2.0.git;a=commit;h=c5cc92d54d2e33b9...
Is there maybe another call to tty open() that was not covered by alarm()?
The issue happens on several machines, but I don't know what triggers this. Reloading syslog-ng fixes the problem.
That fix was applied to the usertty() driver, are you using that? But yes, as you guessed right, the open() call was not expected to block, thus it is not protected by the alarm() call. I didn't know that even open might block. The patch below makes the open() call nonblocking, which means that if the tty is blocked, the message will be dropped. But this is probably the only sane thing to do. diff --git a/src/afuser.c b/src/afuser.c index e3aa50a..245cdcf 100644 --- a/src/afuser.c +++ b/src/afuser.c @@ -95,7 +95,7 @@ afuser_dd_queue(LogPipe *s, LogMessage *msg, const LogPathOptions *path_options) else line[0] = 0; strncat(line, ut->ut_line, sizeof(line)); - fd = open(line, O_NOCTTY | O_APPEND | O_WRONLY); + fd = open(line, O_NOCTTY | O_APPEND | O_WRONLY | O_NONBLOCK); if (fd != -1) { alarm_set(10); I've also committed this to syslog-ng 3.0 tree as this patch: commit 75c5000a4d6ea6ab04e4f2158e6008dec0a1cdcd Author: Balazs Scheidler <bazsi@balabit.hu> Date: Sun Jun 7 11:47:51 2009 +0200 [afuser] open the tty in nonblocking mode to avoid blocking on serial lines Reported-By: Jean-Baptiste Quenot -- Bazsi
2009/6/7 Balazs Scheidler <bazsi@balabit.hu>:
That fix was applied to the usertty() driver, are you using that?
Yes I use the default config from Ubuntu: destination du_all { usertty("*"); };
But yes, as you guessed right, the open() call was not expected to block, thus it is not protected by the alarm() call. I didn't know that even open might block.
The patch below makes the open() call nonblocking, which means that if the tty is blocked, the message will be dropped. But this is probably the only sane thing to do.
diff --git a/src/afuser.c b/src/afuser.c index e3aa50a..245cdcf 100644 --- a/src/afuser.c +++ b/src/afuser.c @@ -95,7 +95,7 @@ afuser_dd_queue(LogPipe *s, LogMessage *msg, const LogPathOptions *path_options) else line[0] = 0; strncat(line, ut->ut_line, sizeof(line)); - fd = open(line, O_NOCTTY | O_APPEND | O_WRONLY); + fd = open(line, O_NOCTTY | O_APPEND | O_WRONLY | O_NONBLOCK); if (fd != -1) { alarm_set(10);
I've also committed this to syslog-ng 3.0 tree as this patch:
commit 75c5000a4d6ea6ab04e4f2158e6008dec0a1cdcd Author: Balazs Scheidler <bazsi@balabit.hu> Date: Sun Jun 7 11:47:51 2009 +0200
[afuser] open the tty in nonblocking mode to avoid blocking on serial lines
Reported-By: Jean-Baptiste Quenot
Great! I will try the patch. As I don't control the hardware of my dedicated servers it's hard to know why the serial line is blocked, and I don't even have access to those ttys, I'm only using the pseudo-devices pts/* allocated when connecting through SSH. Thanks for you great work! -- Jean-Baptiste Quenot http://jbq.caraldi.com/
participants (2)
-
Balazs Scheidler
-
Jean-Baptiste Quenot