Syslog-ng appears to hang for local requests to log
I just joined the list after seeing in the archive the syslog-ng causing other proccesses to freeze I've seen the same issue in syslog-ng-2.0.2. I also usualy see some error about exciding the maximum number of open files for root. if i happen to be loged in to the box as root I've been able to fix it by restarting syslog-ng. if ive been loged in as my normal useri can usualy run a sudo lsof but when i count the number of open files its onwhere near the number set in /etc/secutiy/limits.conf. i also ocasionaly see this in the logs right before it happens kernel: warning: many lost ticks. kernel: Your time source seems to be instable or some driver is hogging interupts kernel: rip default_idle+0x20/0x23
Paul Robert Marino wrote:
I just joined the list after seeing in the archive the syslog-ng causing other proccesses to freeze I've seen the same issue in syslog-ng-2.0.2. I also usualy see some error about exciding the maximum number of open files for root. if i happen to be loged in to the box as root I've been able to fix it by restarting syslog-ng. if ive been loged in as my normal useri can usualy run a sudo lsof but when i count the number of open files its onwhere near the number set in /etc/secutiy/limits.conf. i also ocasionaly see this in the logs right before it happens I'm now pretty sure that my problems are not related to file handles -- syslog-ng seems to have a builtin limit of 1024 open file handles and nev er goes over according to lsof.
I have just upgraded to 2.0.5 but another thing has just occurred to me. I have a destination fifo which is used intermittently. Fifo has an 8KB buffer and I just wondered what happens when this fills. Do writes from syslog-ng fail? If so does syslog-ng then try and buffer output itself? If so that would explain what we are seeing. If the new version does not fix the issue I will try disabling the fifo destination. Russell.
I have a fifo in my config as well and its not always there when syslog-ng tries to write to it. I think you may be onto atleast part of the problem but I don't think its the whole story
On Mon, 2007-08-06 at 11:56 +1200, Russell Fulton wrote:
Paul Robert Marino wrote:
I just joined the list after seeing in the archive the syslog-ng causing other proccesses to freeze I've seen the same issue in syslog-ng-2.0.2. I also usualy see some error about exciding the maximum number of open files for root. if i happen to be loged in to the box as root I've been able to fix it by restarting syslog-ng. if ive been loged in as my normal useri can usualy run a sudo lsof but when i count the number of open files its onwhere near the number set in /etc/secutiy/limits.conf. i also ocasionaly see this in the logs right before it happens I'm now pretty sure that my problems are not related to file handles -- syslog-ng seems to have a builtin limit of 1024 open file handles and nev er goes over according to lsof.
I have just upgraded to 2.0.5 but another thing has just occurred to me. I have a destination fifo which is used intermittently. Fifo has an 8KB buffer and I just wondered what happens when this fills. Do writes from syslog-ng fail? If so does syslog-ng then try and buffer output itself? If so that would explain what we are seeing.
If the new version does not fix the issue I will try disabling the fifo destination.
It depends on the flow-control setting of the log-path with the fifo destination. If flow-control is enabled and the fifo is not read, then the input side of syslog-ng will be suspended, thus the applications trying to send messages will be blocked. That's how flow-control behaves. See the chapter on flow-control in the manual. -- Bazsi
Balazs Scheidler wrote:
On Mon, 2007-08-06 at 11:56 +1200, Russell Fulton wrote:
I have just upgraded to 2.0.5 but another thing has just occurred to me. I have a destination fifo which is used intermittently. Fifo has an 8KB buffer and I just wondered what happens when this fills. Do writes from syslog-ng fail? If so does syslog-ng then try and buffer output itself? If so that would explain what we are seeing.
If the new version does not fix the issue I will try disabling the fifo destination.
It depends on the flow-control setting of the log-path with the fifo destination.
If flow-control is enabled and the fifo is not read, then the input side of syslog-ng will be suspended, thus the applications trying to send messages will be blocked.
That's how flow-control behaves. See the chapter on flow-control in the manual.
Wow! I had not seen the new manual! Great! OK, I've read the section on flow control and now believe that I understand what happens when flow-control is set. I am still a bit confused about what happens by default. In my case I have two destinations -- normal files and a fifo. Everything goes to the file destination and a selection also goes to the fifo which is used for 'real time' analysis (which is currently under development and not running all the time). Clearly my sources are not blocking when the output buffer is full because logs continue to be written to disk. I assume that in the absence of flow-control syslog-ng discards records once the output buffer becomes full. Perhaps this could be stated explicitly for idiots like me. Is this interpretation correct? Russell
On Tue, 2007-08-07 at 06:27 +1200, Russell Fulton wrote:
Balazs Scheidler wrote:
On Mon, 2007-08-06 at 11:56 +1200, Russell Fulton wrote:
I have just upgraded to 2.0.5 but another thing has just occurred to me. I have a destination fifo which is used intermittently. Fifo has an 8KB buffer and I just wondered what happens when this fills. Do writes from syslog-ng fail? If so does syslog-ng then try and buffer output itself? If so that would explain what we are seeing.
If the new version does not fix the issue I will try disabling the fifo destination.
It depends on the flow-control setting of the log-path with the fifo destination.
If flow-control is enabled and the fifo is not read, then the input side of syslog-ng will be suspended, thus the applications trying to send messages will be blocked.
That's how flow-control behaves. See the chapter on flow-control in the manual.
Wow! I had not seen the new manual! Great!
OK, I've read the section on flow control and now believe that I understand what happens when flow-control is set. I am still a bit confused about what happens by default.
In my case I have two destinations -- normal files and a fifo. Everything goes to the file destination and a selection also goes to the fifo which is used for 'real time' analysis (which is currently under development and not running all the time). Clearly my sources are not blocking when the output buffer is full because logs continue to be written to disk. I assume that in the absence of flow-control syslog-ng discards records once the output buffer becomes full. Perhaps this could be stated explicitly for idiots like me.
Is this interpretation correct?
Yes, absolutely. The bit about dropping messages when the buffer is full might be missing from the manual for the very reason it did this all the time. (until flow-control was introduced). I'm Cc-ing our documentation team to consider adding a note about this to that chapter. -- Bazsi
participants (3)
-
Balazs Scheidler
-
Paul Robert Marino
-
Russell Fulton