syslog-ng stopped working yesterday
Hi, I have a wierd problem, yesterday around 14:00 CET syslog-ng just stops working. The daemon runs, but allmost nothing is writtes to the logs. I have used syslog-ng for some years now, and it never let me down. When running syslog-ng -dv the last things it tell me is: Objects alive: 203, garbage collected: 1 AF_INET client connected from 80.63.205.142, port 3630 io.c: Preparing fd 14 for reading And then it stops logging, but seeing with tcpdump that i receive tons of syslog, and even local logging with "logger" doesn't work. Installing a new syslog-ng on another machine does the same, which confuses me. Any good ideas ? -- Med venlig hilsen / Best regards Gorm J. Siiger - mail: gjs@sonnit.dk ------------------------------------ SonnIT, Sonnerupvej 83B, 4682 Tureby Tel. 70274777 - http://www.sonnit.dk
On Tue, 2004-12-28 at 10:30 +0100, Gorm J. Siiger wrote:
Hi,
I have a wierd problem, yesterday around 14:00 CET syslog-ng just stops working. The daemon runs, but allmost nothing is writtes to the logs.
I have used syslog-ng for some years now, and it never let me down.
When running syslog-ng -dv the last things it tell me is:
Objects alive: 203, garbage collected: 1 AF_INET client connected from 80.63.205.142, port 3630 io.c: Preparing fd 14 for reading
And then it stops logging, but seeing with tcpdump that i receive tons of syslog, and even local logging with "logger" doesn't work.
Installing a new syslog-ng on another machine does the same, which confuses me.
syslog-ng probably blocks on something, maybe on a DNS query. can you strace syslog-ng while it is blocked like this? -- Bazsi
Also, can you include the config portions that handle your logging to disk? - billn On Tue, 28 Dec 2004, Balazs Scheidler wrote:
On Tue, 2004-12-28 at 10:30 +0100, Gorm J. Siiger wrote:
Hi,
I have a wierd problem, yesterday around 14:00 CET syslog-ng just stops working. The daemon runs, but allmost nothing is writtes to the logs.
I have used syslog-ng for some years now, and it never let me down.
When running syslog-ng -dv the last things it tell me is:
Objects alive: 203, garbage collected: 1 AF_INET client connected from 80.63.205.142, port 3630 io.c: Preparing fd 14 for reading
And then it stops logging, but seeing with tcpdump that i receive tons of syslog, and even local logging with "logger" doesn't work.
Installing a new syslog-ng on another machine does the same, which confuses me.
syslog-ng probably blocks on something, maybe on a DNS query. can you strace syslog-ng while it is blocked like this?
syslog-ng probably blocks on something, maybe on a DNS query. can you strace syslog-ng while it is blocked like this?
I found the problem, the reverse dns zone for one particular host was classless delegated, and that name-server was configured wrong. Pretty interesting that syslog-ng fails that bad when dns doesn't work. Once the problem was fixed everything went back to normal. -- Med venlig hilsen / Best regards Gorm J. Siiger - mail: gjs@sonnit.dk ------------------------------------ SonnIT, Sonnerupvej 83B, 4682 Tureby Tel. 70274777 - http://www.sonnit.dk
On Wed, 2004-12-29 at 09:30 +0100, Gorm J. Siiger wrote:
syslog-ng probably blocks on something, maybe on a DNS query. can you strace syslog-ng while it is blocked like this?
I found the problem, the reverse dns zone for one particular host was classless delegated, and that name-server was configured wrong.
Pretty interesting that syslog-ng fails that bad when dns doesn't work.
Once the problem was fixed everything went back to normal.
you can always disable resolving names from DNS, but what else could you do? if I'd do non-blocking DNS queries and would continue processing on other messages while waiting for a given message to resolve, then message ordering will be bad. An other solution might be to add non-DNS name lookups, e.g. resolve names from a private mechanism to make it possible to use names in messages and use IP addresses if it fails, but never block. What do you think about this solution? -- Bazsi
--On Thursday, December 30, 2004 6:36 PM +0100 Balazs Scheidler <bazsi@balabit.hu> wrote:
you can always disable resolving names from DNS, but what else could you do? if I'd do non-blocking DNS queries and would continue processing on other messages while waiting for a given message to resolve, then message ordering will be bad.
An other solution might be to add non-DNS name lookups, e.g. resolve names from a private mechanism to make it possible to use names in messages and use IP addresses if it fails, but never block.
What do you think about this solution?
I'd suggest using an async DNS lookup with configurable timeout. This allows you to have multiple DNS queries in-flight at once (assuming you're willing to buffer, but syslog-ng does that already, at least on output). e.g.: recv line 1 start line 1 lookup recv line 2 start line 2 lookup get line 2 answer ... line 1 lookup times out write line 1 (with IP address) write line 2 (with FQDN) This allows me to have names where possible, but tweak the timeout for my DNS setup (I may decide that if it doesn't resolve in 1 second, it isn't going to) -- Carson
participants (4)
-
Balazs Scheidler
-
Bill Nash
-
Carson Gaspar
-
Gorm J. Siiger