[syslog-ng] strange thing

[Balazs Scheidler]

Hi,

> yesterday I experienced a very strange problem with syslog-ng which I'd like
> to report. Maybe someone here has a clue what might have caused this. I am
> not able to reproduce that problem (on a different machine and I don't want
> to try it on the machnie I had experienced it since this fast the main
> server of my company.
> 
> Well, what happened? I was playing around with syslog-ng. I had added a
> program(); destination but for some reason it seemed not to work as i wanted
> (due to a name resolution problem, there always was the IP instead of the
> hostname in the logfiles so my script did not parse the input correctly, but
> that not relevant). In order to see what was going on I added a "destination
> home { udp(1.2.3.4); };" and a "log (source(net); destination(home); };" At
> home, from where i was working I had a simple VB program running on my win
> workstation which was listening on port 514 udp and putting everything it
> recieved in a log-window. However, no messages appaered. So I decided to
> write the messages to a log file wich should be more reliable than sending
> them about a dialup connection, so I modified the log statement to "log
> (source(net); destination(home); destination(all); };" where all was
> "destination all { file("/var/log/allmessages"); };". From that moment on
> (i.e. after the HUP) the whole system went to sleep. Every process trying to
> use syslog blocked. Within a few seconds I had some hundred pop3d and
> sendmail tasks running, my own ssh was blocked since I tried to issued a
> logger command. I was not able to telnet or ssh to this host since both
> daemon tried to log when I connected. Luckily enough someone else at my
> company still had an open telnet. I called him and advised him to remove the
> malicious lines from the config and send syslog-ng a SIGHUP. No Effect. Only
> a SIGKILL was able to help us out of this strange situation. Within seconds
> all the daemons went back to work again.
> 
> I really have no idea what might have caused this and I am not able to give
> you more details. As I said, I was not able to reproduce this situation on a
> test server. I am an developer myself and I perfectly know that this
> description is nearly useless because it is lacking facts, but I am not able
> to deliever some and I thought that it might be better to state that their
> might be a problem so that if someone else will report something similar it
> might help to make it easier to puzzle the whole thing together.
> 
> Please note that the system is very old (Kernel 2.0.35, still libc) and has
> an uptime of more than 300 days now without having ECC ram. This system
> became "to important to be upgraded". It will be replaced in the near future
> with a new machine so that we do not have to take it down but could switch
> softly. Maybe I can reproduce that situation then and send you some straces
> etc.

An strace dump or something could really help here. As it seems syslog-ng
blocked on something (a DNS request maybe?), thus couldn't accept
connections on /dev/log.

Newer libc's allow using unix-dgram /dev/log, try using that, client
programs will never block then.

-- 
Bazsi
PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1
     url: http://www.balabit.hu/pgpkey.txt