[syslog-ng]syslog-ng hanging bringing machine in trouble

[Roberto Nibali]

Hello Peter,

> thanks for fast answer and some
> information-retrieving-hints-on-next-hang.

You're welcome, let's hope we get something useful out of it.

>>Is it reproducable without OWL? Only test it if you can easily do
>>it, if it's a productive machine, I suspect the downtime is too big
>>to do heuristic tests.
>  
> Sorry, can't do that.

Thought so, well, proc-fs output, netstat, lsof and strace will reveil 
the problem if it is connected with OWL. BTW could you list (if it's not 
too big) all the OWL features you've enabled in your running kernel? Not 
that I suspect it to really have an influence on syslog-ng but safe is 
safe ;).

>>You do not need to run klogd if you've configured syslog-ng
>>accordingly unless you need address decoding.
>  
> Hmm, I'm using a given initscript to start syslog-ng:
> 
> start() {
>         echo -n $"Starting system logger: "
>         daemon syslog-ng $SYSLOGD_OPTIONS -f /etc/syslog-ng.conf
>         RETVAL=$?
>         echo
>         echo -n $"Starting kernel logger: "
>         daemon klogd $KLOGD_OPTIONS
>         echo
>         [ $RETVAL -eq 0 ] && touch /var/lock/subsys/syslog-ng
>         return $RETVAL
> }

Fine, just uncomment the three lines concerning klogd, you should still 
get kernel messages.

<OT>
Another thing: Whoever wrote that script part for start() should 
seriously reconsider reading a good shell book or the advance bash 
programming guide.
</OT>

> Does this mean that starting klogd isn't required?

Not really. In the config snipped you posted before you had a 
file("/proc/kmsg") defined as a source in s_local. I just hope you've 
got a d_local where you write those messages into.

>>>Last times I saw also some CROND entries by "ps -ax", one with stat
>>>"D".
>>
>>crond can be in D state sometimes, nothing to worry about ;).
>  
> But they gone after stopping ldap. And everytime seeing in hanging
> case. Also the number of such cron processes increases, looks like
> they cannot do something in case of hanging.

Ok, then I wait with further assumptions until I see some debug output.

>>I would say no but I'm not sure here, I would also suspect it
>>depends on the version of cron deployed on your machine.
>  
> vixie-cron-3.0.1-64

The I suppose it should stop logging. How about if you send a SIGHUP to 
the cron? pkill -HUP cron.

I would also be interested in the process status (signals, state, ...) 
of syslog-ng, cron and whatever seems to be hanging:

cat /proc/${PID_OF_PROCESS}/status

> Does a lsof | grep crond help? I see only some libs, pipes and
> sockets.

Yes, maybe you should also send along the output of:

lsof -c cron -c syslog-ng

> BTW: after the first hangs, I also enable UDP logging to a remote
> host, but logging will stop, here, too.

lsof -n -i udp:${PORT_WHERE_YOU_SEND_THE_LOGS_TO}

Best regards,
Roberto Nibali, ratz
-- 
echo '[q]sa[ln0=aln256%Pln256/snlbx]sb3135071790101768542287578439snlbxq'|dc