[syslog-ng] One destination stopped logging?

[Hari Sekhon]

Yes I appreciate that you need debug info, but like I said it started 
working again after I restarted syslog-ng so there isn't really any 
opportunity for me strace the actual problem.

I know about getting logs from chroots, but this service is not 
chrooted, it should use the same logging socket as everything else.

Is it possible that one blocked destination could affect another, last I 
heard from you on this was that each destination was a separate line and 
if one blocked it would only block that one destination, ie each 
destination is independent, is that right?

If I can make this error reproducable then I'll try to get you some 
straces etc. and possibly a core dump.

Thanks


-h

Hari Sekhon



Balazs Scheidler wrote:
> On Mon, 2007-07-02 at 11:11 +0100, Hari Sekhon wrote:
>   
>> Hi,
>>
>> I've got syslog-ng on my mail server and today I noticed while trying to 
>> debug something else that there was nothing in the maillog. I thought 
>> this was very odd as there should be lots of stuff, so I checked 
>> syslog-ng was running. It was but I decided to restart it and then the 
>> maillog started to fill up again.
>>
>> Now it's been several days since it's written anything to the mail log, 
>> but it has continued writing to the /var/log/messages destination during 
>> that time.
>>
>> Here is my config:
>>
>> options {
>>     chain_hostnames(off);
>>     sync(0);
>>     stats(43200);
>>     log_fifo_size(30000);
>> };
>> source src { unix-stream("/dev/log" max-connections(1000)); internal(); 
>> pipe("/proc/kmsg"); };
>> destination messages { file("/var/log/messages"); };
>> destination d_net { tcp("ip_to_logserver" port(logserver_port) ); };
>> destination maillog { file("/var/log/maillog"); };
>> destination mailerr { file("/var/log/mail.err"); };
>> filter f_mail { facility(mail); };
>> filter f_mailerr { facility(mail) and level(err); };
>> filter f_notmailjunk { not (program("postfix/*") and not level(err)); };
>> log { source(src); filter(f_mail); destination(maillog); };
>> log { source(src); filter(f_mailerr); destination(mailerr); };
>> log { source(src); filter(f_notmailjunk); destination(messages); 
>> destination(d_net); };
>>
>>
>> Possible idea: My logserver was flooded as I was doing something else on 
>> it and it's likely that logs didn't get through (I know, I know - I 
>> wanted a separate machine for this of course but my boss holds the 
>> money...). Is it possible that the logger filled up locally backlogged 
>> messages for the logserver and this caused logs to be lost for the other 
>> destination maillog as syslog-ng was full or something?
>>
>> Does this sound like it or not at all?
>>
>> I am using syslog-ng 1.6.11 by the way.
>>     
>
> syslog-ng uses a completely nonblocking I/O model and it should process
> messages as long as it is not blocked on something external. If it is
> blocked, then it will do nothing at all.
>
> You stated that it was doing everything, except writing messages to your
> mail log.
>
> An strace dump of the syslog-ng process while it was stalled like this
> could have helped somewhat. Your config says that you have STATS
> enabled, can you check if it indicated message loss?
>
> As I see you are using postfix, postfix is often chroot-ed (Debian
> uses /var/spool/postfix for example), and you are not reading
> "/var/spool/postfix/dev/log". This might be a possible problem.
>
> Personally I've never seen anything like this. (but hey, that's what
> bugreports are for :)
>
> The best practice when something like this happens is to gather all
> information possible, like:
>  * strace syslog-ng (with long strings, e.g. -s 256)
>  * strace the sending process
>  * check the syslog transport medium if applicable (e.g. via tcpdump)
>  * let the syslog-ng process dump core (by enabling core files and
> sending a QUIT signal)
>
> If messages flow into syslog-ng (confirmed using tcpdump or by stracing
> the sender process), and it actually receives the messages (confirmed by
> straceing syslog-ng and checking whether it gets this message via recv),
> and still it does not write it into any of its destinations while it
> should have done so (confirmed by checking the configuration file/log
> files), then it is a bug in syslog-ng.
>
> The problem is that we have not yet confirmed the first couple of steps.
>
>