[syslog-ng] One destination stopped logging?

[Balazs Scheidler]

On Mon, 2007-07-02 at 11:11 +0100, Hari Sekhon wrote:
> Hi,
> 
> I've got syslog-ng on my mail server and today I noticed while trying to 
> debug something else that there was nothing in the maillog. I thought 
> this was very odd as there should be lots of stuff, so I checked 
> syslog-ng was running. It was but I decided to restart it and then the 
> maillog started to fill up again.
> 
> Now it's been several days since it's written anything to the mail log, 
> but it has continued writing to the /var/log/messages destination during 
> that time.
> 
> Here is my config:
> 
> options {
>     chain_hostnames(off);
>     sync(0);
>     stats(43200);
>     log_fifo_size(30000);
> };
> source src { unix-stream("/dev/log" max-connections(1000)); internal(); 
> pipe("/proc/kmsg"); };
> destination messages { file("/var/log/messages"); };
> destination d_net { tcp("ip_to_logserver" port(logserver_port) ); };
> destination maillog { file("/var/log/maillog"); };
> destination mailerr { file("/var/log/mail.err"); };
> filter f_mail { facility(mail); };
> filter f_mailerr { facility(mail) and level(err); };
> filter f_notmailjunk { not (program("postfix/*") and not level(err)); };
> log { source(src); filter(f_mail); destination(maillog); };
> log { source(src); filter(f_mailerr); destination(mailerr); };
> log { source(src); filter(f_notmailjunk); destination(messages); 
> destination(d_net); };
> 
> 
> Possible idea: My logserver was flooded as I was doing something else on 
> it and it's likely that logs didn't get through (I know, I know - I 
> wanted a separate machine for this of course but my boss holds the 
> money...). Is it possible that the logger filled up locally backlogged 
> messages for the logserver and this caused logs to be lost for the other 
> destination maillog as syslog-ng was full or something?
> 
> Does this sound like it or not at all?
> 
> I am using syslog-ng 1.6.11 by the way.

syslog-ng uses a completely nonblocking I/O model and it should process
messages as long as it is not blocked on something external. If it is
blocked, then it will do nothing at all.

You stated that it was doing everything, except writing messages to your
mail log.

An strace dump of the syslog-ng process while it was stalled like this
could have helped somewhat. Your config says that you have STATS
enabled, can you check if it indicated message loss?

As I see you are using postfix, postfix is often chroot-ed (Debian
uses /var/spool/postfix for example), and you are not reading
"/var/spool/postfix/dev/log". This might be a possible problem.

Personally I've never seen anything like this. (but hey, that's what
bugreports are for :)

The best practice when something like this happens is to gather all
information possible, like:
 * strace syslog-ng (with long strings, e.g. -s 256)
 * strace the sending process
 * check the syslog transport medium if applicable (e.g. via tcpdump)
 * let the syslog-ng process dump core (by enabling core files and
sending a QUIT signal)

If messages flow into syslog-ng (confirmed using tcpdump or by stracing
the sender process), and it actually receives the messages (confirmed by
straceing syslog-ng and checking whether it gets this message via recv),
and still it does not write it into any of its destinations while it
should have done so (confirmed by checking the configuration file/log
files), then it is a bug in syslog-ng.

The problem is that we have not yet confirmed the first couple of steps.

-- 
Bazsi