[syslog-ng] One destination stopped logging?

[Evan Rempel]

In the syslong-ng 1.6.x series, there is no "reopen" mchanism for disk based files
that become closed. The question is how they get closed.

If they are closed due to an idle timeout (all syslog destinations do this I think),
then when a new message to that destination is processed, the destination will be reopened,
even disk files.

If the disk file was closed due to an error (IO error of some kind), then the file is
never reopened, unless the destination goes through an idle timeout and reopen sequence.

In all cases a reload/restart of syslog-ng causes all destinations to be closed and reopened.

I have seen cases where a busy destination (ours was mail, just like yours) becomes closed due
to a full filesystem. No other destinations became closed because they did not have messages
processed during the interval when the filesystem was full. Something occurs to free up some space
on the filesystem, so new messages all get processed correctly, however, the mail destination never
became idel, but was never opened again.

I would really like to have file destinations handled just like network destination and
adhere to the reopen configuration setting.

I am not sure how the syslog-ng 2.0.x series behaves in these circumstances.

Evan Rempel.


Hari Sekhon wrote:
> Hi,
> 
> I've got syslog-ng on my mail server and today I noticed while trying to 
> debug something else that there was nothing in the maillog. I thought 
> this was very odd as there should be lots of stuff, so I checked 
> syslog-ng was running. It was but I decided to restart it and then the 
> maillog started to fill up again.
> 
> Now it's been several days since it's written anything to the mail log, 
> but it has continued writing to the /var/log/messages destination during 
> that time.
> 
> Here is my config:
> 
> options {
>    chain_hostnames(off);
>    sync(0);
>    stats(43200);
>    log_fifo_size(30000);
> };
> source src { unix-stream("/dev/log" max-connections(1000)); internal(); 
> pipe("/proc/kmsg"); };
> destination messages { file("/var/log/messages"); };
> destination d_net { tcp("ip_to_logserver" port(logserver_port) ); };
> destination maillog { file("/var/log/maillog"); };
> destination mailerr { file("/var/log/mail.err"); };
> filter f_mail { facility(mail); };
> filter f_mailerr { facility(mail) and level(err); };
> filter f_notmailjunk { not (program("postfix/*") and not level(err)); };
> log { source(src); filter(f_mail); destination(maillog); };
> log { source(src); filter(f_mailerr); destination(mailerr); };
> log { source(src); filter(f_notmailjunk); destination(messages); 
> destination(d_net); };
> 
> 
> Possible idea: My logserver was flooded as I was doing something else on 
> it and it's likely that logs didn't get through (I know, I know - I 
> wanted a separate machine for this of course but my boss holds the 
> money...). Is it possible that the logger filled up locally backlogged 
> messages for the logserver and this caused logs to be lost for the other 
> destination maillog as syslog-ng was full or something?
> 
> Does this sound like it or not at all?
> 
> I am using syslog-ng 1.6.11 by the way.
> 
> Thanks
> 
> Hari
>