[syslog-ng] Lost messages

[Balazs Scheidler]

On Fri, 2007-12-21 at 09:14 +0100, Andrew Séguin wrote:
> Hello,
> 
> I'm sorry if I'm missing something obvious, admittedly I've only had
> the chance to quickly search through the manuals and mailing list and
> not read them as deep as I'd like to, yet.
> 
> I'm working in a project where we were considering replacing syslog to
> take advantage of reliable transmission over tcp. Performance is a
> factor for us, so we wrote a short program that simply loops and logs
> a counter and time stamp (to avoid "last message repeated x
> messages"). We ran it with syslog (saving locally to a file, and then
> sending remotely to another station which is logging to file) and got
> some reference numbers for the hardware/OS (Solaris 10). We then
> started the same test with syslog-ng. Performance didn't get to be an
> issue: under the pressure of the performance test, only some 5-600,000
> lines are logged although 1,000,000 were sent! 
> 
> We tried a few tweaks to the configuration file for buffering (see
> below), but it hasn't helped unfortunately. Is there a way to avoid
> this problem? Will we encounter this same problem on the remote host
> (considering the remote/logging host is planned to be accepting
> messages from two servers with a lot of traffic) 
> 
> Thanks for any tips/info!
> Andrew Seguin
> 
> 
> ps: here is the configuration file we have used...
> 
> #
> # Syslog-ng example configuration file for Solaris
> #
> # Copyright (c) 1999 Balazs Scheidler
> # $Id: syslog-ng.conf.solaris,v 1.2 1999/11/15 12:30:41 bazsi Exp $
> #
> # Solaris 2.5.1 and below uses the STREAMS driver, above extends it
> # with doors. For 2.5.1 remove the door() option from the source
> declaration.
> #
>  
> options {
>         sync (0);
>         log_fifo_size (65535);
>         gc_idle_threshold(30); gc_busy_threshold(3000);
> };
>  
> source local { sun-streams("/dev/log" door("/etc/.syslog_door"));
> internal(); };
>  
> destination all { file("/tmp/messages-ng" log_fifo_size(60000)); };
>  
> filter filter_local6 { facility(local6); };
> log { source(local); filter(filter_local6); destination(all); };

The syslog-ng version would be a useful information. 

What you need to find out where the lossage occurs, it might happen on
the /dev/log device, or inside syslog-ng.

To find out whether it's the latter case, please check the "Log
statistics" message (or STATS in syslog-ng 1.6.x). If the drop counters
are zero, then it is the streams device which is dropping messages.

I don't remember all the STREAMS details whether it can lose messages,
but before digging any further it'd be useful to know where the messages
get actually lost.

-- 
Bazsi