[syslog-ng] Lost messages

[Andrew Séguin]

Thank you for the quick reply.

Sorry for not mention, since are starting this now, we used the latest
version, so syslog-ng 2.0.6...

We looked at the stats (set to 10s), and saw the following:

$ ./syslog-ng -v -e -F -f /tmp/syslog-ng.conf.solaris
syslog-ng starting up; version='2.0.6'
Log statistics; processed='center(queued)=0',
processed='center(received)=0', processed='destination(all)=0',
processed='source(local)=0'
Log statistics; processed='center(queued)=0',
processed='center(received)=1', processed='destination(all)=0',
processed='source(local)=1'
Log statistics; processed='center(queued)=0',
processed='center(received)=1', processed='destination(all)=0',
processed='source(local)=1'
Initializing destination file writer; template='/tmp/messages-ng',
filename='/tmp/messages-ng'
Log statistics; processed='center(queued)=46862',
processed='center(received)=46863', processed='destination(all)=46862',
processed='source(local)=46863'
Log statistics; processed='center(queued)=136634',
processed='center(received)=136635', processed='destination(all)=136634',
processed='source(local)=136635'
<...>
Log statistics; processed='center(queued)=578629',
processed='center(received)=578630', processed='destination(all)=578629',
processed='source(local)=578630'
Log statistics; processed='center(queued)=578629',
processed='center(received)=578630', processed='destination(all)=578629',
processed='source(local)=578630'
Log statistics; processed='center(queued)=578629',
processed='center(received)=578630', processed='destination(all)=578629',
processed='source(local)=578630'

Our test program (same host, a Sun Fire v440 running Solaris 10) had tried
to log 1000000 messages. I guess this means the loss is before it reached
Syslog-ng?

Do you have any idea how we can get around this? The more logging-intensive
applications, have their sources available to us, a few however don't, so
maybe combining some other source with /dev/log could help?

Any and all ideas are welcome!

Best regards,
Andrew

On Dec 21, 2007 10:09 AM, Balazs Scheidler <bazsi at balabit.hu> wrote:

>
> On Fri, 2007-12-21 at 09:14 +0100, Andrew Séguin wrote:
> > Hello,
> >
> > I'm sorry if I'm missing something obvious, admittedly I've only had
> > the chance to quickly search through the manuals and mailing list and
> > not read them as deep as I'd like to, yet.
> >
> > I'm working in a project where we were considering replacing syslog to
> > take advantage of reliable transmission over tcp. Performance is a
> > factor for us, so we wrote a short program that simply loops and logs
> > a counter and time stamp (to avoid "last message repeated x
> > messages"). We ran it with syslog (saving locally to a file, and then
> > sending remotely to another station which is logging to file) and got
> > some reference numbers for the hardware/OS (Solaris 10). We then
> > started the same test with syslog-ng. Performance didn't get to be an
> > issue: under the pressure of the performance test, only some 5-600,000
> > lines are logged although 1,000,000 were sent!
> >
> > We tried a few tweaks to the configuration file for buffering (see
> > below), but it hasn't helped unfortunately. Is there a way to avoid
> > this problem? Will we encounter this same problem on the remote host
> > (considering the remote/logging host is planned to be accepting
> > messages from two servers with a lot of traffic)
> >
> > Thanks for any tips/info!
> > Andrew Seguin
> >
> >
> > ps: here is the configuration file we have used...
> >
> > #
> > # Syslog-ng example configuration file for Solaris
> > #
> > # Copyright (c) 1999 Balazs Scheidler
> > # $Id: syslog-ng.conf.solaris,v 1.2 1999/11/15 12:30:41 bazsi Exp $
> > #
> > # Solaris 2.5.1 and below uses the STREAMS driver, above extends it
> > # with doors. For 2.5.1 remove the door() option from the source
> > declaration.
> > #
> >
> > options {
> >         sync (0);
> >         log_fifo_size (65535);
> >         gc_idle_threshold(30); gc_busy_threshold(3000);
> > };
> >
> > source local { sun-streams("/dev/log" door("/etc/.syslog_door"));
> > internal(); };
> >
> > destination all { file("/tmp/messages-ng" log_fifo_size(60000)); };
> >
> > filter filter_local6 { facility(local6); };
> > log { source(local); filter(filter_local6); destination(all); };
>
> The syslog-ng version would be a useful information.
>
> What you need to find out where the lossage occurs, it might happen on
> the /dev/log device, or inside syslog-ng.
>
> To find out whether it's the latter case, please check the "Log
> statistics" message (or STATS in syslog-ng 1.6.x). If the drop counters
> are zero, then it is the streams device which is dropping messages.
>
> I don't remember all the STREAMS details whether it can lose messages,
> but before digging any further it'd be useful to know where the messages
> get actually lost.
>
> --
> Bazsi
>
> _______________________________________________
> syslog-ng maillist  -  syslog-ng at lists.balabit.hu
> https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20071221/fb9b5958/attachment.htm