Out of curiosity, how many messages per second was the stock syslogd able to process with minimal loss?  What method did you employ to determine loss?  I am setting up a similar logging solution with NG using the db-parser module which takes considerable CPU.  I plan on using Cisco server load balancing to round-robin load balance on N number of syslog nodes to achieve zero loss, but I'm concerned that detecting loss will be difficult.

--Martin

On Sat, May 30, 2009 at 12:43 PM, Jan Schaumann <jschauma@netmeister.org> wrote:
Sandor Geller <Sandor.Geller@morganstanley.com> wrote:

> This is somewhat expected as syslog-ng parses incoming messages. So my
> I guess is that syslog-ng can't drain fast enough the receive buffer,
> and the kernel simply drops messages not fitting in the buffer.

Exactly.

> It would be good to know whether the source side or the destination
> side is the limiting factor. As you're using local files myguess is
> the former.

I'm quite sure the source side is the problem.  Ie, I/O to the file on
disk ought to be reasonably fast (otherwise stock syslogd would have the
same problems).  As you noted, the additional processing that syslog-ng
does for every message it receives seems to cause it to not be able to
process them fast enough to drain the buffers.


> >        flags(flow-control)
> >
> > in the log definition.
>
> AFAIK with files/ UDP flow-control is a no-op.

Ah, good to know.

> Unfortunately this can't happen. You can use the 'no-parse' option to
> skip initial parsing the messages which could improve performance.
> This means you can't use the template above as the variables won't get
> defined.

I'll have to give that a try, if only to determine what, if any,
performance difference it causes.

> Generally when it comes to parsing then syslog-ng could be
> CPU-limited. In this case you should consider deploying multiple
> syslog servers, and share the load. Ideally flow-controlling could be
> turned on the client side as well (using TCP).

Yes, those are the long-term plans. :-)  Well, we can't switch all
clients to TCP, since many of them are network/storage devices etc. only
capable of logging via UDP.

For the time being, though, I need to lay the ground work of getting
syslog-ng as a suitable replacement for the stock syslogd used on our
servers.

Thanks for your help.  I'll keep poking at this...

-Jan

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html