[syslog-ng] udp drops
mcholste at gmail.com
Sun May 31 13:59:45 CEST 2009
Out of curiosity, how many messages per second was the stock syslogd able to
process with minimal loss? What method did you employ to determine loss? I
am setting up a similar logging solution with NG using the db-parser module
which takes considerable CPU. I plan on using Cisco server load balancing
to round-robin load balance on N number of syslog nodes to achieve zero
loss, but I'm concerned that detecting loss will be difficult.
On Sat, May 30, 2009 at 12:43 PM, Jan Schaumann <jschauma at netmeister.org>wrote:
> Sandor Geller <Sandor.Geller at morganstanley.com> wrote:
> > This is somewhat expected as syslog-ng parses incoming messages. So my
> > I guess is that syslog-ng can't drain fast enough the receive buffer,
> > and the kernel simply drops messages not fitting in the buffer.
> > It would be good to know whether the source side or the destination
> > side is the limiting factor. As you're using local files myguess is
> > the former.
> I'm quite sure the source side is the problem. Ie, I/O to the file on
> disk ought to be reasonably fast (otherwise stock syslogd would have the
> same problems). As you noted, the additional processing that syslog-ng
> does for every message it receives seems to cause it to not be able to
> process them fast enough to drain the buffers.
> > > flags(flow-control)
> > >
> > > in the log definition.
> > AFAIK with files/ UDP flow-control is a no-op.
> Ah, good to know.
> > Unfortunately this can't happen. You can use the 'no-parse' option to
> > skip initial parsing the messages which could improve performance.
> > This means you can't use the template above as the variables won't get
> > defined.
> I'll have to give that a try, if only to determine what, if any,
> performance difference it causes.
> > Generally when it comes to parsing then syslog-ng could be
> > CPU-limited. In this case you should consider deploying multiple
> > syslog servers, and share the load. Ideally flow-controlling could be
> > turned on the client side as well (using TCP).
> Yes, those are the long-term plans. :-) Well, we can't switch all
> clients to TCP, since many of them are network/storage devices etc. only
> capable of logging via UDP.
> For the time being, though, I need to lay the ground work of getting
> syslog-ng as a suitable replacement for the stock syslogd used on our
> Thanks for your help. I'll keep poking at this...
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the syslog-ng