[syslog-ng] udp drops

[Jan Schaumann]

Sandor Geller <Sandor.Geller at morganstanley.com> wrote:
 
> This is somewhat expected as syslog-ng parses incoming messages. So my
> I guess is that syslog-ng can't drain fast enough the receive buffer,
> and the kernel simply drops messages not fitting in the buffer.

Exactly.

> It would be good to know whether the source side or the destination
> side is the limiting factor. As you're using local files myguess is
> the former.

I'm quite sure the source side is the problem.  Ie, I/O to the file on
disk ought to be reasonably fast (otherwise stock syslogd would have the
same problems).  As you noted, the additional processing that syslog-ng
does for every message it receives seems to cause it to not be able to
process them fast enough to drain the buffers.


> >        flags(flow-control)
> >
> > in the log definition.
> 
> AFAIK with files/ UDP flow-control is a no-op.

Ah, good to know.

> Unfortunately this can't happen. You can use the 'no-parse' option to
> skip initial parsing the messages which could improve performance.
> This means you can't use the template above as the variables won't get
> defined.

I'll have to give that a try, if only to determine what, if any,
performance difference it causes.

> Generally when it comes to parsing then syslog-ng could be
> CPU-limited. In this case you should consider deploying multiple
> syslog servers, and share the load. Ideally flow-controlling could be
> turned on the client side as well (using TCP).

Yes, those are the long-term plans. :-)  Well, we can't switch all
clients to TCP, since many of them are network/storage devices etc. only
capable of logging via UDP.

For the time being, though, I need to lay the ground work of getting
syslog-ng as a suitable replacement for the stock syslogd used on our
servers.

Thanks for your help.  I'll keep poking at this...

-Jan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 186 bytes
Desc: not available
Url : http://lists.balabit.hu/pipermail/syslog-ng/attachments/20090530/0eb2888a/attachment.pgp