[syslog-ng] UDP packet loss with syslog-ng

[Martin Holste]

Ah, so it's not a real file, it's part of awk.  That explains it.

Ok, maybe even easier than running tcpdump for DNS would be to just
swap $FULLHOST_FROM with $SOURCEIP and see if that improves things.
Two more things to look at: what is the CPU % when it's running, and
if you strace it what syscalls does it seem to be doing the most?
Gettimeofday should be in there quite a bit, but sometimes calls you
didn't expect jump out and show what's blocking.

You really shouldn't have to post-process with syslog-ng; there's got
to be something wrong.  I also find your raw socket numbers to be
lower than I'd expect.  I would expect 0 loss through 10k mps at
least.

On Fri, Oct 15, 2010 at 4:22 PM, Lars Kellogg-Stedman <lars at oddbit.com> wrote:
>> Actually, I missed what you were doing with awk because I don't think
>> I've ever seen /inet before. Are you on FreeBSD?  My experience (and
>> cited performance numbers) is all on Linux.
>
> Me too.  Awk has supported network connectivity for a decade or so, I
> think (since v3.1).  Using netcat, the results are pretty much the
> same.  At 2000 msgs/sec I get around 10% loss, which is a lot better
> than syslog-ng.
>
> One of the possibilities I'm looking at is putting something like
> netcat (or socklog, http://smarden.org/socklog/) in front of syslog-ng
> and spooling the messages from the network directly to disk, and then
> post-processing them with syslog-ng.  This would solve the performance
> problem, since these simpler tools appear to have no problem
> supporting higher logging rates, and we've got disk space to spare.
> Injecting the messages into syslog-ng in a useful fashion might be a
> bit of a challenge.
>
> Our alternate plan is just to use tcp syslog across the board (which
> avoids the loss problem), although I'm concerned that this may impact
> the connection-tracking firewalls in use around our environment.
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>