Ah, so it's not a real file, it's part of awk. That explains it. Ok, maybe even easier than running tcpdump for DNS would be to just swap $FULLHOST_FROM with $SOURCEIP and see if that improves things. Two more things to look at: what is the CPU % when it's running, and if you strace it what syscalls does it seem to be doing the most? Gettimeofday should be in there quite a bit, but sometimes calls you didn't expect jump out and show what's blocking. You really shouldn't have to post-process with syslog-ng; there's got to be something wrong. I also find your raw socket numbers to be lower than I'd expect. I would expect 0 loss through 10k mps at least. On Fri, Oct 15, 2010 at 4:22 PM, Lars Kellogg-Stedman <lars@oddbit.com> wrote:
Actually, I missed what you were doing with awk because I don't think I've ever seen /inet before. Are you on FreeBSD? My experience (and cited performance numbers) is all on Linux.
Me too. Awk has supported network connectivity for a decade or so, I think (since v3.1). Using netcat, the results are pretty much the same. At 2000 msgs/sec I get around 10% loss, which is a lot better than syslog-ng.
One of the possibilities I'm looking at is putting something like netcat (or socklog, http://smarden.org/socklog/) in front of syslog-ng and spooling the messages from the network directly to disk, and then post-processing them with syslog-ng. This would solve the performance problem, since these simpler tools appear to have no problem supporting higher logging rates, and we've got disk space to spare. Injecting the messages into syslog-ng in a useful fashion might be a bit of a challenge.
Our alternate plan is just to use tcp syslog across the board (which avoids the loss problem), although I'm concerned that this may impact the connection-tracking firewalls in use around our environment. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html