[syslog-ng] 1.6.5 performance

[Scott C]

I am currently experiencing an odd performance issue with syslog-ng 1.6.5.  I use several load-balanced relays which receive syslog traffic via both udp and tcp.  They forward all traffic to the back-end collection hosts exclusively via tcp.  Overall, this environment collects around 60GB of log data per day.  All of the relays have identical Solaris 8 builds, as well as identical syslog-ng configurations.  Here is the pertinent snippet from a configuration file:
 
options { stats(300); sync(0); time_reopen(10); log_fifo_size(3000000); gc_busy_threshold(200000); check_hostname(no); keep_hostname(yes); };
 
As you can see, my fifo queue is enormous.  Despite that fact, syslog-ng on any given relay will begin dropping messages at an alarming rate (600-700 per second) when combined inbound and outbound tcp traffic increases beyond a certain threshold.  Said threshold seems to be a bit low to me, though.  We're talking somewhere in the neighborhood of 700 tcp segments per second.  The corresponding inbound UDP dgrams are usually between 200 and 400 per second.  It should be noted that the IP stack is handling the load just fine.  Also, CPU utilization rarely exceeds 60 or 70 percent.
 
The collection hosts are fairly beefy systems that are barely breathing even during peak load.  I've watched both the IP stack and syslog-ng on these systems, and they're keeping up just fine.
 
I know the FIFO queue size probably seems unreasonable here, but the lower I make it, the worse the drops.  Of course, the higher I make it, the more memory it eats up.  And in some cases, I've seen syslog-ng eventually eat it all up and the scanner kicks in.
 
But what's really most peculiar in this scenario is the fact that the numbers simply don't add up.  Why does syslog-ng appear (on the surface) to be dropping a very large percentage of the messages that it receives?  I realize that it's not, but the numbers tell a different story.  And how could it possibly drop so many messages when the FIFO queue is configured to buffer three million lines?  How preposterous!
 
Am I way off base here?





		
---------------------------------
 Yahoo! FareChase - Search multiple travel sites in one click.  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20051021/dce6bf0a/attachment.htm