[syslog-ng] Not all messages getting to program destination

[Bill Anderson]

On Apr 13, 2011, at 5:28 PM, Matthew Hall wrote:

> On Wed, Apr 13, 2011 at 03:28:18PM -0500, Martin Holste wrote:
>> Hm, you really shouldn't be dropping with a simple script.  How many
>> msgs/second?
> 
> Try recording a few hundred MB and replaying through the script.
> 
> Print a time stamp for every 10K processed and look for spike / dip with a spreadsheet. Try a profiler, etc.

I had done all of that (though using several GB of data), hence the confusion. Average traffic for this script is 200-400/s, with peaks of 1500/s

> 
> Maybe something in the script is not stable under load.
> 
> Or maybe syslog-ng is getting too many UDPs with too small of a socket buffer and losing them.

If that were the case Id expect those messages to also not make it into the log file. Then again, I'm using TCP for these. ;)

After a couple days of effort I've eliminated the problem. Turns out I'd get bursts of over 1200/second for a while and things apparently just piled up, and once behind it started dropping. I've increased the log_fifo_size to 21k, moved some of the conditional logic into syslog, and reduced the fields in the message to just those that are absolutely required by the script,  and the problem has disappeared entirely. The first two reduced it, the last finished it off entirely.

Related question: is there a reliable way to reset the statistics counters?



Cheers,
Bill

--
Bill Anderson, RHCE
Linux Systems Engineer
bill.anderson at bodybuilding.com