[syslog-ng] Not all messages getting to program destination

[Bill Anderson]

On Apr 13, 2011, at 9:52 AM, Martin Holste wrote:

> Swap out the python script with a different python script which simply
> writes to a file and verify that looks the way it should (matches your
> grep).  That should narrow things down.

It ran overnight and now I am actually getting dropped counters so at least that makes sense, though as to why I wasn't before I dunno. Unfortunately the dropped counters vary greatly from 0 most minutes to ~26k. Looks like I may need to recode in c or c++. That's far too much variance for my tastes. Alternatively some digging into flow-control might work. Or just a python script that dumps the message to either my rabbitmq system or the redis cache, and have the python script pull from there. It's strange that the same python script can handle far more than that rate on stdin normally. In fact if I grep the minutes out of the logfile that I get the >20k dropped changes and pipe them in, it handles it in just a few seconds. 

Perhaps flow control may be the more sane route. Any tips in that direction?


Thanks,
Bill

Cheers,
Bill

--
Bill Anderson, RHCE
Linux Systems Engineer
bill.anderson at bodybuilding.com