I havent followed this up, but this got integrated and is on mainline. The additional functionality is noted but not yet there. On Nov 27, 2013 11:22 AM, "Balazs Scheidler" <bazsi@balabit.hu> wrote:
Hi,
I have now updated my branch. The new functionality (extended statistics log message and log-fifo-size() reporting through stats) is not yet created, however a set of code reorganization would make this change much easier.
So the current patch set (published on the same branch, rebased since last time) uses the stats-freq() timer to prune unneeded counters, but only if they have already been reported by the statistics log message.
This solves the losing-data issue that was discussed earlier.
So this time, it is more of a code-review thing, could those interested in code review this in the coming days, prior to integration?
Thanks.
On Wed, 2013-11-20 at 14:47 +0100, Gergely Nagy wrote:
Balazs Scheidler <bazsi77@gmail.com> writes:
On Mon, 2013-11-18 at 18:22 +0100, Gergely Nagy wrote:
Evan Rempel <erempel@uvic.ca> writes:
Perhaps the same logic can be used to identify those counters that have expired meaning that the stats-lifetime() has been exceeded,but instead of dropping the stat counter at that time, the counter is just flagged for dropping.
Then it becomes the job of the statistics logging, based on the stats-freq() value, to log AND drop those counters that have been flagged as having exceeded the stats-lifetime().
This way a counter is never dropped until it has exceeded the stats-lifetime() AND has been logged.
That makes a lot of sense, but syslog-ng-ctl stats should not count (or perhaps only with a flag), precisely to avoid the scenario you outlined too.
This really makes sense, however my issue is that stats-freq() is often set to 0 which disables the Log statistics message currently. Relying the cleanup mechanism on that would disable cleanup altogether in a lot of setups.
hmm... maybe we should use stats-freq() if that's nonzero and stats-lifetime() when it is zero. This way the cleanup mechanism would kick in, without the risk of losing messages.
syslog-ng-ctl stats wouldn't prune counters.
This way, stats-freq() would be the one which actually clears counters, we get rid of an additional iteration over the set of counters.
This paves the way for an improved log statistics message that could be sent into graphite/logstash/whatever graphing application via standard syslog-ng destination drivers, and all this without the risk of losing counters.
What do you think?
Sounds very good to me. This is a big step towards what I need, and is simple enough to understand too.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq