[syslog-ng] [review request] 3.6queue/f/stats-reset

[Balazs Scheidler]

Hi,

I have now updated my branch. The new functionality (extended statistics
log message and log-fifo-size() reporting through stats) is not yet
created, however a set of code reorganization would make this change
much easier.

So the current patch set (published on the same branch, rebased since
last time) uses the stats-freq() timer to prune unneeded counters, but
only if they have already been reported by the statistics log message.

This solves the losing-data issue that was discussed earlier.

So this time, it is more of a code-review thing, could those interested
in code review this in the coming days, prior to integration?

Thanks.

On Wed, 2013-11-20 at 14:47 +0100, Gergely Nagy wrote:
> Balazs Scheidler <bazsi77 at gmail.com> writes:
> 
> > On Mon, 2013-11-18 at 18:22 +0100, Gergely Nagy wrote:
> >> Evan Rempel <erempel at uvic.ca> writes:
> >> 
> >> > Perhaps the same logic can be used to identify those counters that have expired meaning that the stats-lifetime() has been exceeded,but
> >> > instead of dropping the stat counter at that time, the counter is just flagged for dropping.
> >> >
> >> > Then it becomes the job of the statistics logging, based on the stats-freq() value, to log AND drop those counters that
> >> > have been flagged as having exceeded the stats-lifetime().
> >> >
> >> > This way a counter is never dropped until it has exceeded the
> >> > stats-lifetime() AND has been logged.
> >> 
> >> That makes a lot of sense, but syslog-ng-ctl stats should not count (or
> >> perhaps only with a flag), precisely to avoid the scenario you outlined
> >> too.
> >> 
> >
> > This really makes sense, however my issue is that stats-freq() is often
> > set to 0 which disables the Log statistics message currently. Relying
> > the cleanup mechanism on that would disable cleanup altogether in a lot
> > of setups.
> >
> > hmm... maybe we should use stats-freq() if that's nonzero and
> > stats-lifetime() when it is zero. This way the cleanup mechanism would
> > kick in, without the risk of losing messages.
> >
> > syslog-ng-ctl stats wouldn't prune counters.
> >
> > This way, stats-freq() would be the one which actually clears counters,
> > we get rid of an additional iteration over the set of counters.
> >
> > This paves the way for an improved log statistics message that could be
> > sent into graphite/logstash/whatever graphing application via standard
> > syslog-ng destination drivers, and all this without the risk of losing
> > counters.
> >
> > What do you think?
> 
> Sounds very good to me. This is a big step towards what I need, and is
> simple enough to understand too.
>