[syslog-ng] [review request] 3.6queue/f/stats-reset

[Gergely Nagy]

Balazs Scheidler <bazsi77 at gmail.com> writes:

>> > The current patch series introduces a stats-lifetime() option and
>> > syslog-ng starts inspecting counters every once in a while and prune
>> > those that are updated earlier than stats-lifetime().
>> 
>> So, this means that if I don't query the stats just before clearing,
>> I'll potentially loose information? I mean, considering stats-freq(60)
>> stats-lifetime(10), if I have a source that has messages available every
>> 14 seconds, I'll loose some of  them.
>
> yeah, stats-lifetime() should be set to a value to at least stats-freq()
> (or the poll frequency of an external polling application) to avoid data
> loss.

That's not good.. we should fix this situation soonish, as this is much
more complicated than my brain is willing to understand on a monday
morning.

>> Would it be possible to dump the stats just before forgetting them?
>> Even better, dump only the ones we're about to prune out. That'd make
>> things a bit safer, I think.
>> 
>
> I'd like to make it possible to generate a structured log message when
> stats-freq() happens. that can be the "dump" operation.
>
> I'm thinking along the lines of
>
> ${.syslog-ng.stats[0].type}        == "processed"
> ${.syslog-ng.stats[0].source}      == "src.file"
> ${.syslog-ng.stats[0].instance}    == "input.log"
> ${.syslog-ng.stats[0].value}       == "232323"
> ${.syslog-ng.stats[0].value_delta} == "3"
>
> And when formatted using $(format-json) this could become an actual
> array.
>
> we would need array support in name-value pairs for that though.
>
> The point of the format is to make it as easy to generate graphs using
> graphite or similar tools.
>
> Once we have the structured format, we might as well reuse that to dump
> pruned entries once pruning happens.
>
> What do you think?

This sounds good to me. I don't really care about the format of the dump
at the moment, but the dump should happen.

As for array support: just use [$index] as convention, and we'll teach
most of the things that can potentially use it, to handle it (which
means value-pairs). I had a PoC of that at one point, but I
overcomplicated the code. I'll have another go sometime later, but I
don't think that should hold up this feature.

Just generate a new internal message with the layout you proposed above,
we'll care about properly formatting it later, I would say.

-- 
|8]