Balazs Scheidler <bazsi77@gmail.com> writes:
The current patch series introduces a stats-lifetime() option and syslog-ng starts inspecting counters every once in a while and prune those that are updated earlier than stats-lifetime().
So, this means that if I don't query the stats just before clearing, I'll potentially loose information? I mean, considering stats-freq(60) stats-lifetime(10), if I have a source that has messages available every 14 seconds, I'll loose some of them.
yeah, stats-lifetime() should be set to a value to at least stats-freq() (or the poll frequency of an external polling application) to avoid data loss.
That's not good.. we should fix this situation soonish, as this is much more complicated than my brain is willing to understand on a monday morning.
Would it be possible to dump the stats just before forgetting them? Even better, dump only the ones we're about to prune out. That'd make things a bit safer, I think.
I'd like to make it possible to generate a structured log message when stats-freq() happens. that can be the "dump" operation.
I'm thinking along the lines of
${.syslog-ng.stats[0].type} == "processed" ${.syslog-ng.stats[0].source} == "src.file" ${.syslog-ng.stats[0].instance} == "input.log" ${.syslog-ng.stats[0].value} == "232323" ${.syslog-ng.stats[0].value_delta} == "3"
And when formatted using $(format-json) this could become an actual array.
we would need array support in name-value pairs for that though.
The point of the format is to make it as easy to generate graphs using graphite or similar tools.
Once we have the structured format, we might as well reuse that to dump pruned entries once pruning happens.
What do you think?
This sounds good to me. I don't really care about the format of the dump at the moment, but the dump should happen. As for array support: just use [$index] as convention, and we'll teach most of the things that can potentially use it, to handle it (which means value-pairs). I had a PoC of that at one point, but I overcomplicated the code. I'll have another go sometime later, but I don't think that should hold up this feature. Just generate a new internal message with the layout you proposed above, we'll care about properly formatting it later, I would say. -- |8]