[syslog-ng] [review request] 3.6queue/f/stats-reset

[Evan Rempel]

Perhaps the same logic can be used to identify those counters that have expired meaning that the stats-lifetime() has been exceeded,but
instead of dropping the stat counter at that time, the counter is just flagged for dropping.

Then it becomes the job of the statistics logging, based on the stats-freq() value, to log AND drop those counters that
have been flagged as having exceeded the stats-lifetime().

This way a counter is never dropped until it has exceeded the stats-lifetime() AND has been logged.

This approach will cause me problems because I use the syslog-ng-ctl to get the detailed statistics, AND use
the stats-freq() log line, routed to a program destination to invoke the syslog-ng-ctl. By that time the statistic would have been
dropped :-(

Perhaps the detailed statistics, like those returned from syslog-ng-ctl could be logged internally by syslog-ng, rather than the very long
"Log statistics; ..." log line that is logged today. Perhaps a stats-multiline([0|1]) setting. That means that my program destination
can use the internally logged stream rather than having to invoke syslog-ng-ctl to get the statistics.

Evan.

On 11/18/2013 03:51 AM, Balazs Scheidler wrote:
> On Mon, 2013-11-18 at 12:22 +0100, Gergely Nagy wrote:
>> Balazs Scheidler <bazsi at balabit.hu> writes:
>>
>>> User-visible changes:
>>> =====================
>>>
>>> syslog-ng is keeping track on a number of statistics counters that can
>>> be used to track what is happening.
>>>
>>> The detail level of these counters is controlled by the stats-level()
>>> global option, which defaults to zero.
>>>
>>> syslog-ng reports these counters periodically every stats-freq()
>>> seconds, or preferably on-demand by invoking "syslog-ng-ctl stats" on
>>> the command line.
>>>
>>> At stats-level(3) a number of "dynamic" counters are enabled, which
>>> track the number of messages on a per $HOST, per $PROGRAM and per
>>> $HOST_FROM basis. This means that you can follow how many messages a
>>> given $PROGRAM is generating in a period.
>>>
>>> The issue with dynamic counters is that they can consume a lot of memory
>>> if  the rate of new $PRORGRAM/$HOST values is high.
>>>
>>> Previously the cleanup happened in a limited way at configuration reload
>>> (e.g. SIGHUP) time, which is not really useful for controlling memory
>>> usage by this feature.
>>>
>>> The current patch series introduces a stats-lifetime() option and
>>> syslog-ng starts inspecting counters every once in a while and prune
>>> those that are updated earlier than stats-lifetime().
>>
>> So, this means that if I don't query the stats just before clearing,
>> I'll potentially loose information? I mean, considering stats-freq(60)
>> stats-lifetime(10), if I have a source that has messages available every
>> 14 seconds, I'll loose some of  them.
>
> yeah, stats-lifetime() should be set to a value to at least stats-freq()
> (or the poll frequency of an external polling application) to avoid data
> loss.
>
>>
>> Would it be possible to dump the stats just before forgetting them?
>> Even better, dump only the ones we're about to prune out. That'd make
>> things a bit safer, I think.
>>
>
> I'd like to make it possible to generate a structured log message when
> stats-freq() happens. that can be the "dump" operation.
>
> I'm thinking along the lines of
>
> ${.syslog-ng.stats[0].type}        == "processed"
> ${.syslog-ng.stats[0].source}      == "src.file"
> ${.syslog-ng.stats[0].instance}    == "input.log"
> ${.syslog-ng.stats[0].value}       == "232323"
> ${.syslog-ng.stats[0].value_delta} == "3"
>
> And when formatted using $(format-json) this could become an actual
> array.
>
> we would need array support in name-value pairs for that though.
>
> The point of the format is to make it as easy to generate graphs using
> graphite or similar tools.
>
> Once we have the structured format, we might as well reuse that to dump
> pruned entries once pruning happens.
>
> What do you think?
>
>


-- 
Evan Rempel                                      erempel at uvic.ca
Senior Systems Administrator                        250.721.7691
Data Centre Services, University Systems, University of Victoria