Perhaps the same logic can be used to identify those counters that have expired meaning that the stats-lifetime() has been exceeded,but instead of dropping the stat counter at that time, the counter is just flagged for dropping. Then it becomes the job of the statistics logging, based on the stats-freq() value, to log AND drop those counters that have been flagged as having exceeded the stats-lifetime(). This way a counter is never dropped until it has exceeded the stats-lifetime() AND has been logged. This approach will cause me problems because I use the syslog-ng-ctl to get the detailed statistics, AND use the stats-freq() log line, routed to a program destination to invoke the syslog-ng-ctl. By that time the statistic would have been dropped :-( Perhaps the detailed statistics, like those returned from syslog-ng-ctl could be logged internally by syslog-ng, rather than the very long "Log statistics; ..." log line that is logged today. Perhaps a stats-multiline([0|1]) setting. That means that my program destination can use the internally logged stream rather than having to invoke syslog-ng-ctl to get the statistics. Evan. On 11/18/2013 03:51 AM, Balazs Scheidler wrote:
On Mon, 2013-11-18 at 12:22 +0100, Gergely Nagy wrote:
Balazs Scheidler <bazsi@balabit.hu> writes:
User-visible changes: =====================
syslog-ng is keeping track on a number of statistics counters that can be used to track what is happening.
The detail level of these counters is controlled by the stats-level() global option, which defaults to zero.
syslog-ng reports these counters periodically every stats-freq() seconds, or preferably on-demand by invoking "syslog-ng-ctl stats" on the command line.
At stats-level(3) a number of "dynamic" counters are enabled, which track the number of messages on a per $HOST, per $PROGRAM and per $HOST_FROM basis. This means that you can follow how many messages a given $PROGRAM is generating in a period.
The issue with dynamic counters is that they can consume a lot of memory if the rate of new $PRORGRAM/$HOST values is high.
Previously the cleanup happened in a limited way at configuration reload (e.g. SIGHUP) time, which is not really useful for controlling memory usage by this feature.
The current patch series introduces a stats-lifetime() option and syslog-ng starts inspecting counters every once in a while and prune those that are updated earlier than stats-lifetime().
So, this means that if I don't query the stats just before clearing, I'll potentially loose information? I mean, considering stats-freq(60) stats-lifetime(10), if I have a source that has messages available every 14 seconds, I'll loose some of them.
yeah, stats-lifetime() should be set to a value to at least stats-freq() (or the poll frequency of an external polling application) to avoid data loss.
Would it be possible to dump the stats just before forgetting them? Even better, dump only the ones we're about to prune out. That'd make things a bit safer, I think.
I'd like to make it possible to generate a structured log message when stats-freq() happens. that can be the "dump" operation.
I'm thinking along the lines of
${.syslog-ng.stats[0].type} == "processed" ${.syslog-ng.stats[0].source} == "src.file" ${.syslog-ng.stats[0].instance} == "input.log" ${.syslog-ng.stats[0].value} == "232323" ${.syslog-ng.stats[0].value_delta} == "3"
And when formatted using $(format-json) this could become an actual array.
we would need array support in name-value pairs for that though.
The point of the format is to make it as easy to generate graphs using graphite or similar tools.
Once we have the structured format, we might as well reuse that to dump pruned entries once pruning happens.
What do you think?
-- Evan Rempel erempel@uvic.ca Senior Systems Administrator 250.721.7691 Data Centre Services, University Systems, University of Victoria