[syslog-ng] [review request] 3.6queue/f/stats-reset

[Fekete Róbert]

Hi, 

I am not sure I understand correctly how this feature will work. For example, I set the stats-lifetime() option to 60 seconds, and start syslog-ng. At that moment, all counters are 0.
After some time, say, 70 seconds after start, I use the syslog-ng-ctl stats command. What will be the output?
The number of messages received since the last time the counters were zeroed (that is, in the last 10 seconds)? Or the number of messages received in the last full cycle (0-60 seconds)? If the former, does the stats message indicate that the number apply to the last 10 seconds?

Also, if I want to aggregate the statistics, do I have to use the stats-freq() option, then filter and parse the stats message (so I get the numbers for the last 60 minutes), or is it possible to get this info reliably using the syslog-ng-ctl-stats command somehow?

Regards,

Robert
 
On Saturday, November 16, 2013 14:50 CET, Balazs Scheidler <bazsi at balabit.hu> wrote: 
 
> Hi,
> 
> I'm about to integrate a stats-counter related change and prior to doing
> that I'd like to solicit feedback.
> 
> There are both user-visible changes and developer visible changes,
> feedback on both is sought.
> 
> Code related comments are preferred on github, functionality comments
> are best sent via email in response to this one.
> 
> Here's the branch:
> 
> https://github.com/balabit/syslog-ng/tree/3.6queue/f/stats-reset
> 
> Thanks in advance.
> 
> User-visible changes:
> =====================
> 
> syslog-ng is keeping track on a number of statistics counters that can
> be used to track what is happening.
> 
> The detail level of these counters is controlled by the stats-level()
> global option, which defaults to zero.
> 
> syslog-ng reports these counters periodically every stats-freq()
> seconds, or preferably on-demand by invoking "syslog-ng-ctl stats" on
> the command line.
> 
> At stats-level(3) a number of "dynamic" counters are enabled, which
> track the number of messages on a per $HOST, per $PROGRAM and per
> $HOST_FROM basis. This means that you can follow how many messages a
> given $PROGRAM is generating in a period.
> 
> The issue with dynamic counters is that they can consume a lot of memory
> if  the rate of new $PRORGRAM/$HOST values is high.
> 
> Previously the cleanup happened in a limited way at configuration reload
> (e.g. SIGHUP) time, which is not really useful for controlling memory
> usage by this feature.
> 
> The current patch series introduces a stats-lifetime() option and
> syslog-ng starts inspecting counters every once in a while and prune
> those that are updated earlier than stats-lifetime().
> 
> Not-so visible changes
> ======================
> 
> I have fixed a few bugs and renamed a function to make it easier to
> understand. Details are in the git history.
> 
> 
> Balazs Scheidler (7):
>   logsource: use stats_instant_inc_dynamic_counter() for HOST counters
>   logsource: track time for PROGRAM counters
>   mainloop: remove old orphanage based stats cleanup
>   stats: add stats_prune_old_counters()
>   stats-timer: implement stats_lifetime option
>   cfg: add stats-lifetime() global option
>   stats: rename stats_instance_inc_dynamic_counter() to
>     stats_register_and_increment_dynamic_counter()
> 
>  lib/cfg-grammar.y | 16 ++++++++-------
>  lib/cfg-parser.c  |  1 +
>  lib/cfg.c         |  3 ++-
>  lib/cfg.h         |  1 +
>  lib/logsource.c   | 15 +++-----------
>  lib/mainloop.c    |  1 -
>  lib/stats-timer.c | 61 +++++++++++++++++++++++++++++++++++++++----------------
>  lib/stats-timer.h |  2 +-
>  lib/stats.c       | 52 +++++++++++++++++++++++++++++++++++++++--------
>  lib/stats.h       |  7 +++----
>  10 files changed, 108 insertions(+), 51 deletions(-)
> 
> 
> -- 
> Bazsi
>