[syslog-ng] [review request] 3.6queue/f/stats-reset

[Balazs Scheidler]

Hi,

I'm about to integrate a stats-counter related change and prior to doing
that I'd like to solicit feedback.

There are both user-visible changes and developer visible changes,
feedback on both is sought.

Code related comments are preferred on github, functionality comments
are best sent via email in response to this one.

Here's the branch:

https://github.com/balabit/syslog-ng/tree/3.6queue/f/stats-reset

Thanks in advance.

User-visible changes:
=====================

syslog-ng is keeping track on a number of statistics counters that can
be used to track what is happening.

The detail level of these counters is controlled by the stats-level()
global option, which defaults to zero.

syslog-ng reports these counters periodically every stats-freq()
seconds, or preferably on-demand by invoking "syslog-ng-ctl stats" on
the command line.

At stats-level(3) a number of "dynamic" counters are enabled, which
track the number of messages on a per $HOST, per $PROGRAM and per
$HOST_FROM basis. This means that you can follow how many messages a
given $PROGRAM is generating in a period.

The issue with dynamic counters is that they can consume a lot of memory
if  the rate of new $PRORGRAM/$HOST values is high.

Previously the cleanup happened in a limited way at configuration reload
(e.g. SIGHUP) time, which is not really useful for controlling memory
usage by this feature.

The current patch series introduces a stats-lifetime() option and
syslog-ng starts inspecting counters every once in a while and prune
those that are updated earlier than stats-lifetime().

Not-so visible changes
======================

I have fixed a few bugs and renamed a function to make it easier to
understand. Details are in the git history.


Balazs Scheidler (7):
  logsource: use stats_instant_inc_dynamic_counter() for HOST counters
  logsource: track time for PROGRAM counters
  mainloop: remove old orphanage based stats cleanup
  stats: add stats_prune_old_counters()
  stats-timer: implement stats_lifetime option
  cfg: add stats-lifetime() global option
  stats: rename stats_instance_inc_dynamic_counter() to
    stats_register_and_increment_dynamic_counter()

 lib/cfg-grammar.y | 16 ++++++++-------
 lib/cfg-parser.c  |  1 +
 lib/cfg.c         |  3 ++-
 lib/cfg.h         |  1 +
 lib/logsource.c   | 15 +++-----------
 lib/mainloop.c    |  1 -
 lib/stats-timer.c | 61 +++++++++++++++++++++++++++++++++++++++----------------
 lib/stats-timer.h |  2 +-
 lib/stats.c       | 52 +++++++++++++++++++++++++++++++++++++++--------
 lib/stats.h       |  7 +++----
 10 files changed, 108 insertions(+), 51 deletions(-)


-- 
Bazsi