[syslog-ng] [review request] 3.6queue/f/stats-reset

[Balazs Scheidler]

Hi,

Counters are not zeroed. They get pruned iff they are not updated for
stats-lifetime() seconds. And only the dynamic kind.

This means that you have to query them within the lifetime period to avoid
losing their value.

Also, I'm thinking about sending stats as a well structured message with
name value pairs, so you don't need to write custom scripts to query/store
counters, you can use syslog-ng machinery for that.
On Nov 17, 2013 7:49 PM, "Fekete Róbert" <frobert at balabit.hu> wrote:

> Hi,
>
> I am not sure I understand correctly how this feature will work. For
> example, I set the stats-lifetime() option to 60 seconds, and start
> syslog-ng. At that moment, all counters are 0.
> After some time, say, 70 seconds after start, I use the syslog-ng-ctl
> stats command. What will be the output?
> The number of messages received since the last time the counters were
> zeroed (that is, in the last 10 seconds)? Or the number of messages
> received in the last full cycle (0-60 seconds)? If the former, does the
> stats message indicate that the number apply to the last 10 seconds?
>
> Also, if I want to aggregate the statistics, do I have to use the
> stats-freq() option, then filter and parse the stats message (so I get the
> numbers for the last 60 minutes), or is it possible to get this info
> reliably using the syslog-ng-ctl-stats command somehow?
>
> Regards,
>
> Robert
>
> On Saturday, November 16, 2013 14:50 CET, Balazs Scheidler <
> bazsi at balabit.hu> wrote:
>
> > Hi,
> >
> > I'm about to integrate a stats-counter related change and prior to doing
> > that I'd like to solicit feedback.
> >
> > There are both user-visible changes and developer visible changes,
> > feedback on both is sought.
> >
> > Code related comments are preferred on github, functionality comments
> > are best sent via email in response to this one.
> >
> > Here's the branch:
> >
> > https://github.com/balabit/syslog-ng/tree/3.6queue/f/stats-reset
> >
> > Thanks in advance.
> >
> > User-visible changes:
> > =====================
> >
> > syslog-ng is keeping track on a number of statistics counters that can
> > be used to track what is happening.
> >
> > The detail level of these counters is controlled by the stats-level()
> > global option, which defaults to zero.
> >
> > syslog-ng reports these counters periodically every stats-freq()
> > seconds, or preferably on-demand by invoking "syslog-ng-ctl stats" on
> > the command line.
> >
> > At stats-level(3) a number of "dynamic" counters are enabled, which
> > track the number of messages on a per $HOST, per $PROGRAM and per
> > $HOST_FROM basis. This means that you can follow how many messages a
> > given $PROGRAM is generating in a period.
> >
> > The issue with dynamic counters is that they can consume a lot of memory
> > if  the rate of new $PRORGRAM/$HOST values is high.
> >
> > Previously the cleanup happened in a limited way at configuration reload
> > (e.g. SIGHUP) time, which is not really useful for controlling memory
> > usage by this feature.
> >
> > The current patch series introduces a stats-lifetime() option and
> > syslog-ng starts inspecting counters every once in a while and prune
> > those that are updated earlier than stats-lifetime().
> >
> > Not-so visible changes
> > ======================
> >
> > I have fixed a few bugs and renamed a function to make it easier to
> > understand. Details are in the git history.
> >
> >
> > Balazs Scheidler (7):
> >   logsource: use stats_instant_inc_dynamic_counter() for HOST counters
> >   logsource: track time for PROGRAM counters
> >   mainloop: remove old orphanage based stats cleanup
> >   stats: add stats_prune_old_counters()
> >   stats-timer: implement stats_lifetime option
> >   cfg: add stats-lifetime() global option
> >   stats: rename stats_instance_inc_dynamic_counter() to
> >     stats_register_and_increment_dynamic_counter()
> >
> >  lib/cfg-grammar.y | 16 ++++++++-------
> >  lib/cfg-parser.c  |  1 +
> >  lib/cfg.c         |  3 ++-
> >  lib/cfg.h         |  1 +
> >  lib/logsource.c   | 15 +++-----------
> >  lib/mainloop.c    |  1 -
> >  lib/stats-timer.c | 61
> +++++++++++++++++++++++++++++++++++++++----------------
> >  lib/stats-timer.h |  2 +-
> >  lib/stats.c       | 52 +++++++++++++++++++++++++++++++++++++++--------
> >  lib/stats.h       |  7 +++----
> >  10 files changed, 108 insertions(+), 51 deletions(-)
> >
> >
> > --
> > Bazsi
> >
>
>
>
>
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20131118/3fb1c524/attachment.htm