<p dir="ltr">Hi,</p>
<p dir="ltr">Counters are not zeroed. They get pruned iff they are not updated for stats-lifetime() seconds. And only the dynamic kind.</p>
<p dir="ltr">This means that you have to query them within the lifetime period to avoid losing their value.</p>
<p dir="ltr">Also, I'm thinking about sending stats as a well structured message with name value pairs, so you don't need to write custom scripts to query/store counters, you can use syslog-ng machinery for that.</p>
<div class="gmail_quote">On Nov 17, 2013 7:49 PM, "Fekete Róbert" <<a href="mailto:frobert@balabit.hu">frobert@balabit.hu</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi,<br>
<br>
I am not sure I understand correctly how this feature will work. For example, I set the stats-lifetime() option to 60 seconds, and start syslog-ng. At that moment, all counters are 0.<br>
After some time, say, 70 seconds after start, I use the syslog-ng-ctl stats command. What will be the output?<br>
The number of messages received since the last time the counters were zeroed (that is, in the last 10 seconds)? Or the number of messages received in the last full cycle (0-60 seconds)? If the former, does the stats message indicate that the number apply to the last 10 seconds?<br>
<br>
Also, if I want to aggregate the statistics, do I have to use the stats-freq() option, then filter and parse the stats message (so I get the numbers for the last 60 minutes), or is it possible to get this info reliably using the syslog-ng-ctl-stats command somehow?<br>
<br>
Regards,<br>
<br>
Robert<br>
<br>
On Saturday, November 16, 2013 14:50 CET, Balazs Scheidler <<a href="mailto:bazsi@balabit.hu">bazsi@balabit.hu</a>> wrote:<br>
<br>
> Hi,<br>
><br>
> I'm about to integrate a stats-counter related change and prior to doing<br>
> that I'd like to solicit feedback.<br>
><br>
> There are both user-visible changes and developer visible changes,<br>
> feedback on both is sought.<br>
><br>
> Code related comments are preferred on github, functionality comments<br>
> are best sent via email in response to this one.<br>
><br>
> Here's the branch:<br>
><br>
> <a href="https://github.com/balabit/syslog-ng/tree/3.6queue/f/stats-reset" target="_blank">https://github.com/balabit/syslog-ng/tree/3.6queue/f/stats-reset</a><br>
><br>
> Thanks in advance.<br>
><br>
> User-visible changes:<br>
> =====================<br>
><br>
> syslog-ng is keeping track on a number of statistics counters that can<br>
> be used to track what is happening.<br>
><br>
> The detail level of these counters is controlled by the stats-level()<br>
> global option, which defaults to zero.<br>
><br>
> syslog-ng reports these counters periodically every stats-freq()<br>
> seconds, or preferably on-demand by invoking "syslog-ng-ctl stats" on<br>
> the command line.<br>
><br>
> At stats-level(3) a number of "dynamic" counters are enabled, which<br>
> track the number of messages on a per $HOST, per $PROGRAM and per<br>
> $HOST_FROM basis. This means that you can follow how many messages a<br>
> given $PROGRAM is generating in a period.<br>
><br>
> The issue with dynamic counters is that they can consume a lot of memory<br>
> if the rate of new $PRORGRAM/$HOST values is high.<br>
><br>
> Previously the cleanup happened in a limited way at configuration reload<br>
> (e.g. SIGHUP) time, which is not really useful for controlling memory<br>
> usage by this feature.<br>
><br>
> The current patch series introduces a stats-lifetime() option and<br>
> syslog-ng starts inspecting counters every once in a while and prune<br>
> those that are updated earlier than stats-lifetime().<br>
><br>
> Not-so visible changes<br>
> ======================<br>
><br>
> I have fixed a few bugs and renamed a function to make it easier to<br>
> understand. Details are in the git history.<br>
><br>
><br>
> Balazs Scheidler (7):<br>
> logsource: use stats_instant_inc_dynamic_counter() for HOST counters<br>
> logsource: track time for PROGRAM counters<br>
> mainloop: remove old orphanage based stats cleanup<br>
> stats: add stats_prune_old_counters()<br>
> stats-timer: implement stats_lifetime option<br>
> cfg: add stats-lifetime() global option<br>
> stats: rename stats_instance_inc_dynamic_counter() to<br>
> stats_register_and_increment_dynamic_counter()<br>
><br>
> lib/cfg-grammar.y | 16 ++++++++-------<br>
> lib/cfg-parser.c | 1 +<br>
> lib/cfg.c | 3 ++-<br>
> lib/cfg.h | 1 +<br>
> lib/logsource.c | 15 +++-----------<br>
> lib/mainloop.c | 1 -<br>
> lib/stats-timer.c | 61 +++++++++++++++++++++++++++++++++++++++----------------<br>
> lib/stats-timer.h | 2 +-<br>
> lib/stats.c | 52 +++++++++++++++++++++++++++++++++++++++--------<br>
> lib/stats.h | 7 +++----<br>
> 10 files changed, 108 insertions(+), 51 deletions(-)<br>
><br>
><br>
> --<br>
> Bazsi<br>
><br>
<br>
<br>
<br>
<br>
<br>
<br>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote></div>