[syslog-ng] Statistics summary

[Anton Koldaev]

Balazs, I hope you can answer the following questions:
1. Is that formula correct?
*SUM(`syslog-ng-ctl stats | grep source | grep processed`) ==
SUM(`syslog-ng-ctl stats | grep source | grep processed`)*

If I'm not dropping any messages by 'flags(final)' without sending to any
destination and also I'm not sending the same logs to multiple destinations.

Where SUM - summarize messages count in each line.

2. I don't need 'dst.name...' and 'src.name...' to get the summ, right?
Destination = dst.name1 + dst.name2 + ..., is that correct?

3. What should I check if I see more 'source' processed messages than
'destination' and there are no dropped messages?

4. What other global things is it good to monitor using syslog stats?
Are there are any methods to see if syslog-ng has stopped reading the
source?

For now I'm going to monitor:
1. Message/per second rate - alert if it decreased significantly
2. The number of stored messages - alert if the queue is always full
(log_fifo_size)
3. The number of dropped messages
4. Timestamps of the latest message - alert if the timestamp is tool old.
... any other ideas?




On Fri, Dec 28, 2012 at 12:57 PM, Anton Koldaev <koldaevav at gmail.com> wrote:

> Thanks Balazs for the explanation.
> Looks like they all are not very important for me to monitor.
>
>
> On Fri, Dec 28, 2012 at 9:44 AM, Balazs Scheidler <bazsi77 at gmail.com>wrote:
>
>> **
>>
>> ----- Original message -----
>> > Thanks Daniel. I'm already using "syslog-ng-ctl stats" with different
>> > stats levels. I'm just confused with "center()":
>> > *% sudo syslog-ng-ctl stats | grep center
>> >
>> >                          *
>> > *center;;received;a;processed;0*
>> > *center;;queued;a;processed;0*
>>
>> center has been broken since 3.2, because there's no 'center' anymore.
>> IIRC I have removed that counter since then. If there's a usecase for
>> center I may give a thought how to revive it.
>>
>> > *
>> > *
>> > Also maybe balabit guys are able to give us some information on global
>> > stats:
>> > *% sudo syslog-ng-ctl stats | grep global *
>> > *global;sdata_updates;;a;processed;0*
>>
>> this one counts the number of 'slow' updates to structured data entries
>> during processing. being zero means that you basically never add sdata, but
>> either not change it at all, or only change entries that are already
>> present.
>>
>> > *global;payload_reallocs;;a;processed;760*
>>
>> this counts the number of reallocs of the message payload. syslog-ng
>> sizes the allocated buffer with a simple heuristics in the hope that
>> parsing, rewrite rules will not cause it to grow. in your case syslog-ng
>> had to do a realloc for 760 messages. if this happens to be close to all
>> messages you processed, it's the cause for performance degradation.
>>
>> if it's a minority then you probably don't have to care.
>>
>> if the first one is true, I'd like to know about it.
>>
>> right now the alllocated size is twice the length of the incoming
>> message.
>>
>>
>> > *global;msg_clones;;a;processed;0*
>> >
>>
>> again a performance monitoring counter, that tracks the number of clone
>> operations.
>>
>> >
>> > On Thu, Dec 27, 2012 at 6:07 PM, Daniel Neubacher
>> > <daniel.neubacher at xing.com
>> > > wrote:
>> >
>> > > I guess u need „syslog-ng-ctl stats”.  But u need to configure
>> > > “stats_level(2);” in your options to get this running nicely.****
>> > >
>> > > And if you want to get logs per second I’ve attached my little bash
>> > > script: ****
>> > >
>> > > #!/bin/bash****
>> > >
>> > > ** **
>> > >
>> > > while true****
>> > >
>> > > do ****
>> > >
>> > > for i in $(syslog-ng-ctl stats | grep src.tcp | grep proc |
>> > > cut -d ";" -f6) ****
>> > >
>> > > do ****
>> > >
>> > > let tc1+=$i ****
>> > >
>> > > done ****
>> > >
>> > > let lps=tc1-tc2****
>> > >
>> > > test -z $tc2 || echo $lps****
>> > >
>> > > tc2=$tc1****
>> > >
>> > > tc1=0****
>> > >
>> > > sleep 1****
>> > >
>> > > done****
>> > >
>> > > ** **
>> > >
>> > > *Von:* syslog-ng-bounces at lists.balabit.hu [mailto:
>> > > syslog-ng-bounces at lists.balabit.hu] *Im Auftrag von *Anton Koldaev
>> > > *Gesendet:* Donnerstag, 27. Dezember 2012 10:18
>> > > *An:* Syslog-ng users' and developers' mailing list
>> > > *Betreff:* [syslog-ng] Statistics summary****
>> > >
>> > > ** **
>> > >
>> > > Is there a nice way to get total numbers of processed/stored/dropped
>> > > messages in syslog-ng v3.3?****
>> > >
>> > > ** **
>> > >
>> > > From docs:****
>> > >
>> > > > The *center(received)* entry shows the total number of messages
>> > > received from every configured sources.
>> > > ****
>> > >
>> > > ** **
>> > >
>> > > But in my stats center() shows zero almost for every host. I see zero
>> > > in some other examples in docs too.****
>> > >
>> > > ** **
>> > >
>> > > --
>> > > Best regards,
>> > > Koldaev Anton ****
>> > >
>> > >
>> > >
>> ______________________________________________________________________________
>>
>> > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> > > Documentation:
>> > > http://www.balabit.com/support/documentation/?product=syslog-ng
>> > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>> > >
>> > >
>> > >
>> >
>> >
>> > --
>> > Best regards,
>> > Koldaev Anton
>>
>>
>
>
> --
> Best regards,
> Koldaev Anton
>



-- 
Best regards,
Koldaev Anton
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20121230/23afb5a8/attachment.htm