<div dir="ltr"><span style="font-family:arial,sans-serif;font-size:13px">>> *global;payload_reallocs;;a;</span><span style="font-family:arial,sans-serif;font-size:13px">processed;760* </span><br style="font-family:arial,sans-serif;font-size:13px">
<br style="font-family:arial,sans-serif;font-size:13px"><span style="font-family:arial,sans-serif;font-size:13px">> this counts the number of reallocs of the message payload. syslog-ng sizes the allocated buffer with a simple heuristics in the hope that parsing, rewrite rules will not cause it to grow. in your case syslog-ng had to do a realloc for 760 messages. if this happens to be close to all messages you processed, it's the cause for performance degradation. </span><br style="font-family:arial,sans-serif;font-size:13px">
<br><span style="font-family:arial,sans-serif;font-size:13px">> if the first one is true, I'd like to know about it. </span><br><div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div style>
<span style="font-family:arial,sans-serif;font-size:13px">At one of my clients hosts(I don't have stats monitoring for all of them yet) I have about 45 msg/sec sent by syslog-ng client to syslog-ng server and about 4msg/sec reallocated. Does it sound like a problem?</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Sun, Dec 30, 2012 at 11:31 PM, Anton Koldaev <span dir="ltr"><<a href="mailto:koldaevav@gmail.com" target="_blank">koldaevav@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Balazs, I hope you can answer the following questions:<div>1. Is that formula correct?<br><div><b>SUM(`syslog-ng-ctl stats | grep source | grep processed`) == SUM(`syslog-ng-ctl stats | grep source | grep processed`)</b></div>
<div><br></div><div>If I'm not dropping any messages by 'flags(final)' without sending to any destination and also I'm not sending the same logs to multiple destinations.</div><div><br></div>
<div>Where SUM - summarize messages count in each line.</div><div><br></div><div>2. I don't need 'dst.name...' and 'src.name...' to get the summ, right?</div><div>Destination = dst.name1 + dst.name2 + ..., is that correct?</div>
<div><br></div><div>3. What should I check if I see more 'source' processed messages than 'destination' and there are no dropped messages?</div><div><br></div><div>4. What other global things is it good to monitor using syslog stats?</div>
<div>Are there are any methods to see if syslog-ng has stopped reading the source?</div><div><br></div><div>For now I'm going to monitor:</div><div>1. Message/per second rate - alert if it decreased significantly</div>
<div>2. The number of stored messages - alert if the queue is always full (log_fifo_size)</div><div>3. The number of dropped messages</div><div>4. Timestamps of the latest message - alert if the timestamp is tool old.</div>
<div>... any other ideas?</div></div><div><br></div><div><br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Dec 28, 2012 at 12:57 PM, Anton Koldaev <span dir="ltr"><<a href="mailto:koldaevav@gmail.com" target="_blank">koldaevav@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thanks Balazs for the explanation.<div>Looks like they all are not very important for me to monitor.</div>
</div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Dec 28, 2012 at 9:44 AM, Balazs Scheidler <span dir="ltr"><<a href="mailto:bazsi77@gmail.com" target="_blank">bazsi77@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><u></u>
<div>
<p></p><div>----- Original message -----
<br>> Thanks Daniel. I'm already using "syslog-ng-ctl stats" with different
<br>> stats levels. I'm just confused with "center()":
<br></div>> *% sudo syslog-ng-ctl stats | grep center
<br>>
<br>> *
<br>> *center;;received;a;processed;0*
<br>> *center;;queued;a;processed;0*
<br>
<br>center has been broken since 3.2, because there's no 'center' anymore. IIRC I have removed that counter since then. If there's a usecase for center I may give a thought how to revive it.
<br>
<br>> *
<br>> *
<br><div>> Also maybe balabit guys are able to give us some information on global
<br>> stats:
<br></div>> *% sudo syslog-ng-ctl stats | grep global *
<br>> *global;sdata_updates;;a;processed;0*
<br>
<br>this one counts the number of 'slow' updates to structured data entries during processing. being zero means that you basically never add sdata, but either not change it at all, or only change entries that are already present.
<br>
<br>> *global;payload_reallocs;;a;processed;760*
<br>
<br>this counts the number of reallocs of the message payload. syslog-ng sizes the allocated buffer with a simple heuristics in the hope that parsing, rewrite rules will not cause it to grow. in your case syslog-ng had to do a realloc for 760 messages. if this happens to be close to all messages you processed, it's the cause for performance degradation.
<br>
<br>if it's a minority then you probably don't have to care.
<br>
<br>if the first one is true, I'd like to know about it.
<br>
<br>right now the alllocated size is twice the length of the incoming message.
<br>
<br>
<br>> *global;msg_clones;;a;processed;0*
<br>>
<br>
<br>again a performance monitoring counter, that tracks the number of clone operations.
<br><div>
<br>>
<br>> On Thu, Dec 27, 2012 at 6:07 PM, Daniel Neubacher
<br>> <<a href="mailto:daniel.neubacher@xing.com" target="_blank">daniel.neubacher@xing.com</a>
<br>> > wrote:
<br>>
<br>> > I guess u need „syslog-ng-ctl stats”. But u need to configure
<br></div>> > “stats_level(2);” in your options to get this running nicely.****
<br><div>> >
<br>> > And if you want to get logs per second I’ve attached my little bash
<br></div>> > script: ****
<br>> >
<br>> > #!/bin/bash****
<br>> >
<br>> > ** **
<br>> >
<br>> > while true****
<br>> >
<br>> > do ****
<br><div>> >
<br>> > for i in $(syslog-ng-ctl stats | grep src.tcp | grep proc |
<br></div>> > cut -d ";" -f6) ****
<br>> >
<br>> > do ****
<br>> >
<br>> > let tc1+=$i ****
<br>> >
<br>> > done ****
<br>> >
<br>> > let lps=tc1-tc2****
<br>> >
<br>> > test -z $tc2 || echo $lps****
<br>> >
<br>> > tc2=$tc1****
<br>> >
<br>> > tc1=0****
<br>> >
<br>> > sleep 1****
<br>> >
<br>> > done****
<br>> >
<br>> > ** **
<br>> >
<br>> > *Von:* <a href="mailto:syslog-ng-bounces@lists.balabit.hu" target="_blank">syslog-ng-bounces@lists.balabit.hu</a> [mailto:
<br>> > <a href="mailto:syslog-ng-bounces@lists.balabit.hu" target="_blank">syslog-ng-bounces@lists.balabit.hu</a>] *Im Auftrag von *Anton Koldaev
<br>> > *Gesendet:* Donnerstag, 27. Dezember 2012 10:18
<br>> > *An:* Syslog-ng users' and developers' mailing list
<br>> > *Betreff:* [syslog-ng] Statistics summary****
<br>> >
<br>> > ** **
<br><div>> >
<br>> > Is there a nice way to get total numbers of processed/stored/dropped
<br></div>> > messages in syslog-ng v3.3?****
<br>> >
<br>> > ** **
<br>> >
<br>> > From docs:****
<br>> >
<br>> > > The *center(received)* entry shows the total number of messages
<br><div>> > received from every configured sources.
<br></div>> > ****
<br>> >
<br>> > ** **
<br><div>> >
<br>> > But in my stats center() shows zero almost for every host. I see zero
<br></div>> > in some other examples in docs too.****
<br>> >
<br>> > ** **
<br>> >
<br>> > --
<br>> > Best regards,
<br>> > Koldaev Anton ****
<br><div>> >
<br>> >
<br>> > ______________________________________________________________________________
<br>> > Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a>
<br>> > Documentation:
<br>> > <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a>
<br>> > FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a>
<br>> >
<br>> >
<br>> >
<br>>
<br>>
<br>> --
<br>> Best regards,
<br>> Koldaev Anton
<br><br></div><p></p>
</div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>Best regards,<br>Koldaev Anton
</div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>Best regards,<br>Koldaev Anton
</div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>Best regards,<br>Koldaev Anton
</div>