<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="generator" content="Osso Notes">
<title></title></head>
<body>
<p>----- Original message -----
<br>> > > *global;payload_reallocs;;a;processed;760*
<br>>
<br>> > this counts the number of reallocs of the message payload. syslog-ng
<br>> sizes the allocated buffer with a simple heuristics in the hope that
<br>> parsing, rewrite rules will not cause it to grow. in your case syslog-ng
<br>> had to do a realloc for 760 messages. if this happens to be close to all
<br>> messages you processed, it's the cause for performance degradation.
<br>>
<br>> > if the first one is true, I'd like to know about it.
<br>>
<br>> At one of my clients hosts(I don't have stats monitoring for all of them
<br>> yet) I have about 45 msg/sec sent by syslog-ng client to syslog-ng server
<br>> and about 4msg/sec reallocated. Does it sound like a problem?
<br>>
<br>
<br>not immediate problem, I just would like to know your use-case. what are these messages and what you do to them.
<br>
<br>I was thinking about a more complicated heuristic when I implemented this code, but then I decided to defer them until I saw actual uses where something better is needed
<br>
<br>>
<br>>
<br>> On Sun, Dec 30, 2012 at 11:31 PM, Anton Koldaev <<a href="mailto:koldaevav@gmail.com">koldaevav@gmail.com</a>>
<br>> wrote:
<br>>
<br>> > Balazs, I hope you can answer the following questions:
<br>> > 1. Is that formula correct?
<br>> > *SUM(`syslog-ng-ctl stats | grep source | grep processed`) ==
<br>> > SUM(`syslog-ng-ctl stats | grep source | grep processed`)*
<br>> >
<br>> > If I'm not dropping any messages by 'flags(final)' without sending to
<br>> > any destination and also I'm not sending the same logs to multiple
<br>> > destinations.
<br>> >
<br>> > Where SUM - summarize messages count in each line.
<br>> >
<br>> > 2. I don't need 'dst.name...' and 'src.name...' to get the summ, right?
<br>> > Destination = dst.name1 + dst.name2 + ..., is that correct?
<br>> >
<br>> > 3. What should I check if I see more 'source' processed messages than
<br>> > 'destination' and there are no dropped messages?
<br>> >
<br>> > 4. What other global things is it good to monitor using syslog stats?
<br>> > Are there are any methods to see if syslog-ng has stopped reading the
<br>> > source?
<br>> >
<br>> > For now I'm going to monitor:
<br>> > 1. Message/per second rate - alert if it decreased significantly
<br>> > 2. The number of stored messages - alert if the queue is always full
<br>> > (log_fifo_size)
<br>> > 3. The number of dropped messages
<br>> > 4. Timestamps of the latest message - alert if the timestamp is tool
<br>> > old. ... any other ideas?
<br>> >
<br>> >
<br>> >
<br>> >
<br>> > On Fri, Dec 28, 2012 at 12:57 PM, Anton Koldaev
<br>> > <<a href="mailto:koldaevav@gmail.com">koldaevav@gmail.com</a>>wrote:
<br>> >
<br>> > > Thanks Balazs for the explanation.
<br>> > > Looks like they all are not very important for me to monitor.
<br>> > >
<br>> > >
<br>> > > On Fri, Dec 28, 2012 at 9:44 AM, Balazs Scheidler
<br>> > > <<a href="mailto:bazsi77@gmail.com">bazsi77@gmail.com</a>>wrote:
<br>> > >
<br>> > > > **
<br>> > > >
<br>> > > > ----- Original message -----
<br>> > > > > Thanks Daniel. I'm already using "syslog-ng-ctl stats" with
<br>> > > > > different stats levels. I'm just confused with "center()":
<br>> > > > > *% sudo syslog-ng-ctl stats | grep center
<br>> > > > >
<br>> > > > > *
<br>> > > > > *center;;received;a;processed;0*
<br>> > > > > *center;;queued;a;processed;0*
<br>> > > >
<br>> > > > center has been broken since 3.2, because there's no 'center'
<br>> > > > anymore. IIRC I have removed that counter since then. If there's a
<br>> > > > usecase for center I may give a thought how to revive it.
<br>> > > >
<br>> > > > > *
<br>> > > > > *
<br>> > > > > Also maybe balabit guys are able to give us some information on
<br>> > > > > global stats:
<br>> > > > > *% sudo syslog-ng-ctl stats | grep global *
<br>> > > > > *global;sdata_updates;;a;processed;0*
<br>> > > >
<br>> > > > this one counts the number of 'slow' updates to structured data
<br>> > > > entries during processing. being zero means that you basically
<br>> > > > never add sdata, but either not change it at all, or only change
<br>> > > > entries that are already present.
<br>> > > >
<br>> > > > > *global;payload_reallocs;;a;processed;760*
<br>> > > >
<br>> > > > this counts the number of reallocs of the message payload.
<br>> > > > syslog-ng sizes the allocated buffer with a simple heuristics in
<br>> > > > the hope that parsing, rewrite rules will not cause it to grow. in
<br>> > > > your case syslog-ng had to do a realloc for 760 messages. if this
<br>> > > > happens to be close to all messages you processed, it's the cause
<br>> > > > for performance degradation.
<br>> > > >
<br>> > > > if it's a minority then you probably don't have to care.
<br>> > > >
<br>> > > > if the first one is true, I'd like to know about it.
<br>> > > >
<br>> > > > right now the alllocated size is twice the length of the incoming
<br>> > > > message.
<br>> > > >
<br>> > > >
<br>> > > > > *global;msg_clones;;a;processed;0*
<br>> > > > >
<br>> > > >
<br>> > > > again a performance monitoring counter, that tracks the number of
<br>> > > > clone operations.
<br>> > > >
<br>> > > > >
<br>> > > > > On Thu, Dec 27, 2012 at 6:07 PM, Daniel Neubacher
<br>> > > > > <<a href="mailto:daniel.neubacher@xing.com">daniel.neubacher@xing.com</a>
<br>> > > > > > wrote:
<br>> > > > >
<br>> > > > > > I guess u need „syslog-ng-ctl stats”.  But u need to configure
<br>> > > > > > “stats_level(2);” in your options to get this running
<br>> > > > > > nicely.****
<br>> > > > > >
<br>> > > > > > And if you want to get logs per second I’ve attached my little
<br>> > > > > > bash script: ****
<br>> > > > > >
<br>> > > > > > #!/bin/bash****
<br>> > > > > >
<br>> > > > > > ** **
<br>> > > > > >
<br>> > > > > > while true****
<br>> > > > > >
<br>> > > > > > do ****
<br>> > > > > >
<br>> > > > > > for i in $(syslog-ng-ctl stats | grep src.tcp | grep proc |
<br>> > > > > > cut -d ";" -f6) ****
<br>> > > > > >
<br>> > > > > > do ****
<br>> > > > > >
<br>> > > > > > let tc1+=$i ****
<br>> > > > > >
<br>> > > > > > done ****
<br>> > > > > >
<br>> > > > > > let lps=tc1-tc2****
<br>> > > > > >
<br>> > > > > > test -z $tc2 || echo $lps****
<br>> > > > > >
<br>> > > > > > tc2=$tc1****
<br>> > > > > >
<br>> > > > > > tc1=0****
<br>> > > > > >
<br>> > > > > > sleep 1****
<br>> > > > > >
<br>> > > > > > done****
<br>> > > > > >
<br>> > > > > > ** **
<br>> > > > > >
<br>> > > > > > *Von:* <a href="mailto:syslog-ng-bounces@lists.balabit.hu">syslog-ng-bounces@lists.balabit.hu</a> [mailto:
<br>> > > > > > <a href="mailto:syslog-ng-bounces@lists.balabit.hu">syslog-ng-bounces@lists.balabit.hu</a>] *Im Auftrag von *Anton
<br>> > > > > > Koldaev *Gesendet:* Donnerstag, 27. Dezember 2012 10:18
<br>> > > > > > *An:* Syslog-ng users' and developers' mailing list
<br>> > > > > > *Betreff:* [syslog-ng] Statistics summary****
<br>> > > > > >
<br>> > > > > > ** **
<br>> > > > > >
<br>> > > > > > Is there a nice way to get total numbers of
<br>> > > > > > processed/stored/dropped messages in syslog-ng v3.3?****
<br>> > > > > >
<br>> > > > > > ** **
<br>> > > > > >
<br>> > > > > > From docs:****
<br>> > > > > >
<br>> > > > > > > The *center(received)* entry shows the total number of
<br>> > > > > > > messages
<br>> > > > > > received from every configured sources.
<br>> > > > > > ****
<br>> > > > > >
<br>> > > > > > ** **
<br>> > > > > >
<br>> > > > > > But in my stats center() shows zero almost for every host. I
<br>> > > > > > see
<br>> > > > zero
<br>> > > > > > in some other examples in docs too.****
<br>> > > > > >
<br>> > > > > > ** **
<br>> > > > > >
<br>> > > > > > --
<br>> > > > > > Best regards,
<br>> > > > > > Koldaev Anton ****
<br>> > > > > >
<br>> > > > > >
<br>> > > > > >
<br>> > > > ______________________________________________________________________________
<br>> > > >
<br>> > > > > > Member info:
<br>> > > > > > <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a>
<br>> > > > > > Documentation:
<br>> > > > > > <a href="http://www.balabit.com/support/documentation/?product=syslog-ng">http://www.balabit.com/support/documentation/?product=syslog-ng</a>
<br>> > > > > > FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq">http://www.balabit.com/wiki/syslog-ng-faq</a>
<br>> > > > > >
<br>> > > > > >
<br>> > > > > >
<br>> > > > >
<br>> > > > >
<br>> > > > > --
<br>> > > > > Best regards,
<br>> > > > > Koldaev Anton
<br>> > > >
<br>> > > >
<br>> > >
<br>> > >
<br>> > > --
<br>> > > Best regards,
<br>> > > Koldaev Anton
<br>> > >
<br>> >
<br>> >
<br>> >
<br>> > --
<br>> > Best regards,
<br>> > Koldaev Anton
<br>> >
<br>>
<br>>
<br>>
<br>> --
<br>> Best regards,
<br>> Koldaev Anton
<br><br></p>
</body>
</html>