[syslog-ng] syslog-ng stats

[Ioan Indreias]

Hello Bazsi,

Thank a lot for the info provided - I was happy that no dropped
messages are reported - however I am now a bit worried about the
payload_reallocs values.

Could we do something to lower it?

We have started a monitor script (in Cacti) and log the number of
processed messages (received from the network) - the average for the
last 3 days is 700 msg/sec with some peaks near 1000. Because I am
expecting that these numbers will slowly increase I have started to
check the STATS values.

Just to complete my report, the average number of MySQL questions is
near 150 qps. We are using 2 templates to route the MySQL queries to a
pipe file and a "consumer" script which actually run the queries.
Probably there are some rooms to optimize the configuration used, this
list being a valuable source of information for us.

Best regards,
Ioan

On Sun, Mar 20, 2011 at 8:32 PM, Balazs Scheidler <bazsi at balabit.hu> wrote:
> On Fri, 2011-03-18 at 15:17 +0200, Ioan Indreias wrote:
>> Hello,
>>
>> Running with syslog-ng OSE 3.2.1 and wondering what exactly mean some
>> of the parameters by STATS.
>>
>> Please find below an extract of the output we get (with the default
>> stats_level) , hoping to receive some hints from you.
>>
>> global > payload_reallocs = ?
>
> The number of times the payload structure of a LogMessage needed to be
> reallocated. A high number of this probably means that you have a lot of
> name-value pairs, and the preallocation is not sufficient for your
> setup.
>
> It is not a big issue, however it causes the performance to be dropped
> somewhat.
>
> The heuristics to allocate a payload is: the length of the incoming
> message mulitplied by 2, and this is where all name-value pairs (MSG,
> PID, PROGRAM, and user-defined ones too) need to be stored. Each
> name-value pair has some overhead too.
>
> Basically if this is number is large, or even if it's non-zero, I'd like
> to know about it.
>
>
>> global > msg_clones = ?
>
> The number of times a message needed to be cloned. messages need to be
> cloned in case a log path is changing the message (with a parser or
> rewrite rule for instance), since syslog-ng ensures that parallel
> branches of the log processing tree (root is the source, branches
> described by the log statements, forks is when the same message needs to
> go multiple directions) receives the same message.
>
>> global > sdata_updates = ?
>
> Counts the number of times a new structured data member was added to a
> message, causing the SDATA array to be updated. This is a slower
> operation, therefore it is counted to make it possible to diagnose
> performance issues.
>
>> why center > received is 0 ? From OSE 3.2 manual, Chapter 14, it
>> should be the "total number of messages received from every configured
>> sources."
>
> This is a bug, that I never got around to fixing yet.
>
>>
>> BR,
>> Ioan
>>
>> ###
>> [root at myhost TOOLS]# /opt/syslog-ng/sbin/syslog-ng-ctl stats
>> SourceName;SourceId;SourceInstance;State;Type;Number
>> global;payload_reallocs;;a;processed;1643977
>> source;s_net;;a;processed;615301996
>> global;msg_clones;;a;processed;1230603992
>> destination;d_mysql1;;a;processed;247656
>> destination;d_mysql2;;a;processed;1354498
>> global;sdata_updates;;a;processed;0
>> center;;received;a;processed;0
>> center;;queued;a;processed;0
>> ###
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.campin.net/syslog-ng/faq.html
>>
>>
>
> --
> Bazsi
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>