[syslog-ng] payload_reallocs twice the count number of messages.

[Balazs Scheidler]

hi,

in that case it might make sense to add a parameter to control initial allocation size or create a heuristic to automatically adjust that.

did you notice performance issues? how many messages are you processing? did the cpu usage of syslog-ng increase dramatically at one point as you were adding more and more name-value pairs?

thanks for the info.

----- Original message -----
> Back in December there was discussion of the payload_reallocs statistic.
> 
> https://lists.balabit.hu/pipermail/syslog-ng/2012-December/019842.html
> 
> 
>   >
>   >> *global;payload_reallocs;;a;processed;760*
>   >
>   >this counts the number of reallocs of the message payload. syslog-ng
> sizes the allocated buffer >with a simple heuristics in the hope that
> parsing, rewrite rules will not cause it to grow. >in your case
> syslog-ng had to do a realloc for 760 messages. if this happens to be
> close to >all messages you processed, it's the cause for performance
> degradation. > >if it's a minority then you probably don't have to care.
>   >
>   >if the first one is true, I'd like to know about it.
>   >
>   >right now the allocated size is twice the length of the incoming
> message. >
> 
> 
> Well, You wanted to know if this happens for nearly all of the messages
> required realloc
> 
> Syslog-ng OSE 3.3.7
> 
> The d_archive destination receives all of our messages;
> 
> global;payload_reallocs;;a;processed;61142004
> destination;d_archive;;a;processed;31650382
> 
> about 15 seconds later
> 
> global;payload_reallocs;;a;processed;61197495
> destination;d_archive;;a;processed;31680143
> 
> This means that for
> 
> # messages = 29761
> # reallocs = 55491
> 
> or approximately 2 reallocs for each message.
> 
> We make heavy use of patternDB to apply meta data to messages,
> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20130207/bb7c9574/attachment.htm