hi, in that case it might make sense to add a parameter to control initial allocation size or create a heuristic to automatically adjust that. did you notice performance issues? how many messages are you processing? did the cpu usage of syslog-ng increase dramatically at one point as you were adding more and more name-value pairs? thanks for the info. ----- Original message -----
Back in December there was discussion of the payload_reallocs statistic.
https://lists.balabit.hu/pipermail/syslog-ng/2012-December/019842.html
> >> *global;payload_reallocs;;a;processed;760* > >this counts the number of reallocs of the message payload. syslog-ng sizes the allocated buffer >with a simple heuristics in the hope that parsing, rewrite rules will not cause it to grow. >in your case syslog-ng had to do a realloc for 760 messages. if this happens to be close to >all messages you processed, it's the cause for performance degradation. > >if it's a minority then you probably don't have to care. > >if the first one is true, I'd like to know about it. > >right now the allocated size is twice the length of the incoming message. >
Well, You wanted to know if this happens for nearly all of the messages required realloc
Syslog-ng OSE 3.3.7
The d_archive destination receives all of our messages;
global;payload_reallocs;;a;processed;61142004 destination;d_archive;;a;processed;31650382
about 15 seconds later
global;payload_reallocs;;a;processed;61197495 destination;d_archive;;a;processed;31680143
This means that for
# messages = 29761 # reallocs = 55491
or approximately 2 reallocs for each message.
We make heavy use of patternDB to apply meta data to messages,