<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="generator" content="Osso Notes">
<title></title></head>
<body>
<p>hi,
<br>
<br>in that case it might make sense to add a parameter to control initial allocation size or create a heuristic to automatically adjust that.
<br>
<br>did you notice performance issues? how many messages are you processing? did the cpu usage of syslog-ng increase dramatically at one point as you were adding more and more name-value pairs?
<br>
<br>thanks for the info.
<br>
<br>----- Original message -----
<br>> Back in December there was discussion of the payload_reallocs statistic.
<br>>
<br>> <a href="https://lists.balabit.hu/pipermail/syslog-ng/2012-December/019842.html">https://lists.balabit.hu/pipermail/syslog-ng/2012-December/019842.html</a>
<br>>
<br>>
<br>>  >
<br>>  >> *global;payload_reallocs;;a;processed;760*
<br>>  >
<br>>  >this counts the number of reallocs of the message payload. syslog-ng
<br>> sizes the allocated buffer >with a simple heuristics in the hope that
<br>> parsing, rewrite rules will not cause it to grow. >in your case
<br>> syslog-ng had to do a realloc for 760 messages. if this happens to be
<br>> close to >all messages you processed, it's the cause for performance
<br>> degradation. > >if it's a minority then you probably don't have to care.
<br>>  >
<br>>  >if the first one is true, I'd like to know about it.
<br>>  >
<br>>  >right now the allocated size is twice the length of the incoming
<br>> message. >
<br>>
<br>>
<br>> Well, You wanted to know if this happens for nearly all of the messages
<br>> required realloc
<br>>
<br>> Syslog-ng OSE 3.3.7
<br>>
<br>> The d_archive destination receives all of our messages;
<br>>
<br>> global;payload_reallocs;;a;processed;61142004
<br>> destination;d_archive;;a;processed;31650382
<br>>
<br>> about 15 seconds later
<br>>
<br>> global;payload_reallocs;;a;processed;61197495
<br>> destination;d_archive;;a;processed;31680143
<br>>
<br>> This means that for
<br>>
<br>> # messages = 29761
<br>> # reallocs = 55491
<br>>
<br>> or approximately 2 reallocs for each message.
<br>>
<br>> We make heavy use of patternDB to apply meta data to messages,
<br>>
<br>>
<br>>
<br><br></p>
</body>
</html>