[syslog-ng]Re: syslog-ng's memory footprint growing constantly

[Craig J Constantine]

...on 7/27/01 1:02 AM, the following was received from 
syslog-ng-admin@lists.balabit.hu

>Date: Thu, 26 Jul 2001 10:25:08 +0200
>From: Balazs Scheidler <bazsi@balabit.hu>
>To: syslog-ng@lists.balabit.hu
>Subject: Re: syslog-ng's memory footprint growing constantly
>
>On Wed, Jul 25, 2001 at 04:17:06PM -0400, Craig J Constantine wrote:
>> Hi,
>>   Basically, the memory size of the process simply grows and grows:
>> USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
>> root     13899  0.1 19.3 25412 24764 ?       S    Jul24   2:05 syslog-ng
>> That was snapped after about 4 hours of run time. If I let it run for 
>> days... well, it once got to 90000 VSZ before the box choked for lack of 
>> RAM. For gauging how heavily worked syslog-ng is, the line counts for all 
>> the files it writes (this is one day of logs) are:
[rest of my original, snipped]
>
>There have been memory leaks, but they were fixed in 1.4.8. To help me to
>debug your problem, please compile syslog-ng _AND_ libol with --with-debug
>enabled (configure time option), and run syslog-ng with --debug --verbose
>switches, and save the resulting output. I'm mainly interested in lines
>starting with "Objects alive ... garbage collected...".
>
>If the number after "objects alive" increases then there's some spurious
>link within the data structure (the gc frees only unlinked objects), if it
>doesn't it is some kind of string data (for example messages).

I recompiled libol/syslog-ng from fresh tarballs using the configure 
flag. Then I fired it up with runtime debug arguments and captured 
everything to a file. I started the run around 9am, and left it go for a 
good half an hour.

The resulting file was, well, truly huge. 8^) Let me know if you'd like 
to see it, I can drop it on a server where you can grab it via anon FTP. 
Likewise if you want to see my config file again.

...around 9:07 syslog-ng was:
USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
root     30271  0.3  0.7  1652 1000 pts/1    S    08:54   0:02 
./syslog-ng.debug
(ie, tiny :)
The "Objects alive" was at 293.

...around 9:37 syslog-ng was:
USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
root     30271  0.3  1.1  2104 1452 pts/1    S    08:54   0:09 
./syslog-ng.debug
The "Objects alive" was at 299.

...around 10:07:
USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
root     30271  0.3  1.4  2552 1900 pts/1    S    08:54   0:16 
./syslog-ng.debug
The "Objects alive" was at 299.

...around 10:37:
USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
root     30271  0.4  1.9  3124 2472 pts/1    S    08:54   0:24 
./syslog-ng.debug
The "Objects alive" was still at 299.

...around 11:07:
USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
root     30271  0.4  2.3  3720 3068 pts/1    S    08:54   0:35 
./syslog-ng.debug
The "Objects alive" was still at 299.

...looks like the objects count is stable, but the memory size is still 
growing. I sent it a SIGQUIT and scooped up the core file. It looks like 
a large portion (more than half) of the image is memory that held plain 
text messages at some point. Notably the string piece "hosts.org_dir" 
appears often enough that I visually noticed it.

Thoughts? If you want to see the core file, I can make it available via 
anon FTP as well.

-Craig