[syslog-ng] Reload does not free up RAM

[Evan Rempel]

Balazs Scheidler wrote:
> On Sun, 2008-10-05 at 13:49 -0700, Evan Rempel wrote:
>> Balazs Scheidler wrote:
>>>>>>>> We have cases where a syslog-ng buffers quite a bit of data (2GB or so) before
>>>>>>>> it is able to flush the data to the destination. Currently, the only way to "free up" that RAM
>>>>>>>> is to restart syslog-ng, which means that some log messages are lost during the restart
>>>>>>>> window.
>>> Both PE and OSE frees up disk/memory when possible. If they don't, then
>>> that's probably a leak.

No, it isn't a memory leak as the process can be months old and only grows when
it buffers messages.

>>>
>>> So if your syslog-ng process is 2GB in size, and you didn't specify an
>>> overly large log_fifo_size(), it's probably because of a leak.
 >>
>> I have a large log_fifo_size() but the process starts small, grows when it needs
>> to buffer, and does NOT get small again when the destination is available.
> 
> Hmm... that might be related to the malloc implementation. Which
> platform are you using?
> 
>>>> 2. The buffer space should be thrown away only after the message
>>>>     has been delivered to the destination.
>>> I see, this is what should happen now.
 >>
>> This does not appear to be happening.
>> I am using OSE 2.0.8 Have there been memory fixes/changes since then?
> 
> Not that I know of.
> 
>> Is there a way to get buffer count statistics out? How many messages are
>> in each fifo buffer?
> 
> In the experimental 3.0 branch, yes there is. In current 2.0, I'm afraid
> there's none.
> 
> Is there a limit on the growth of syslog-ng? I mean if it gets large
> once and then empties its buffers, it stays large, but does it start
> growing again, or it stays steadily at the size it was before?

I have not specifically tested this scenario, but we do track processes
and memory usage, and it appears that once the space is allocated, it does
not grow unless a larger set of messages gets buffered.


> Again, the platform matters a lot, how the local malloc implementation
> takes care about fragmentation, whether it shrinks the process once
> memory is freed.

This is running on Redhat AS 3 (2.4 kernel) and Redhat AS 4 (2.6 kernel)

On the Redhat 4 system the glibc packages installed are

glibc-common-2.3.4-2.36
glibc-devel-2.3.4-2.36
glibc-headers-2.3.4-2.36
glib2-2.4.7-1
glibc-kernheaders-2.4-9.1.100.EL


> The problem is, that if the local malloc implementation is wrong the
> only thing I could do is to try to use a different allocator (e.g. use
> GNU malloc on your platform), but that's not a simple change.

Understood.

-- 
Evan Rempel