[syslog-ng] Reload does not free up RAM
erempel at uvic.ca
Mon Oct 6 20:38:20 CEST 2008
Balazs Scheidler wrote:
> On Sun, 2008-10-05 at 13:49 -0700, Evan Rempel wrote:
>> Balazs Scheidler wrote:
>>>>>>>> We have cases where a syslog-ng buffers quite a bit of data (2GB or so) before
>>>>>>>> it is able to flush the data to the destination. Currently, the only way to "free up" that RAM
>>>>>>>> is to restart syslog-ng, which means that some log messages are lost during the restart
>>> Both PE and OSE frees up disk/memory when possible. If they don't, then
>>> that's probably a leak.
No, it isn't a memory leak as the process can be months old and only grows when
it buffers messages.
>>> So if your syslog-ng process is 2GB in size, and you didn't specify an
>>> overly large log_fifo_size(), it's probably because of a leak.
>> I have a large log_fifo_size() but the process starts small, grows when it needs
>> to buffer, and does NOT get small again when the destination is available.
> Hmm... that might be related to the malloc implementation. Which
> platform are you using?
>>>> 2. The buffer space should be thrown away only after the message
>>>> has been delivered to the destination.
>>> I see, this is what should happen now.
>> This does not appear to be happening.
>> I am using OSE 2.0.8 Have there been memory fixes/changes since then?
> Not that I know of.
>> Is there a way to get buffer count statistics out? How many messages are
>> in each fifo buffer?
> In the experimental 3.0 branch, yes there is. In current 2.0, I'm afraid
> there's none.
> Is there a limit on the growth of syslog-ng? I mean if it gets large
> once and then empties its buffers, it stays large, but does it start
> growing again, or it stays steadily at the size it was before?
I have not specifically tested this scenario, but we do track processes
and memory usage, and it appears that once the space is allocated, it does
not grow unless a larger set of messages gets buffered.
> Again, the platform matters a lot, how the local malloc implementation
> takes care about fragmentation, whether it shrinks the process once
> memory is freed.
This is running on Redhat AS 3 (2.4 kernel) and Redhat AS 4 (2.6 kernel)
On the Redhat 4 system the glibc packages installed are
> The problem is, that if the local malloc implementation is wrong the
> only thing I could do is to try to use a different allocator (e.g. use
> GNU malloc on your platform), but that's not a simple change.
More information about the syslog-ng