[syslog-ng] Reload does not free up RAM

[Balazs Scheidler]

On Sun, 2008-10-05 at 13:49 -0700, Evan Rempel wrote:
> Balazs Scheidler wrote:
> >>>>>> We have cases where a syslog-ng buffers quite a bit of data (2GB or so) before
> >>>>>> it is able to flush the data to the destination. Currently, the only way to "free up" that RAM
> >>>>>> is to restart syslog-ng, which means that some log messages are lost during the restart
> >>>>>> window.
> > 
> > Both PE and OSE frees up disk/memory when possible. If they don't, then
> > that's probably a leak.
> > 
> > So if your syslog-ng process is 2GB in size, and you didn't specify an
> > overly large log_fifo_size(), it's probably because of a leak.
> 
> I have a large log_fifo_size() but the process starts small, grows when it needs
> to buffer, and does NOT get small again when the destination is available.

Hmm... that might be related to the malloc implementation. Which
platform are you using?

> 
> >> 2. The buffer space should be thrown away only after the message
> >>     has been delivered to the destination.
> > 
> > I see, this is what should happen now.
> 
> This does not appear to be happening.
> I am using OSE 2.0.8 Have there been memory fixes/changes since then?

Not that I know of.

> Is there a way to get buffer count statistics out? How many messages are
> in each fifo buffer?

In the experimental 3.0 branch, yes there is. In current 2.0, I'm afraid
there's none.

Is there a limit on the growth of syslog-ng? I mean if it gets large
once and then empties its buffers, it stays large, but does it start
growing again, or it stays steadily at the size it was before?

Again, the platform matters a lot, how the local malloc implementation
takes care about fragmentation, whether it shrinks the process once
memory is freed.

The problem is, that if the local malloc implementation is wrong the
only thing I could do is to try to use a different allocator (e.g. use
GNU malloc on your platform), but that's not a simple change.

-- 
Bazsi