[syslog-ng] Reload does not free up RAM

[Balazs Scheidler]

Hi,

I might have misunderstood you a bit first, I'm trying to respond to
multiple messages at the same time, so my answers are inlined.

On Sat, 2008-10-04 at 08:26 -0700, Evan Rempel wrote:
> Balazs Scheidler wrote:
> > On Thu, 2008-10-02 at 08:43 -0700, Evan Rempel wrote:
> >> Balazs Scheidler wrote:
> >>> On Wed, 2008-10-01 at 09:54 -0700, Evan Rempel wrote:
> >>>> We have cases where a syslog-ng buffers quite a bit of data (2GB or so) before
> >>>> it is able to flush the data to the destination. Currently, the only way to "free up" that RAM
> >>>> is to restart syslog-ng, which means that some log messages are lost during the restart
> >>>> window.
> >>>>
> >>>> I open for discussion the idea that on a reload that syslog-ng flush and free all buffer
> >>>> space and essentially start from a "just started" state again.
> >>> Well, this is currently a PE feature, and BalaBit decided not to open
> >>> source it this time with 3.0 as that's one of the primary incentives to
> >>> buy the commercial version.
> >>>
> >>> PE with its persistent disk buffers is saving queue contents upon
> >>> restart right into the disk buffer file. And also, syslog-ng does not
> >>> buffer so much data in memory as it first uses the disk.
> >> The disk buffering offers a lot more than this.
> >>
> >> Persistent buffers around a syslog restart
> >> Persistent buffers around a system restart (think power outage)
> >> Much larger buffering (potentially hundreds of GBytes) with minimal
> >> system impact.
> >>
> >> I would hope that PE version would shrink its disk buffers once they
> >> are flushed, just like I am hoping that OSE would shrink its RAM buffers
> >> once they are flushed.

Both PE and OSE frees up disk/memory when possible. If they don't, then
that's probably a leak.

So if your syslog-ng process is 2GB in size, and you didn't specify an
overly large log_fifo_size(), it's probably because of a leak.

> >>
> >> I don't think that RAM usage AFTER buffers are flushed is much of an
> >> incentive to purchase the PE. The persistence and size of the buffers
> >> is the incentive.
> >>
> > 
> > E.g. you want to throw away the buffer contents but without restarts?
> > that should be doable.
> 
> Just to verify the points.
> 
> 1. I don't want to drop/loose messages with this change.

If your buffer contains 2GB of messages at the time when HUP is received
I need to put those messages somewhere. The PE version puts messages in
the log fifo to the disk, the OSE version keeps them in memory.

> 2. The buffer space should be thrown away only after the message
>     has been delivered to the destination.

I see, this is what should happen now.

> 3. This should be done at least on a reload as a minimum.
>     Since all of the sources are closed during a reload, I thought that
>     the buffers could be flushed and all buffer space thrown away
>     at that time. This is really an implementation decision that is
>     really your choice, and does not matter to me.

It should be doing it during operation as well, not just on reloads.

> 4. Freeing and allocating memory is an expensive operation, so
>     I don't want it done for each message, but doing it without the
>     need for a reload would be ideal.
> 
>     Algorithms that allocate memory in blocks, and free one block
>     when two are unused always keeps 1 block available for
>     use and you don't get into a busy allocate/drop that repeats
>     because one message comes and goes.

There are parts in syslog-ng which use chunk based allocation, but
according to my measures the glibc malloc/free is almost as fast as the
algorithm in g_slice_alloc(), which is the sub-chunk allocation library
of glib.

Of course using a different OS, these numbers might be different, but
glibc seems to be quite good in its dynamic memory allocation routines.


-- 
Bazsi