[syslog-ng] Reload does not free up RAM

[Evan Rempel]

Balazs Scheidler wrote:
> On Thu, 2008-10-02 at 08:43 -0700, Evan Rempel wrote:
>> Balazs Scheidler wrote:
>>> On Wed, 2008-10-01 at 09:54 -0700, Evan Rempel wrote:
>>>> We have cases where a syslog-ng buffers quite a bit of data (2GB or so) before
>>>> it is able to flush the data to the destination. Currently, the only way to "free up" that RAM
>>>> is to restart syslog-ng, which means that some log messages are lost during the restart
>>>> window.
>>>>
>>>> I open for discussion the idea that on a reload that syslog-ng flush and free all buffer
>>>> space and essentially start from a "just started" state again.
>>> Well, this is currently a PE feature, and BalaBit decided not to open
>>> source it this time with 3.0 as that's one of the primary incentives to
>>> buy the commercial version.
>>>
>>> PE with its persistent disk buffers is saving queue contents upon
>>> restart right into the disk buffer file. And also, syslog-ng does not
>>> buffer so much data in memory as it first uses the disk.
>> The disk buffering offers a lot more than this.
>>
>> Persistent buffers around a syslog restart
>> Persistent buffers around a system restart (think power outage)
>> Much larger buffering (potentially hundreds of GBytes) with minimal
>> system impact.
>>
>> I would hope that PE version would shrink its disk buffers once they
>> are flushed, just like I am hoping that OSE would shrink its RAM buffers
>> once they are flushed.
>>
>> I don't think that RAM usage AFTER buffers are flushed is much of an
>> incentive to purchase the PE. The persistence and size of the buffers
>> is the incentive.
>>
> 
> E.g. you want to throw away the buffer contents but without restarts?
> that should be doable.

Just to verify the points.

1. I don't want to drop/loose messages with this change.
2. The buffer space should be thrown away only after the message
    has been delivered to the destination.
3. This should be done at least on a reload as a minimum.
    Since all of the sources are closed during a reload, I thought that
    the buffers could be flushed and all buffer space thrown away
    at that time. This is really an implementation decision that is
    really your choice, and does not matter to me.
4. Freeing and allocating memory is an expensive operation, so
    I don't want it done for each message, but doing it without the
    need for a reload would be ideal.

    Algorithms that allocate memory in blocks, and free one block
    when two are unused always keeps 1 block available for
    use and you don't get into a busy allocate/drop that repeats
    because one message comes and goes.

Thanks for entertaining this discussion.


Evan.