Re: [syslog-ng] Reload does not free up RAM
Balazs Scheidler wrote:
On Thu, 2008-10-02 at 08:43 -0700, Evan Rempel wrote:
Balazs Scheidler wrote:
On Wed, 2008-10-01 at 09:54 -0700, Evan Rempel wrote:
We have cases where a syslog-ng buffers quite a bit of data (2GB or so) before it is able to flush the data to the destination. Currently, the only way to "free up" that RAM is to restart syslog-ng, which means that some log messages are lost during the restart window.
I open for discussion the idea that on a reload that syslog-ng flush and free all buffer space and essentially start from a "just started" state again. Well, this is currently a PE feature, and BalaBit decided not to open source it this time with 3.0 as that's one of the primary incentives to buy the commercial version.
PE with its persistent disk buffers is saving queue contents upon restart right into the disk buffer file. And also, syslog-ng does not buffer so much data in memory as it first uses the disk. The disk buffering offers a lot more than this.
Persistent buffers around a syslog restart Persistent buffers around a system restart (think power outage) Much larger buffering (potentially hundreds of GBytes) with minimal system impact.
I would hope that PE version would shrink its disk buffers once they are flushed, just like I am hoping that OSE would shrink its RAM buffers once they are flushed.
I don't think that RAM usage AFTER buffers are flushed is much of an incentive to purchase the PE. The persistence and size of the buffers is the incentive.
E.g. you want to throw away the buffer contents but without restarts? that should be doable.
Just to verify the points. 1. I don't want to drop/loose messages with this change. 2. The buffer space should be thrown away only after the message has been delivered to the destination. 3. This should be done at least on a reload as a minimum. Since all of the sources are closed during a reload, I thought that the buffers could be flushed and all buffer space thrown away at that time. This is really an implementation decision that is really your choice, and does not matter to me. 4. Freeing and allocating memory is an expensive operation, so I don't want it done for each message, but doing it without the need for a reload would be ideal. Algorithms that allocate memory in blocks, and free one block when two are unused always keeps 1 block available for use and you don't get into a busy allocate/drop that repeats because one message comes and goes. Thanks for entertaining this discussion. Evan.
Hi, I might have misunderstood you a bit first, I'm trying to respond to multiple messages at the same time, so my answers are inlined. On Sat, 2008-10-04 at 08:26 -0700, Evan Rempel wrote:
Balazs Scheidler wrote:
On Thu, 2008-10-02 at 08:43 -0700, Evan Rempel wrote:
Balazs Scheidler wrote:
On Wed, 2008-10-01 at 09:54 -0700, Evan Rempel wrote:
We have cases where a syslog-ng buffers quite a bit of data (2GB or so) before it is able to flush the data to the destination. Currently, the only way to "free up" that RAM is to restart syslog-ng, which means that some log messages are lost during the restart window.
I open for discussion the idea that on a reload that syslog-ng flush and free all buffer space and essentially start from a "just started" state again. Well, this is currently a PE feature, and BalaBit decided not to open source it this time with 3.0 as that's one of the primary incentives to buy the commercial version.
PE with its persistent disk buffers is saving queue contents upon restart right into the disk buffer file. And also, syslog-ng does not buffer so much data in memory as it first uses the disk. The disk buffering offers a lot more than this.
Persistent buffers around a syslog restart Persistent buffers around a system restart (think power outage) Much larger buffering (potentially hundreds of GBytes) with minimal system impact.
I would hope that PE version would shrink its disk buffers once they are flushed, just like I am hoping that OSE would shrink its RAM buffers once they are flushed.
Both PE and OSE frees up disk/memory when possible. If they don't, then that's probably a leak. So if your syslog-ng process is 2GB in size, and you didn't specify an overly large log_fifo_size(), it's probably because of a leak.
I don't think that RAM usage AFTER buffers are flushed is much of an incentive to purchase the PE. The persistence and size of the buffers is the incentive.
E.g. you want to throw away the buffer contents but without restarts? that should be doable.
Just to verify the points.
1. I don't want to drop/loose messages with this change.
If your buffer contains 2GB of messages at the time when HUP is received I need to put those messages somewhere. The PE version puts messages in the log fifo to the disk, the OSE version keeps them in memory.
2. The buffer space should be thrown away only after the message has been delivered to the destination.
I see, this is what should happen now.
3. This should be done at least on a reload as a minimum. Since all of the sources are closed during a reload, I thought that the buffers could be flushed and all buffer space thrown away at that time. This is really an implementation decision that is really your choice, and does not matter to me.
It should be doing it during operation as well, not just on reloads.
4. Freeing and allocating memory is an expensive operation, so I don't want it done for each message, but doing it without the need for a reload would be ideal.
Algorithms that allocate memory in blocks, and free one block when two are unused always keeps 1 block available for use and you don't get into a busy allocate/drop that repeats because one message comes and goes.
There are parts in syslog-ng which use chunk based allocation, but according to my measures the glibc malloc/free is almost as fast as the algorithm in g_slice_alloc(), which is the sub-chunk allocation library of glib. Of course using a different OS, these numbers might be different, but glibc seems to be quite good in its dynamic memory allocation routines. -- Bazsi
participants (2)
-
Balazs Scheidler
-
Evan Rempel