[syslog-ng] Syslog-ng shipping logs through AMQP with huge memory leaks

[Michal Purzynski]

I went ahead and installed syslog-ng from github's master with the changes
from the PR. Results are nothing short of impressive.

18503 root      20   0 2859872  43580   5800 S  39.9  0.1  12:33.79
syslog-ng

:-)

Syslog-ng started with fierce, taking 4.5GB and 460% (!!) of CPU then it
settled down and the memory usage started shrinking quickly.

We are forwarding 3500 eps with 38% or so of a single CPU and 44MB of RAM.


On Wed, Mar 28, 2018 at 11:46 AM, Michal Purzynski <michal at mozilla.com>
wrote:

> Awesome! Do you want me to test it? I guess I can fetch the PR and deploy
> the RPM in place of the affected package we have now.
>
> On Wed, Mar 28, 2018 at 9:04 AM, Scheidler, Balázs <
> balazs.scheidler at balabit.com> wrote:
>
>> There's a potential fix here: https://github.com/balabit/sys
>> log-ng/pull/1946
>>
>>
>> --
>> Bazsi
>>
>> On Wed, Mar 28, 2018 at 10:29 AM, Nagy, Gábor <gabor.nagy at balabit.com>
>> wrote:
>>
>>> I'm resending this letter to the mail list as we hit the message size
>>> limit. :)
>>>
>>> Hello Michal!
>>>
>>> I have some good news.
>>> We have found the reason for the increasing memory usage of syslog-ng in
>>> amqp() driver.
>>> It is due to our internal memory handling method (aka scratch_buffers)
>>> which we use in performance critical paths to prevent the overhead of
>>> memory allocations/deallocations of GStrings.
>>> It is using memory from a pre-allocated pool instead of
>>> allocating/freeing.
>>>
>>> As I have described before, the memory usage only increases if the
>>> number of queued messages reaches a new maximum.
>>> It is not per message and does not happen every for every queued message.
>>>
>>> We have several options how to fix this issue and will soon share one.
>>>
>>>
>>> Regards,
>>> Gabor
>>>
>>>
>>> ____________________________________________________________
>>> __________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation: http://www.balabit.com/support
>>> /documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>>
>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=
>> syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20180328/0fce6bf7/attachment.html>