[syslog-ng] Syslog-ng shipping logs through AMQP with huge memory leaks

[Scheidler, Balázs]

You should thank to Gabor and the Balabit syslog-ng team :)

One additional ask if i may:

There are a number of allocations in the amqp destinations i don't like,
which could be responsible for that high cpu usage. But to be sure can you
pls send a fresh perf record output? Would be great to have it with
debugging symbols and call graph.

Thanks in advance

On Mar 28, 2018 16:47, "Michal Purzynski" <michal at mozilla.com> wrote:

> I went ahead and installed syslog-ng from github's master with the changes
> from the PR. Results are nothing short of impressive.
>
> 18503 root      20   0 2859872  43580   5800 S  39.9  0.1  12:33.79
> syslog-ng
>
> :-)
>
> Syslog-ng started with fierce, taking 4.5GB and 460% (!!) of CPU then it
> settled down and the memory usage started shrinking quickly.
>
> We are forwarding 3500 eps with 38% or so of a single CPU and 44MB of RAM.
>
>
> On Wed, Mar 28, 2018 at 11:46 AM, Michal Purzynski <michal at mozilla.com>
> wrote:
>
>> Awesome! Do you want me to test it? I guess I can fetch the PR and deploy
>> the RPM in place of the affected package we have now.
>>
>> On Wed, Mar 28, 2018 at 9:04 AM, Scheidler, Balázs <
>> balazs.scheidler at balabit.com> wrote:
>>
>>> There's a potential fix here: https://github.com/balabit/sys
>>> log-ng/pull/1946
>>>
>>>
>>> --
>>> Bazsi
>>>
>>> On Wed, Mar 28, 2018 at 10:29 AM, Nagy, Gábor <gabor.nagy at balabit.com>
>>> wrote:
>>>
>>>> I'm resending this letter to the mail list as we hit the message size
>>>> limit. :)
>>>>
>>>> Hello Michal!
>>>>
>>>> I have some good news.
>>>> We have found the reason for the increasing memory usage of syslog-ng
>>>> in amqp() driver.
>>>> It is due to our internal memory handling method (aka scratch_buffers)
>>>> which we use in performance critical paths to prevent the overhead of
>>>> memory allocations/deallocations of GStrings.
>>>> It is using memory from a pre-allocated pool instead of
>>>> allocating/freeing.
>>>>
>>>> As I have described before, the memory usage only increases if the
>>>> number of queued messages reaches a new maximum.
>>>> It is not per message and does not happen every for every queued
>>>> message.
>>>>
>>>> We have several options how to fix this issue and will soon share one.
>>>>
>>>>
>>>> Regards,
>>>> Gabor
>>>>
>>>>
>>>> ____________________________________________________________
>>>> __________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation: http://www.balabit.com/support
>>>> /documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>>>
>>>
>>> ____________________________________________________________
>>> __________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation: http://www.balabit.com/support
>>> /documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>>
>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20180328/744f2936/attachment-0001.html>