I went ahead and installed syslog-ng from github's master with the changes from the PR. Results are nothing short of impressive.

18503 root      20   0 2859872  43580   5800 S  39.9  0.1  12:33.79 syslog-ng                                                 

:-)

Syslog-ng started with fierce, taking 4.5GB and 460% (!!) of CPU then it settled down and the memory usage started shrinking quickly.

We are forwarding 3500 eps with 38% or so of a single CPU and 44MB of RAM.
 

On Wed, Mar 28, 2018 at 11:46 AM, Michal Purzynski <michal@mozilla.com> wrote:
Awesome! Do you want me to test it? I guess I can fetch the PR and deploy the RPM in place of the affected package we have now.

On Wed, Mar 28, 2018 at 9:04 AM, Scheidler, Balázs <balazs.scheidler@balabit.com> wrote:

--
Bazsi

On Wed, Mar 28, 2018 at 10:29 AM, Nagy, Gábor <gabor.nagy@balabit.com> wrote:
I'm resending this letter to the mail list as we hit the message size limit. :)

Hello Michal!

I have some good news.
We have found the reason for the increasing memory usage of syslog-ng in amqp() driver.
It is due to our internal memory handling method (aka scratch_buffers) which we use in performance critical paths to prevent the overhead of memory allocations/deallocations of GStrings.
It is using memory from a pre-allocated pool instead of allocating/freeing.

As I have described before, the memory usage only increases if the number of queued messages reaches a new maximum.
It is not per message and does not happen every for every queued message.

We have several options how to fix this issue and will soon share one.


Regards,
Gabor


______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq




______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq