Do stats cause memory leak
We are running git head download post 3.9.1 and the memory footprint seems to continually grow. Looking at the statistics for queued messages I found none, but discovered that we have nearly 22,000 statistic sets for a total of 68195 tracked statistics. % /usr/local/sbin/syslog-ng-ctl stats | wc -l 68195 It it possible that all of these statistics take up a non-trivial amount of RAM? USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1489 79.2 61.5 12134140 10139760 ? Ssl Apr24 41581:22 /usr/local/sbin/syslog-ng That is a 12GB footprint after 1 month of uptime. Evan.
Hi, https://github.com/balabit/syslog-ng/pull/1505 seems to address memory problems. If you use my git head packages, I can add this patch for you and see if it resolves your problems. Bye, Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik On Wed, May 31, 2017 at 4:28 AM, Evan Rempel <erempel@uvic.ca> wrote:
We are running git head download post 3.9.1 and the memory footprint seems to continually grow.
Looking at the statistics for queued messages I found none, but discovered that we have nearly 22,000 statistic sets for a total of 68195 tracked statistics. % /usr/local/sbin/syslog-ng-ctl stats | wc -l 68195
It it possible that all of these statistics take up a non-trivial amount of RAM?
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1489 79.2 61.5 12134140 10139760 ? Ssl Apr24 41581:22 /usr/local/sbin/syslog-ng
That is a 12GB footprint after 1 month of uptime.
Evan. ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
There is also this issue that might be related (don't know if it is solved by the above PR or not): https://github.com/balabit/syslog-ng/issues/1494 On Wed, May 31, 2017 at 8:28 AM, Czanik, Péter <peter.czanik@balabit.com> wrote:
Hi,
https://github.com/balabit/syslog-ng/pull/1505 seems to address memory problems. If you use my git head packages, I can add this patch for you and see if it resolves your problems.
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik
On Wed, May 31, 2017 at 4:28 AM, Evan Rempel <erempel@uvic.ca> wrote:
We are running git head download post 3.9.1 and the memory footprint seems to continually grow.
Looking at the statistics for queued messages I found none, but discovered that we have nearly 22,000 statistic sets for a total of 68195 tracked statistics. % /usr/local/sbin/syslog-ng-ctl stats | wc -l 68195
It it possible that all of these statistics take up a non-trivial amount of RAM?
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1489 79.2 61.5 12134140 10139760 ? Ssl Apr24 41581:22 /usr/local/sbin/syslog-ng
That is a 12GB footprint after 1 month of uptime.
Evan. ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
My guess that it is related to #1517 instead, albeit stats counters were worked on recently. If #1517 does not fix this, please report it again. -- Bazsi On Wed, May 31, 2017 at 9:02 AM, Fekete, Róbert <robert.fekete@balabit.com> wrote:
There is also this issue that might be related (don't know if it is solved by the above PR or not): https://github.com/balabit/syslog-ng/issues/1494
On Wed, May 31, 2017 at 8:28 AM, Czanik, Péter <peter.czanik@balabit.com> wrote:
Hi,
https://github.com/balabit/syslog-ng/pull/1505 seems to address memory problems. If you use my git head packages, I can add this patch for you and see if it resolves your problems.
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik
On Wed, May 31, 2017 at 4:28 AM, Evan Rempel <erempel@uvic.ca> wrote:
We are running git head download post 3.9.1 and the memory footprint seems to continually grow.
Looking at the statistics for queued messages I found none, but discovered that we have nearly 22,000 statistic sets for a total of 68195 tracked statistics. % /usr/local/sbin/syslog-ng-ctl stats | wc -l 68195
It it possible that all of these statistics take up a non-trivial amount of RAM?
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1489 79.2 61.5 12134140 <(1)%20213%204140> 10139760 <(1)%20013%209760> ? Ssl Apr24 41581:22 /usr/local/sbin/syslog-ng
That is a 12GB footprint after 1 month of uptime.
Evan. ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
So it looks like the latest git head no longer compiles on Redhat 6. It is using g_atomic_pointer_and which requires glib2 2.30+ and Redhat 6 only has glib2 2.28 Looks like I will not be able to test this :-( nor get this issue fixed in our infrastructure. Evan. On 06/03/2017 02:08 PM, Scheidler, Balázs wrote:
My guess that it is related to #1517 instead, albeit stats counters were worked on recently. If #1517 does not fix this, please report it again.
-- Bazsi
On Wed, May 31, 2017 at 9:02 AM, Fekete, Róbert <robert.fekete@balabit.com <mailto:robert.fekete@balabit.com>> wrote:
There is also this issue that might be related (don't know if it is solved by the above PR or not): https://github.com/balabit/syslog-ng/issues/1494 <https://github.com/balabit/syslog-ng/issues/1494>
On Wed, May 31, 2017 at 8:28 AM, Czanik, Péter <peter.czanik@balabit.com <mailto:peter.czanik@balabit.com>> wrote:
Hi,
https://github.com/balabit/syslog-ng/pull/1505 <https://github.com/balabit/syslog-ng/pull/1505> seems to address memory problems. If you use my git head packages, I can add this patch for you and see if it resolves your problems.
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com <mailto:peter.czanik@balabit.com>> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ <https://www.balabit.com/blog/author/peterczanik/> https://twitter.com/PCzanik
On Wed, May 31, 2017 at 4:28 AM, Evan Rempel <erempel@uvic.ca <mailto:erempel@uvic.ca>> wrote:
We are running git head download post 3.9.1 and the memory footprint seems to continually grow.
Looking at the statistics for queued messages I found none, but discovered that we have nearly 22,000 statistic sets for a total of 68195 tracked statistics. % /usr/local/sbin/syslog-ng-ctl stats | wc -l 68195
It it possible that all of these statistics take up a non-trivial amount of RAM?
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1489 79.2 61.5 12134140 <tel:%281%29%20213%204140> 10139760 <tel:%281%29%20013%209760> ? Ssl Apr24 41581:22 /usr/local/sbin/syslog-ng
That is a 12GB footprint after 1 month of uptime.
Evan.
SLES 11 is also affected: https://github.com/balabit/syslog-ng/issues/1403#issuecomment-290195851 Both SLES 11 and RHEL 6 are ancient but still used at many places. Actually about a year ago both still had more installations than SLES 12 or RHEL 7. Bye, Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik On Sun, Jun 4, 2017 at 1:07 AM, Evan Rempel <erempel@uvic.ca> wrote:
So it looks like the latest git head no longer compiles on Redhat 6. It is using g_atomic_pointer_and which requires glib2 2.30+ and Redhat 6 only has glib2 2.28
Looks like I will not be able to test this :-( nor get this issue fixed in our infrastructure.
Evan.
On 06/03/2017 02:08 PM, Scheidler, Balázs wrote:
My guess that it is related to #1517 instead, albeit stats counters were worked on recently. If #1517 does not fix this, please report it again.
-- Bazsi
On Wed, May 31, 2017 at 9:02 AM, Fekete, Róbert <robert.fekete@balabit.com
wrote:
There is also this issue that might be related (don't know if it is solved by the above PR or not): https://github.com/balab it/syslog-ng/issues/1494
On Wed, May 31, 2017 at 8:28 AM, Czanik, Péter <peter.czanik@balabit.com> wrote:
Hi,
https://github.com/balabit/syslog-ng/pull/1505 seems to address memory problems. If you use my git head packages, I can add this patch for you and see if it resolves your problems.
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik
On Wed, May 31, 2017 at 4:28 AM, Evan Rempel <erempel@uvic.ca> wrote:
We are running git head download post 3.9.1 and the memory footprint seems to continually grow.
Looking at the statistics for queued messages I found none, but discovered that we have nearly 22,000 statistic sets for a total of 68195 tracked statistics. % /usr/local/sbin/syslog-ng-ctl stats | wc -l 68195
It it possible that all of these statistics take up a non-trivial amount of RAM?
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1489 79.2 61.5 12134140 <%281%29%20213%204140> 10139760 <%281%29%20013%209760> ? Ssl Apr24 41581:22 /usr/local/sbin/syslog-ng
That is a 12GB footprint after 1 month of uptime.
Evan.
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
I'll check if that dependency can be removed. On Jun 4, 2017 9:04 AM, "Czanik, Péter" <peter.czanik@balabit.com> wrote: SLES 11 is also affected: https://github.com/balabit/syslog-ng/issues/1403# issuecomment-290195851 Both SLES 11 and RHEL 6 are ancient but still used at many places. Actually about a year ago both still had more installations than SLES 12 or RHEL 7. Bye, Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik On Sun, Jun 4, 2017 at 1:07 AM, Evan Rempel <erempel@uvic.ca> wrote:
So it looks like the latest git head no longer compiles on Redhat 6. It is using g_atomic_pointer_and which requires glib2 2.30+ and Redhat 6 only has glib2 2.28
Looks like I will not be able to test this :-( nor get this issue fixed in our infrastructure.
Evan.
On 06/03/2017 02:08 PM, Scheidler, Balázs wrote:
My guess that it is related to #1517 instead, albeit stats counters were worked on recently. If #1517 does not fix this, please report it again.
-- Bazsi
On Wed, May 31, 2017 at 9:02 AM, Fekete, Róbert <robert.fekete@balabit.com
wrote:
There is also this issue that might be related (don't know if it is solved by the above PR or not): https://github.com/balab it/syslog-ng/issues/1494
On Wed, May 31, 2017 at 8:28 AM, Czanik, Péter <peter.czanik@balabit.com> wrote:
Hi,
https://github.com/balabit/syslog-ng/pull/1505 seems to address memory problems. If you use my git head packages, I can add this patch for you and see if it resolves your problems.
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik
On Wed, May 31, 2017 at 4:28 AM, Evan Rempel <erempel@uvic.ca> wrote:
We are running git head download post 3.9.1 and the memory footprint seems to continually grow.
Looking at the statistics for queued messages I found none, but discovered that we have nearly 22,000 statistic sets for a total of 68195 tracked statistics. % /usr/local/sbin/syslog-ng-ctl stats | wc -l 68195
It it possible that all of these statistics take up a non-trivial amount of RAM?
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1489 79.2 61.5 12134140 <%281%29%20213%204140> 10139760 <%281%29%20013%209760> ? Ssl Apr24 41581:22 /usr/local/sbin/syslog-ng
That is a 12GB footprint after 1 month of uptime.
Evan.
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Sorry that I mistyped the library function. It should be g_atomic_pointer_add (not g_atomic_pointer_and). On 06/05/2017 10:33 AM, Scheidler, Balázs wrote:
I'll check if that dependency can be removed.
On Jun 4, 2017 9:04 AM, "Czanik, Péter" <peter.czanik@balabit.com <mailto:peter.czanik@balabit.com>> wrote:
SLES 11 is also affected: https://github.com/balabit/syslog-ng/issues/1403#issuecomment-290195851 <https://github.com/balabit/syslog-ng/issues/1403#issuecomment-290195851>
Both SLES 11 and RHEL 6 are ancient but still used at many places. Actually about a year ago both still had more installations than SLES 12 or RHEL 7.
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com <mailto:peter.czanik@balabit.com>> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ <https://www.balabit.com/blog/author/peterczanik/> https://twitter.com/PCzanik
On Sun, Jun 4, 2017 at 1:07 AM, Evan Rempel <erempel@uvic.ca <mailto:erempel@uvic.ca>> wrote:
So it looks like the latest git head no longer compiles on Redhat 6. It is using g_atomic_pointer_and which requires glib2 2.30+ and Redhat 6 only has glib2 2.28
Looks like I will not be able to test this :-( nor get this issue fixed in our infrastructure.
Evan.
On 06/03/2017 02:08 PM, Scheidler, Balázs wrote:
My guess that it is related to #1517 instead, albeit stats counters were worked on recently. If #1517 does not fix this, please report it again.
-- Bazsi
On Wed, May 31, 2017 at 9:02 AM, Fekete, Róbert <robert.fekete@balabit.com <mailto:robert.fekete@balabit.com>> wrote:
There is also this issue that might be related (don't know if it is solved by the above PR or not): https://github.com/balabit/syslog-ng/issues/1494 <https://github.com/balabit/syslog-ng/issues/1494>
On Wed, May 31, 2017 at 8:28 AM, Czanik, Péter <peter.czanik@balabit.com <mailto:peter.czanik@balabit.com>> wrote:
Hi,
https://github.com/balabit/syslog-ng/pull/1505 <https://github.com/balabit/syslog-ng/pull/1505> seems to address memory problems. If you use my git head packages, I can add this patch for you and see if it resolves your problems.
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com <mailto:peter.czanik@balabit.com>> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ <https://www.balabit.com/blog/author/peterczanik/> https://twitter.com/PCzanik
On Wed, May 31, 2017 at 4:28 AM, Evan Rempel <erempel@uvic.ca <mailto:erempel@uvic.ca>> wrote:
We are running git head download post 3.9.1 and the memory footprint seems to continually grow.
Looking at the statistics for queued messages I found none, but discovered that we have nearly 22,000 statistic sets for a total of 68195 tracked statistics. % /usr/local/sbin/syslog-ng-ctl stats | wc -l 68195
It it possible that all of these statistics take up a non-trivial amount of RAM?
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1489 79.2 61.5 12134140 <tel:%281%29%20213%204140> 10139760 <tel:%281%29%20013%209760> ? Ssl Apr24 41581:22 /usr/local/sbin/syslog-ng
That is a 12GB footprint after 1 month of uptime.
Evan.
Hi, On Mon, Jun 5, 2017 at 7:33 PM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
I'll check if that dependency can be removed.
That would be great! Until now at least the core syslog-ng compiled on older platforms. So, even if many of the dependencies for some of the news features were not available, at least the core could be kept up-to-date. Bye, Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik
On Jun 4, 2017 9:04 AM, "Czanik, Péter" <peter.czanik@balabit.com> wrote:
SLES 11 is also affected: https://github.com/balabit/sys log-ng/issues/1403#issuecomment-290195851
Both SLES 11 and RHEL 6 are ancient but still used at many places. Actually about a year ago both still had more installations than SLES 12 or RHEL 7.
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik
On Sun, Jun 4, 2017 at 1:07 AM, Evan Rempel <erempel@uvic.ca> wrote:
So it looks like the latest git head no longer compiles on Redhat 6. It is using g_atomic_pointer_and which requires glib2 2.30+ and Redhat 6 only has glib2 2.28
Looks like I will not be able to test this :-( nor get this issue fixed in our infrastructure.
Evan.
On 06/03/2017 02:08 PM, Scheidler, Balázs wrote:
My guess that it is related to #1517 instead, albeit stats counters were worked on recently. If #1517 does not fix this, please report it again.
-- Bazsi
On Wed, May 31, 2017 at 9:02 AM, Fekete, Róbert < robert.fekete@balabit.com> wrote:
There is also this issue that might be related (don't know if it is solved by the above PR or not): https://github.com/balab it/syslog-ng/issues/1494
On Wed, May 31, 2017 at 8:28 AM, Czanik, Péter <peter.czanik@balabit.com
wrote:
Hi,
https://github.com/balabit/syslog-ng/pull/1505 seems to address memory problems. If you use my git head packages, I can add this patch for you and see if it resolves your problems.
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik
On Wed, May 31, 2017 at 4:28 AM, Evan Rempel <erempel@uvic.ca> wrote:
We are running git head download post 3.9.1 and the memory footprint seems to continually grow.
Looking at the statistics for queued messages I found none, but discovered that we have nearly 22,000 statistic sets for a total of 68195 tracked statistics. % /usr/local/sbin/syslog-ng-ctl stats | wc -l 68195
It it possible that all of these statistics take up a non-trivial amount of RAM?
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1489 79.2 61.5 12134140 <%281%29%20213%204140> 10139760 <%281%29%20013%209760> ? Ssl Apr24 41581:22 /usr/local/sbin/syslog-ng
That is a 12GB footprint after 1 month of uptime.
Evan.
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (4)
-
Czanik, Péter
-
Evan Rempel
-
Fekete, Róbert
-
Scheidler, Balázs