Hi, I'm using syslog-ng 3.16.1 with wildcard-file as source (let's call it "syslog-agent), which sends log messages to another syslog-ng acting as a relay. I've noticed that syslog-agent instances RAM consumption keeps increasing until they leave no free memory in the cluster (each server has 64G of RAM). In my use case, new folders & files are created constantly under base-dir folder, but every 2 days obsolete folders & files are deleted. I assumed that syslog-ng would free some RAM every time those folders & files are deleted, but it doesn't, not even if I run a syslog-ng-ctl reload operation. log-fifo-size value is 345000, so I assume it can't be a buffer situation since 345000 messages can't occupy 40-50 GB of memory. I've performed the following test to reproduce the situation in small scale: - Launch a syslog-agent with a wildcard-file source reading from "/tmp/test/" base-dir. syslog-agent RAM usage is about 125M. - Run a simple script to create complex folder hierarchy under /tmp/test and some files with 5000 log messages to read from. - Wait until syslog-agent RAM usage gets 1GB - Stop script execution and wait until syslog-agent has send all logs to relay. - Delete everything under /tmp/test and execute syslog-ng-ctl reload operation - 24 hours after, syslog-agent RAM usage still is 1GB I've used heaptrack tool as a try to find a memory leak in syslog-ng, you can see in the attached image that iv_list_empty function in iv_list.h file is where most of the RAM usage is. How do I get syslog-ng to free RAM? Or is it a memory leak? Thanks in advance. -- | Jose Angel Santiago [image: Logo_signature2.png] <http://www.stratio.com/> Vía de las dos Castillas, 33, Ática 4, 3ª Planta 28224 Pozuelo de Alarcón, Madrid, Spain +34 918 286 473 | www.stratio.com <https://twitter.com/stratiobd> <https://www.linkedin.com/company/stratiobd> <https://www.youtube.com/c/StratioBD>
Hello, Thanks for the thorough investigation! We've checked it and found a memory leak in directory monitoring. I've pushed the fix to my fork and created a merge request about it: https://github.com/gaborznagy/syslog-ng/commits/fix-wildcard-memleak https://github.com/balabit/syslog-ng/pull/2261 Can you verify that this fixes your problem, please?
log-fifo-size value is 345000, so I assume it can't be a buffer situation since 345000 messages can't occupy 40-50 GB of memory. Well our default log-msg-size is 64kB so if all messages have a size of log-msg-size then it can be around ~21GB when syslog-ng is buffering.
Best Regards, Gabor On Tue, Sep 4, 2018 at 10:50 AM Jose Angel Santiago <jasantiago@stratio.com> wrote:
Hi,
I'm using syslog-ng 3.16.1 with wildcard-file as source (let's call it "syslog-agent), which sends log messages to another syslog-ng acting as a relay.
I've noticed that syslog-agent instances RAM consumption keeps increasing until they leave no free memory in the cluster (each server has 64G of RAM). In my use case, new folders & files are created constantly under base-dir folder, but every 2 days obsolete folders & files are deleted. I assumed that syslog-ng would free some RAM every time those folders & files are deleted, but it doesn't, not even if I run a syslog-ng-ctl reload operation.
log-fifo-size value is 345000, so I assume it can't be a buffer situation since 345000 messages can't occupy 40-50 GB of memory.
I've performed the following test to reproduce the situation in small scale:
- Launch a syslog-agent with a wildcard-file source reading from "/tmp/test/" base-dir. syslog-agent RAM usage is about 125M. - Run a simple script to create complex folder hierarchy under /tmp/test and some files with 5000 log messages to read from. - Wait until syslog-agent RAM usage gets 1GB - Stop script execution and wait until syslog-agent has send all logs to relay. - Delete everything under /tmp/test and execute syslog-ng-ctl reload operation - 24 hours after, syslog-agent RAM usage still is 1GB
I've used heaptrack tool as a try to find a memory leak in syslog-ng, you can see in the attached image that iv_list_empty function in iv_list.h file is where most of the RAM usage is.
How do I get syslog-ng to free RAM? Or is it a memory leak?
Thanks in advance.
--
| Jose Angel Santiago
[image: Logo_signature2.png] <http://www.stratio.com/>
Vía de las dos Castillas, 33, Ática 4, 3ª Planta
28224 Pozuelo de Alarcón, Madrid, Spain
+34 918 286 473 | www.stratio.com <https://twitter.com/stratiobd> <https://www.linkedin.com/company/stratiobd> <https://www.youtube.com/c/StratioBD>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, I was testing your fix in several environments before replying, unfortunately syslog-ng stills shows a huge RAM consumption in our use of case. We have two wildcard-file sources defined in our syslog-ng conf file, both of them have the same base-dir reference, but one source has a filename-pattern and the other source has a different one (I think we can't use the same source and several filename-patterns because of the lack of regular expression support). We execute the deletion of obsolete files and folder under base-dir once every 2 days, and we create a lot of new folders and files constantly while using our platform, but the growth of these log files (the ones affected by the filename-pattern) has been pretty low in this test, about 10-20 msgs per second total. This is some data of the fix test: Syslog-ng was consuming 21,7G of RAM with 10600 files affected by the filename-pattern and 20000 folders under base-dir (not a single folder were deleted yet so this is the expected behaviour of syslog-ng, and we find it extremely unaffordable) One hour later, right before the cleaning process was executed, syslog-ng was consuming 35G of RAM, and it didn't decrease after the cleaning, not even a little, with just 380 files affected by the filename-pattern and 2200 folders under base-dir. There is no message in the syslog-agent queue and all the messages are received by the relay (which ingest messages in an elasticsearch) For the record, we do noticed a small RAM usage decrease in a previous test in a lab environment when we deleted every folder & file under base-dir (from 870M to 550M). It seems syslog-ng wildcard-source is not optimized when it has too handle thousands of folders and filename-patterns under base-dir. Have you ever test the wildcard-source with a use of case similar to ours? Can I send you any more information which may help to find out how to optimize it? I wish I can help with the code but I have no idea of C language :-( Thanks in advance. 2018-09-04 18:03 GMT+02:00 Nagy, Gábor <gabor.nagy@oneidentity.com>:
Hello,
Thanks for the thorough investigation! We've checked it and found a memory leak in directory monitoring.
I've pushed the fix to my fork and created a merge request about it: https://github.com/gaborznagy/syslog-ng/commits/fix-wildcard-memleak https://github.com/balabit/syslog-ng/pull/2261 Can you verify that this fixes your problem, please?
log-fifo-size value is 345000, so I assume it can't be a buffer situation since 345000 messages can't occupy 40-50 GB of memory. Well our default log-msg-size is 64kB so if all messages have a size of log-msg-size then it can be around ~21GB when syslog-ng is buffering.
Best Regards, Gabor
On Tue, Sep 4, 2018 at 10:50 AM Jose Angel Santiago < jasantiago@stratio.com> wrote:
Hi,
I'm using syslog-ng 3.16.1 with wildcard-file as source (let's call it "syslog-agent), which sends log messages to another syslog-ng acting as a relay.
I've noticed that syslog-agent instances RAM consumption keeps increasing until they leave no free memory in the cluster (each server has 64G of RAM). In my use case, new folders & files are created constantly under base-dir folder, but every 2 days obsolete folders & files are deleted. I assumed that syslog-ng would free some RAM every time those folders & files are deleted, but it doesn't, not even if I run a syslog-ng-ctl reload operation.
log-fifo-size value is 345000, so I assume it can't be a buffer situation since 345000 messages can't occupy 40-50 GB of memory.
I've performed the following test to reproduce the situation in small scale:
- Launch a syslog-agent with a wildcard-file source reading from "/tmp/test/" base-dir. syslog-agent RAM usage is about 125M. - Run a simple script to create complex folder hierarchy under /tmp/test and some files with 5000 log messages to read from. - Wait until syslog-agent RAM usage gets 1GB - Stop script execution and wait until syslog-agent has send all logs to relay. - Delete everything under /tmp/test and execute syslog-ng-ctl reload operation - 24 hours after, syslog-agent RAM usage still is 1GB
I've used heaptrack tool as a try to find a memory leak in syslog-ng, you can see in the attached image that iv_list_empty function in iv_list.h file is where most of the RAM usage is.
How do I get syslog-ng to free RAM? Or is it a memory leak?
Thanks in advance.
--
| Jose Angel Santiago
[image: Logo_signature2.png] <http://www.stratio.com/>
Vía de las dos Castillas, 33, Ática 4, 3ª Planta
28224 Pozuelo de Alarcón, Madrid, Spain
+34 918 286 473 | www.stratio.com <https://twitter.com/stratiobd> <https://www.linkedin.com/company/stratiobd> <https://www.youtube.com/c/StratioBD> ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- | Jose Angel Santiago [image: Logo_signature2.png] <http://www.stratio.com/> Vía de las dos Castillas, 33, Ática 4, 3ª Planta 28224 Pozuelo de Alarcón, Madrid, Spain +34 918 286 473 | www.stratio.com <https://twitter.com/stratiobd> <https://www.linkedin.com/company/stratiobd> <https://www.youtube.com/c/StratioBD>
Reviewing your commit I've found out that the method *_handle_file_deleted* seems not to free memory when a monitored file has been deleted, according to the changes you made in *_handler_handler_directory_deleted.* Could it be the cause of the memory leak? Thanks. 2018-09-06 11:50 GMT+02:00 Jose Angel Santiago <jasantiago@stratio.com>:
Hi,
I was testing your fix in several environments before replying, unfortunately syslog-ng stills shows a huge RAM consumption in our use of case.
We have two wildcard-file sources defined in our syslog-ng conf file, both of them have the same base-dir reference, but one source has a filename-pattern and the other source has a different one (I think we can't use the same source and several filename-patterns because of the lack of regular expression support). We execute the deletion of obsolete files and folder under base-dir once every 2 days, and we create a lot of new folders and files constantly while using our platform, but the growth of these log files (the ones affected by the filename-pattern) has been pretty low in this test, about 10-20 msgs per second total.
This is some data of the fix test:
Syslog-ng was consuming 21,7G of RAM with 10600 files affected by the filename-pattern and 20000 folders under base-dir (not a single folder were deleted yet so this is the expected behaviour of syslog-ng, and we find it extremely unaffordable)
One hour later, right before the cleaning process was executed, syslog-ng was consuming 35G of RAM, and it didn't decrease after the cleaning, not even a little, with just 380 files affected by the filename-pattern and 2200 folders under base-dir.
There is no message in the syslog-agent queue and all the messages are received by the relay (which ingest messages in an elasticsearch)
For the record, we do noticed a small RAM usage decrease in a previous test in a lab environment when we deleted every folder & file under base-dir (from 870M to 550M).
It seems syslog-ng wildcard-source is not optimized when it has too handle thousands of folders and filename-patterns under base-dir. Have you ever test the wildcard-source with a use of case similar to ours?
Can I send you any more information which may help to find out how to optimize it? I wish I can help with the code but I have no idea of C language :-(
Thanks in advance.
2018-09-04 18:03 GMT+02:00 Nagy, Gábor <gabor.nagy@oneidentity.com>:
Hello,
Thanks for the thorough investigation! We've checked it and found a memory leak in directory monitoring.
I've pushed the fix to my fork and created a merge request about it: https://github.com/gaborznagy/syslog-ng/commits/fix-wildcard-memleak https://github.com/balabit/syslog-ng/pull/2261 Can you verify that this fixes your problem, please?
log-fifo-size value is 345000, so I assume it can't be a buffer situation since 345000 messages can't occupy 40-50 GB of memory. Well our default log-msg-size is 64kB so if all messages have a size of log-msg-size then it can be around ~21GB when syslog-ng is buffering.
Best Regards, Gabor
On Tue, Sep 4, 2018 at 10:50 AM Jose Angel Santiago < jasantiago@stratio.com> wrote:
Hi,
I'm using syslog-ng 3.16.1 with wildcard-file as source (let's call it "syslog-agent), which sends log messages to another syslog-ng acting as a relay.
I've noticed that syslog-agent instances RAM consumption keeps increasing until they leave no free memory in the cluster (each server has 64G of RAM). In my use case, new folders & files are created constantly under base-dir folder, but every 2 days obsolete folders & files are deleted. I assumed that syslog-ng would free some RAM every time those folders & files are deleted, but it doesn't, not even if I run a syslog-ng-ctl reload operation.
log-fifo-size value is 345000, so I assume it can't be a buffer situation since 345000 messages can't occupy 40-50 GB of memory.
I've performed the following test to reproduce the situation in small scale:
- Launch a syslog-agent with a wildcard-file source reading from "/tmp/test/" base-dir. syslog-agent RAM usage is about 125M. - Run a simple script to create complex folder hierarchy under /tmp/test and some files with 5000 log messages to read from. - Wait until syslog-agent RAM usage gets 1GB - Stop script execution and wait until syslog-agent has send all logs to relay. - Delete everything under /tmp/test and execute syslog-ng-ctl reload operation - 24 hours after, syslog-agent RAM usage still is 1GB
I've used heaptrack tool as a try to find a memory leak in syslog-ng, you can see in the attached image that iv_list_empty function in iv_list.h file is where most of the RAM usage is.
How do I get syslog-ng to free RAM? Or is it a memory leak?
Thanks in advance.
--
| Jose Angel Santiago
[image: Logo_signature2.png] <http://www.stratio.com/>
Vía de las dos Castillas, 33, Ática 4, 3ª Planta
28224 Pozuelo de Alarcón, Madrid, Spain
+34 918 286 473 | www.stratio.com <https://twitter.com/stratiobd> <https://www.linkedin.com/company/stratiobd> <https://www.youtube.com/c/StratioBD> ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
--
| Jose Angel Santiago
[image: Logo_signature2.png] <http://www.stratio.com/>
Vía de las dos Castillas, 33, Ática 4, 3ª Planta
28224 Pozuelo de Alarcón, Madrid, Spain
+34 918 286 473 | www.stratio.com <https://twitter.com/stratiobd> <https://www.linkedin.com/company/stratiobd> <https://www.youtube.com/c/StratioBD>
-- | Jose Angel Santiago [image: Logo_signature2.png] <http://www.stratio.com/> Vía de las dos Castillas, 33, Ática 4, 3ª Planta 28224 Pozuelo de Alarcón, Madrid, Spain +34 918 286 473 | www.stratio.com <https://twitter.com/stratiobd> <https://www.linkedin.com/company/stratiobd> <https://www.youtube.com/c/StratioBD>
Hi, The same free policy cannot work with _handle_file_deleted, because we cannot free the readers immediately. There is a race between file delete and syslog-ng reading the data. It can happen that one deletes a file before syslog-ng reads all its contents, which would cause message loss. In that case we just signal the deletion event to the reader (log_pipe_notify(&reader->super, NC_FILE_DELETED, NULL); and when the reader reads EOF at a later time, that's when the reader object is freed. Can you send the output of pmap <pid of syslog-ng> with the leak? That may give us a clue where to look with the leak. Also can you send your configuration, or at least the part about the wildcard filesources? The reproduction might depend on max_files or monitor_method parameter, or maybe other. Thanks, Antal On Thu, Sep 6, 2018 at 12:00 PM Jose Angel Santiago <jasantiago@stratio.com> wrote:
Reviewing your commit I've found out that the method *_handle_file_deleted* seems not to free memory when a monitored file has been deleted, according to the changes you made in *_handler_handler_directory_deleted.* Could it be the cause of the memory leak?
Thanks.
2018-09-06 11:50 GMT+02:00 Jose Angel Santiago <jasantiago@stratio.com>:
Hi,
I was testing your fix in several environments before replying, unfortunately syslog-ng stills shows a huge RAM consumption in our use of case.
We have two wildcard-file sources defined in our syslog-ng conf file, both of them have the same base-dir reference, but one source has a filename-pattern and the other source has a different one (I think we can't use the same source and several filename-patterns because of the lack of regular expression support). We execute the deletion of obsolete files and folder under base-dir once every 2 days, and we create a lot of new folders and files constantly while using our platform, but the growth of these log files (the ones affected by the filename-pattern) has been pretty low in this test, about 10-20 msgs per second total.
This is some data of the fix test:
Syslog-ng was consuming 21,7G of RAM with 10600 files affected by the filename-pattern and 20000 folders under base-dir (not a single folder were deleted yet so this is the expected behaviour of syslog-ng, and we find it extremely unaffordable)
One hour later, right before the cleaning process was executed, syslog-ng was consuming 35G of RAM, and it didn't decrease after the cleaning, not even a little, with just 380 files affected by the filename-pattern and 2200 folders under base-dir.
There is no message in the syslog-agent queue and all the messages are received by the relay (which ingest messages in an elasticsearch)
For the record, we do noticed a small RAM usage decrease in a previous test in a lab environment when we deleted every folder & file under base-dir (from 870M to 550M).
It seems syslog-ng wildcard-source is not optimized when it has too handle thousands of folders and filename-patterns under base-dir. Have you ever test the wildcard-source with a use of case similar to ours?
Can I send you any more information which may help to find out how to optimize it? I wish I can help with the code but I have no idea of C language :-(
Thanks in advance.
2018-09-04 18:03 GMT+02:00 Nagy, Gábor <gabor.nagy@oneidentity.com>:
Hello,
Thanks for the thorough investigation! We've checked it and found a memory leak in directory monitoring.
I've pushed the fix to my fork and created a merge request about it: https://github.com/gaborznagy/syslog-ng/commits/fix-wildcard-memleak https://github.com/balabit/syslog-ng/pull/2261 Can you verify that this fixes your problem, please?
log-fifo-size value is 345000, so I assume it can't be a buffer situation since 345000 messages can't occupy 40-50 GB of memory. Well our default log-msg-size is 64kB so if all messages have a size of log-msg-size then it can be around ~21GB when syslog-ng is buffering.
Best Regards, Gabor
On Tue, Sep 4, 2018 at 10:50 AM Jose Angel Santiago < jasantiago@stratio.com> wrote:
Hi,
I'm using syslog-ng 3.16.1 with wildcard-file as source (let's call it "syslog-agent), which sends log messages to another syslog-ng acting as a relay.
I've noticed that syslog-agent instances RAM consumption keeps increasing until they leave no free memory in the cluster (each server has 64G of RAM). In my use case, new folders & files are created constantly under base-dir folder, but every 2 days obsolete folders & files are deleted. I assumed that syslog-ng would free some RAM every time those folders & files are deleted, but it doesn't, not even if I run a syslog-ng-ctl reload operation.
log-fifo-size value is 345000, so I assume it can't be a buffer situation since 345000 messages can't occupy 40-50 GB of memory.
I've performed the following test to reproduce the situation in small scale:
- Launch a syslog-agent with a wildcard-file source reading from "/tmp/test/" base-dir. syslog-agent RAM usage is about 125M. - Run a simple script to create complex folder hierarchy under /tmp/test and some files with 5000 log messages to read from. - Wait until syslog-agent RAM usage gets 1GB - Stop script execution and wait until syslog-agent has send all logs to relay. - Delete everything under /tmp/test and execute syslog-ng-ctl reload operation - 24 hours after, syslog-agent RAM usage still is 1GB
I've used heaptrack tool as a try to find a memory leak in syslog-ng, you can see in the attached image that iv_list_empty function in iv_list.h file is where most of the RAM usage is.
How do I get syslog-ng to free RAM? Or is it a memory leak?
Thanks in advance.
--
| Jose Angel Santiago
[image: Logo_signature2.png] <http://www.stratio.com/>
Vía de las dos Castillas, 33, Ática 4, 3ª Planta
28224 Pozuelo de Alarcón, Madrid, Spain
+34 918 286 473 <+34%20918%2028%2064%2073> | www.stratio.com <https://twitter.com/stratiobd> <https://www.linkedin.com/company/stratiobd> <https://www.youtube.com/c/StratioBD>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
--
| Jose Angel Santiago
[image: Logo_signature2.png] <http://www.stratio.com/>
Vía de las dos Castillas, 33, Ática 4, 3ª Planta
28224 Pozuelo de Alarcón, Madrid, Spain
+34 918 286 473 <+34%20918%2028%2064%2073> | www.stratio.com <https://twitter.com/stratiobd> <https://www.linkedin.com/company/stratiobd> <https://www.youtube.com/c/StratioBD>
--
| Jose Angel Santiago
[image: Logo_signature2.png] <http://www.stratio.com/>
Vía de las dos Castillas, 33, Ática 4, 3ª Planta
28224 Pozuelo de Alarcón, Madrid, Spain
+34 918 286 473 <+34%20918%2028%2064%2073> | www.stratio.com <https://twitter.com/stratiobd> <https://www.linkedin.com/company/stratiobd> <https://www.youtube.com/c/StratioBD>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, I had to restart syslog-ng process because it was affecting other services in the same host. I've checked another host with was running with the fix too, it has 23000 files monitored and 38700 folders under base-dir, and now it's starting to eat the RAM, the cleaning process will be executed in about an hour (I'm sorry I can't force to do it before) PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 96 root 20 0 *5.609g 4.274g* 0.006g R *101.3* *6.8 * 1263:46 syslog-ng Trace of pmap: 96: /syslog-ng/exec/sbin/syslog-ng -F --no-caps -f /etc/syslog-ng/syslog-ng.conf -p /var/syslog-ng/syslog-ng.pid -R /var/syslog-ng/syslog-ng.persist -c /var/syslog-ng/syslog-ng.ctl 0000000000400000 12K r-x-- syslog-ng 0000000000602000 4K r---- syslog-ng 0000000000603000 4K rw--- syslog-ng 00000000015bc000 5689360K rw--- [ anon ] 00007ff9c8000000 19588K rw--- [ anon ] 00007ff9c9321000 45948K ----- [ anon ] 00007ff9cc000000 4288K rw--- [ anon ] 00007ff9cc430000 61248K ----- [ anon ] 00007ff9d0000000 26844K rw--- [ anon ] 00007ff9d1a37000 38692K ----- [ anon ] 00007ff9d4000000 10280K rw--- [ anon ] 00007ff9d4a0a000 55256K ----- [ anon ] 00007ff9dc000000 64604K rw--- [ anon ] 00007ff9dff17000 932K ----- [ anon ] 00007ff9e0000000 65508K rw--- [ anon ] 00007ff9e3ff9000 28K ----- [ anon ] 00007ff9e8000000 4480K rw--- [ anon ] 00007ff9e8460000 61056K ----- [ anon ] 00007ff9f0000000 65488K rw--- [ anon ] 00007ff9f3ff4000 48K ----- [ anon ] 00007ff9f8000000 14716K rw--- [ anon ] 00007ff9f8e5f000 50820K ----- [ anon ] 00007ffa00000000 18252K rw--- [ anon ] 00007ffa011d3000 47284K ----- [ anon ] 00007ffa08000000 65536K rw--- [ anon ] 00007ffa0c000000 30460K rw--- [ anon ] 00007ffa0ddbf000 35076K ----- [ anon ] 00007ffa10000000 65484K rw--- [ anon ] 00007ffa13ff3000 52K ----- [ anon ] 00007ffa167fd000 4K ----- [ anon ] 00007ffa167fe000 8192K rw--- [ anon ] 00007ffa18000000 7920K rw--- [ anon ] 00007ffa187bc000 57616K ----- [ anon ] 00007ffa1cffa000 4K ----- [ anon ] 00007ffa1cffb000 8192K rw--- [ anon ] 00007ffa1f7ff000 4K ----- [ anon ] 00007ffa1f800000 8192K rw--- [ anon ] 00007ffa20000000 65536K rw--- [ anon ] 00007ffa24000000 14764K rw--- [ anon ] 00007ffa24e6b000 50772K ----- [ anon ] 00007ffa28000000 24348K rw--- [ anon ] 00007ffa297c7000 41188K ----- [ anon ] 00007ffa2c000000 40916K rw--- [ anon ] 00007ffa2e7f5000 24620K ----- [ anon ] 00007ffa30000000 17684K rw--- [ anon ] 00007ffa31145000 47852K ----- [ anon ] 00007ffa34000000 10988K rw--- [ anon ] 00007ffa34abb000 54548K ----- [ anon ] 00007ffa38000000 20464K rw--- [ anon ] 00007ffa393fc000 45072K ----- [ anon ] 00007ffa3c316000 4K ----- [ anon ] 00007ffa3c317000 8192K rw--- [ anon ] 00007ffa3ea78000 4K ----- [ anon ] 00007ffa3ea79000 8192K rw--- [ anon ] 00007ffa3fbee000 7360K rw-s- syslog-ng.persist 00007ffa4031e000 92K r-x-- libresolv-2.23.so 00007ffa40335000 2048K ----- libresolv-2.23.so 00007ffa40535000 4K r---- libresolv-2.23.so 00007ffa40536000 4K rw--- libresolv-2.23.so 00007ffa40537000 8K rw--- [ anon ] 00007ffa40539000 20K r-x-- libnss_dns-2.23.so 00007ffa4053e000 2048K ----- libnss_dns-2.23.so 00007ffa4073e000 4K r---- libnss_dns-2.23.so 00007ffa4073f000 4K rw--- libnss_dns-2.23.so 00007ffa40740000 3328K rw--- [ anon ] 00007ffa40ac0000 256K rw--- [ anon ] 00007ffa40b40000 512K rw--- [ anon ] 00007ffa40c00000 256K rw--- [ anon ] 00007ffa40c6c000 256K rw--- [ anon ] 00007ffa40d6c000 44K r-x-- libsyslogformat.so 00007ffa40d77000 2044K ----- libsyslogformat.so 00007ffa40f76000 4K r---- libsyslogformat.so 00007ffa40f77000 4K rw--- libsyslogformat.so 00007ffa40f78000 4612K rw--- [ anon ] 00007ffa413f9000 24K r-x-- _hashlib.x86_64-linux-gnu.so 00007ffa413ff000 2044K ----- _hashlib.x86_64-linux-gnu.so 00007ffa415fe000 4K r---- _hashlib.x86_64-linux-gnu.so 00007ffa415ff000 4K rw--- _hashlib.x86_64-linux-gnu.so 00007ffa41600000 256K rw--- [ anon ] 00007ffa41640000 48K r-x-- _json.x86_64-linux-gnu.so 00007ffa4164c000 2044K ----- _json.x86_64-linux-gnu.so 00007ffa4184b000 4K r---- _json.x86_64-linux-gnu.so 00007ffa4184c000 4K rw--- _json.x86_64-linux-gnu.so 00007ffa4184d000 84K r-x-- _ssl.x86_64-linux-gnu.so 00007ffa41862000 2044K ----- _ssl.x86_64-linux-gnu.so 00007ffa41a61000 4K r---- _ssl.x86_64-linux-gnu.so 00007ffa41a62000 16K rw--- _ssl.x86_64-linux-gnu.so 00007ffa41a66000 768K rw--- [ anon ] 00007ffa41b57000 256K rw--- [ anon ] 00007ffa41b97000 1056K r-x-- libm-2.23.so 00007ffa41c9f000 2044K ----- libm-2.23.so 00007ffa41e9e000 4K r---- libm-2.23.so 00007ffa41e9f000 4K rw--- libm-2.23.so 00007ffa41ea0000 8K r-x-- libutil-2.23.so 00007ffa41ea2000 2044K ----- libutil-2.23.so 00007ffa420a1000 4K r---- libutil-2.23.so 00007ffa420a2000 4K rw--- libutil-2.23.so 00007ffa420a3000 100K r-x-- libz.so.1.2.8 00007ffa420bc000 2044K ----- libz.so.1.2.8 00007ffa422bb000 4K r---- libz.so.1.2.8 00007ffa422bc000 4K rw--- libz.so.1.2.8 00007ffa422bd000 3016K r-x-- libpython2.7.so.1.0 00007ffa425af000 2048K ----- libpython2.7.so.1.0 00007ffa427af000 8K r---- libpython2.7.so.1.0 00007ffa427b1000 476K rw--- libpython2.7.so.1.0 00007ffa42828000 140K rw--- [ anon ] 00007ffa4284b000 72K r-x-- libmod-python.so 00007ffa4285d000 2044K ----- libmod-python.so 00007ffa42a5c000 4K r---- libmod-python.so 00007ffa42a5d000 4K rw--- libmod-python.so 00007ffa42a5e000 4K rw--- [ anon ] 00007ffa42a5f000 32K r-x-- libwrap.so.0.7.6 00007ffa42a67000 2044K ----- libwrap.so.0.7.6 00007ffa42c66000 4K r---- libwrap.so.0.7.6 00007ffa42c67000 4K rw--- libwrap.so.0.7.6 00007ffa42c68000 4K rw--- [ anon ] 00007ffa42c69000 88K r-x-- libnet.so.1.7.0 00007ffa42c7f000 2044K ----- libnet.so.1.7.0 00007ffa42e7e000 4K r---- libnet.so.1.7.0 00007ffa42e7f000 4K rw--- libnet.so.1.7.0 00007ffa42e80000 8K rw--- [ anon ] 00007ffa42e82000 132K r-x-- libafsocket.so 00007ffa42ea3000 2048K ----- libafsocket.so 00007ffa430a3000 4K r---- libafsocket.so 00007ffa430a4000 8K rw--- libafsocket.so 00007ffa430a6000 4K rw--- [ anon ] 00007ffa430a7000 108K r-x-- libaffile.so 00007ffa430c2000 2044K ----- libaffile.so 00007ffa432c1000 4K r---- libaffile.so 00007ffa432c2000 4K rw--- libaffile.so 00007ffa432c3000 4K rw--- [ anon ] 00007ffa432c4000 8K r-x-- libconfgen.so 00007ffa432c6000 2044K ----- libconfgen.so 00007ffa434c5000 4K r---- libconfgen.so 00007ffa434c6000 4K rw--- libconfgen.so 00007ffa434c7000 36K r-x-- libbasicfuncs.so 00007ffa434d0000 2048K ----- libbasicfuncs.so 00007ffa436d0000 4K r---- libbasicfuncs.so 00007ffa436d1000 8K rw--- libbasicfuncs.so 00007ffa436d3000 32K r-x-- libappmodel.so 00007ffa436db000 2048K ----- libappmodel.so 00007ffa438db000 4K r---- libappmodel.so 00007ffa438dc000 4K rw--- libappmodel.so 00007ffa438dd000 40K r-x-- libjson-c.so.2.0.0 00007ffa438e7000 2044K ----- libjson-c.so.2.0.0 00007ffa43ae6000 4K r---- libjson-c.so.2.0.0 00007ffa43ae7000 4K rw--- libjson-c.so.2.0.0 00007ffa43ae8000 40K r-x-- libjson-plugin.so 00007ffa43af2000 2048K ----- libjson-plugin.so 00007ffa43cf2000 4K r---- libjson-plugin.so 00007ffa43cf3000 4K rw--- libjson-plugin.so 00007ffa43cf4000 44K r-x-- libnss_nis-2.23.so 00007ffa43cff000 2044K ----- libnss_nis-2.23.so 00007ffa43efe000 4K r---- libnss_nis-2.23.so 00007ffa43eff000 4K rw--- libnss_nis-2.23.so 00007ffa43f00000 88K r-x-- libnsl-2.23.so 00007ffa43f16000 2044K ----- libnsl-2.23.so 00007ffa44115000 4K r---- libnsl-2.23.so 00007ffa44116000 4K rw--- libnsl-2.23.so 00007ffa44117000 8K rw--- [ anon ] 00007ffa44119000 32K r-x-- libnss_compat-2.23.so 00007ffa44121000 2044K ----- libnss_compat-2.23.so 00007ffa44320000 4K r---- libnss_compat-2.23.so 00007ffa44321000 4K rw--- libnss_compat-2.23.so 00007ffa44322000 44K r-x-- libnss_files-2.23.so 00007ffa4432d000 2044K ----- libnss_files-2.23.so 00007ffa4452c000 4K r---- libnss_files-2.23.so 00007ffa4452d000 4K rw--- libnss_files-2.23.so 00007ffa4452e000 24K rw--- [ anon ] 00007ffa44534000 72K r-x-- libgpg-error.so.0.17.0 00007ffa44546000 2048K ----- libgpg-error.so.0.17.0 00007ffa44746000 4K r---- libgpg-error.so.0.17.0 00007ffa44747000 4K rw--- libgpg-error.so.0.17.0 00007ffa44748000 864K r-x-- libgcrypt.so.20.0.5 00007ffa44820000 2044K ----- libgcrypt.so.20.0.5 00007ffa44a1f000 4K r---- libgcrypt.so.20.0.5 00007ffa44a20000 32K rw--- libgcrypt.so.20.0.5 00007ffa44a28000 4K rw--- [ anon ] 00007ffa44a29000 132K r-x-- liblzma.so.5.0.0 00007ffa44a4a000 2044K ----- liblzma.so.5.0.0 00007ffa44c49000 4K r---- liblzma.so.5.0.0 00007ffa44c4a000 4K rw--- liblzma.so.5.0.0 00007ffa44c4b000 124K r-x-- libselinux.so.1 00007ffa44c6a000 2044K ----- libselinux.so.1 00007ffa44e69000 4K r---- libselinux.so.1 00007ffa44e6a000 4K rw--- libselinux.so.1 00007ffa44e6b000 8K rw--- [ anon ] 00007ffa44e6d000 2152K r-x-- libcrypto.so.1.0.0 00007ffa45087000 2044K ----- libcrypto.so.1.0.0 00007ffa45286000 112K r---- libcrypto.so.1.0.0 00007ffa452a2000 48K rw--- libcrypto.so.1.0.0 00007ffa452ae000 12K rw--- [ anon ] 00007ffa452b1000 376K r-x-- libssl.so.1.0.0 00007ffa4530f000 2048K ----- libssl.so.1.0.0 00007ffa4550f000 16K r---- libssl.so.1.0.0 00007ffa45513000 28K rw--- libssl.so.1.0.0 00007ffa4551a000 12K r-x-- libsecret-storage.so.0.0.0 00007ffa4551d000 2044K ----- libsecret-storage.so.0.0.0 00007ffa4571c000 4K r---- libsecret-storage.so.0.0.0 00007ffa4571d000 4K rw--- libsecret-storage.so.0.0.0 00007ffa4571e000 12K r-x-- libdl-2.23.so 00007ffa45721000 2044K ----- libdl-2.23.so 00007ffa45920000 4K r---- libdl-2.23.so 00007ffa45921000 4K rw--- libdl-2.23.so 00007ffa45922000 440K r-x-- libpcre.so.3.13.2 00007ffa45990000 2048K ----- libpcre.so.3.13.2 00007ffa45b90000 4K r---- libpcre.so.3.13.2 00007ffa45b91000 4K rw--- libpcre.so.3.13.2 00007ffa45b92000 16K r-x-- libcap.so.2.24 00007ffa45b96000 2048K ----- libcap.so.2.24 00007ffa45d96000 4K r---- libcap.so.2.24 00007ffa45d97000 4K rw--- libcap.so.2.24 00007ffa45d98000 12K r-x-- libevtlog-3.17.so.0.0.0 00007ffa45d9b000 2048K ----- libevtlog-3.17.so.0.0.0 00007ffa45f9b000 4K r---- libevtlog-3.17.so.0.0.0 00007ffa45f9c000 4K rw--- libevtlog-3.17.so.0.0.0 00007ffa45f9d000 4K r-x-- libgthread-2.0.so.0.4800.2 00007ffa45f9e000 2044K ----- libgthread-2.0.so.0.4800.2 00007ffa4619d000 4K r---- libgthread-2.0.so.0.4800.2 00007ffa4619e000 4K rw--- libgthread-2.0.so.0.4800.2 00007ffa4619f000 12K r-x-- libgmodule-2.0.so.0.4800.2 00007ffa461a2000 2044K ----- libgmodule-2.0.so.0.4800.2 00007ffa463a1000 4K r---- libgmodule-2.0.so.0.4800.2 00007ffa463a2000 4K rw--- libgmodule-2.0.so.0.4800.2 00007ffa463a3000 28K r-x-- librt-2.23.so 00007ffa463aa000 2044K ----- librt-2.23.so 00007ffa465a9000 4K r---- librt-2.23.so 00007ffa465aa000 4K rw--- librt-2.23.so 00007ffa465ab000 1792K r-x-- libc-2.23.so 00007ffa4676b000 2044K ----- libc-2.23.so 00007ffa4696a000 16K r---- libc-2.23.so 00007ffa4696e000 8K rw--- libc-2.23.so 00007ffa46970000 16K rw--- [ anon ] 00007ffa46974000 96K r-x-- libpthread-2.23.so 00007ffa4698c000 2044K ----- libpthread-2.23.so 00007ffa46b8b000 4K r---- libpthread-2.23.so 00007ffa46b8c000 4K rw--- libpthread-2.23.so 00007ffa46b8d000 16K rw--- [ anon ] 00007ffa46b91000 1084K r-x-- libglib-2.0.so.0.4800.2 00007ffa46ca0000 2044K ----- libglib-2.0.so.0.4800.2 00007ffa46e9f000 4K r---- libglib-2.0.so.0.4800.2 00007ffa46ea0000 4K rw--- libglib-2.0.so.0.4800.2 00007ffa46ea1000 4K rw--- [ anon ] 00007ffa46ea2000 768K r-x-- libsyslog-ng-3.17.so.0.0.0 00007ffa46f62000 2044K ----- libsyslog-ng-3.17.so.0.0.0 00007ffa47161000 16K r---- libsyslog-ng-3.17.so.0.0.0 00007ffa47165000 24K rw--- libsyslog-ng-3.17.so.0.0.0 00007ffa4716b000 8K rw--- [ anon ] 00007ffa4716d000 152K r-x-- ld-2.23.so 00007ffa471ba000 256K rw--- [ anon ] 00007ffa472a2000 256K rw--- [ anon ] 00007ffa472e2000 64K rwx-- [ anon ] 00007ffa472f2000 28K r--s- gconv-modules.cache 00007ffa472f9000 36K rw--- [ anon ] 00007ffa47302000 512K r-x-- libsystemd.so.0.14.0 00007ffa47382000 12K r---- libsystemd.so.0.14.0 00007ffa47385000 4K rw--- libsystemd.so.0.14.0 00007ffa47386000 20K rw--- [ anon ] 00007ffa47390000 8K rw--- [ anon ] 00007ffa47392000 4K r---- ld-2.23.so 00007ffa47393000 4K rw--- ld-2.23.so 00007ffa47394000 4K rw--- [ anon ] 00007ffe81489000 132K rw--- [ stack ] 00007ffe81564000 8K r-x-- [ anon ] ffffffffff600000 4K r-x-- [ anon ] total 7226528K Configuration: @version: 3.16.1 @module mod-java @include "/etc/syslog-ng/scl.conf" options { chain_hostnames(off); use-dns(no); use-fqdn(no); owner("syslog-ng"); group("stratio"); perm(0640); stats_freq(0); bad-hostname("^gconfd$"); flush-lines(100); log-fifo-size(345000); }; ######################## # Filters ######################## # Here's come the filter options. With this rules, we can set which # message go where. filter f_dbg { level(debug); }; filter f_info { level(info); }; filter f_notice { level(notice); }; filter f_warn { level(warn); }; filter f_err { level(err); }; filter f_crit { level(crit .. emerg); }; filter f_debug { level(debug) and not facility(auth, authpriv, news, mail); }; filter f_error { level(err .. emerg) ; }; filter f_messages { level(info,notice,warn) and not facility(auth,authpriv,cron,daemon,mail,news); }; filter f_auth { facility(auth, authpriv) and not filter(f_debug); }; filter f_cron { facility(cron) and not filter(f_debug); }; filter f_daemon { facility(daemon) and not filter(f_debug); }; filter f_kern { facility(kern) and not filter(f_debug); }; filter f_lpr { facility(lpr) and not filter(f_debug); }; filter f_local { facility(local0, local1, local3, local4, local5, local6, local7) and not filter(f_debug); }; filter f_mail { facility(mail) and not filter(f_debug); }; filter f_news { facility(news) and not filter(f_debug); }; filter f_syslog3 { not facility(auth, authpriv, mail) and not filter(f_debug); }; filter f_user { facility(user) and not filter(f_debug); }; filter f_text_only { not match( ".*[\\x80-\\xFF].*" value("MESSAGE") ); }; ### # Sources source s_stdout_files { wildcard-file( base-dir("/var/lib/slaves/") filename-pattern("stdout") max-files(3400) follow-freq(0.5) monitor-method(poll) log-fetch-limit(100) log-iw-size(345000) flags(no-parse) recursive(yes) multi-line-mode(indented) ); }; source s_stderr_files { wildcard-file( base-dir("/var/lib/slaves/") filename-pattern("stderr") max-files(3400) follow-freq(0.5) monitor-method(poll) log-fetch-limit(100) log-iw-size(345000) flags(no-parse) recursive(yes) multi-line-mode(indented) ); }; Thanks. 2018-09-06 12:59 GMT+02:00 Nemes, Antal <antal.nemes@oneidentity.com>:
Hi,
The same free policy cannot work with _handle_file_deleted, because we cannot free the readers immediately. There is a race between file delete and syslog-ng reading the data. It can happen that one deletes a file before syslog-ng reads all its contents, which would cause message loss. In that case we just signal the deletion event to the reader (log_pipe_notify(&reader->super, NC_FILE_DELETED, NULL); and when the reader reads EOF at a later time, that's when the reader object is freed.
Can you send the output of pmap <pid of syslog-ng> with the leak? That may give us a clue where to look with the leak.
Also can you send your configuration, or at least the part about the wildcard filesources? The reproduction might depend on max_files or monitor_method parameter, or maybe other.
Thanks, Antal
On Thu, Sep 6, 2018 at 12:00 PM Jose Angel Santiago < jasantiago@stratio.com> wrote:
Reviewing your commit I've found out that the method *_handle_file_deleted* seems not to free memory when a monitored file has been deleted, according to the changes you made in *_handler_handler_directory_deleted.* Could it be the cause of the memory leak?
Thanks.
2018-09-06 11:50 GMT+02:00 Jose Angel Santiago <jasantiago@stratio.com>:
Hi,
I was testing your fix in several environments before replying, unfortunately syslog-ng stills shows a huge RAM consumption in our use of case.
We have two wildcard-file sources defined in our syslog-ng conf file, both of them have the same base-dir reference, but one source has a filename-pattern and the other source has a different one (I think we can't use the same source and several filename-patterns because of the lack of regular expression support). We execute the deletion of obsolete files and folder under base-dir once every 2 days, and we create a lot of new folders and files constantly while using our platform, but the growth of these log files (the ones affected by the filename-pattern) has been pretty low in this test, about 10-20 msgs per second total.
This is some data of the fix test:
Syslog-ng was consuming 21,7G of RAM with 10600 files affected by the filename-pattern and 20000 folders under base-dir (not a single folder were deleted yet so this is the expected behaviour of syslog-ng, and we find it extremely unaffordable)
One hour later, right before the cleaning process was executed, syslog-ng was consuming 35G of RAM, and it didn't decrease after the cleaning, not even a little, with just 380 files affected by the filename-pattern and 2200 folders under base-dir.
There is no message in the syslog-agent queue and all the messages are received by the relay (which ingest messages in an elasticsearch)
For the record, we do noticed a small RAM usage decrease in a previous test in a lab environment when we deleted every folder & file under base-dir (from 870M to 550M).
It seems syslog-ng wildcard-source is not optimized when it has too handle thousands of folders and filename-patterns under base-dir. Have you ever test the wildcard-source with a use of case similar to ours?
Can I send you any more information which may help to find out how to optimize it? I wish I can help with the code but I have no idea of C language :-(
Thanks in advance.
2018-09-04 18:03 GMT+02:00 Nagy, Gábor <gabor.nagy@oneidentity.com>:
Hello,
Thanks for the thorough investigation! We've checked it and found a memory leak in directory monitoring.
I've pushed the fix to my fork and created a merge request about it: https://github.com/gaborznagy/syslog-ng/commits/fix-wildcard-memleak https://github.com/balabit/syslog-ng/pull/2261 Can you verify that this fixes your problem, please?
log-fifo-size value is 345000, so I assume it can't be a buffer situation since 345000 messages can't occupy 40-50 GB of memory. Well our default log-msg-size is 64kB so if all messages have a size of log-msg-size then it can be around ~21GB when syslog-ng is buffering.
Best Regards, Gabor
On Tue, Sep 4, 2018 at 10:50 AM Jose Angel Santiago < jasantiago@stratio.com> wrote:
Hi,
I'm using syslog-ng 3.16.1 with wildcard-file as source (let's call it "syslog-agent), which sends log messages to another syslog-ng acting as a relay.
I've noticed that syslog-agent instances RAM consumption keeps increasing until they leave no free memory in the cluster (each server has 64G of RAM). In my use case, new folders & files are created constantly under base-dir folder, but every 2 days obsolete folders & files are deleted. I assumed that syslog-ng would free some RAM every time those folders & files are deleted, but it doesn't, not even if I run a syslog-ng-ctl reload operation.
log-fifo-size value is 345000, so I assume it can't be a buffer situation since 345000 messages can't occupy 40-50 GB of memory.
I've performed the following test to reproduce the situation in small scale:
- Launch a syslog-agent with a wildcard-file source reading from "/tmp/test/" base-dir. syslog-agent RAM usage is about 125M. - Run a simple script to create complex folder hierarchy under /tmp/test and some files with 5000 log messages to read from. - Wait until syslog-agent RAM usage gets 1GB - Stop script execution and wait until syslog-agent has send all logs to relay. - Delete everything under /tmp/test and execute syslog-ng-ctl reload operation - 24 hours after, syslog-agent RAM usage still is 1GB
I've used heaptrack tool as a try to find a memory leak in syslog-ng, you can see in the attached image that iv_list_empty function in iv_list.h file is where most of the RAM usage is.
How do I get syslog-ng to free RAM? Or is it a memory leak?
Thanks in advance.
--
| Jose Angel Santiago
[image: Logo_signature2.png] <http://www.stratio.com/>
Vía de las dos Castillas, 33, Ática 4, 3ª Planta
28224 Pozuelo de Alarcón, Madrid, Spain
+34 918 286 473 <+34%20918%2028%2064%2073> | www.stratio.com <https://twitter.com/stratiobd> <https://www.linkedin.com/company/stratiobd> <https://www.youtube.com/c/StratioBD> ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
--
| Jose Angel Santiago
[image: Logo_signature2.png] <http://www.stratio.com/>
Vía de las dos Castillas, 33, Ática 4, 3ª Planta
28224 Pozuelo de Alarcón, Madrid, Spain
+34 918 286 473 <+34%20918%2028%2064%2073> | www.stratio.com <https://twitter.com/stratiobd> <https://www.linkedin.com/company/stratiobd> <https://www.youtube.com/c/StratioBD>
--
| Jose Angel Santiago
[image: Logo_signature2.png] <http://www.stratio.com/>
Vía de las dos Castillas, 33, Ática 4, 3ª Planta
28224 Pozuelo de Alarcón, Madrid, Spain
+34 918 286 473 <+34%20918%2028%2064%2073> | www.stratio.com <https://twitter.com/stratiobd> <https://www.linkedin.com/company/stratiobd> <https://www.youtube.com/c/StratioBD> ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- | Jose Angel Santiago [image: Logo_signature2.png] <http://www.stratio.com/> Vía de las dos Castillas, 33, Ática 4, 3ª Planta 28224 Pozuelo de Alarcón, Madrid, Spain +34 918 286 473 | www.stratio.com <https://twitter.com/stratiobd> <https://www.linkedin.com/company/stratiobd> <https://www.youtube.com/c/StratioBD>
Hi, We found another leak related to monitor-method(poll). When the tracked directory is deleted, glib allocates a GError structure for "Error opening directory \'...\': No such file or directory", that was not freed by syslog-ng. This can cause considerable memory leak when high number of directories tracked and deleted. We created a pull request to fix it: here you can track the progress: https://github.com/balabit/syslog-ng/pull/2267 Br, Antal On Thu, Sep 6, 2018 at 1:15 PM Jose Angel Santiago <jasantiago@stratio.com> wrote:
Hi,
I had to restart syslog-ng process because it was affecting other services in the same host.
I've checked another host with was running with the fix too, it has 23000 files monitored and 38700 folders under base-dir, and now it's starting to eat the RAM, the cleaning process will be executed in about an hour (I'm sorry I can't force to do it before)
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 96 root 20 0 *5.609g 4.274g* 0.006g R *101.3* *6.8 * 1263:46 syslog-ng
Trace of pmap:
96: /syslog-ng/exec/sbin/syslog-ng -F --no-caps -f /etc/syslog-ng/syslog-ng.conf -p /var/syslog-ng/syslog-ng.pid -R /var/syslog-ng/syslog-ng.persist -c /var/syslog-ng/syslog-ng.ctl 0000000000400000 12K r-x-- syslog-ng 0000000000602000 4K r---- syslog-ng 0000000000603000 4K rw--- syslog-ng 00000000015bc000 5689360K rw--- [ anon ] 00007ff9c8000000 19588K rw--- [ anon ] 00007ff9c9321000 45948K ----- [ anon ] 00007ff9cc000000 4288K rw--- [ anon ] 00007ff9cc430000 61248K ----- [ anon ] 00007ff9d0000000 26844K rw--- [ anon ] 00007ff9d1a37000 38692K ----- [ anon ] 00007ff9d4000000 10280K rw--- [ anon ] 00007ff9d4a0a000 55256K ----- [ anon ] 00007ff9dc000000 64604K rw--- [ anon ] 00007ff9dff17000 932K ----- [ anon ] 00007ff9e0000000 65508K rw--- [ anon ] 00007ff9e3ff9000 28K ----- [ anon ] 00007ff9e8000000 4480K rw--- [ anon ] 00007ff9e8460000 61056K ----- [ anon ] 00007ff9f0000000 65488K rw--- [ anon ] 00007ff9f3ff4000 48K ----- [ anon ] 00007ff9f8000000 14716K rw--- [ anon ] 00007ff9f8e5f000 50820K ----- [ anon ] 00007ffa00000000 18252K rw--- [ anon ] 00007ffa011d3000 47284K ----- [ anon ] 00007ffa08000000 65536K rw--- [ anon ] 00007ffa0c000000 30460K rw--- [ anon ] 00007ffa0ddbf000 35076K ----- [ anon ] 00007ffa10000000 65484K rw--- [ anon ] 00007ffa13ff3000 52K ----- [ anon ] 00007ffa167fd000 4K ----- [ anon ] 00007ffa167fe000 8192K rw--- [ anon ] 00007ffa18000000 7920K rw--- [ anon ] 00007ffa187bc000 57616K ----- [ anon ] 00007ffa1cffa000 4K ----- [ anon ] 00007ffa1cffb000 8192K rw--- [ anon ] 00007ffa1f7ff000 4K ----- [ anon ] 00007ffa1f800000 8192K rw--- [ anon ] 00007ffa20000000 65536K rw--- [ anon ] 00007ffa24000000 14764K rw--- [ anon ] 00007ffa24e6b000 50772K ----- [ anon ] 00007ffa28000000 24348K rw--- [ anon ] 00007ffa297c7000 41188K ----- [ anon ] 00007ffa2c000000 40916K rw--- [ anon ] 00007ffa2e7f5000 24620K ----- [ anon ] 00007ffa30000000 17684K rw--- [ anon ] 00007ffa31145000 47852K ----- [ anon ] 00007ffa34000000 10988K rw--- [ anon ] 00007ffa34abb000 54548K ----- [ anon ] 00007ffa38000000 20464K rw--- [ anon ] 00007ffa393fc000 45072K ----- [ anon ] 00007ffa3c316000 4K ----- [ anon ] 00007ffa3c317000 8192K rw--- [ anon ] 00007ffa3ea78000 4K ----- [ anon ] 00007ffa3ea79000 8192K rw--- [ anon ] 00007ffa3fbee000 7360K rw-s- syslog-ng.persist 00007ffa4031e000 92K r-x-- libresolv-2.23.so 00007ffa40335000 2048K ----- libresolv-2.23.so 00007ffa40535000 4K r---- libresolv-2.23.so 00007ffa40536000 4K rw--- libresolv-2.23.so 00007ffa40537000 8K rw--- [ anon ] 00007ffa40539000 20K r-x-- libnss_dns-2.23.so 00007ffa4053e000 2048K ----- libnss_dns-2.23.so 00007ffa4073e000 4K r---- libnss_dns-2.23.so 00007ffa4073f000 4K rw--- libnss_dns-2.23.so 00007ffa40740000 3328K rw--- [ anon ] 00007ffa40ac0000 256K rw--- [ anon ] 00007ffa40b40000 512K rw--- [ anon ] 00007ffa40c00000 256K rw--- [ anon ] 00007ffa40c6c000 256K rw--- [ anon ] 00007ffa40d6c000 44K r-x-- libsyslogformat.so 00007ffa40d77000 2044K ----- libsyslogformat.so 00007ffa40f76000 4K r---- libsyslogformat.so 00007ffa40f77000 4K rw--- libsyslogformat.so 00007ffa40f78000 4612K rw--- [ anon ] 00007ffa413f9000 24K r-x-- _hashlib.x86_64-linux-gnu.so 00007ffa413ff000 2044K ----- _hashlib.x86_64-linux-gnu.so 00007ffa415fe000 4K r---- _hashlib.x86_64-linux-gnu.so 00007ffa415ff000 4K rw--- _hashlib.x86_64-linux-gnu.so 00007ffa41600000 256K rw--- [ anon ] 00007ffa41640000 48K r-x-- _json.x86_64-linux-gnu.so 00007ffa4164c000 2044K ----- _json.x86_64-linux-gnu.so 00007ffa4184b000 4K r---- _json.x86_64-linux-gnu.so 00007ffa4184c000 4K rw--- _json.x86_64-linux-gnu.so 00007ffa4184d000 84K r-x-- _ssl.x86_64-linux-gnu.so 00007ffa41862000 2044K ----- _ssl.x86_64-linux-gnu.so 00007ffa41a61000 4K r---- _ssl.x86_64-linux-gnu.so 00007ffa41a62000 16K rw--- _ssl.x86_64-linux-gnu.so 00007ffa41a66000 768K rw--- [ anon ] 00007ffa41b57000 256K rw--- [ anon ] 00007ffa41b97000 1056K r-x-- libm-2.23.so 00007ffa41c9f000 2044K ----- libm-2.23.so 00007ffa41e9e000 4K r---- libm-2.23.so 00007ffa41e9f000 4K rw--- libm-2.23.so 00007ffa41ea0000 8K r-x-- libutil-2.23.so 00007ffa41ea2000 2044K ----- libutil-2.23.so 00007ffa420a1000 4K r---- libutil-2.23.so 00007ffa420a2000 4K rw--- libutil-2.23.so 00007ffa420a3000 100K r-x-- libz.so.1.2.8 00007ffa420bc000 2044K ----- libz.so.1.2.8 00007ffa422bb000 4K r---- libz.so.1.2.8 00007ffa422bc000 4K rw--- libz.so.1.2.8 00007ffa422bd000 3016K r-x-- libpython2.7.so.1.0 00007ffa425af000 2048K ----- libpython2.7.so.1.0 00007ffa427af000 8K r---- libpython2.7.so.1.0 00007ffa427b1000 476K rw--- libpython2.7.so.1.0 00007ffa42828000 140K rw--- [ anon ] 00007ffa4284b000 72K r-x-- libmod-python.so 00007ffa4285d000 2044K ----- libmod-python.so 00007ffa42a5c000 4K r---- libmod-python.so 00007ffa42a5d000 4K rw--- libmod-python.so 00007ffa42a5e000 4K rw--- [ anon ] 00007ffa42a5f000 32K r-x-- libwrap.so.0.7.6 00007ffa42a67000 2044K ----- libwrap.so.0.7.6 00007ffa42c66000 4K r---- libwrap.so.0.7.6 00007ffa42c67000 4K rw--- libwrap.so.0.7.6 00007ffa42c68000 4K rw--- [ anon ] 00007ffa42c69000 88K r-x-- libnet.so.1.7.0 00007ffa42c7f000 2044K ----- libnet.so.1.7.0 00007ffa42e7e000 4K r---- libnet.so.1.7.0 00007ffa42e7f000 4K rw--- libnet.so.1.7.0 00007ffa42e80000 8K rw--- [ anon ] 00007ffa42e82000 132K r-x-- libafsocket.so 00007ffa42ea3000 2048K ----- libafsocket.so 00007ffa430a3000 4K r---- libafsocket.so 00007ffa430a4000 8K rw--- libafsocket.so 00007ffa430a6000 4K rw--- [ anon ] 00007ffa430a7000 108K r-x-- libaffile.so 00007ffa430c2000 2044K ----- libaffile.so 00007ffa432c1000 4K r---- libaffile.so 00007ffa432c2000 4K rw--- libaffile.so 00007ffa432c3000 4K rw--- [ anon ] 00007ffa432c4000 8K r-x-- libconfgen.so 00007ffa432c6000 2044K ----- libconfgen.so 00007ffa434c5000 4K r---- libconfgen.so 00007ffa434c6000 4K rw--- libconfgen.so 00007ffa434c7000 36K r-x-- libbasicfuncs.so 00007ffa434d0000 2048K ----- libbasicfuncs.so 00007ffa436d0000 4K r---- libbasicfuncs.so 00007ffa436d1000 8K rw--- libbasicfuncs.so 00007ffa436d3000 32K r-x-- libappmodel.so 00007ffa436db000 2048K ----- libappmodel.so 00007ffa438db000 4K r---- libappmodel.so 00007ffa438dc000 4K rw--- libappmodel.so 00007ffa438dd000 40K r-x-- libjson-c.so.2.0.0 00007ffa438e7000 2044K ----- libjson-c.so.2.0.0 00007ffa43ae6000 4K r---- libjson-c.so.2.0.0 00007ffa43ae7000 4K rw--- libjson-c.so.2.0.0 00007ffa43ae8000 40K r-x-- libjson-plugin.so 00007ffa43af2000 2048K ----- libjson-plugin.so 00007ffa43cf2000 4K r---- libjson-plugin.so 00007ffa43cf3000 4K rw--- libjson-plugin.so 00007ffa43cf4000 44K r-x-- libnss_nis-2.23.so 00007ffa43cff000 2044K ----- libnss_nis-2.23.so 00007ffa43efe000 4K r---- libnss_nis-2.23.so 00007ffa43eff000 4K rw--- libnss_nis-2.23.so 00007ffa43f00000 88K r-x-- libnsl-2.23.so 00007ffa43f16000 2044K ----- libnsl-2.23.so 00007ffa44115000 4K r---- libnsl-2.23.so 00007ffa44116000 4K rw--- libnsl-2.23.so 00007ffa44117000 8K rw--- [ anon ] 00007ffa44119000 32K r-x-- libnss_compat-2.23.so 00007ffa44121000 2044K ----- libnss_compat-2.23.so 00007ffa44320000 4K r---- libnss_compat-2.23.so 00007ffa44321000 4K rw--- libnss_compat-2.23.so 00007ffa44322000 44K r-x-- libnss_files-2.23.so 00007ffa4432d000 2044K ----- libnss_files-2.23.so 00007ffa4452c000 4K r---- libnss_files-2.23.so 00007ffa4452d000 4K rw--- libnss_files-2.23.so 00007ffa4452e000 24K rw--- [ anon ] 00007ffa44534000 72K r-x-- libgpg-error.so.0.17.0 00007ffa44546000 2048K ----- libgpg-error.so.0.17.0 00007ffa44746000 4K r---- libgpg-error.so.0.17.0 00007ffa44747000 4K rw--- libgpg-error.so.0.17.0 00007ffa44748000 864K r-x-- libgcrypt.so.20.0.5 00007ffa44820000 2044K ----- libgcrypt.so.20.0.5 00007ffa44a1f000 4K r---- libgcrypt.so.20.0.5 00007ffa44a20000 32K rw--- libgcrypt.so.20.0.5 00007ffa44a28000 4K rw--- [ anon ] 00007ffa44a29000 132K r-x-- liblzma.so.5.0.0 00007ffa44a4a000 2044K ----- liblzma.so.5.0.0 00007ffa44c49000 4K r---- liblzma.so.5.0.0 00007ffa44c4a000 4K rw--- liblzma.so.5.0.0 00007ffa44c4b000 124K r-x-- libselinux.so.1 00007ffa44c6a000 2044K ----- libselinux.so.1 00007ffa44e69000 4K r---- libselinux.so.1 00007ffa44e6a000 4K rw--- libselinux.so.1 00007ffa44e6b000 8K rw--- [ anon ] 00007ffa44e6d000 2152K r-x-- libcrypto.so.1.0.0 00007ffa45087000 2044K ----- libcrypto.so.1.0.0 00007ffa45286000 112K r---- libcrypto.so.1.0.0 00007ffa452a2000 48K rw--- libcrypto.so.1.0.0 00007ffa452ae000 12K rw--- [ anon ] 00007ffa452b1000 376K r-x-- libssl.so.1.0.0 00007ffa4530f000 2048K ----- libssl.so.1.0.0 00007ffa4550f000 16K r---- libssl.so.1.0.0 00007ffa45513000 28K rw--- libssl.so.1.0.0 00007ffa4551a000 12K r-x-- libsecret-storage.so.0.0.0 00007ffa4551d000 2044K ----- libsecret-storage.so.0.0.0 00007ffa4571c000 4K r---- libsecret-storage.so.0.0.0 00007ffa4571d000 4K rw--- libsecret-storage.so.0.0.0 00007ffa4571e000 12K r-x-- libdl-2.23.so 00007ffa45721000 2044K ----- libdl-2.23.so 00007ffa45920000 4K r---- libdl-2.23.so 00007ffa45921000 4K rw--- libdl-2.23.so 00007ffa45922000 440K r-x-- libpcre.so.3.13.2 00007ffa45990000 2048K ----- libpcre.so.3.13.2 00007ffa45b90000 4K r---- libpcre.so.3.13.2 00007ffa45b91000 4K rw--- libpcre.so.3.13.2 00007ffa45b92000 16K r-x-- libcap.so.2.24 00007ffa45b96000 2048K ----- libcap.so.2.24 00007ffa45d96000 4K r---- libcap.so.2.24 00007ffa45d97000 4K rw--- libcap.so.2.24 00007ffa45d98000 12K r-x-- libevtlog-3.17.so.0.0.0 00007ffa45d9b000 2048K ----- libevtlog-3.17.so.0.0.0 00007ffa45f9b000 4K r---- libevtlog-3.17.so.0.0.0 00007ffa45f9c000 4K rw--- libevtlog-3.17.so.0.0.0 00007ffa45f9d000 4K r-x-- libgthread-2.0.so.0.4800.2 00007ffa45f9e000 2044K ----- libgthread-2.0.so.0.4800.2 00007ffa4619d000 4K r---- libgthread-2.0.so.0.4800.2 00007ffa4619e000 4K rw--- libgthread-2.0.so.0.4800.2 00007ffa4619f000 12K r-x-- libgmodule-2.0.so.0.4800.2 00007ffa461a2000 2044K ----- libgmodule-2.0.so.0.4800.2 00007ffa463a1000 4K r---- libgmodule-2.0.so.0.4800.2 00007ffa463a2000 4K rw--- libgmodule-2.0.so.0.4800.2 00007ffa463a3000 28K r-x-- librt-2.23.so 00007ffa463aa000 2044K ----- librt-2.23.so 00007ffa465a9000 4K r---- librt-2.23.so 00007ffa465aa000 4K rw--- librt-2.23.so 00007ffa465ab000 1792K r-x-- libc-2.23.so 00007ffa4676b000 2044K ----- libc-2.23.so 00007ffa4696a000 16K r---- libc-2.23.so 00007ffa4696e000 8K rw--- libc-2.23.so 00007ffa46970000 16K rw--- [ anon ] 00007ffa46974000 96K r-x-- libpthread-2.23.so 00007ffa4698c000 2044K ----- libpthread-2.23.so 00007ffa46b8b000 4K r---- libpthread-2.23.so 00007ffa46b8c000 4K rw--- libpthread-2.23.so 00007ffa46b8d000 16K rw--- [ anon ] 00007ffa46b91000 1084K r-x-- libglib-2.0.so.0.4800.2 00007ffa46ca0000 2044K ----- libglib-2.0.so.0.4800.2 00007ffa46e9f000 4K r---- libglib-2.0.so.0.4800.2 00007ffa46ea0000 4K rw--- libglib-2.0.so.0.4800.2 00007ffa46ea1000 4K rw--- [ anon ] 00007ffa46ea2000 768K r-x-- libsyslog-ng-3.17.so.0.0.0 00007ffa46f62000 2044K ----- libsyslog-ng-3.17.so.0.0.0 00007ffa47161000 16K r---- libsyslog-ng-3.17.so.0.0.0 00007ffa47165000 24K rw--- libsyslog-ng-3.17.so.0.0.0 00007ffa4716b000 8K rw--- [ anon ] 00007ffa4716d000 152K r-x-- ld-2.23.so 00007ffa471ba000 256K rw--- [ anon ] 00007ffa472a2000 256K rw--- [ anon ] 00007ffa472e2000 64K rwx-- [ anon ] 00007ffa472f2000 28K r--s- gconv-modules.cache 00007ffa472f9000 36K rw--- [ anon ] 00007ffa47302000 512K r-x-- libsystemd.so.0.14.0 00007ffa47382000 12K r---- libsystemd.so.0.14.0 00007ffa47385000 4K rw--- libsystemd.so.0.14.0 00007ffa47386000 20K rw--- [ anon ] 00007ffa47390000 8K rw--- [ anon ] 00007ffa47392000 4K r---- ld-2.23.so 00007ffa47393000 4K rw--- ld-2.23.so 00007ffa47394000 4K rw--- [ anon ] 00007ffe81489000 132K rw--- [ stack ] 00007ffe81564000 8K r-x-- [ anon ] ffffffffff600000 4K r-x-- [ anon ] total 7226528K
Configuration:
@version: 3.16.1 @module mod-java @include "/etc/syslog-ng/scl.conf"
options { chain_hostnames(off); use-dns(no); use-fqdn(no); owner("syslog-ng"); group("stratio"); perm(0640); stats_freq(0); bad-hostname("^gconfd$"); flush-lines(100); log-fifo-size(345000); };
######################## # Filters ######################## # Here's come the filter options. With this rules, we can set which # message go where.
filter f_dbg { level(debug); }; filter f_info { level(info); }; filter f_notice { level(notice); }; filter f_warn { level(warn); }; filter f_err { level(err); }; filter f_crit { level(crit .. emerg); };
filter f_debug { level(debug) and not facility(auth, authpriv, news, mail); }; filter f_error { level(err .. emerg) ; }; filter f_messages { level(info,notice,warn) and not facility(auth,authpriv,cron,daemon,mail,news); };
filter f_auth { facility(auth, authpriv) and not filter(f_debug); }; filter f_cron { facility(cron) and not filter(f_debug); }; filter f_daemon { facility(daemon) and not filter(f_debug); }; filter f_kern { facility(kern) and not filter(f_debug); }; filter f_lpr { facility(lpr) and not filter(f_debug); }; filter f_local { facility(local0, local1, local3, local4, local5, local6, local7) and not filter(f_debug); }; filter f_mail { facility(mail) and not filter(f_debug); }; filter f_news { facility(news) and not filter(f_debug); }; filter f_syslog3 { not facility(auth, authpriv, mail) and not filter(f_debug); }; filter f_user { facility(user) and not filter(f_debug); };
filter f_text_only { not match( ".*[\\x80-\\xFF].*" value("MESSAGE") ); };
### # Sources
source s_stdout_files { wildcard-file( base-dir("/var/lib/slaves/") filename-pattern("stdout") max-files(3400) follow-freq(0.5) monitor-method(poll) log-fetch-limit(100) log-iw-size(345000) flags(no-parse) recursive(yes) multi-line-mode(indented) ); };
source s_stderr_files { wildcard-file( base-dir("/var/lib/slaves/") filename-pattern("stderr") max-files(3400) follow-freq(0.5) monitor-method(poll) log-fetch-limit(100) log-iw-size(345000) flags(no-parse) recursive(yes) multi-line-mode(indented) ); };
Thanks.
2018-09-06 12:59 GMT+02:00 Nemes, Antal <antal.nemes@oneidentity.com>:
Hi,
The same free policy cannot work with _handle_file_deleted, because we cannot free the readers immediately. There is a race between file delete and syslog-ng reading the data. It can happen that one deletes a file before syslog-ng reads all its contents, which would cause message loss. In that case we just signal the deletion event to the reader (log_pipe_notify(&reader->super, NC_FILE_DELETED, NULL); and when the reader reads EOF at a later time, that's when the reader object is freed.
Can you send the output of pmap <pid of syslog-ng> with the leak? That may give us a clue where to look with the leak.
Also can you send your configuration, or at least the part about the wildcard filesources? The reproduction might depend on max_files or monitor_method parameter, or maybe other.
Thanks, Antal
On Thu, Sep 6, 2018 at 12:00 PM Jose Angel Santiago < jasantiago@stratio.com> wrote:
Reviewing your commit I've found out that the method *_handle_file_deleted* seems not to free memory when a monitored file has been deleted, according to the changes you made in *_handler_handler_directory_deleted.* Could it be the cause of the memory leak?
Thanks.
2018-09-06 11:50 GMT+02:00 Jose Angel Santiago <jasantiago@stratio.com>:
Hi,
I was testing your fix in several environments before replying, unfortunately syslog-ng stills shows a huge RAM consumption in our use of case.
We have two wildcard-file sources defined in our syslog-ng conf file, both of them have the same base-dir reference, but one source has a filename-pattern and the other source has a different one (I think we can't use the same source and several filename-patterns because of the lack of regular expression support). We execute the deletion of obsolete files and folder under base-dir once every 2 days, and we create a lot of new folders and files constantly while using our platform, but the growth of these log files (the ones affected by the filename-pattern) has been pretty low in this test, about 10-20 msgs per second total.
This is some data of the fix test:
Syslog-ng was consuming 21,7G of RAM with 10600 files affected by the filename-pattern and 20000 folders under base-dir (not a single folder were deleted yet so this is the expected behaviour of syslog-ng, and we find it extremely unaffordable)
One hour later, right before the cleaning process was executed, syslog-ng was consuming 35G of RAM, and it didn't decrease after the cleaning, not even a little, with just 380 files affected by the filename-pattern and 2200 folders under base-dir.
There is no message in the syslog-agent queue and all the messages are received by the relay (which ingest messages in an elasticsearch)
For the record, we do noticed a small RAM usage decrease in a previous test in a lab environment when we deleted every folder & file under base-dir (from 870M to 550M).
It seems syslog-ng wildcard-source is not optimized when it has too handle thousands of folders and filename-patterns under base-dir. Have you ever test the wildcard-source with a use of case similar to ours?
Can I send you any more information which may help to find out how to optimize it? I wish I can help with the code but I have no idea of C language :-(
Thanks in advance.
2018-09-04 18:03 GMT+02:00 Nagy, Gábor <gabor.nagy@oneidentity.com>:
Hello,
Thanks for the thorough investigation! We've checked it and found a memory leak in directory monitoring.
I've pushed the fix to my fork and created a merge request about it: https://github.com/gaborznagy/syslog-ng/commits/fix-wildcard-memleak https://github.com/balabit/syslog-ng/pull/2261 Can you verify that this fixes your problem, please?
log-fifo-size value is 345000, so I assume it can't be a buffer situation since 345000 messages can't occupy 40-50 GB of memory. Well our default log-msg-size is 64kB so if all messages have a size of log-msg-size then it can be around ~21GB when syslog-ng is buffering.
Best Regards, Gabor
On Tue, Sep 4, 2018 at 10:50 AM Jose Angel Santiago < jasantiago@stratio.com> wrote:
Hi,
I'm using syslog-ng 3.16.1 with wildcard-file as source (let's call it "syslog-agent), which sends log messages to another syslog-ng acting as a relay.
I've noticed that syslog-agent instances RAM consumption keeps increasing until they leave no free memory in the cluster (each server has 64G of RAM). In my use case, new folders & files are created constantly under base-dir folder, but every 2 days obsolete folders & files are deleted. I assumed that syslog-ng would free some RAM every time those folders & files are deleted, but it doesn't, not even if I run a syslog-ng-ctl reload operation.
log-fifo-size value is 345000, so I assume it can't be a buffer situation since 345000 messages can't occupy 40-50 GB of memory.
I've performed the following test to reproduce the situation in small scale:
- Launch a syslog-agent with a wildcard-file source reading from "/tmp/test/" base-dir. syslog-agent RAM usage is about 125M. - Run a simple script to create complex folder hierarchy under /tmp/test and some files with 5000 log messages to read from. - Wait until syslog-agent RAM usage gets 1GB - Stop script execution and wait until syslog-agent has send all logs to relay. - Delete everything under /tmp/test and execute syslog-ng-ctl reload operation - 24 hours after, syslog-agent RAM usage still is 1GB
I've used heaptrack tool as a try to find a memory leak in syslog-ng, you can see in the attached image that iv_list_empty function in iv_list.h file is where most of the RAM usage is.
How do I get syslog-ng to free RAM? Or is it a memory leak?
Thanks in advance.
--
| Jose Angel Santiago
[image: Logo_signature2.png] <http://www.stratio.com/>
Vía de las dos Castillas, 33, Ática 4, 3ª Planta
28224 Pozuelo de Alarcón, Madrid, Spain
+34 918 286 473 <+34%20918%2028%2064%2073> | www.stratio.com <https://twitter.com/stratiobd> <https://www.linkedin.com/company/stratiobd> <https://www.youtube.com/c/StratioBD>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
--
| Jose Angel Santiago
[image: Logo_signature2.png] <http://www.stratio.com/>
Vía de las dos Castillas, 33, Ática 4, 3ª Planta
28224 Pozuelo de Alarcón, Madrid, Spain
+34 918 286 473 <+34%20918%2028%2064%2073> | www.stratio.com <https://twitter.com/stratiobd> <https://www.linkedin.com/company/stratiobd> <https://www.youtube.com/c/StratioBD>
--
| Jose Angel Santiago
[image: Logo_signature2.png] <http://www.stratio.com/>
Vía de las dos Castillas, 33, Ática 4, 3ª Planta
28224 Pozuelo de Alarcón, Madrid, Spain
+34 918 286 473 <+34%20918%2028%2064%2073> | www.stratio.com <https://twitter.com/stratiobd> <https://www.linkedin.com/company/stratiobd> <https://www.youtube.com/c/StratioBD>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
--
| Jose Angel Santiago
[image: Logo_signature2.png] <http://www.stratio.com/>
Vía de las dos Castillas, 33, Ática 4, 3ª Planta
28224 Pozuelo de Alarcón, Madrid, Spain
+34 918 286 473 <+34%20918%2028%2064%2073> | www.stratio.com <https://twitter.com/stratiobd> <https://www.linkedin.com/company/stratiobd> <https://www.youtube.com/c/StratioBD>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, We've tested these 2 fixes along with the previous one during this weekend in our demo environment and now the RAM usage seems to be pretty fair, so I guess fixes are working great! Thanks for your quick & effective response. 2018-09-06 16:45 GMT+02:00 Nemes, Antal <antal.nemes@oneidentity.com>:
Hi,
We found another leak related to monitor-method(poll). When the tracked directory is deleted, glib allocates a GError structure for "Error opening directory \'...\': No such file or directory", that was not freed by syslog-ng. This can cause considerable memory leak when high number of directories tracked and deleted. We created a pull request to fix it: here you can track the progress: https://github.com/balabit/syslog-ng/pull/2267
Br, Antal
On Thu, Sep 6, 2018 at 1:15 PM Jose Angel Santiago <jasantiago@stratio.com> wrote:
Hi,
I had to restart syslog-ng process because it was affecting other services in the same host.
I've checked another host with was running with the fix too, it has 23000 files monitored and 38700 folders under base-dir, and now it's starting to eat the RAM, the cleaning process will be executed in about an hour (I'm sorry I can't force to do it before)
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 96 root 20 0 *5.609g 4.274g* 0.006g R *101.3* *6.8 * 1263:46 syslog-ng
Trace of pmap:
96: /syslog-ng/exec/sbin/syslog-ng -F --no-caps -f /etc/syslog-ng/syslog-ng.conf -p /var/syslog-ng/syslog-ng.pid -R /var/syslog-ng/syslog-ng.persist -c /var/syslog-ng/syslog-ng.ctl 0000000000400000 12K r-x-- syslog-ng 0000000000602000 4K r---- syslog-ng 0000000000603000 4K rw--- syslog-ng 00000000015bc000 5689360K rw--- [ anon ] 00007ff9c8000000 19588K rw--- [ anon ] 00007ff9c9321000 45948K ----- [ anon ] 00007ff9cc000000 4288K rw--- [ anon ] 00007ff9cc430000 61248K ----- [ anon ] 00007ff9d0000000 26844K rw--- [ anon ] 00007ff9d1a37000 38692K ----- [ anon ] 00007ff9d4000000 10280K rw--- [ anon ] 00007ff9d4a0a000 55256K ----- [ anon ] 00007ff9dc000000 64604K rw--- [ anon ] 00007ff9dff17000 932K ----- [ anon ] 00007ff9e0000000 65508K rw--- [ anon ] 00007ff9e3ff9000 28K ----- [ anon ] 00007ff9e8000000 4480K rw--- [ anon ] 00007ff9e8460000 61056K ----- [ anon ] 00007ff9f0000000 65488K rw--- [ anon ] 00007ff9f3ff4000 48K ----- [ anon ] 00007ff9f8000000 14716K rw--- [ anon ] 00007ff9f8e5f000 50820K ----- [ anon ] 00007ffa00000000 18252K rw--- [ anon ] 00007ffa011d3000 47284K ----- [ anon ] 00007ffa08000000 65536K rw--- [ anon ] 00007ffa0c000000 30460K rw--- [ anon ] 00007ffa0ddbf000 35076K ----- [ anon ] 00007ffa10000000 65484K rw--- [ anon ] 00007ffa13ff3000 52K ----- [ anon ] 00007ffa167fd000 4K ----- [ anon ] 00007ffa167fe000 8192K rw--- [ anon ] 00007ffa18000000 7920K rw--- [ anon ] 00007ffa187bc000 57616K ----- [ anon ] 00007ffa1cffa000 4K ----- [ anon ] 00007ffa1cffb000 8192K rw--- [ anon ] 00007ffa1f7ff000 4K ----- [ anon ] 00007ffa1f800000 8192K rw--- [ anon ] 00007ffa20000000 65536K rw--- [ anon ] 00007ffa24000000 14764K rw--- [ anon ] 00007ffa24e6b000 50772K ----- [ anon ] 00007ffa28000000 24348K rw--- [ anon ] 00007ffa297c7000 41188K ----- [ anon ] 00007ffa2c000000 40916K rw--- [ anon ] 00007ffa2e7f5000 24620K ----- [ anon ] 00007ffa30000000 17684K rw--- [ anon ] 00007ffa31145000 47852K ----- [ anon ] 00007ffa34000000 10988K rw--- [ anon ] 00007ffa34abb000 54548K ----- [ anon ] 00007ffa38000000 20464K rw--- [ anon ] 00007ffa393fc000 45072K ----- [ anon ] 00007ffa3c316000 4K ----- [ anon ] 00007ffa3c317000 8192K rw--- [ anon ] 00007ffa3ea78000 4K ----- [ anon ] 00007ffa3ea79000 8192K rw--- [ anon ] 00007ffa3fbee000 7360K rw-s- syslog-ng.persist 00007ffa4031e000 92K r-x-- libresolv-2.23.so 00007ffa40335000 2048K ----- libresolv-2.23.so 00007ffa40535000 4K r---- libresolv-2.23.so 00007ffa40536000 4K rw--- libresolv-2.23.so 00007ffa40537000 8K rw--- [ anon ] 00007ffa40539000 20K r-x-- libnss_dns-2.23.so 00007ffa4053e000 2048K ----- libnss_dns-2.23.so 00007ffa4073e000 4K r---- libnss_dns-2.23.so 00007ffa4073f000 4K rw--- libnss_dns-2.23.so 00007ffa40740000 3328K rw--- [ anon ] 00007ffa40ac0000 256K rw--- [ anon ] 00007ffa40b40000 512K rw--- [ anon ] 00007ffa40c00000 256K rw--- [ anon ] 00007ffa40c6c000 256K rw--- [ anon ] 00007ffa40d6c000 44K r-x-- libsyslogformat.so 00007ffa40d77000 2044K ----- libsyslogformat.so 00007ffa40f76000 4K r---- libsyslogformat.so 00007ffa40f77000 4K rw--- libsyslogformat.so 00007ffa40f78000 4612K rw--- [ anon ] 00007ffa413f9000 24K r-x-- _hashlib.x86_64-linux-gnu.so 00007ffa413ff000 2044K ----- _hashlib.x86_64-linux-gnu.so 00007ffa415fe000 4K r---- _hashlib.x86_64-linux-gnu.so 00007ffa415ff000 4K rw--- _hashlib.x86_64-linux-gnu.so 00007ffa41600000 256K rw--- [ anon ] 00007ffa41640000 48K r-x-- _json.x86_64-linux-gnu.so 00007ffa4164c000 2044K ----- _json.x86_64-linux-gnu.so 00007ffa4184b000 4K r---- _json.x86_64-linux-gnu.so 00007ffa4184c000 4K rw--- _json.x86_64-linux-gnu.so 00007ffa4184d000 84K r-x-- _ssl.x86_64-linux-gnu.so 00007ffa41862000 2044K ----- _ssl.x86_64-linux-gnu.so 00007ffa41a61000 4K r---- _ssl.x86_64-linux-gnu.so 00007ffa41a62000 16K rw--- _ssl.x86_64-linux-gnu.so 00007ffa41a66000 768K rw--- [ anon ] 00007ffa41b57000 256K rw--- [ anon ] 00007ffa41b97000 1056K r-x-- libm-2.23.so 00007ffa41c9f000 2044K ----- libm-2.23.so 00007ffa41e9e000 4K r---- libm-2.23.so 00007ffa41e9f000 4K rw--- libm-2.23.so 00007ffa41ea0000 8K r-x-- libutil-2.23.so 00007ffa41ea2000 2044K ----- libutil-2.23.so 00007ffa420a1000 4K r---- libutil-2.23.so 00007ffa420a2000 4K rw--- libutil-2.23.so 00007ffa420a3000 100K r-x-- libz.so.1.2.8 00007ffa420bc000 2044K ----- libz.so.1.2.8 00007ffa422bb000 4K r---- libz.so.1.2.8 00007ffa422bc000 4K rw--- libz.so.1.2.8 00007ffa422bd000 3016K r-x-- libpython2.7.so.1.0 00007ffa425af000 2048K ----- libpython2.7.so.1.0 00007ffa427af000 8K r---- libpython2.7.so.1.0 00007ffa427b1000 476K rw--- libpython2.7.so.1.0 00007ffa42828000 140K rw--- [ anon ] 00007ffa4284b000 72K r-x-- libmod-python.so 00007ffa4285d000 2044K ----- libmod-python.so 00007ffa42a5c000 4K r---- libmod-python.so 00007ffa42a5d000 4K rw--- libmod-python.so 00007ffa42a5e000 4K rw--- [ anon ] 00007ffa42a5f000 32K r-x-- libwrap.so.0.7.6 00007ffa42a67000 2044K ----- libwrap.so.0.7.6 00007ffa42c66000 4K r---- libwrap.so.0.7.6 00007ffa42c67000 4K rw--- libwrap.so.0.7.6 00007ffa42c68000 4K rw--- [ anon ] 00007ffa42c69000 88K r-x-- libnet.so.1.7.0 00007ffa42c7f000 2044K ----- libnet.so.1.7.0 00007ffa42e7e000 4K r---- libnet.so.1.7.0 00007ffa42e7f000 4K rw--- libnet.so.1.7.0 00007ffa42e80000 8K rw--- [ anon ] 00007ffa42e82000 132K r-x-- libafsocket.so 00007ffa42ea3000 2048K ----- libafsocket.so 00007ffa430a3000 4K r---- libafsocket.so 00007ffa430a4000 8K rw--- libafsocket.so 00007ffa430a6000 4K rw--- [ anon ] 00007ffa430a7000 108K r-x-- libaffile.so 00007ffa430c2000 2044K ----- libaffile.so 00007ffa432c1000 4K r---- libaffile.so 00007ffa432c2000 4K rw--- libaffile.so 00007ffa432c3000 4K rw--- [ anon ] 00007ffa432c4000 8K r-x-- libconfgen.so 00007ffa432c6000 2044K ----- libconfgen.so 00007ffa434c5000 4K r---- libconfgen.so 00007ffa434c6000 4K rw--- libconfgen.so 00007ffa434c7000 36K r-x-- libbasicfuncs.so 00007ffa434d0000 2048K ----- libbasicfuncs.so 00007ffa436d0000 4K r---- libbasicfuncs.so 00007ffa436d1000 8K rw--- libbasicfuncs.so 00007ffa436d3000 32K r-x-- libappmodel.so 00007ffa436db000 2048K ----- libappmodel.so 00007ffa438db000 4K r---- libappmodel.so 00007ffa438dc000 4K rw--- libappmodel.so 00007ffa438dd000 40K r-x-- libjson-c.so.2.0.0 00007ffa438e7000 2044K ----- libjson-c.so.2.0.0 00007ffa43ae6000 4K r---- libjson-c.so.2.0.0 00007ffa43ae7000 4K rw--- libjson-c.so.2.0.0 00007ffa43ae8000 40K r-x-- libjson-plugin.so 00007ffa43af2000 2048K ----- libjson-plugin.so 00007ffa43cf2000 4K r---- libjson-plugin.so 00007ffa43cf3000 4K rw--- libjson-plugin.so 00007ffa43cf4000 44K r-x-- libnss_nis-2.23.so 00007ffa43cff000 2044K ----- libnss_nis-2.23.so 00007ffa43efe000 4K r---- libnss_nis-2.23.so 00007ffa43eff000 4K rw--- libnss_nis-2.23.so 00007ffa43f00000 88K r-x-- libnsl-2.23.so 00007ffa43f16000 2044K ----- libnsl-2.23.so 00007ffa44115000 4K r---- libnsl-2.23.so 00007ffa44116000 4K rw--- libnsl-2.23.so 00007ffa44117000 8K rw--- [ anon ] 00007ffa44119000 32K r-x-- libnss_compat-2.23.so 00007ffa44121000 2044K ----- libnss_compat-2.23.so 00007ffa44320000 4K r---- libnss_compat-2.23.so 00007ffa44321000 4K rw--- libnss_compat-2.23.so 00007ffa44322000 44K r-x-- libnss_files-2.23.so 00007ffa4432d000 2044K ----- libnss_files-2.23.so 00007ffa4452c000 4K r---- libnss_files-2.23.so 00007ffa4452d000 4K rw--- libnss_files-2.23.so 00007ffa4452e000 24K rw--- [ anon ] 00007ffa44534000 72K r-x-- libgpg-error.so.0.17.0 00007ffa44546000 2048K ----- libgpg-error.so.0.17.0 00007ffa44746000 4K r---- libgpg-error.so.0.17.0 00007ffa44747000 4K rw--- libgpg-error.so.0.17.0 00007ffa44748000 864K r-x-- libgcrypt.so.20.0.5 00007ffa44820000 2044K ----- libgcrypt.so.20.0.5 00007ffa44a1f000 4K r---- libgcrypt.so.20.0.5 00007ffa44a20000 32K rw--- libgcrypt.so.20.0.5 00007ffa44a28000 4K rw--- [ anon ] 00007ffa44a29000 132K r-x-- liblzma.so.5.0.0 00007ffa44a4a000 2044K ----- liblzma.so.5.0.0 00007ffa44c49000 4K r---- liblzma.so.5.0.0 00007ffa44c4a000 4K rw--- liblzma.so.5.0.0 00007ffa44c4b000 124K r-x-- libselinux.so.1 00007ffa44c6a000 2044K ----- libselinux.so.1 00007ffa44e69000 4K r---- libselinux.so.1 00007ffa44e6a000 4K rw--- libselinux.so.1 00007ffa44e6b000 8K rw--- [ anon ] 00007ffa44e6d000 2152K r-x-- libcrypto.so.1.0.0 00007ffa45087000 2044K ----- libcrypto.so.1.0.0 00007ffa45286000 112K r---- libcrypto.so.1.0.0 00007ffa452a2000 48K rw--- libcrypto.so.1.0.0 00007ffa452ae000 12K rw--- [ anon ] 00007ffa452b1000 376K r-x-- libssl.so.1.0.0 00007ffa4530f000 2048K ----- libssl.so.1.0.0 00007ffa4550f000 16K r---- libssl.so.1.0.0 00007ffa45513000 28K rw--- libssl.so.1.0.0 00007ffa4551a000 12K r-x-- libsecret-storage.so.0.0.0 00007ffa4551d000 2044K ----- libsecret-storage.so.0.0.0 00007ffa4571c000 4K r---- libsecret-storage.so.0.0.0 00007ffa4571d000 4K rw--- libsecret-storage.so.0.0.0 00007ffa4571e000 12K r-x-- libdl-2.23.so 00007ffa45721000 2044K ----- libdl-2.23.so 00007ffa45920000 4K r---- libdl-2.23.so 00007ffa45921000 4K rw--- libdl-2.23.so 00007ffa45922000 440K r-x-- libpcre.so.3.13.2 00007ffa45990000 2048K ----- libpcre.so.3.13.2 00007ffa45b90000 4K r---- libpcre.so.3.13.2 00007ffa45b91000 4K rw--- libpcre.so.3.13.2 00007ffa45b92000 16K r-x-- libcap.so.2.24 00007ffa45b96000 2048K ----- libcap.so.2.24 00007ffa45d96000 4K r---- libcap.so.2.24 00007ffa45d97000 4K rw--- libcap.so.2.24 00007ffa45d98000 12K r-x-- libevtlog-3.17.so.0.0.0 00007ffa45d9b000 2048K ----- libevtlog-3.17.so.0.0.0 00007ffa45f9b000 4K r---- libevtlog-3.17.so.0.0.0 00007ffa45f9c000 4K rw--- libevtlog-3.17.so.0.0.0 00007ffa45f9d000 4K r-x-- libgthread-2.0.so.0.4800.2 00007ffa45f9e000 2044K ----- libgthread-2.0.so.0.4800.2 00007ffa4619d000 4K r---- libgthread-2.0.so.0.4800.2 00007ffa4619e000 4K rw--- libgthread-2.0.so.0.4800.2 00007ffa4619f000 12K r-x-- libgmodule-2.0.so.0.4800.2 00007ffa461a2000 2044K ----- libgmodule-2.0.so.0.4800.2 00007ffa463a1000 4K r---- libgmodule-2.0.so.0.4800.2 00007ffa463a2000 4K rw--- libgmodule-2.0.so.0.4800.2 00007ffa463a3000 28K r-x-- librt-2.23.so 00007ffa463aa000 2044K ----- librt-2.23.so 00007ffa465a9000 4K r---- librt-2.23.so 00007ffa465aa000 4K rw--- librt-2.23.so 00007ffa465ab000 1792K r-x-- libc-2.23.so 00007ffa4676b000 2044K ----- libc-2.23.so 00007ffa4696a000 16K r---- libc-2.23.so 00007ffa4696e000 8K rw--- libc-2.23.so 00007ffa46970000 16K rw--- [ anon ] 00007ffa46974000 96K r-x-- libpthread-2.23.so 00007ffa4698c000 2044K ----- libpthread-2.23.so 00007ffa46b8b000 4K r---- libpthread-2.23.so 00007ffa46b8c000 4K rw--- libpthread-2.23.so 00007ffa46b8d000 16K rw--- [ anon ] 00007ffa46b91000 1084K r-x-- libglib-2.0.so.0.4800.2 00007ffa46ca0000 2044K ----- libglib-2.0.so.0.4800.2 00007ffa46e9f000 4K r---- libglib-2.0.so.0.4800.2 00007ffa46ea0000 4K rw--- libglib-2.0.so.0.4800.2 00007ffa46ea1000 4K rw--- [ anon ] 00007ffa46ea2000 768K r-x-- libsyslog-ng-3.17.so.0.0.0 00007ffa46f62000 2044K ----- libsyslog-ng-3.17.so.0.0.0 00007ffa47161000 16K r---- libsyslog-ng-3.17.so.0.0.0 00007ffa47165000 24K rw--- libsyslog-ng-3.17.so.0.0.0 00007ffa4716b000 8K rw--- [ anon ] 00007ffa4716d000 152K r-x-- ld-2.23.so 00007ffa471ba000 256K rw--- [ anon ] 00007ffa472a2000 256K rw--- [ anon ] 00007ffa472e2000 64K rwx-- [ anon ] 00007ffa472f2000 28K r--s- gconv-modules.cache 00007ffa472f9000 36K rw--- [ anon ] 00007ffa47302000 512K r-x-- libsystemd.so.0.14.0 00007ffa47382000 12K r---- libsystemd.so.0.14.0 00007ffa47385000 4K rw--- libsystemd.so.0.14.0 00007ffa47386000 20K rw--- [ anon ] 00007ffa47390000 8K rw--- [ anon ] 00007ffa47392000 4K r---- ld-2.23.so 00007ffa47393000 4K rw--- ld-2.23.so 00007ffa47394000 4K rw--- [ anon ] 00007ffe81489000 132K rw--- [ stack ] 00007ffe81564000 8K r-x-- [ anon ] ffffffffff600000 4K r-x-- [ anon ] total 7226528K
Configuration:
@version: 3.16.1 @module mod-java @include "/etc/syslog-ng/scl.conf"
options { chain_hostnames(off); use-dns(no); use-fqdn(no); owner("syslog-ng"); group("stratio"); perm(0640); stats_freq(0); bad-hostname("^gconfd$"); flush-lines(100); log-fifo-size(345000); };
######################## # Filters ######################## # Here's come the filter options. With this rules, we can set which # message go where.
filter f_dbg { level(debug); }; filter f_info { level(info); }; filter f_notice { level(notice); }; filter f_warn { level(warn); }; filter f_err { level(err); }; filter f_crit { level(crit .. emerg); };
filter f_debug { level(debug) and not facility(auth, authpriv, news, mail); }; filter f_error { level(err .. emerg) ; }; filter f_messages { level(info,notice,warn) and not facility(auth,authpriv,cron,daemon,mail,news); };
filter f_auth { facility(auth, authpriv) and not filter(f_debug); }; filter f_cron { facility(cron) and not filter(f_debug); }; filter f_daemon { facility(daemon) and not filter(f_debug); }; filter f_kern { facility(kern) and not filter(f_debug); }; filter f_lpr { facility(lpr) and not filter(f_debug); }; filter f_local { facility(local0, local1, local3, local4, local5, local6, local7) and not filter(f_debug); }; filter f_mail { facility(mail) and not filter(f_debug); }; filter f_news { facility(news) and not filter(f_debug); }; filter f_syslog3 { not facility(auth, authpriv, mail) and not filter(f_debug); }; filter f_user { facility(user) and not filter(f_debug); };
filter f_text_only { not match( ".*[\\x80-\\xFF].*" value("MESSAGE") ); };
### # Sources
source s_stdout_files { wildcard-file( base-dir("/var/lib/slaves/") filename-pattern("stdout") max-files(3400) follow-freq(0.5) monitor-method(poll) log-fetch-limit(100) log-iw-size(345000) flags(no-parse) recursive(yes) multi-line-mode(indented) ); };
source s_stderr_files { wildcard-file( base-dir("/var/lib/slaves/") filename-pattern("stderr") max-files(3400) follow-freq(0.5) monitor-method(poll) log-fetch-limit(100) log-iw-size(345000) flags(no-parse) recursive(yes) multi-line-mode(indented) ); };
Thanks.
2018-09-06 12:59 GMT+02:00 Nemes, Antal <antal.nemes@oneidentity.com>:
Hi,
The same free policy cannot work with _handle_file_deleted, because we cannot free the readers immediately. There is a race between file delete and syslog-ng reading the data. It can happen that one deletes a file before syslog-ng reads all its contents, which would cause message loss. In that case we just signal the deletion event to the reader (log_pipe_notify(&reader->super, NC_FILE_DELETED, NULL); and when the reader reads EOF at a later time, that's when the reader object is freed.
Can you send the output of pmap <pid of syslog-ng> with the leak? That may give us a clue where to look with the leak.
Also can you send your configuration, or at least the part about the wildcard filesources? The reproduction might depend on max_files or monitor_method parameter, or maybe other.
Thanks, Antal
On Thu, Sep 6, 2018 at 12:00 PM Jose Angel Santiago < jasantiago@stratio.com> wrote:
Reviewing your commit I've found out that the method *_handle_file_deleted* seems not to free memory when a monitored file has been deleted, according to the changes you made in *_handler_handler_directory_deleted.* Could it be the cause of the memory leak?
Thanks.
2018-09-06 11:50 GMT+02:00 Jose Angel Santiago <jasantiago@stratio.com> :
Hi,
I was testing your fix in several environments before replying, unfortunately syslog-ng stills shows a huge RAM consumption in our use of case.
We have two wildcard-file sources defined in our syslog-ng conf file, both of them have the same base-dir reference, but one source has a filename-pattern and the other source has a different one (I think we can't use the same source and several filename-patterns because of the lack of regular expression support). We execute the deletion of obsolete files and folder under base-dir once every 2 days, and we create a lot of new folders and files constantly while using our platform, but the growth of these log files (the ones affected by the filename-pattern) has been pretty low in this test, about 10-20 msgs per second total.
This is some data of the fix test:
Syslog-ng was consuming 21,7G of RAM with 10600 files affected by the filename-pattern and 20000 folders under base-dir (not a single folder were deleted yet so this is the expected behaviour of syslog-ng, and we find it extremely unaffordable)
One hour later, right before the cleaning process was executed, syslog-ng was consuming 35G of RAM, and it didn't decrease after the cleaning, not even a little, with just 380 files affected by the filename-pattern and 2200 folders under base-dir.
There is no message in the syslog-agent queue and all the messages are received by the relay (which ingest messages in an elasticsearch)
For the record, we do noticed a small RAM usage decrease in a previous test in a lab environment when we deleted every folder & file under base-dir (from 870M to 550M).
It seems syslog-ng wildcard-source is not optimized when it has too handle thousands of folders and filename-patterns under base-dir. Have you ever test the wildcard-source with a use of case similar to ours?
Can I send you any more information which may help to find out how to optimize it? I wish I can help with the code but I have no idea of C language :-(
Thanks in advance.
2018-09-04 18:03 GMT+02:00 Nagy, Gábor <gabor.nagy@oneidentity.com>:
Hello,
Thanks for the thorough investigation! We've checked it and found a memory leak in directory monitoring.
I've pushed the fix to my fork and created a merge request about it: https://github.com/gaborznagy/syslog-ng/commits/fix-wildcard-memleak https://github.com/balabit/syslog-ng/pull/2261 Can you verify that this fixes your problem, please?
> log-fifo-size value is 345000, so I assume it can't be a buffer situation since 345000 messages can't occupy 40-50 GB of memory. Well our default log-msg-size is 64kB so if all messages have a size of log-msg-size then it can be around ~21GB when syslog-ng is buffering.
Best Regards, Gabor
On Tue, Sep 4, 2018 at 10:50 AM Jose Angel Santiago < jasantiago@stratio.com> wrote:
> Hi, > > I'm using syslog-ng 3.16.1 with wildcard-file as source (let's call > it "syslog-agent), which sends log messages to another syslog-ng acting as > a relay. > > I've noticed that syslog-agent instances RAM consumption keeps > increasing until they leave no free memory in the cluster (each server has > 64G of RAM). In my use case, new folders & files are created constantly > under base-dir folder, but every 2 days obsolete folders & files are > deleted. I assumed that syslog-ng would free some RAM every time those > folders & files are deleted, but it doesn't, not even if I run a > syslog-ng-ctl reload operation. > > log-fifo-size value is 345000, so I assume it can't be a buffer > situation since 345000 messages can't occupy 40-50 GB of memory. > > I've performed the following test to reproduce the situation in > small scale: > > - Launch a syslog-agent with a wildcard-file source reading from > "/tmp/test/" base-dir. syslog-agent RAM usage is about 125M. > - Run a simple script to create complex folder hierarchy under > /tmp/test and some files with 5000 log messages to read from. > - Wait until syslog-agent RAM usage gets 1GB > - Stop script execution and wait until syslog-agent has send all > logs to relay. > - Delete everything under /tmp/test and execute syslog-ng-ctl reload > operation > - 24 hours after, syslog-agent RAM usage still is 1GB > > I've used heaptrack tool as a try to find a memory leak in > syslog-ng, you can see in the attached image that iv_list_empty function in > iv_list.h file is where most of the RAM usage is. > > How do I get syslog-ng to free RAM? Or is it a memory leak? > > Thanks in advance. > > > -- > > | Jose Angel Santiago > > [image: Logo_signature2.png] <http://www.stratio.com/> > > Vía de las dos Castillas, 33, Ática 4, 3ª Planta > > 28224 Pozuelo de Alarcón, Madrid, Spain > > +34 918 286 473 <+34%20918%2028%2064%2073> | www.stratio.com > <https://twitter.com/stratiobd> > <https://www.linkedin.com/company/stratiobd> > <https://www.youtube.com/c/StratioBD> > ____________________________________________________________ > __________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/? > product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
--
| Jose Angel Santiago
[image: Logo_signature2.png] <http://www.stratio.com/>
Vía de las dos Castillas, 33, Ática 4, 3ª Planta
28224 Pozuelo de Alarcón, Madrid, Spain
+34 918 286 473 <+34%20918%2028%2064%2073> | www.stratio.com <https://twitter.com/stratiobd> <https://www.linkedin.com/company/stratiobd> <https://www.youtube.com/c/StratioBD>
--
| Jose Angel Santiago
[image: Logo_signature2.png] <http://www.stratio.com/>
Vía de las dos Castillas, 33, Ática 4, 3ª Planta
28224 Pozuelo de Alarcón, Madrid, Spain
+34 918 286 473 <+34%20918%2028%2064%2073> | www.stratio.com <https://twitter.com/stratiobd> <https://www.linkedin.com/company/stratiobd> <https://www.youtube.com/c/StratioBD> ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
--
| Jose Angel Santiago
[image: Logo_signature2.png] <http://www.stratio.com/>
Vía de las dos Castillas, 33, Ática 4, 3ª Planta
28224 Pozuelo de Alarcón, Madrid, Spain
+34 918 286 473 <+34%20918%2028%2064%2073> | www.stratio.com <https://twitter.com/stratiobd> <https://www.linkedin.com/company/stratiobd> <https://www.youtube.com/c/StratioBD> ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- | Jose Angel Santiago [image: Logo_signature2.png] <http://www.stratio.com/> Vía de las dos Castillas, 33, Ática 4, 3ª Planta 28224 Pozuelo de Alarcón, Madrid, Spain +34 918 286 473 | www.stratio.com <https://twitter.com/stratiobd> <https://www.linkedin.com/company/stratiobd> <https://www.youtube.com/c/StratioBD>
participants (3)
-
Jose Angel Santiago
-
Nagy, Gábor
-
Nemes, Antal