3.3.4 anyone else notice a memory leak
I run a number of syslog-ng processes, and I am noticing that the processes that read from a pipe, and write to a program destination all seem to have a memory leak in them. The rate of leak seems to be dependent on the volume of log lines (not surprising). Has anyone else noticed this. My config is; @version: 3.3 # # UVic syslog-ng configuration options { log_fifo_size(100000); use_fqdn(yes); keep_hostname(yes); chain_hostnames(yes); time_reap(60); time_reopen(5); flush_lines(1000); flush_timeout(1000); }; source local { pipe("/var/log/syslog.pipes/syslogstats" log_iw_size(90000) log_fetch_limit(500) flags(no-parse)); }; source int { internal(); }; destination localsyslog { tcp("localhost" port(1514) localip(localhost) log_fifo_size(50000) template("<$PRI>$S_DATE syslogstats@$HOST syslogstats: $MSGONLY\n") template_escape(no) ); }; destination syslog_stats { program("/home/sysprog/syslog-apps/syslog_stats/syslogstats" template("$MESSAGE\n") template_escape(no) ); }; log { source(local); destination(syslog_stats); flags(flow-control); }; log { source(int); destination(localsyslog); };
It looks like it must be writing to a program destination, or writing to a pipe that leaks the memory. The only syslog-ng processes I have that do not leak memory are; - read from a pipe and writes to files. - read from ksm /dev/log, write to files and tcp destination The rest leak; - Read from network, write to many pipes - Read from pipe and write to program Evan. ________________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Evan Rempel [erempel@uvic.ca] Sent: Tuesday, February 07, 2012 9:35 PM To: Syslog-ng users' and developers' mailing list Subject: [syslog-ng] 3.3.4 anyone else notice a memory leak I run a number of syslog-ng processes, and I am noticing that the processes that read from a pipe, and write to a program destination all seem to have a memory leak in them. The rate of leak seems to be dependent on the volume of log lines (not surprising). Has anyone else noticed this. My config is; @version: 3.3 # # UVic syslog-ng configuration options { log_fifo_size(100000); use_fqdn(yes); keep_hostname(yes); chain_hostnames(yes); time_reap(60); time_reopen(5); flush_lines(1000); flush_timeout(1000); }; source local { pipe("/var/log/syslog.pipes/syslogstats" log_iw_size(90000) log_fetch_limit(500) flags(no-parse)); }; source int { internal(); }; destination localsyslog { tcp("localhost" port(1514) localip(localhost) log_fifo_size(50000) template("<$PRI>$S_DATE syslogstats@$HOST syslogstats: $MSGONLY\n") template_escape(no) ); }; destination syslog_stats { program("/home/sysprog/syslog-apps/syslog_stats/syslogstats" template("$MESSAGE\n") template_escape(no) ); }; log { source(local); destination(syslog_stats); flags(flow-control); }; log { source(int); destination(localsyslog); }; ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hm, I'm not seeing the leak in ELSA (which uses program()) and I'm using 3.3.4 from the latest tarball (not git). On Tue, Feb 7, 2012 at 11:49 PM, Evan Rempel <erempel@uvic.ca> wrote:
It looks like it must be writing to a program destination, or writing to a pipe that leaks the memory.
The only syslog-ng processes I have that do not leak memory are;
- read from a pipe and writes to files. - read from ksm /dev/log, write to files and tcp destination
The rest leak;
- Read from network, write to many pipes - Read from pipe and write to program
Evan. ________________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Evan Rempel [erempel@uvic.ca] Sent: Tuesday, February 07, 2012 9:35 PM To: Syslog-ng users' and developers' mailing list Subject: [syslog-ng] 3.3.4 anyone else notice a memory leak
I run a number of syslog-ng processes, and I am noticing that the processes that read from a pipe, and write to a program destination all seem to have a memory leak in them.
The rate of leak seems to be dependent on the volume of log lines (not surprising).
Has anyone else noticed this.
My config is;
@version: 3.3
# # UVic syslog-ng configuration
options { log_fifo_size(100000); use_fqdn(yes); keep_hostname(yes); chain_hostnames(yes); time_reap(60); time_reopen(5); flush_lines(1000); flush_timeout(1000); };
source local { pipe("/var/log/syslog.pipes/syslogstats" log_iw_size(90000) log_fetch_limit(500) flags(no-parse)); }; source int { internal(); };
destination localsyslog { tcp("localhost" port(1514) localip(localhost) log_fifo_size(50000) template("<$PRI>$S_DATE syslogstats@$HOST syslogstats: $MSGONLY\n") template_escape(no) ); };
destination syslog_stats { program("/home/sysprog/syslog-apps/syslog_stats/syslogstats" template("$MESSAGE\n") template_escape(no) ); };
log { source(local); destination(syslog_stats); flags(flow-control); }; log { source(int); destination(localsyslog); }; ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hmm. Mine is build from the tar ball as well. I am fairly certain that there is a memory leak. How do I help track it down? My syslog-ng statistic line shows that nothing was dropped (implying that not much was queued) and that nothing was stored (does that mean than nothing ever needed to be queued?). 2012-02-09T08:34:14-08:00 flexfilter_alert@linnet.comp.uvic.ca/linnet.comp.uvic.ca syslog.info flexfilter_alert: Log statistics; dropped='dst.tcp(localsyslog#0,localhost:1514)=0', processed='dst.tcp(localsyslog#0,localhost:1514)=78', stored='dst.tcp(localsyslog#0,localhost:1514)=0', processed='src.internal(int#0)=78', stamp='src.internal(int#0)=1328804653', processed='destination(localsyslog)=78', processed='source(local)=45390409', processed='destination(flexfilter)=45390409', processed='global(payload_reallocs)=53024', processed='global(msg_clones)=0', processed='source(int)=78', processed='center(queued)=0', processed='global(sdata_updates)=0', processed='center(received)=0', dropped='dst.program(flexfilter#0,/home1l/sysprog/flexfilter/flexfilter /home1l/sysprog/syslog-apps/alert/fullsyslog.xml)=0', processed='dst.program(flexfilter#0,/home1l/sysprog/flexfilter/flexfilter /home1l/sysprog/syslog-apps/alert/fullsyslog.xml)=45390409', stored='dst.program(flexfilter#0,/home1l/sysprog/flexfilter/flexfilter /home1l/sysprog/syslog-apps/alert/fullsyslog.xml)=0' The memory (virtual size) shows DATE TIME USER PID PPID NI PRI CPU VSZ ELAPSED TIME COMMAND 20120209 000101 root 11545 11544 0 19 - 601332 04:17:01 00:00:57 /usr/local/sbin/syslog-ng 20120209 001601 root 11545 11544 0 19 - 632528 04:32:01 00:01:00 /usr/local/sbin/syslog-ng 20120209 003101 root 11545 11544 0 19 - 660760 04:47:01 00:01:04 /usr/local/sbin/syslog-ng 20120209 004601 root 11545 11544 0 19 - 684580 05:02:01 00:01:07 /usr/local/sbin/syslog-ng 20120209 010101 root 11545 11544 0 19 - 707024 05:17:01 00:01:09 /usr/local/sbin/syslog-ng 20120209 011601 root 11545 11544 0 19 - 729292 05:32:01 00:01:11 /usr/local/sbin/syslog-ng 20120209 013101 root 11545 11544 0 19 - 752112 05:47:01 00:01:14 /usr/local/sbin/syslog-ng 20120209 014601 root 11545 11544 0 19 - 773556 06:02:01 00:01:16 /usr/local/sbin/syslog-ng 20120209 020101 root 11545 11544 0 19 - 797420 06:17:01 00:01:19 /usr/local/sbin/syslog-ng 20120209 021601 root 11545 11544 0 19 - 825760 06:32:01 00:01:22 /usr/local/sbin/syslog-ng 20120209 023101 root 11545 11544 0 19 - 847680 06:47:01 00:01:25 /usr/local/sbin/syslog-ng 20120209 024601 root 11545 11544 0 19 - 872712 07:02:01 00:01:28 /usr/local/sbin/syslog-ng 20120209 030101 root 11545 11544 0 19 - 898816 07:17:01 00:01:31 /usr/local/sbin/syslog-ng 20120209 031601 root 11545 11544 0 19 - 925232 07:32:01 00:01:34 /usr/local/sbin/syslog-ng 20120209 033101 root 11545 11544 0 19 - 949924 07:47:01 00:01:37 /usr/local/sbin/syslog-ng 20120209 034601 root 11545 11544 0 19 - 976864 08:02:01 00:01:40 /usr/local/sbin/syslog-ng 20120209 040101 root 11545 11544 0 19 - 1003048 08:17:01 00:01:43 /usr/local/sbin/syslog-ng 20120209 041601 root 11545 11544 0 19 - 1027700 08:32:01 00:01:46 /usr/local/sbin/syslog-ng 20120209 043101 root 11545 11544 0 19 - 1054552 08:47:01 00:01:49 /usr/local/sbin/syslog-ng 20120209 044601 root 11545 11544 0 19 - 1080776 09:02:01 00:01:52 /usr/local/sbin/syslog-ng 20120209 050101 root 11545 11544 0 19 - 1105228 09:17:01 00:01:55 /usr/local/sbin/syslog-ng 20120209 051601 root 11545 11544 0 19 - 1132676 09:32:01 00:01:58 /usr/local/sbin/syslog-ng 20120209 053101 root 11545 11544 0 19 - 1159172 09:47:01 00:02:01 /usr/local/sbin/syslog-ng 20120209 054601 root 11545 11544 0 19 - 1185140 10:02:01 00:02:05 /usr/local/sbin/syslog-ng 20120209 060101 root 11545 11544 0 19 - 1212596 10:17:01 00:02:08 /usr/local/sbin/syslog-ng 20120209 061601 root 11545 11544 0 19 - 1244612 10:32:01 00:02:12 /usr/local/sbin/syslog-ng 20120209 063101 root 11545 11544 0 19 - 1278132 10:47:01 00:02:16 /usr/local/sbin/syslog-ng 20120209 064601 root 11545 11544 0 19 - 1312416 11:02:01 00:02:20 /usr/local/sbin/syslog-ng 20120209 070101 root 11545 11544 0 19 - 1343452 11:17:01 00:02:24 /usr/local/sbin/syslog-ng 20120209 071601 root 11545 11544 0 19 - 1377180 11:32:01 00:02:28 /usr/local/sbin/syslog-ng 20120209 073101 root 11545 11544 0 19 - 1408764 11:47:01 00:02:32 /usr/local/sbin/syslog-ng 20120209 074602 root 11545 11544 0 19 - 1438876 12:02:02 00:02:35 /usr/local/sbin/syslog-ng 20120209 080101 root 11545 11544 0 19 - 1475788 12:17:01 00:02:40 /usr/local/sbin/syslog-ng 20120209 081601 root 11545 11544 0 19 - 1511372 12:32:01 00:02:44 /usr/local/sbin/syslog-ng 20120209 083101 root 11545 11544 0 19 - 1554716 12:47:01 00:02:49 /usr/local/sbin/syslog-ng So you can see that the memory footprint grows constantly. Evan. ________________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Martin Holste [mcholste@gmail.com] Sent: Wednesday, February 08, 2012 8:58 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] 3.3.4 anyone else notice a memory leak Hm, I'm not seeing the leak in ELSA (which uses program()) and I'm using 3.3.4 from the latest tarball (not git). On Tue, Feb 7, 2012 at 11:49 PM, Evan Rempel <erempel@uvic.ca> wrote:
It looks like it must be writing to a program destination, or writing to a pipe that leaks the memory.
The only syslog-ng processes I have that do not leak memory are;
- read from a pipe and writes to files. - read from ksm /dev/log, write to files and tcp destination
The rest leak;
- Read from network, write to many pipes - Read from pipe and write to program
Evan. ________________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Evan Rempel [erempel@uvic.ca] Sent: Tuesday, February 07, 2012 9:35 PM To: Syslog-ng users' and developers' mailing list Subject: [syslog-ng] 3.3.4 anyone else notice a memory leak
I run a number of syslog-ng processes, and I am noticing that the processes that read from a pipe, and write to a program destination all seem to have a memory leak in them.
The rate of leak seems to be dependent on the volume of log lines (not surprising).
Has anyone else noticed this.
My config is;
@version: 3.3
# # UVic syslog-ng configuration
options { log_fifo_size(100000); use_fqdn(yes); keep_hostname(yes); chain_hostnames(yes); time_reap(60); time_reopen(5); flush_lines(1000); flush_timeout(1000); };
source local { pipe("/var/log/syslog.pipes/syslogstats" log_iw_size(90000) log_fetch_limit(500) flags(no-parse)); }; source int { internal(); };
destination localsyslog { tcp("localhost" port(1514) localip(localhost) log_fifo_size(50000) template("<$PRI>$S_DATE syslogstats@$HOST syslogstats: $MSGONLY\n") template_escape(no) ); };
destination syslog_stats { program("/home/sysprog/syslog-apps/syslog_stats/syslogstats" template("$MESSAGE\n") template_escape(no) ); };
log { source(local); destination(syslog_stats); flags(flow-control); }; log { source(int); destination(localsyslog); }; ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Advice from Gergely Nagy in a previous case: <quote Gergely> Hrm. Well, valgrind would be best, but that slows things down to almost a grounding halt. (But in case you want to try that: G_SLICE=always-malloc valgrind --leak-check=full -v --show-reachable-yes --log-file=valgrind.log syslog-ng -F would be the command to run ;) Other than that, a config could be useful, so I can try and reproduce the problem (or at least identify possible areas where the leak might be coming from, and go from there), and see if I can fix it. </quote> On Thu, Feb 9, 2012 at 6:35 PM, Evan Rempel <erempel@uvic.ca> wrote:
Hmm. Mine is build from the tar ball as well.
I am fairly certain that there is a memory leak. How do I help track it down?
My syslog-ng statistic line shows that nothing was dropped (implying that not much was queued) and that nothing was stored (does that mean than nothing ever needed to be queued?).
2012-02-09T08:34:14-08:00 flexfilter_alert@linnet.comp.uvic.ca/linnet.comp.uvic.ca syslog.info flexfilter_alert: Log statistics; dropped='dst.tcp(localsyslog#0,localhost:1514)=0', processed='dst.tcp(localsyslog#0,localhost:1514)=78', stored='dst.tcp(localsyslog#0,localhost:1514)=0', processed='src.internal(int#0)=78', stamp='src.internal(int#0)=1328804653', processed='destination(localsyslog)=78', processed='source(local)=45390409', processed='destination(flexfilter)=45390409', processed='global(payload_reallocs)=53024', processed='global(msg_clones)=0', processed='source(int)=78', processed='center(queued)=0', processed='global(sdata_updates)=0', processed='center(received)=0', dropped='dst.program(flexfilter#0,/home1l/sysprog/flexfilter/flexfilter /home1l/sysprog/syslog-apps/alert/fullsyslog.xml)=0', processed='dst.program(flexfilter#0,/home1l/sysprog/flexfilter/flexfilter /home1l/sysprog/syslog-apps/alert/fullsyslog.xml)=45390409', stored='dst.program(flexfilter#0,/home1l/sysprog/flexfilter/flexfilter /home1l/sysprog/syslog-apps/alert/fullsyslog.xml)=0'
The memory (virtual size) shows
DATE TIME USER PID PPID NI PRI CPU VSZ ELAPSED TIME COMMAND 20120209 000101 root 11545 11544 0 19 - 601332 04:17:01 00:00:57 /usr/local/sbin/syslog-ng 20120209 001601 root 11545 11544 0 19 - 632528 04:32:01 00:01:00 /usr/local/sbin/syslog-ng 20120209 003101 root 11545 11544 0 19 - 660760 04:47:01 00:01:04 /usr/local/sbin/syslog-ng 20120209 004601 root 11545 11544 0 19 - 684580 05:02:01 00:01:07 /usr/local/sbin/syslog-ng 20120209 010101 root 11545 11544 0 19 - 707024 05:17:01 00:01:09 /usr/local/sbin/syslog-ng 20120209 011601 root 11545 11544 0 19 - 729292 05:32:01 00:01:11 /usr/local/sbin/syslog-ng 20120209 013101 root 11545 11544 0 19 - 752112 05:47:01 00:01:14 /usr/local/sbin/syslog-ng 20120209 014601 root 11545 11544 0 19 - 773556 06:02:01 00:01:16 /usr/local/sbin/syslog-ng 20120209 020101 root 11545 11544 0 19 - 797420 06:17:01 00:01:19 /usr/local/sbin/syslog-ng 20120209 021601 root 11545 11544 0 19 - 825760 06:32:01 00:01:22 /usr/local/sbin/syslog-ng 20120209 023101 root 11545 11544 0 19 - 847680 06:47:01 00:01:25 /usr/local/sbin/syslog-ng 20120209 024601 root 11545 11544 0 19 - 872712 07:02:01 00:01:28 /usr/local/sbin/syslog-ng 20120209 030101 root 11545 11544 0 19 - 898816 07:17:01 00:01:31 /usr/local/sbin/syslog-ng 20120209 031601 root 11545 11544 0 19 - 925232 07:32:01 00:01:34 /usr/local/sbin/syslog-ng 20120209 033101 root 11545 11544 0 19 - 949924 07:47:01 00:01:37 /usr/local/sbin/syslog-ng 20120209 034601 root 11545 11544 0 19 - 976864 08:02:01 00:01:40 /usr/local/sbin/syslog-ng 20120209 040101 root 11545 11544 0 19 - 1003048 08:17:01 00:01:43 /usr/local/sbin/syslog-ng 20120209 041601 root 11545 11544 0 19 - 1027700 08:32:01 00:01:46 /usr/local/sbin/syslog-ng 20120209 043101 root 11545 11544 0 19 - 1054552 08:47:01 00:01:49 /usr/local/sbin/syslog-ng 20120209 044601 root 11545 11544 0 19 - 1080776 09:02:01 00:01:52 /usr/local/sbin/syslog-ng 20120209 050101 root 11545 11544 0 19 - 1105228 09:17:01 00:01:55 /usr/local/sbin/syslog-ng 20120209 051601 root 11545 11544 0 19 - 1132676 09:32:01 00:01:58 /usr/local/sbin/syslog-ng 20120209 053101 root 11545 11544 0 19 - 1159172 09:47:01 00:02:01 /usr/local/sbin/syslog-ng 20120209 054601 root 11545 11544 0 19 - 1185140 10:02:01 00:02:05 /usr/local/sbin/syslog-ng 20120209 060101 root 11545 11544 0 19 - 1212596 10:17:01 00:02:08 /usr/local/sbin/syslog-ng 20120209 061601 root 11545 11544 0 19 - 1244612 10:32:01 00:02:12 /usr/local/sbin/syslog-ng 20120209 063101 root 11545 11544 0 19 - 1278132 10:47:01 00:02:16 /usr/local/sbin/syslog-ng 20120209 064601 root 11545 11544 0 19 - 1312416 11:02:01 00:02:20 /usr/local/sbin/syslog-ng 20120209 070101 root 11545 11544 0 19 - 1343452 11:17:01 00:02:24 /usr/local/sbin/syslog-ng 20120209 071601 root 11545 11544 0 19 - 1377180 11:32:01 00:02:28 /usr/local/sbin/syslog-ng 20120209 073101 root 11545 11544 0 19 - 1408764 11:47:01 00:02:32 /usr/local/sbin/syslog-ng 20120209 074602 root 11545 11544 0 19 - 1438876 12:02:02 00:02:35 /usr/local/sbin/syslog-ng 20120209 080101 root 11545 11544 0 19 - 1475788 12:17:01 00:02:40 /usr/local/sbin/syslog-ng 20120209 081601 root 11545 11544 0 19 - 1511372 12:32:01 00:02:44 /usr/local/sbin/syslog-ng 20120209 083101 root 11545 11544 0 19 - 1554716 12:47:01 00:02:49 /usr/local/sbin/syslog-ng
So you can see that the memory footprint grows constantly.
Evan. ________________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Martin Holste [mcholste@gmail.com] Sent: Wednesday, February 08, 2012 8:58 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] 3.3.4 anyone else notice a memory leak
Hm, I'm not seeing the leak in ELSA (which uses program()) and I'm using 3.3.4 from the latest tarball (not git).
On Tue, Feb 7, 2012 at 11:49 PM, Evan Rempel <erempel@uvic.ca> wrote:
It looks like it must be writing to a program destination, or writing to a pipe that leaks the memory.
The only syslog-ng processes I have that do not leak memory are;
- read from a pipe and writes to files. - read from ksm /dev/log, write to files and tcp destination
The rest leak;
- Read from network, write to many pipes - Read from pipe and write to program
Evan. ________________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Evan Rempel [erempel@uvic.ca] Sent: Tuesday, February 07, 2012 9:35 PM To: Syslog-ng users' and developers' mailing list Subject: [syslog-ng] 3.3.4 anyone else notice a memory leak
I run a number of syslog-ng processes, and I am noticing that the processes that read from a pipe, and write to a program destination all seem to have a memory leak in them.
The rate of leak seems to be dependent on the volume of log lines (not surprising).
Has anyone else noticed this.
My config is;
@version: 3.3
# # UVic syslog-ng configuration
options { log_fifo_size(100000); use_fqdn(yes); keep_hostname(yes); chain_hostnames(yes); time_reap(60); time_reopen(5); flush_lines(1000); flush_timeout(1000); };
source local { pipe("/var/log/syslog.pipes/syslogstats" log_iw_size(90000) log_fetch_limit(500) flags(no-parse)); }; source int { internal(); };
destination localsyslog { tcp("localhost" port(1514) localip(localhost) log_fifo_size(50000) template("<$PRI>$S_DATE syslogstats@$HOST syslogstats: $MSGONLY\n") template_escape(no) ); };
destination syslog_stats { program("/home/sysprog/syslog-apps/syslog_stats/syslogstats" template("$MESSAGE\n") template_escape(no) ); };
log { source(local); destination(syslog_stats); flags(flow-control); }; log { source(int); destination(localsyslog); }; ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Evan Rempel <erempel@uvic.ca> writes:
I am fairly certain that there is a memory leak. How do I help track it down?
Looking at your config, I don't recall any known problems in those areas, but I'll try to reproduce the problem. Meanwhile, if you have time, running syslog-ng under valgrind would be very, very useful. If you do that, please set the G_SLICE environment variable to always-malloc, otherwise due to glib's allocator, the results will be severley screwed and hardly usable. -- |8]
Gergely Nagy wrote:
Evan Rempel <erempel@uvic.ca> writes:
I am fairly certain that there is a memory leak. How do I help track it down?
Looking at your config, I don't recall any known problems in those areas, but I'll try to reproduce the problem.
Meanwhile, if you have time, running syslog-ng under valgrind would be very, very useful. If you do that, please set the G_SLICE environment variable to always-malloc, otherwise due to glib's allocator, the results will be severley screwed and hardly usable.
Yup, I can reproduce it. The output of valgrind does not appear to show the leak. If I run syslog-ng and do NOT send any data to it (it processes no log lines) the output of valgrind is the same as if it processes 100,000 log lines, however, when I "dump" 100,000 log lines % wc /home1l/erempel/log 107590 1233928 18956930 /home1l/erempel/log into the pipe (cat /home1l/erempel/log > pipe) the memory footprint of syslog-ng grows by 3MB, every time I dump in the log lines. I have attached the config file and two valgrind outputs. -- Evan Rempel erempel@uvic.ca Senior Systems Administrator 250.721.7691 Unix Services, University Systems, University of Victoria [root@catamount ~]# G_SLICE=always-malloc [root@catamount ~]# export G_SLICE [root@catamount ~]# /usr/bin/valgrind /usr/local/sbin/syslog-ng --cfgfile=/usr/local/etc/syslog-ng/syslog-ng.erempel.conf --persist-file=/var/local/syslog-ng.erempel.persist --pidfile=/var/run/syslog-ng.erempel.pid --foreground ==19346== Memcheck, a memory error detector ==19346== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al. ==19346== Using Valgrind-3.6.0 and LibVEX; rerun with -h for copyright info ==19346== Command: /usr/local/sbin/syslog-ng --cfgfile=/usr/local/etc/syslog-ng/syslog-ng.erempel.conf --persist-file=/var/local/syslog-ng.erempel.persist --pidfile=/var/run/syslog-ng.erempel.pid --foreground ==19346== syslog-ng: Error setting file number limit; limit='4096'; error='Operation not permitted' ==19346== Syscall param epoll_ctl(event) points to uninitialised byte(s) ==19346== at 0x64F3ECA: epoll_ctl (in /lib64/libc-2.12.so) ==19346== by 0x4E7F8FF: iv_epoll_pollable (iv_method_epoll.c:153) ==19346== by 0x4E6356B: log_writer_start_watches (logwriter.c:426) ==19346== by 0x4E62CFD: log_writer_reopen (logwriter.c:1229) ==19346== by 0x6DC25FD: afprogram_dd_reopen (afprog.c:280) ==19346== by 0x4E5096A: log_dest_group_init (logpipe.h:239) ==19346== by 0x4E4C166: log_center_init (logpipe.h:239) ==19346== by 0x4E4D3EF: cfg_init (cfg.c:255) ==19346== by 0x4E664BC: main_loop_init (mainloop.c:481) ==19346== by 0x401584: main (in /usr/local/sbin/syslog-ng) ==19346== Address 0x7ff000128 is on thread 1's stack ==19346== ^C==19346== Invalid read of size 8 ==19346== at 0x4E51A81: log_dest_driver_deinit_method (driver.c:172) ==19346== by 0x6DC284C: afprogram_dd_deinit (afprog.c:344) ==19346== by 0x4E50CAC: log_dest_group_deinit (logpipe.h:254) ==19346== by 0x4E4C575: log_center_deinit (logpipe.h:254) ==19346== by 0x4E659EF: main_loop_exit_finish (mainloop.c:585) ==19346== by 0x4E80EE8: iv_run_timers (iv_timer.c:345) ==19346== by 0x4E7F417: iv_main (iv_main.c:252) ==19346== by 0x4E6570D: main_loop_run (mainloop.c:731) ==19346== by 0x4015B0: main (in /usr/local/sbin/syslog-ng) ==19346== Address 0x684a928 is 8 bytes inside a block of size 24 free'd ==19346== at 0x4C2695D: free (vg_replace_malloc.c:366) ==19346== by 0x590EFDF: g_list_remove (in /lib64/libglib-2.0.so.0.2200.5) ==19346== by 0x4E51A5F: log_dest_driver_deinit_method (driver.h:190) ==19346== by 0x6DC284C: afprogram_dd_deinit (afprog.c:344) ==19346== by 0x4E50CAC: log_dest_group_deinit (logpipe.h:254) ==19346== by 0x4E4C575: log_center_deinit (logpipe.h:254) ==19346== by 0x4E659EF: main_loop_exit_finish (mainloop.c:585) ==19346== by 0x4E80EE8: iv_run_timers (iv_timer.c:345) ==19346== by 0x4E7F417: iv_main (iv_main.c:252) ==19346== by 0x4E6570D: main_loop_run (mainloop.c:731) ==19346== by 0x4015B0: main (in /usr/local/sbin/syslog-ng) ==19346== ==19346== Invalid read of size 8 ==19346== at 0x4E51A81: log_dest_driver_deinit_method (driver.c:172) ==19346== by 0x6FD07DA: afsocket_dd_deinit (afsocket.c:1221) ==19346== by 0x4E50CAC: log_dest_group_deinit (logpipe.h:254) ==19346== by 0x4E4C575: log_center_deinit (logpipe.h:254) ==19346== by 0x4E659EF: main_loop_exit_finish (mainloop.c:585) ==19346== by 0x4E80EE8: iv_run_timers (iv_timer.c:345) ==19346== by 0x4E7F417: iv_main (iv_main.c:252) ==19346== by 0x4E6570D: main_loop_run (mainloop.c:731) ==19346== by 0x4015B0: main (in /usr/local/sbin/syslog-ng) ==19346== Address 0x684f358 is 8 bytes inside a block of size 24 free'd ==19346== at 0x4C2695D: free (vg_replace_malloc.c:366) ==19346== by 0x590EFDF: g_list_remove (in /lib64/libglib-2.0.so.0.2200.5) ==19346== by 0x4E51A5F: log_dest_driver_deinit_method (driver.h:190) ==19346== by 0x6FD07DA: afsocket_dd_deinit (afsocket.c:1221) ==19346== by 0x4E50CAC: log_dest_group_deinit (logpipe.h:254) ==19346== by 0x4E4C575: log_center_deinit (logpipe.h:254) ==19346== by 0x4E659EF: main_loop_exit_finish (mainloop.c:585) ==19346== by 0x4E80EE8: iv_run_timers (iv_timer.c:345) ==19346== by 0x4E7F417: iv_main (iv_main.c:252) ==19346== by 0x4E6570D: main_loop_run (mainloop.c:731) ==19346== by 0x4015B0: main (in /usr/local/sbin/syslog-ng) ==19346== ==19346== ==19346== HEAP SUMMARY: ==19346== in use at exit: 169,334 bytes in 3,407 blocks ==19346== total heap usage: 226,539 allocs, 223,132 frees, 178,250,594 bytes allocated ==19346== ==19346== LEAK SUMMARY: ==19346== definitely lost: 197 bytes in 2 blocks ==19346== indirectly lost: 1,823 bytes in 22 blocks ==19346== possibly lost: 0 bytes in 0 blocks ==19346== still reachable: 167,314 bytes in 3,383 blocks ==19346== suppressed: 0 bytes in 0 blocks ==19346== Rerun with --leak-check=full to see details of leaked memory ==19346== ==19346== For counts of detected and suppressed errors, rerun with: -v ==19346== Use --track-origins=yes to see where uninitialised values come from ==19346== ERROR SUMMARY: 4 errors from 3 contexts (suppressed: 66 from 9) [root@catamount ~]# /usr/bin/valgrind --leak-check=full /usr/local/sbin/syslog-ng --cfgfile=/usr/local/etc/syslog-ng/syslog-ng.erempel.conf --persist-file=/var/local/syslog-ng.erempel.persist --pidfile=/var/run/syslog-ng.erempel.pid --foreground ==19487== Memcheck, a memory error detector ==19487== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al. ==19487== Using Valgrind-3.6.0 and LibVEX; rerun with -h for copyright info ==19487== Command: /usr/local/sbin/syslog-ng --cfgfile=/usr/local/etc/syslog-ng/syslog-ng.erempel.conf --persist-file=/var/local/syslog-ng.erempel.persist --pidfile=/var/run/syslog-ng.erempel.pid --foreground ==19487== syslog-ng: Error setting file number limit; limit='4096'; error='Operation not permitted' ==19487== Syscall param epoll_ctl(event) points to uninitialised byte(s) ==19487== at 0x64F3ECA: epoll_ctl (in /lib64/libc-2.12.so) ==19487== by 0x4E7F8FF: iv_epoll_pollable (iv_method_epoll.c:153) ==19487== by 0x4E6356B: log_writer_start_watches (logwriter.c:426) ==19487== by 0x4E62CFD: log_writer_reopen (logwriter.c:1229) ==19487== by 0x6DC25FD: afprogram_dd_reopen (afprog.c:280) ==19487== by 0x4E5096A: log_dest_group_init (logpipe.h:239) ==19487== by 0x4E4C166: log_center_init (logpipe.h:239) ==19487== by 0x4E4D3EF: cfg_init (cfg.c:255) ==19487== by 0x4E664BC: main_loop_init (mainloop.c:481) ==19487== by 0x401584: main (in /usr/local/sbin/syslog-ng) ==19487== Address 0x7ff000128 is on thread 1's stack ==19487== ^C==19487== Invalid read of size 8 ==19487== at 0x4E51A81: log_dest_driver_deinit_method (driver.c:172) ==19487== by 0x6DC284C: afprogram_dd_deinit (afprog.c:344) ==19487== by 0x4E50CAC: log_dest_group_deinit (logpipe.h:254) ==19487== by 0x4E4C575: log_center_deinit (logpipe.h:254) ==19487== by 0x4E659EF: main_loop_exit_finish (mainloop.c:585) ==19487== by 0x4E80EE8: iv_run_timers (iv_timer.c:345) ==19487== by 0x4E7F417: iv_main (iv_main.c:252) ==19487== by 0x4E6570D: main_loop_run (mainloop.c:731) ==19487== by 0x4015B0: main (in /usr/local/sbin/syslog-ng) ==19487== Address 0x684a928 is 8 bytes inside a block of size 24 free'd ==19487== at 0x4C2695D: free (vg_replace_malloc.c:366) ==19487== by 0x590EFDF: g_list_remove (in /lib64/libglib-2.0.so.0.2200.5) ==19487== by 0x4E51A5F: log_dest_driver_deinit_method (driver.h:190) ==19487== by 0x6DC284C: afprogram_dd_deinit (afprog.c:344) ==19487== by 0x4E50CAC: log_dest_group_deinit (logpipe.h:254) ==19487== by 0x4E4C575: log_center_deinit (logpipe.h:254) ==19487== by 0x4E659EF: main_loop_exit_finish (mainloop.c:585) ==19487== by 0x4E80EE8: iv_run_timers (iv_timer.c:345) ==19487== by 0x4E7F417: iv_main (iv_main.c:252) ==19487== by 0x4E6570D: main_loop_run (mainloop.c:731) ==19487== by 0x4015B0: main (in /usr/local/sbin/syslog-ng) ==19487== ==19487== Invalid read of size 8 ==19487== at 0x4E51A81: log_dest_driver_deinit_method (driver.c:172) ==19487== by 0x6FD07DA: afsocket_dd_deinit (afsocket.c:1221) ==19487== by 0x4E50CAC: log_dest_group_deinit (logpipe.h:254) ==19487== by 0x4E4C575: log_center_deinit (logpipe.h:254) ==19487== by 0x4E659EF: main_loop_exit_finish (mainloop.c:585) ==19487== by 0x4E80EE8: iv_run_timers (iv_timer.c:345) ==19487== by 0x4E7F417: iv_main (iv_main.c:252) ==19487== by 0x4E6570D: main_loop_run (mainloop.c:731) ==19487== by 0x4015B0: main (in /usr/local/sbin/syslog-ng) ==19487== Address 0x684f358 is 8 bytes inside a block of size 24 free'd ==19487== at 0x4C2695D: free (vg_replace_malloc.c:366) ==19487== by 0x590EFDF: g_list_remove (in /lib64/libglib-2.0.so.0.2200.5) ==19487== by 0x4E51A5F: log_dest_driver_deinit_method (driver.h:190) ==19487== by 0x6FD07DA: afsocket_dd_deinit (afsocket.c:1221) ==19487== by 0x4E50CAC: log_dest_group_deinit (logpipe.h:254) ==19487== by 0x4E4C575: log_center_deinit (logpipe.h:254) ==19487== by 0x4E659EF: main_loop_exit_finish (mainloop.c:585) ==19487== by 0x4E80EE8: iv_run_timers (iv_timer.c:345) ==19487== by 0x4E7F417: iv_main (iv_main.c:252) ==19487== by 0x4E6570D: main_loop_run (mainloop.c:731) ==19487== by 0x4015B0: main (in /usr/local/sbin/syslog-ng) ==19487== ==19487== ==19487== HEAP SUMMARY: ==19487== in use at exit: 169,334 bytes in 3,407 blocks ==19487== total heap usage: 226,539 allocs, 223,132 frees, 178,250,594 bytes allocated ==19487== ==19487== 13 bytes in 1 blocks are definitely lost in loss record 11 of 604 ==19487== at 0x4C26FDE: malloc (vg_replace_malloc.c:236) ==19487== by 0x59197D2: g_malloc (in /lib64/libglib-2.0.so.0.2200.5) ==19487== by 0x5930FCD: g_strdup (in /lib64/libglib-2.0.so.0.2200.5) ==19487== by 0x4E6E999: log_tags_init (tags.c:162) ==19487== by 0x4E4B721: app_startup (apphook.c:120) ==19487== by 0x4E663AE: main_loop_init (mainloop.c:663) ==19487== by 0x401584: main (in /usr/local/sbin/syslog-ng) ==19487== ==19487== 2,007 (184 direct, 1,823 indirect) bytes in 1 blocks are definitely lost in loss record 589 of 604 ==19487== at 0x4C26FDE: malloc (vg_replace_malloc.c:236) ==19487== by 0x59197D2: g_malloc (in /lib64/libglib-2.0.so.0.2200.5) ==19487== by 0x4E544AF: g_process_set_argv_space (gprocess.c:500) ==19487== by 0x40148C: main (in /usr/local/sbin/syslog-ng) ==19487== ==19487== LEAK SUMMARY: ==19487== definitely lost: 197 bytes in 2 blocks ==19487== indirectly lost: 1,823 bytes in 22 blocks ==19487== possibly lost: 0 bytes in 0 blocks ==19487== still reachable: 167,314 bytes in 3,383 blocks ==19487== suppressed: 0 bytes in 0 blocks ==19487== Reachable blocks (those to which a pointer was found) are not shown. ==19487== To see them, rerun with: --leak-check=full --show-reachable=yes ==19487== ==19487== For counts of detected and suppressed errors, rerun with: -v ==19487== Use --track-origins=yes to see where uninitialised values come from ==19487== ERROR SUMMARY: 6 errors from 5 contexts (suppressed: 66 from 9) @version: 3.3 options { log_fifo_size(1000000); # must be at least the biggest value of any log_is_size use_fqdn(yes); keep_hostname(yes); chain_hostnames(yes); time_reap(60); time_reopen(5); flush_lines(1000); flush_timeout(1000); }; source test { pipe("/var/log/syslog.pipes/test" log_fetch_limit(50) log_iw_size(10000) flags(no-parse) ); }; source int { internal(); }; template t_standardfile { template("$S_ISODATE $FULLHOST $FACILITY.$LEVEL $MSGHDR][$PROGRAM][$MESSAGE\n"); template_escape(no); }; destination localsyslog { tcp("localhost" port(1514) localip(catamount.comp.uvic.ca) log_fifo_size(5000) template("<$PRI>$S_ISODATE test@$HOST flexfilter_alert: $MSGONLY\n") template_escape(no) ); }; destination flexfilter { program("/usr/local/sbin/flexfilter /home1l/erempel/alert/fullsyslog.xml" template("$MESSAGE\n") template_escape(no) ); }; destination flexfilter2 { program("/bin/cat > /dev/null" template("$MESSAGE\n") template_escape(no) ); }; log { source(test); destination(flexfilter); flags(flow-control); }; log { source(int); destination(localsyslog); };
Oh ya. Redhat 6 % cat /etc/redhat-release Red Hat Enterprise Linux Server release 6.2 (Santiago) % uname -a Linux xxx 2.6.32-220.4.1.el6.x86_64 #1 SMP Thu Jan 19 14:50:54 EST 2012 x86_64 x86_64 x86_64 GNU/Linux Evan Rempel wrote:
Gergely Nagy wrote:
Evan Rempel <erempel@uvic.ca> writes:
I am fairly certain that there is a memory leak. How do I help track it down? Looking at your config, I don't recall any known problems in those areas, but I'll try to reproduce the problem.
Meanwhile, if you have time, running syslog-ng under valgrind would be very, very useful. If you do that, please set the G_SLICE environment variable to always-malloc, otherwise due to glib's allocator, the results will be severley screwed and hardly usable.
Yup, I can reproduce it. The output of valgrind does not appear to show the leak. If I run syslog-ng and do NOT send any data to it (it processes no log lines) the output of valgrind is the same as if it processes 100,000 log lines, however, when I "dump" 100,000 log lines
% wc /home1l/erempel/log 107590 1233928 18956930 /home1l/erempel/log
into the pipe (cat /home1l/erempel/log > pipe) the memory footprint of syslog-ng grows by 3MB, every time I dump in the log lines.
I have attached the config file and two valgrind outputs.
-- Evan Rempel erempel@uvic.ca Senior Systems Administrator 250.721.7691 Unix Services, University Systems, University of Victoria
On 2012-02-10, Evan Rempel wrote:
I am fairly certain that there is a memory leak. How do I help track it down? [...] Yup, I can reproduce it. The output of valgrind does not appear to show the leak. If I run syslog-ng and do NOT send any data to it (it processes no log lines) the output of valgrind is the same as if it processes 100,000 log lines, however, when I "dump" 100,000 log lines
% wc /home1l/erempel/log 107590 1233928 18956930 /home1l/erempel/log
into the pipe (cat /home1l/erempel/log > pipe) the memory footprint of syslog-ng grows by 3MB, every time I dump in the log lines.
Okay, this might be a long shot, but... can you try to revert commit [1], recompile syslog-ng and reproduce the leak? (grab the tarball, unpack it, get the raw form of [1], and use -R option to reverse the patch; then your usual build procedure) This commit was applied right after 3.3.2 was released, because it supposedly fixed crash with tcp destinations. Bazsi had doubts[2] that it may reintroduce a memory leak... and now I know it *does* reintroduce the leak, because it was leak I originally reported (longish thread here, see [3], [4], [5]) and now I'm seeing it again, with both 3.3.3 and 3.3.4: ==3645== LEAK SUMMARY: ==3645== definitely lost: 3,719,421 bytes in 10,008 blocks ==3645== indirectly lost: 905,917 bytes in 10,014 blocks ==3645== possibly lost: 17,214 bytes in 128 blocks ==3645== still reachable: 80,063 bytes in 3,278 blocks ==3645== suppressed: 0 bytes in 0 blocks I've just re-ran my tests[6] with commit [1] reverted, and the results are good again: ==7885== LEAK SUMMARY: ==7885== definitely lost: 297 bytes in 6 blocks ==7885== indirectly lost: 1,025 bytes in 22 blocks ==7885== possibly lost: 13,527 bytes in 110 blocks ==7885== still reachable: 106,235 bytes in 3,327 blocks ==7885== suppressed: 0 bytes in 0 blocks I understand that my use-case scenario is a bit different than yours, but perhaps the underlying issue is the same? Anyway - Bazsi, Gergely, could you please take a look at this whole thing? Because looks like we either end up with huge memleaks (with [1]) or tcp()-related crashes (without it). Hope this helps a bit. [1] http://git.balabit.hu/?p=bazsi/syslog-ng-3.3.git;a=commitdiff;h=c7070e2a6f1c... [2] https://lists.balabit.hu/pipermail/syslog-ng/2011-November/017697.html [3] https://lists.balabit.hu/pipermail/syslog-ng/2011-September/017269.html [4] https://lists.balabit.hu/pipermail/syslog-ng/2011-September/017275.html [5] https://lists.balabit.hu/pipermail/syslog-ng/2011-September/017338.html [6] https://lists.balabit.hu/pipermail/syslog-ng/2011-October/017555.html Regards, Jakub -- Jakub Jankowski|shasta@toxcorp.com|http://toxcorp.com/ GPG: FCBF F03D 9ADB B768 8B92 BB52 0341 9037 A875 942D
No, that does not help the issue at all. It almost seems to be that the OS is doing a lazy malloc/free and that syslog-ng is free(ing) the memory, but the OS keeps is associated with the application. Then, if syslog-ng needs to buffer some messages again it does a new malloc and the OS gives it a different block of with the end result that the memory footprint continues to grow. I saw this in 3.0 where the RAM footprint would grow when syslog-ng buffered memory, but it would take 3-4 hours after the destination was available and the buffer completely flushed before the RAM footprint would start to shrink and another 3-4 hours before it reached the original size prior to the buffering. I'm going to play with the caching malloc to see if that makes a difference, but with syslog-ng allocating memory blocks of differing sizes I'm not confident that the caching malloc will be able to do anything. Evan. ________________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Jakub Jankowski [shasta@toxcorp.com] Sent: Friday, February 10, 2012 4:10 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] 3.3.4 anyone else notice a memory leak On 2012-02-10, Evan Rempel wrote:
I am fairly certain that there is a memory leak. How do I help track it down? [...] Yup, I can reproduce it. The output of valgrind does not appear to show the leak. If I run syslog-ng and do NOT send any data to it (it processes no log lines) the output of valgrind is the same as if it processes 100,000 log lines, however, when I "dump" 100,000 log lines
% wc /home1l/erempel/log 107590 1233928 18956930 /home1l/erempel/log
into the pipe (cat /home1l/erempel/log > pipe) the memory footprint of syslog-ng grows by 3MB, every time I dump in the log lines.
Okay, this might be a long shot, but... can you try to revert commit [1], recompile syslog-ng and reproduce the leak? (grab the tarball, unpack it, get the raw form of [1], and use -R option to reverse the patch; then your usual build procedure) This commit was applied right after 3.3.2 was released, because it supposedly fixed crash with tcp destinations. Bazsi had doubts[2] that it may reintroduce a memory leak... and now I know it *does* reintroduce the leak, because it was leak I originally reported (longish thread here, see [3], [4], [5]) and now I'm seeing it again, with both 3.3.3 and 3.3.4: ==3645== LEAK SUMMARY: ==3645== definitely lost: 3,719,421 bytes in 10,008 blocks ==3645== indirectly lost: 905,917 bytes in 10,014 blocks ==3645== possibly lost: 17,214 bytes in 128 blocks ==3645== still reachable: 80,063 bytes in 3,278 blocks ==3645== suppressed: 0 bytes in 0 blocks I've just re-ran my tests[6] with commit [1] reverted, and the results are good again: ==7885== LEAK SUMMARY: ==7885== definitely lost: 297 bytes in 6 blocks ==7885== indirectly lost: 1,025 bytes in 22 blocks ==7885== possibly lost: 13,527 bytes in 110 blocks ==7885== still reachable: 106,235 bytes in 3,327 blocks ==7885== suppressed: 0 bytes in 0 blocks I understand that my use-case scenario is a bit different than yours, but perhaps the underlying issue is the same? Anyway - Bazsi, Gergely, could you please take a look at this whole thing? Because looks like we either end up with huge memleaks (with [1]) or tcp()-related crashes (without it). Hope this helps a bit. [1] http://git.balabit.hu/?p=bazsi/syslog-ng-3.3.git;a=commitdiff;h=c7070e2a6f1c... [2] https://lists.balabit.hu/pipermail/syslog-ng/2011-November/017697.html [3] https://lists.balabit.hu/pipermail/syslog-ng/2011-September/017269.html [4] https://lists.balabit.hu/pipermail/syslog-ng/2011-September/017275.html [5] https://lists.balabit.hu/pipermail/syslog-ng/2011-September/017338.html [6] https://lists.balabit.hu/pipermail/syslog-ng/2011-October/017555.html Regards, Jakub -- Jakub Jankowski|shasta@toxcorp.com|http://toxcorp.com/ GPG: FCBF F03D 9ADB B768 8B92 BB52 0341 9037 A875 942D ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
OK, I think I can state that there is a memory problem. I would not classify it as a leak in that valgrind does not show it, but that just means that the original pointer is not lost. It appears that syslog-ng keeps track of all the memory, but it just keeps acquiring more. I complied everything up with the Thread-Caching Malloc http://code.google.com/p/gperftools/ which does its own malloc, and then manages it all within the user space of the application making it much faster. By doing this I remove the question of the OS doing lazy free and the mallocs just piling up waiting for the OS to free them. The memory footprint grows continuously. The other thing that ctmalloc does is provide memory profiling tools. It might me interesting for the syslog-ng developers to use this feature to assist in debugging memory problems like the one I think I have. I just made a pipe, made syslog-ng read from it and write to a program destination of 'cat > /dev/null' Then cat a large file (200,000 lines, I used an exisitng syslog file) to the pipe. Watch the memory grow. Repeat the process and watch the memory grow more. Seems to grow by about 3 MB every time I cat the file. Evan. ________________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Evan Rempel [erempel@uvic.ca] Sent: Saturday, February 11, 2012 11:32 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] 3.3.4 anyone else notice a memory leak No, that does not help the issue at all. It almost seems to be that the OS is doing a lazy malloc/free and that syslog-ng is free(ing) the memory, but the OS keeps is associated with the application. Then, if syslog-ng needs to buffer some messages again it does a new malloc and the OS gives it a different block of with the end result that the memory footprint continues to grow. I saw this in 3.0 where the RAM footprint would grow when syslog-ng buffered memory, but it would take 3-4 hours after the destination was available and the buffer completely flushed before the RAM footprint would start to shrink and another 3-4 hours before it reached the original size prior to the buffering. I'm going to play with the caching malloc to see if that makes a difference, but with syslog-ng allocating memory blocks of differing sizes I'm not confident that the caching malloc will be able to do anything. Evan. ________________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Jakub Jankowski [shasta@toxcorp.com] Sent: Friday, February 10, 2012 4:10 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] 3.3.4 anyone else notice a memory leak On 2012-02-10, Evan Rempel wrote:
I am fairly certain that there is a memory leak. How do I help track it down? [...] Yup, I can reproduce it. The output of valgrind does not appear to show the leak. If I run syslog-ng and do NOT send any data to it (it processes no log lines) the output of valgrind is the same as if it processes 100,000 log lines, however, when I "dump" 100,000 log lines
% wc /home1l/erempel/log 107590 1233928 18956930 /home1l/erempel/log
into the pipe (cat /home1l/erempel/log > pipe) the memory footprint of syslog-ng grows by 3MB, every time I dump in the log lines.
Okay, this might be a long shot, but... can you try to revert commit [1], recompile syslog-ng and reproduce the leak? (grab the tarball, unpack it, get the raw form of [1], and use -R option to reverse the patch; then your usual build procedure) This commit was applied right after 3.3.2 was released, because it supposedly fixed crash with tcp destinations. Bazsi had doubts[2] that it may reintroduce a memory leak... and now I know it *does* reintroduce the leak, because it was leak I originally reported (longish thread here, see [3], [4], [5]) and now I'm seeing it again, with both 3.3.3 and 3.3.4: ==3645== LEAK SUMMARY: ==3645== definitely lost: 3,719,421 bytes in 10,008 blocks ==3645== indirectly lost: 905,917 bytes in 10,014 blocks ==3645== possibly lost: 17,214 bytes in 128 blocks ==3645== still reachable: 80,063 bytes in 3,278 blocks ==3645== suppressed: 0 bytes in 0 blocks I've just re-ran my tests[6] with commit [1] reverted, and the results are good again: ==7885== LEAK SUMMARY: ==7885== definitely lost: 297 bytes in 6 blocks ==7885== indirectly lost: 1,025 bytes in 22 blocks ==7885== possibly lost: 13,527 bytes in 110 blocks ==7885== still reachable: 106,235 bytes in 3,327 blocks ==7885== suppressed: 0 bytes in 0 blocks I understand that my use-case scenario is a bit different than yours, but perhaps the underlying issue is the same? Anyway - Bazsi, Gergely, could you please take a look at this whole thing? Because looks like we either end up with huge memleaks (with [1]) or tcp()-related crashes (without it). Hope this helps a bit. [1] http://git.balabit.hu/?p=bazsi/syslog-ng-3.3.git;a=commitdiff;h=c7070e2a6f1c... [2] https://lists.balabit.hu/pipermail/syslog-ng/2011-November/017697.html [3] https://lists.balabit.hu/pipermail/syslog-ng/2011-September/017269.html [4] https://lists.balabit.hu/pipermail/syslog-ng/2011-September/017275.html [5] https://lists.balabit.hu/pipermail/syslog-ng/2011-September/017338.html [6] https://lists.balabit.hu/pipermail/syslog-ng/2011-October/017555.html Regards, Jakub -- Jakub Jankowski|shasta@toxcorp.com|http://toxcorp.com/ GPG: FCBF F03D 9ADB B768 8B92 BB52 0341 9037 A875 942D ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Evan Rempel <erempel@uvic.ca> writes:
OK, I think I can state that there is a memory problem. I would not classify it as a leak in that valgrind does not show it, but that just means that the original pointer is not lost.
I believe that's what valgrind calls "still reachable".
I just made a pipe, made syslog-ng read from it and write to a program destination of 'cat > /dev/null'
Then cat a large file (200,000 lines, I used an exisitng syslog file) to the pipe. Watch the memory grow. Repeat the process and watch the memory grow more. Seems to grow by about 3 MB every time I cat the file.
Mmhm, that should be easy to reproduce. Thanks for the research so far, I'll look into it soon! -- |8]
On Fri, 2012-02-10 at 14:57 -0800, Evan Rempel wrote:
Gergely Nagy wrote:
Evan Rempel <erempel@uvic.ca> writes:
I am fairly certain that there is a memory leak. How do I help track it down?
Looking at your config, I don't recall any known problems in those areas, but I'll try to reproduce the problem.
Meanwhile, if you have time, running syslog-ng under valgrind would be very, very useful. If you do that, please set the G_SLICE environment variable to always-malloc, otherwise due to glib's allocator, the results will be severley screwed and hardly usable.
Yup, I can reproduce it. The output of valgrind does not appear to show the leak. If I run syslog-ng and do NOT send any data to it (it processes no log lines) the output of valgrind is the same as if it processes 100,000 log lines, however, when I "dump" 100,000 log lines
% wc /home1l/erempel/log 107590 1233928 18956930 /home1l/erempel/log
into the pipe (cat /home1l/erempel/log > pipe) the memory footprint of syslog-ng grows by 3MB, every time I dump in the log lines.
I've tried to reproduce it locally, but without success. Two patches in addition to 3.3.4, but neither seems to be related. $ cat etc/syslog-ng-leak.conf @version: 3.3 # # UVic syslog-ng configuration options { log_fifo_size(100000); use_fqdn(yes); keep_hostname(yes); chain_hostnames(yes); time_reap(60); time_reopen(5); flush_lines(1000); flush_timeout(1000); }; source local { pipe("logloglog" log_iw_size(90000) log_fetch_limit(500) flags(no-parse)); }; source int { internal(); }; destination localsyslog { tcp("localhost" port(1514) localip(localhost) log_fifo_size(50000) template("<$PRI>$S_DATE syslogstats@$HOST syslogstats: $MSGONLY\n") template_escape(no) ); }; destination syslog_stats { program("/bin/cat > /dev/null" template("$MESSAGE\n") template_escape(no) ); }; log { source(local); destination(syslog_stats); flags(flow-control); }; log { source(int); destination(localsyslog); }; The tcp destination wasn't working, but if I understand it doesn't matter in your usecase. $ wc pattern1 109290 643018 5927136 pattern1 $ ps axuw | grep syslog-ng bazsi 23689 10.3 0.1 68016 4260 pts/0 S+ 06:42 0:02 sbin/syslog-ng -Fef etc/syslog-ng-leak.conf bazsi 23741 0.0 0.0 12048 892 pts/1 S+ 06:42 0:00 grep syslog-ng bazsi@bzorp:~/.zwa/install/syslog-ng-ose-3.3$ cat pattern pattern pattern pattern1 > logloglog bazsi@bzorp:~/.zwa/install/syslog-ng-ose-3.3$ ps axuw | grep syslog-ng bazsi 23689 10.8 0.1 68020 4264 pts/0 S+ 06:42 0:03 sbin/syslog-ng -Fef etc/syslog-ng-leak.conf bazsi 23748 0.0 0.0 12048 888 pts/1 S+ 06:42 0:00 grep syslog-ng
I have attached the config file and two valgrind outputs.
plain text document attachment (valgrind-syslog-ng.txt) [root@catamount ~]# G_SLICE=always-malloc [root@catamount ~]# export G_SLICE [root@catamount ~]# /usr/bin/valgrind /usr/local/sbin/syslog-ng --cfgfile=/usr/local/etc/syslog-ng/syslog-ng.erempel.conf --persist-file=/var/local/syslog-ng.erempel.persist --pidfile=/var/run/syslog-ng.erempel.pid --foreground ==19346== Memcheck, a memory error detector ==19346== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al. ==19346== Using Valgrind-3.6.0 and LibVEX; rerun with -h for copyright info ==19346== Command: /usr/local/sbin/syslog-ng --cfgfile=/usr/local/etc/syslog-ng/syslog-ng.erempel.conf --persist-file=/var/local/syslog-ng.erempel.persist --pidfile=/var/run/syslog-ng.erempel.pid --foreground ==19346== syslog-ng: Error setting file number limit; limit='4096'; error='Operation not permitted' ==19346== Syscall param epoll_ctl(event) points to uninitialised byte(s) ==19346== at 0x64F3ECA: epoll_ctl (in /lib64/libc-2.12.so) ==19346== by 0x4E7F8FF: iv_epoll_pollable (iv_method_epoll.c:153) ==19346== by 0x4E6356B: log_writer_start_watches (logwriter.c:426) ==19346== by 0x4E62CFD: log_writer_reopen (logwriter.c:1229) ==19346== by 0x6DC25FD: afprogram_dd_reopen (afprog.c:280) ==19346== by 0x4E5096A: log_dest_group_init (logpipe.h:239) ==19346== by 0x4E4C166: log_center_init (logpipe.h:239) ==19346== by 0x4E4D3EF: cfg_init (cfg.c:255) ==19346== by 0x4E664BC: main_loop_init (mainloop.c:481) ==19346== by 0x401584: main (in /usr/local/sbin/syslog-ng) ==19346== Address 0x7ff000128 is on thread 1's stack ==19346== ^C==19346== Invalid read of size 8 ==19346== at 0x4E51A81: log_dest_driver_deinit_method (driver.c:172) ==19346== by 0x6DC284C: afprogram_dd_deinit (afprog.c:344) ==19346== by 0x4E50CAC: log_dest_group_deinit (logpipe.h:254) ==19346== by 0x4E4C575: log_center_deinit (logpipe.h:254) ==19346== by 0x4E659EF: main_loop_exit_finish (mainloop.c:585) ==19346== by 0x4E80EE8: iv_run_timers (iv_timer.c:345) ==19346== by 0x4E7F417: iv_main (iv_main.c:252) ==19346== by 0x4E6570D: main_loop_run (mainloop.c:731) ==19346== by 0x4015B0: main (in /usr/local/sbin/syslog-ng) ==19346== Address 0x684a928 is 8 bytes inside a block of size 24 free'd ==19346== at 0x4C2695D: free (vg_replace_malloc.c:366) ==19346== by 0x590EFDF: g_list_remove (in /lib64/libglib-2.0.so.0.2200.5) ==19346== by 0x4E51A5F: log_dest_driver_deinit_method (driver.h:190) ==19346== by 0x6DC284C: afprogram_dd_deinit (afprog.c:344) ==19346== by 0x4E50CAC: log_dest_group_deinit (logpipe.h:254) ==19346== by 0x4E4C575: log_center_deinit (logpipe.h:254) ==19346== by 0x4E659EF: main_loop_exit_finish (mainloop.c:585) ==19346== by 0x4E80EE8: iv_run_timers (iv_timer.c:345) ==19346== by 0x4E7F417: iv_main (iv_main.c:252) ==19346== by 0x4E6570D: main_loop_run (mainloop.c:731) ==19346== by 0x4015B0: main (in /usr/local/sbin/syslog-ng) ==19346== ==19346== Invalid read of size 8 ==19346== at 0x4E51A81: log_dest_driver_deinit_method (driver.c:172) ==19346== by 0x6FD07DA: afsocket_dd_deinit (afsocket.c:1221) ==19346== by 0x4E50CAC: log_dest_group_deinit (logpipe.h:254) ==19346== by 0x4E4C575: log_center_deinit (logpipe.h:254) ==19346== by 0x4E659EF: main_loop_exit_finish (mainloop.c:585) ==19346== by 0x4E80EE8: iv_run_timers (iv_timer.c:345) ==19346== by 0x4E7F417: iv_main (iv_main.c:252) ==19346== by 0x4E6570D: main_loop_run (mainloop.c:731) ==19346== by 0x4015B0: main (in /usr/local/sbin/syslog-ng) ==19346== Address 0x684f358 is 8 bytes inside a block of size 24 free'd ==19346== at 0x4C2695D: free (vg_replace_malloc.c:366) ==19346== by 0x590EFDF: g_list_remove (in /lib64/libglib-2.0.so.0.2200.5) ==19346== by 0x4E51A5F: log_dest_driver_deinit_method (driver.h:190) ==19346== by 0x6FD07DA: afsocket_dd_deinit (afsocket.c:1221) ==19346== by 0x4E50CAC: log_dest_group_deinit (logpipe.h:254) ==19346== by 0x4E4C575: log_center_deinit (logpipe.h:254) ==19346== by 0x4E659EF: main_loop_exit_finish (mainloop.c:585) ==19346== by 0x4E80EE8: iv_run_timers (iv_timer.c:345) ==19346== by 0x4E7F417: iv_main (iv_main.c:252) ==19346== by 0x4E6570D: main_loop_run (mainloop.c:731) ==19346== by 0x4015B0: main (in /usr/local/sbin/syslog-ng) ==19346== ==19346== ==19346== HEAP SUMMARY: ==19346== in use at exit: 169,334 bytes in 3,407 blocks ==19346== total heap usage: 226,539 allocs, 223,132 frees, 178,250,594 bytes allocated ==19346== ==19346== LEAK SUMMARY: ==19346== definitely lost: 197 bytes in 2 blocks ==19346== indirectly lost: 1,823 bytes in 22 blocks ==19346== possibly lost: 0 bytes in 0 blocks ==19346== still reachable: 167,314 bytes in 3,383 blocks ==19346== suppressed: 0 bytes in 0 blocks ==19346== Rerun with --leak-check=full to see details of leaked memory ==19346== ==19346== For counts of detected and suppressed errors, rerun with: -v ==19346== Use --track-origins=yes to see where uninitialised values come from ==19346== ERROR SUMMARY: 4 errors from 3 contexts (suppressed: 66 from 9)
plain text document attachment (valgrind-syslog-ng-leak-check.txt) [root@catamount ~]# /usr/bin/valgrind --leak-check=full /usr/local/sbin/syslog-ng --cfgfile=/usr/local/etc/syslog-ng/syslog-ng.erempel.conf --persist-file=/var/local/syslog-ng.erempel.persist --pidfile=/var/run/syslog-ng.erempel.pid --foreground ==19487== Memcheck, a memory error detector ==19487== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al. ==19487== Using Valgrind-3.6.0 and LibVEX; rerun with -h for copyright info ==19487== Command: /usr/local/sbin/syslog-ng --cfgfile=/usr/local/etc/syslog-ng/syslog-ng.erempel.conf --persist-file=/var/local/syslog-ng.erempel.persist --pidfile=/var/run/syslog-ng.erempel.pid --foreground ==19487== syslog-ng: Error setting file number limit; limit='4096'; error='Operation not permitted' ==19487== Syscall param epoll_ctl(event) points to uninitialised byte(s) ==19487== at 0x64F3ECA: epoll_ctl (in /lib64/libc-2.12.so) ==19487== by 0x4E7F8FF: iv_epoll_pollable (iv_method_epoll.c:153) ==19487== by 0x4E6356B: log_writer_start_watches (logwriter.c:426) ==19487== by 0x4E62CFD: log_writer_reopen (logwriter.c:1229) ==19487== by 0x6DC25FD: afprogram_dd_reopen (afprog.c:280) ==19487== by 0x4E5096A: log_dest_group_init (logpipe.h:239) ==19487== by 0x4E4C166: log_center_init (logpipe.h:239) ==19487== by 0x4E4D3EF: cfg_init (cfg.c:255) ==19487== by 0x4E664BC: main_loop_init (mainloop.c:481) ==19487== by 0x401584: main (in /usr/local/sbin/syslog-ng) ==19487== Address 0x7ff000128 is on thread 1's stack ==19487== ^C==19487== Invalid read of size 8 ==19487== at 0x4E51A81: log_dest_driver_deinit_method (driver.c:172) ==19487== by 0x6DC284C: afprogram_dd_deinit (afprog.c:344) ==19487== by 0x4E50CAC: log_dest_group_deinit (logpipe.h:254) ==19487== by 0x4E4C575: log_center_deinit (logpipe.h:254) ==19487== by 0x4E659EF: main_loop_exit_finish (mainloop.c:585) ==19487== by 0x4E80EE8: iv_run_timers (iv_timer.c:345) ==19487== by 0x4E7F417: iv_main (iv_main.c:252) ==19487== by 0x4E6570D: main_loop_run (mainloop.c:731) ==19487== by 0x4015B0: main (in /usr/local/sbin/syslog-ng) ==19487== Address 0x684a928 is 8 bytes inside a block of size 24 free'd ==19487== at 0x4C2695D: free (vg_replace_malloc.c:366) ==19487== by 0x590EFDF: g_list_remove (in /lib64/libglib-2.0.so.0.2200.5) ==19487== by 0x4E51A5F: log_dest_driver_deinit_method (driver.h:190) ==19487== by 0x6DC284C: afprogram_dd_deinit (afprog.c:344) ==19487== by 0x4E50CAC: log_dest_group_deinit (logpipe.h:254) ==19487== by 0x4E4C575: log_center_deinit (logpipe.h:254) ==19487== by 0x4E659EF: main_loop_exit_finish (mainloop.c:585) ==19487== by 0x4E80EE8: iv_run_timers (iv_timer.c:345) ==19487== by 0x4E7F417: iv_main (iv_main.c:252) ==19487== by 0x4E6570D: main_loop_run (mainloop.c:731) ==19487== by 0x4015B0: main (in /usr/local/sbin/syslog-ng) ==19487== ==19487== Invalid read of size 8 ==19487== at 0x4E51A81: log_dest_driver_deinit_method (driver.c:172) ==19487== by 0x6FD07DA: afsocket_dd_deinit (afsocket.c:1221) ==19487== by 0x4E50CAC: log_dest_group_deinit (logpipe.h:254) ==19487== by 0x4E4C575: log_center_deinit (logpipe.h:254) ==19487== by 0x4E659EF: main_loop_exit_finish (mainloop.c:585) ==19487== by 0x4E80EE8: iv_run_timers (iv_timer.c:345) ==19487== by 0x4E7F417: iv_main (iv_main.c:252) ==19487== by 0x4E6570D: main_loop_run (mainloop.c:731) ==19487== by 0x4015B0: main (in /usr/local/sbin/syslog-ng) ==19487== Address 0x684f358 is 8 bytes inside a block of size 24 free'd ==19487== at 0x4C2695D: free (vg_replace_malloc.c:366) ==19487== by 0x590EFDF: g_list_remove (in /lib64/libglib-2.0.so.0.2200.5) ==19487== by 0x4E51A5F: log_dest_driver_deinit_method (driver.h:190) ==19487== by 0x6FD07DA: afsocket_dd_deinit (afsocket.c:1221) ==19487== by 0x4E50CAC: log_dest_group_deinit (logpipe.h:254) ==19487== by 0x4E4C575: log_center_deinit (logpipe.h:254) ==19487== by 0x4E659EF: main_loop_exit_finish (mainloop.c:585) ==19487== by 0x4E80EE8: iv_run_timers (iv_timer.c:345) ==19487== by 0x4E7F417: iv_main (iv_main.c:252) ==19487== by 0x4E6570D: main_loop_run (mainloop.c:731) ==19487== by 0x4015B0: main (in /usr/local/sbin/syslog-ng) ==19487== ==19487== ==19487== HEAP SUMMARY: ==19487== in use at exit: 169,334 bytes in 3,407 blocks ==19487== total heap usage: 226,539 allocs, 223,132 frees, 178,250,594 bytes allocated ==19487== ==19487== 13 bytes in 1 blocks are definitely lost in loss record 11 of 604 ==19487== at 0x4C26FDE: malloc (vg_replace_malloc.c:236) ==19487== by 0x59197D2: g_malloc (in /lib64/libglib-2.0.so.0.2200.5) ==19487== by 0x5930FCD: g_strdup (in /lib64/libglib-2.0.so.0.2200.5) ==19487== by 0x4E6E999: log_tags_init (tags.c:162) ==19487== by 0x4E4B721: app_startup (apphook.c:120) ==19487== by 0x4E663AE: main_loop_init (mainloop.c:663) ==19487== by 0x401584: main (in /usr/local/sbin/syslog-ng) ==19487== ==19487== 2,007 (184 direct, 1,823 indirect) bytes in 1 blocks are definitely lost in loss record 589 of 604 ==19487== at 0x4C26FDE: malloc (vg_replace_malloc.c:236) ==19487== by 0x59197D2: g_malloc (in /lib64/libglib-2.0.so.0.2200.5) ==19487== by 0x4E544AF: g_process_set_argv_space (gprocess.c:500) ==19487== by 0x40148C: main (in /usr/local/sbin/syslog-ng) ==19487== ==19487== LEAK SUMMARY: ==19487== definitely lost: 197 bytes in 2 blocks ==19487== indirectly lost: 1,823 bytes in 22 blocks ==19487== possibly lost: 0 bytes in 0 blocks ==19487== still reachable: 167,314 bytes in 3,383 blocks ==19487== suppressed: 0 bytes in 0 blocks ==19487== Reachable blocks (those to which a pointer was found) are not shown. ==19487== To see them, rerun with: --leak-check=full --show-reachable=yes ==19487== ==19487== For counts of detected and suppressed errors, rerun with: -v ==19487== Use --track-origins=yes to see where uninitialised values come from ==19487== ERROR SUMMARY: 6 errors from 5 contexts (suppressed: 66 from 9)
plain text document attachment (syslog-ng.conf) @version: 3.3
options { log_fifo_size(1000000); # must be at least the biggest value of any log_is_size use_fqdn(yes); keep_hostname(yes); chain_hostnames(yes); time_reap(60); time_reopen(5); flush_lines(1000); flush_timeout(1000); };
source test { pipe("/var/log/syslog.pipes/test" log_fetch_limit(50) log_iw_size(10000) flags(no-parse) ); }; source int { internal(); };
template t_standardfile { template("$S_ISODATE $FULLHOST $FACILITY.$LEVEL $MSGHDR][$PROGRAM][$MESSAGE\n"); template_escape(no); };
destination localsyslog { tcp("localhost" port(1514) localip(catamount.comp.uvic.ca) log_fifo_size(5000) template("<$PRI>$S_ISODATE test@$HOST flexfilter_alert: $MSGONLY\n") template_escape(no) ); }; destination flexfilter { program("/usr/local/sbin/flexfilter /home1l/erempel/alert/fullsyslog.xml" template("$MESSAGE\n") template_escape(no) ); }; destination flexfilter2 { program("/bin/cat > /dev/null" template("$MESSAGE\n") template_escape(no) ); };
log { source(test); destination(flexfilter); flags(flow-control); }; log { source(int); destination(localsyslog); };
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Bazsi
participants (6)
-
Balazs Scheidler
-
Evan Rempel
-
Gergely Nagy
-
Hendrik Visage
-
Jakub Jankowski
-
Martin Holste