Syslog-NG 1.6.6 memory leak when sending UDP logs
I'm running 1.6.6 on Linux Redhat ES 3.0 logging Cisco router logs to local files and it's been running just fine for over a week. I just enabled forwarding for local4 and local7 (with spoof UDP) to 2 additional hosts via UDP and noticed, within a day, that it syslog-NG has consumed all available memory. It continues to log, however, the system starts killing processes off unless I restart syslog-ng. I checked the list, it seems like the closest similar problem I could find was back in 2000. Any advice for fixing/identifying the problem?
On Mon, 2005-02-28 at 09:36 -0500, henry@shoelacecity.com wrote:
I'm running 1.6.6 on Linux Redhat ES 3.0 logging Cisco router logs to local files and it's been running just fine for over a week. I just enabled forwarding for local4 and local7 (with spoof UDP) to 2 additional hosts via UDP and noticed, within a day, that it syslog-NG has consumed all available memory. It continues to log, however, the system starts killing processes off unless I restart syslog-ng.
If I understand you correctly you use syslog-ng to send logs to another host via UDP. Is it a new installation or this installation has worked so far and upgrading to 1.6.6 is what triggered the problem? Are you using address spoofing (spoof-source(yes))? Can you post your configuration file? -- Bazsi
The installation is a fresh 1.6.6 installation on a clean install of Redhat 3.0. It's been running fine for about 2 weeks logging the same amount of traffic to local files only. When I turned on the UDP destinations, it started chewing up memory. The destinations are receiving logs entries properly, and the source address is properly spoofed. Versions of libs: libnet 1.1.2 libol 0.3.14 Here's the syslog-ng config file - the only thing that changed to cause this memory eating behavior is the 3 lines under "# Sets up local destination for Cisco logs". It makes no difference if I am sending to a single host, or multiple, the memory "leak" is still there. I have it currently running under "ElectricFence" looking for malloc problems - so far none, just memory consumption. ------------------------------------------- options { long_hostnames(off); sync(0); }; source src { unix-stream("/dev/log"); internal(); }; source kernsrc { file("/proc/kmsg"); }; source net { udp(port(514)); }; # Sets up local desitation for Cisco logs destination ciscolog { file("/var/log/cisco.log"); }; # Sets up forwarding to other hosts destination loghostd { udp(loghostd spoof_source(yes)); }; destination loghosta { udp(loghosta spoof_source(yes)); }; destination scribe02 { udp(scribe02 spoof_source(yes)); }; # standard localhost logging stuff destination authlog { file("/var/log/auth.log"); }; destination syslog { file("/var/log/syslog"); }; destination cron { file("/var/log/cron.log"); }; destination daemon { file("/var/log/daemon.log"); }; destination kern { file("/var/log/kern.log"); }; destination lpr { file("/var/log/lpr.log"); }; destination user { file("/var/log/user.log"); }; destination uucp { file("/var/log/uucp.log"); }; destination ppp { file("/var/log/ppp.log"); }; destination mail { file("/var/log/mail.log"); }; destination mailinfo { file("/var/log/mail.info"); }; destination mailwarn { file("/var/log/mail.warn"); }; destination mailerr { file("/var/log/mail.err"); }; destination local0 { file("/var/log/local0.log"); }; destination local1 { file("/var/log/local1.log"); }; destination all { file("/var/log/all.log"); }; destination allnet { file("/var/log/allnet.log"); }; destination newscrit { file("/var/log/news/news.crit"); }; destination newserr { file("/var/log/news/news.err"); }; destination newsnotice { file("/var/log/news/news.notice"); }; destination debug { file("/var/log/debug"); }; destination messages { file("/var/log/messages"); }; destination console { usertty("root"); }; destination console_all { file("/dev/tty12"); }; # Filters Cisco events - well, at least local7 and local4 events filter f_cisco { facility(local7,local4);}; filter f_notcisco {not facility(local7, local4); }; filter f_auth { facility(auth); }; filter f_authpriv { facility(auth, authpriv); }; filter f_syslog { not facility(authpriv, mail) and not match(ppp.*LCP); }; filter f_cron { facility(cron); }; filter f_daemon { facility(daemon); }; filter f_kern { facility(kern); }; filter f_lpr { facility(lpr); }; filter f_mail { facility(mail) and not match (imapd); }; filter f_user { facility(user); }; filter f_uucp { facility(cron); }; filter f_ppp { program(ppp); }; filter f_news { facility(news); }; filter f_debug { not facility(auth, authpriv, news, mail) and not match(ppp.*LCP); }; filter f_messages { level(info..warn) and not facility(auth, authpriv, mail, news); }; filter f_emergency { level(emerg); }; filter f_info { level(info); }; filter f_notice { level(notice); }; filter f_warn { level(warn); }; filter f_crit { level(crit); }; filter f_err { level(err); }; # Reflects Cisco SYSLOG packets to remote hosts log { source(net); filter(f_cisco); destination(loghostd); }; log { source(net); filter(f_cisco); destination(loghosta); }; log { source(net); filter(f_cisco); destination(scribe02); }; # Keeps a copy for ourself log { source(net); filter(f_cisco); destination(ciscolog); }; log { source(src); filter(f_authpriv); destination(authlog); }; log { source(src); filter(f_syslog); destination(syslog); }; log { source(src); filter(f_cron); destination(cron); }; log { source(src); filter(f_daemon); destination(daemon); }; log { source(kernsrc); filter(f_kern); destination(kern); }; log { source(src); filter(f_lpr); destination(lpr); }; log { source(src); filter(f_mail); destination(mail); }; log { source(src); filter(f_user); destination(user); }; log { source(src); filter(f_uucp); destination(uucp); }; log { source(src); filter(f_mail); filter(f_info); destination(mailinfo); }; log { source(src); filter(f_mail); filter(f_warn); destination(mailwarn); }; log { source(src); filter(f_mail); filter(f_err); destination(mailerr); }; log { source(src); filter(f_news); filter(f_crit); destination(newscrit); }; log { source(src); filter(f_news); filter(f_err); destination(newserr); }; log { source(src); filter(f_news); filter(f_notice); destination(newsnotice); }; log { source(src); filter(f_debug); destination(debug); }; log { source(src); filter(f_messages); destination(messages); }; log { source(src); filter(f_emergency); destination(console); }; log { source(src); filter(f_ppp); destination(ppp); }; log { source(src); destination(console_all); }; log { source(src); destination(all); }; # Log all non cisco stuff to localfile for debugging log { source(net); filter(f_notcisco); destination(allnet); }; Henry - Balazs Scheidler <bazsi@balabit.hu> Sent by: syslog-ng-admin@lists.balabit.hu 02/28/2005 11:11 AM Please respond to syslog-ng@lists.balabit.hu To syslog-ng@lists.balabit.hu cc Subject Re: [syslog-ng]Syslog-NG 1.6.6 memory leak when sending UDP logs On Mon, 2005-02-28 at 09:36 -0500, henry@shoelacecity.com wrote:
I'm running 1.6.6 on Linux Redhat ES 3.0 logging Cisco router logs to local files and it's been running just fine for over a week. I just enabled forwarding for local4 and local7 (with spoof UDP) to 2 additional hosts via UDP and noticed, within a day, that it syslog-NG has consumed all available memory. It continues to log, however, the system starts killing processes off unless I restart syslog-ng.
If I understand you correctly you use syslog-ng to send logs to another host via UDP. Is it a new installation or this installation has worked so far and upgrading to 1.6.6 is what triggered the problem? Are you using address spoofing (spoof-source(yes))? Can you post your configuration file? -- Bazsi _______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
No sooner did I write this that ElectricFence reported back (after 2 hours of uptime): "ElectricFence: Exiting: mprotect() failed: Cannot allocate memory" The machine is a dual Proc P2 w/512 Mb RAM. syslog-ng initilizes with 10mb, and slowly consumes about a 1 mb per minute. henry@shoelacecity.com Sent by: syslog-ng-admin@lists.balabit.hu 02/28/2005 11:27 AM Please respond to syslog-ng@lists.balabit.hu To syslog-ng@lists.balabit.hu cc Subject Re: [syslog-ng]Syslog-NG 1.6.6 memory leak when sending UDP logs The installation is a fresh 1.6.6 installation on a clean install of Redhat 3.0. It's been running fine for about 2 weeks logging the same amount of traffic to local files only. When I turned on the UDP destinations, it started chewing up memory. The destinations are receiving logs entries properly, and the source address is properly spoofed. Versions of libs: libnet 1.1.2 libol 0.3.14 Here's the syslog-ng config file - the only thing that changed to cause this memory eating behavior is the 3 lines under "# Sets up local destination for Cisco logs". It makes no difference if I am sending to a single host, or multiple, the memory "leak" is still there. I have it currently running under "ElectricFence" looking for malloc problems - so far none, just memory consumption. ------------------------------------------- options { long_hostnames(off); sync(0); }; source src { unix-stream("/dev/log"); internal(); }; source kernsrc { file("/proc/kmsg"); }; source net { udp(port(514)); }; # Sets up local desitation for Cisco logs destination ciscolog { file("/var/log/cisco.log"); }; # Sets up forwarding to other hosts destination loghostd { udp(loghostd spoof_source(yes)); }; destination loghosta { udp(loghosta spoof_source(yes)); }; destination scribe02 { udp(scribe02 spoof_source(yes)); }; # standard localhost logging stuff destination authlog { file("/var/log/auth.log"); }; destination syslog { file("/var/log/syslog"); }; destination cron { file("/var/log/cron.log"); }; destination daemon { file("/var/log/daemon.log"); }; destination kern { file("/var/log/kern.log"); }; destination lpr { file("/var/log/lpr.log"); }; destination user { file("/var/log/user.log"); }; destination uucp { file("/var/log/uucp.log"); }; destination ppp { file("/var/log/ppp.log"); }; destination mail { file("/var/log/mail.log"); }; destination mailinfo { file("/var/log/mail.info"); }; destination mailwarn { file("/var/log/mail.warn"); }; destination mailerr { file("/var/log/mail.err"); }; destination local0 { file("/var/log/local0.log"); }; destination local1 { file("/var/log/local1.log"); }; destination all { file("/var/log/all.log"); }; destination allnet { file("/var/log/allnet.log"); }; destination newscrit { file("/var/log/news/news.crit"); }; destination newserr { file("/var/log/news/news.err"); }; destination newsnotice { file("/var/log/news/news.notice"); }; destination debug { file("/var/log/debug"); }; destination messages { file("/var/log/messages"); }; destination console { usertty("root"); }; destination console_all { file("/dev/tty12"); }; # Filters Cisco events - well, at least local7 and local4 events filter f_cisco { facility(local7,local4);}; filter f_notcisco {not facility(local7, local4); }; filter f_auth { facility(auth); }; filter f_authpriv { facility(auth, authpriv); }; filter f_syslog { not facility(authpriv, mail) and not match(ppp.*LCP); }; filter f_cron { facility(cron); }; filter f_daemon { facility(daemon); }; filter f_kern { facility(kern); }; filter f_lpr { facility(lpr); }; filter f_mail { facility(mail) and not match (imapd); }; filter f_user { facility(user); }; filter f_uucp { facility(cron); }; filter f_ppp { program(ppp); }; filter f_news { facility(news); }; filter f_debug { not facility(auth, authpriv, news, mail) and not match(ppp.*LCP); }; filter f_messages { level(info..warn) and not facility(auth, authpriv, mail, news); }; filter f_emergency { level(emerg); }; filter f_info { level(info); }; filter f_notice { level(notice); }; filter f_warn { level(warn); }; filter f_crit { level(crit); }; filter f_err { level(err); }; # Reflects Cisco SYSLOG packets to remote hosts log { source(net); filter(f_cisco); destination(loghostd); }; log { source(net); filter(f_cisco); destination(loghosta); }; log { source(net); filter(f_cisco); destination(scribe02); }; # Keeps a copy for ourself log { source(net); filter(f_cisco); destination(ciscolog); }; log { source(src); filter(f_authpriv); destination(authlog); }; log { source(src); filter(f_syslog); destination(syslog); }; log { source(src); filter(f_cron); destination(cron); }; log { source(src); filter(f_daemon); destination(daemon); }; log { source(kernsrc); filter(f_kern); destination(kern); }; log { source(src); filter(f_lpr); destination(lpr); }; log { source(src); filter(f_mail); destination(mail); }; log { source(src); filter(f_user); destination(user); }; log { source(src); filter(f_uucp); destination(uucp); }; log { source(src); filter(f_mail); filter(f_info); destination(mailinfo); }; log { source(src); filter(f_mail); filter(f_warn); destination(mailwarn); }; log { source(src); filter(f_mail); filter(f_err); destination(mailerr); }; log { source(src); filter(f_news); filter(f_crit); destination(newscrit); }; log { source(src); filter(f_news); filter(f_err); destination(newserr); }; log { source(src); filter(f_news); filter(f_notice); destination(newsnotice); }; log { source(src); filter(f_debug); destination(debug); }; log { source(src); filter(f_messages); destination(messages); }; log { source(src); filter(f_emergency); destination(console); }; log { source(src); filter(f_ppp); destination(ppp); }; log { source(src); destination(console_all); }; log { source(src); destination(all); }; # Log all non cisco stuff to localfile for debugging log { source(net); filter(f_notcisco); destination(allnet); }; Henry - Balazs Scheidler <bazsi@balabit.hu> Sent by: syslog-ng-admin@lists.balabit.hu 02/28/2005 11:11 AM Please respond to syslog-ng@lists.balabit.hu To syslog-ng@lists.balabit.hu cc Subject Re: [syslog-ng]Syslog-NG 1.6.6 memory leak when sending UDP logs On Mon, 2005-02-28 at 09:36 -0500, henry@shoelacecity.com wrote:
I'm running 1.6.6 on Linux Redhat ES 3.0 logging Cisco router logs to local files and it's been running just fine for over a week. I just enabled forwarding for local4 and local7 (with spoof UDP) to 2 additional hosts via UDP and noticed, within a day, that it syslog-NG has consumed all available memory. It continues to log, however, the system starts killing processes off unless I restart syslog-ng.
If I understand you correctly you use syslog-ng to send logs to another host via UDP. Is it a new installation or this installation has worked so far and upgrading to 1.6.6 is what triggered the problem? Are you using address spoofing (spoof-source(yes))? Can you post your configuration file? -- Bazsi _______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
On Mon, 2005-02-28 at 11:34 -0500, henry@shoelacecity.com wrote:
No sooner did I write this that ElectricFence reported back (after 2 hours of uptime):
"ElectricFence: Exiting: mprotect() failed: Cannot allocate memory"
the workaround probably is to disable spoof-source if you can do that in your infrastructure, it seems to be related to that function, I can have a closer look tomorrow. -- Bazsi
Fair enough. I've disabled udp spoof, but am keeping forwarding turned on - it seems to have plugged the hole. One of the major reasons for using syslong-ng is the spoofing feature. I look forward to getting to the cause. # Sets up forwarding to other hosts destination loghostd { udp(loghostd); }; destination loghosta { udp(loghosta); }; destination scribe02 { udp(scribe02); }; Balazs Scheidler <bazsi@balabit.hu> Sent by: syslog-ng-admin@lists.balabit.hu 02/28/2005 01:28 PM Please respond to syslog-ng@lists.balabit.hu To syslog-ng@lists.balabit.hu cc Subject Re: [syslog-ng]Syslog-NG 1.6.6 memory leak when sending UDP logs On Mon, 2005-02-28 at 11:34 -0500, henry@shoelacecity.com wrote:
No sooner did I write this that ElectricFence reported back (after 2 hours of uptime):
"ElectricFence: Exiting: mprotect() failed: Cannot allocate memory"
the workaround probably is to disable spoof-source if you can do that in your infrastructure, it seems to be related to that function, I can have a closer look tomorrow. -- Bazsi _______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
On Mon, 2005-02-28 at 13:49 -0500, henry@shoelacecity.com wrote:
Fair enough. I've disabled udp spoof, but am keeping forwarding turned on - it seems to have plugged the hole. One of the major reasons for using syslong-ng is the spoofing feature. I look forward to getting to the cause.
Can you check if this simple one-liner solves the problem? (if you don't have scsh, touch afinet.c.x after applying the patch to satisfy build dependencies) diff -u -r1.25.4.6 afinet.c --- afinet.c 5 Aug 2004 11:35:12 -0000 1.25.4.6 +++ afinet.c 28 Feb 2005 19:17:30 -0000 @@ -653,6 +653,7 @@ if (libnet_write(self->lnet_ctx) < 0) { werror("Error sending raw frame, error: %z", libnet_geterror(self->lnet_ctx)); } + ol_string_free(msg_line); } else { fallback_socket: -- Bazsi
That seems to have done the trick - I still see some small amount of memory consumption over time, but nowhere near the rate it was at before your suggestion. I'll monitor it overnight and see if it hits a plateau. But now, I think it's time for some pálinka! Henry - Balazs Scheidler <bazsi@balabit.hu> Sent by: syslog-ng-admin@lists.balabit.hu 02/28/2005 02:19 PM Please respond to syslog-ng@lists.balabit.hu To syslog-ng@lists.balabit.hu cc Subject Re: [syslog-ng]Syslog-NG 1.6.6 memory leak when sending UDP logs On Mon, 2005-02-28 at 13:49 -0500, henry@shoelacecity.com wrote:
Fair enough. I've disabled udp spoof, but am keeping forwarding turned on - it seems to have plugged the hole. One of the major reasons for using syslong-ng is the spoofing feature. I look forward to getting to the cause.
Can you check if this simple one-liner solves the problem? (if you don't have scsh, touch afinet.c.x after applying the patch to satisfy build dependencies) diff -u -r1.25.4.6 afinet.c --- afinet.c 5 Aug 2004 11:35:12 -0000 1.25.4.6 +++ afinet.c 28 Feb 2005 19:17:30 -0000 @@ -653,6 +653,7 @@ if (libnet_write(self->lnet_ctx) < 0) { werror("Error sending raw frame, error: %z", libnet_geterror(self->lnet_ctx)); } + ol_string_free(msg_line); } else { fallback_socket: -- Bazsi _______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
Hello,
Can you check if this simple one-liner solves the problem? (if you don't have scsh, touch afinet.c.x after applying the patch to satisfy build dependencies)
diff -u -r1.25.4.6 afinet.c --- afinet.c 5 Aug 2004 11:35:12 -0000 1.25.4.6 +++ afinet.c 28 Feb 2005 19:17:30 -0000 @@ -653,6 +653,7 @@ if (libnet_write(self->lnet_ctx) < 0) { werror("Error sending raw frame, error: %z", libnet_geterror(self->lnet_ctx)); } + ol_string_free(msg_line); } else { fallback_socket:
It looks very much like a leak. As I've heard last week from our development team, they seem to have seen similar massive leaks when using UDP spoofing. But I need to confirm that first. I'll let you know tomorrow or the latest by next Wednesday. The leak manifested itself when calling syslog() over a perl script or something similar, I don't exactly remember. The gross and I/O intesive solution was to fork()+exec(), but that's what you have SMP machines for, right? :). Thanks, Roberto Nibali, ratz -- echo '[q]sa[ln0=aln256%Pln256/snlbx]sb3135071790101768542287578439snlbxq'|dc
I let it run over night with Balazs' "patch". It's been running for about 20 hours, and it is still leaking memory - in the past 20 hours, it's up to 384 mb consumed. It seems to be a little slower, but that could be subjective on the amount of log traffic. Either way, it still exists. Roberto Nibali <ratz@drugphish.ch> Sent by: syslog-ng-admin@lists.balabit.hu 03/01/2005 03:06 AM Please respond to syslog-ng@lists.balabit.hu To syslog-ng@lists.balabit.hu cc Subject Re: [syslog-ng]Syslog-NG 1.6.6 memory leak when sending UDP logs Hello,
Can you check if this simple one-liner solves the problem? (if you don't have scsh, touch afinet.c.x after applying the patch to satisfy build dependencies)
diff -u -r1.25.4.6 afinet.c --- afinet.c 5 Aug 2004 11:35:12 -0000 1.25.4.6 +++ afinet.c 28 Feb 2005 19:17:30 -0000 @@ -653,6 +653,7 @@ if (libnet_write(self->lnet_ctx) < 0) { werror("Error sending raw frame, error: %z", libnet_geterror(self->lnet_ctx)); } + ol_string_free(msg_line); } else { fallback_socket:
It looks very much like a leak. As I've heard last week from our development team, they seem to have seen similar massive leaks when using UDP spoofing. But I need to confirm that first. I'll let you know tomorrow or the latest by next Wednesday. The leak manifested itself when calling syslog() over a perl script or something similar, I don't exactly remember. The gross and I/O intesive solution was to fork()+exec(), but that's what you have SMP machines for, right? :). Thanks, Roberto Nibali, ratz -- echo '[q]sa[ln0=aln256%Pln256/snlbx]sb3135071790101768542287578439snlbxq'|dc _______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
CORRIGENDUM:
It looks very much like a leak. As I've heard last week from our development team, they seem to have seen similar massive leaks when using UDP spoofing. But I need to confirm that first. I'll let you know tomorrow or the latest by next Wednesday.
Ok, I've checked back again with that developer and ...
The leak manifested itself when calling syslog() over a perl script or something similar, I don't exactly remember. The gross and I/O intesive
Actually it was a bug in the Perl Module Net::RawIP. We needed specially crafted UDP packets for sending them to syslog-ng. So my barking is wrong here and I do _NOT_ see this memory leak. Sorry for the noise and happy debugging :). Regards, Roberto Nibali, ratz -- ------------------------------------------------------------- addr://Rathausgasse 31, CH-5001 Aarau tel://++41 62 823 9355 http://www.terreactive.com fax://++41 62 823 9356 ------------------------------------------------------------- terreActive AG Wir sichern Ihren Erfolg -------------------------------------------------------------
Let me understand this, you were seeing a leak when using a PERL script to send UDP packets using Net::RawIP _to_ syslong_ng. I am still experiencing clear leak behavior when using syslog-ng to send spoffed UDP packets to other syslog-ng's. The syslog-ng sender is leaking, the receiver is exhibitning normal behavior. syslog-ng-admin@lists.balabit.hu wrote on 03/02/2005 08:42:23 AM:
CORRIGENDUM:
It looks very much like a leak. As I've heard last week from our development team, they seem to have seen similar massive leaks when using UDP spoofing. But I need to confirm that first. I'll let you know tomorrow or the latest by next Wednesday.
Ok, I've checked back again with that developer and ...
The leak manifested itself when calling syslog() over a perl script or something similar, I don't exactly remember. The gross and I/O intesive
Actually it was a bug in the Perl Module Net::RawIP. We needed specially crafted UDP packets for sending them to syslog-ng. So my barking is wrong here and I do _NOT_ see this memory leak. Sorry for the noise and happy debugging :).
Regards, Roberto Nibali, ratz -- ------------------------------------------------------------- addr://Rathausgasse 31, CH-5001 Aarau tel://++41 62 823 9355 http://www.terreactive.com fax://++41 62 823 9356 ------------------------------------------------------------- terreActive AG Wir sichern Ihren Erfolg ------------------------------------------------------------- _______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
henry@shoelacecity.com wrote:
Let me understand this, you were seeing a leak when using a PERL script to send UDP packets using Net::RawIP _to_ syslong_ng.
Exact, the perl-related process was leaking, no syslog involved. We had to hand-craft the UDP packets since the spoofing only works partially from our POV.
I am still experiencing clear leak behavior when using syslog-ng to send spoffed UDP packets to other syslog-ng's. The syslog-ng sender is leaking, the receiver is exhibitning normal behavior.
You could valgrind the process ... http://valgrind.kde.org/ Sorry for not being more helpful to you in this matter, Roberto Nibali, ratz -- ------------------------------------------------------------- addr://Rathausgasse 31, CH-5001 Aarau tel://++41 62 823 9355 http://www.terreactive.com fax://++41 62 823 9356 ------------------------------------------------------------- terreActive AG Wir sichern Ihren Erfolg -------------------------------------------------------------
I "ElectricFenced" it instead of valgrinding it. Efence did not detect any "problems" until it "ran out of memeory". I'll give valgrind a shot. Roberto Nibali <ratz@tac.ch> Sent by: syslog-ng-admin@lists.balabit.hu 03/02/2005 10:13 AM Please respond to syslog-ng@lists.balabit.hu To syslog-ng@lists.balabit.hu cc Subject Re: [syslog-ng]Syslog-NG 1.6.6 memory leak when sending UDP logs henry@shoelacecity.com wrote:
Let me understand this, you were seeing a leak when using a PERL script to send UDP packets using Net::RawIP _to_ syslong_ng.
Exact, the perl-related process was leaking, no syslog involved. We had to hand-craft the UDP packets since the spoofing only works partially from our POV.
I am still experiencing clear leak behavior when using syslog-ng to send spoffed UDP packets to other syslog-ng's. The syslog-ng sender is leaking, the receiver is exhibitning normal behavior.
You could valgrind the process ... http://valgrind.kde.org/ Sorry for not being more helpful to you in this matter, Roberto Nibali, ratz -- ------------------------------------------------------------- addr://Rathausgasse 31, CH-5001 Aarau tel://++41 62 823 9355 http://www.terreactive.com fax://++41 62 823 9356 ------------------------------------------------------------- terreActive AG Wir sichern Ihren Erfolg ------------------------------------------------------------- _______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
Ok, so Valgrind came up with something(thanks for the suggestion Robert): With UDP Spoof Turned on: ==27361== Memcheck, a memory error detector for x86-linux. ==27361== Copyright (C) 2002-2004, and GNU GPL'd, by Julian Seward et al. ==27361== Using valgrind-2.2.0, a program supervision framework for x86-linux. ==27361== Copyright (C) 2000-2004, and GNU GPL'd, by Julian Seward et al. ==27361== For more details, rerun with: -v ==27361== io.c: Preparing fd 3 for reading io.c: Preparing fd 4 for reading io.c: listening on fd 5 io.c: connecting using fd 6 io.c: connecting using fd 8 io.c: connecting using fd 8 syslog-ng version 1.6.6 starting io.c: Preparing fd 6 for writing ==27361== Invalid read of size 2 ==27361== at 0x805A987: libnet_in_cksum (in /usr/local/sbin/syslog-ng) ==27361== Address 0x1BA764E2 is 178 bytes inside a block of size 179 alloc'd ==27361== at 0x1B902E28: malloc (vg_replace_malloc.c:131) ==27361== by 0x805912D: libnet_pblock_coalesce (in /usr/local/sbin/syslog-ng) ==27361== by 0x804C063: do_handle_log (destinations.c:103) ==27361== by 0x804B5EC: do_distribute_log (center.c:149) ==27361== by 0x804B02A: do_add_source_name (sources.c:289) ==27361== by 0x804AA8C: do_handle_line (sources.c:75) ==27361== by 0x804ADA5: do_read_line (sources.c:134) ==27361== by 0x8054AF8: read_callback (in /usr/local/sbin/syslog-ng) ==27361== by 0x804A079: main_loop (main.c:253) ==27361== by 0x804A75C: main (main.c:549) io.c: Preparing fd 8 for writing io.c: connecting using fd 11 io.c: connecting using fd 11 With UDP spoof turned off: ==27373== Memcheck, a memory error detector for x86-linux. ==27373== Copyright (C) 2002-2004, and GNU GPL'd, by Julian Seward et al. ==27373== Using valgrind-2.2.0, a program supervision framework for x86-linux. ==27373== Copyright (C) 2000-2004, and GNU GPL'd, by Julian Seward et al. ==27373== For more details, rerun with: -v ==27373== io.c: Preparing fd 3 for reading io.c: Preparing fd 4 for reading io.c: listening on fd 5 io.c: connecting using fd 6 io.c: connecting using fd 7 io.c: connecting using fd 7 syslog-ng version 1.6.6 starting io.c: Preparing fd 6 for writing io.c: Preparing fd 7 for writing Which doesent say too much. I'm using libnet 1.1.2.1. The valgrind message only appears once - and does not appear as the memory leak contiues. I'm no valgrind expert, but I'm guessing it leaks one byte for each UDP packet sent. Not sure why spoofing would cause this inside libnet. Roberto Nibali <ratz@tac.ch> Sent by: syslog-ng-admin@lists.balabit.hu 03/02/2005 10:13 AM Please respond to syslog-ng@lists.balabit.hu To syslog-ng@lists.balabit.hu cc Subject Re: [syslog-ng]Syslog-NG 1.6.6 memory leak when sending UDP logs henry@shoelacecity.com wrote:
Let me understand this, you were seeing a leak when using a PERL script to send UDP packets using Net::RawIP _to_ syslong_ng.
Exact, the perl-related process was leaking, no syslog involved. We had to hand-craft the UDP packets since the spoofing only works partially from our POV.
I am still experiencing clear leak behavior when using syslog-ng to send spoffed UDP packets to other syslog-ng's. The syslog-ng sender is leaking, the receiver is exhibitning normal behavior.
You could valgrind the process ... http://valgrind.kde.org/ Sorry for not being more helpful to you in this matter, Roberto Nibali, ratz -- ------------------------------------------------------------- addr://Rathausgasse 31, CH-5001 Aarau tel://++41 62 823 9355 http://www.terreactive.com fax://++41 62 823 9356 ------------------------------------------------------------- terreActive AG Wir sichern Ihren Erfolg ------------------------------------------------------------- _______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
Hello, I'm not an expert either, but can you start valgrind as follows: valgrind --tool=memcheck --trace-children=yes --leak-check=yes ./syslog-ng If possible maybe you omit forking syslog-ng.
io.c: Preparing fd 6 for writing ==27361== Invalid read of size 2
There seems to be a off-by-one error in a string. This is the result if you do something like follows: char c; char *jj = malloc(10 * sizeof(char)); c = jj[10];
==27361== at 0x805A987: libnet_in_cksum (in /usr/local/sbin/syslog-ng) ==27361== Address 0x1BA764E2 is 178 bytes inside a block of size 179 alloc'd
There seems to be a wrong free, not really a missing one.
==27361== at 0x1B902E28: malloc (vg_replace_malloc.c:131) ==27361== by 0x805912D: libnet_pblock_coalesce (in /usr/local/sbin/syslog-ng) ==27361== by 0x804C063: do_handle_log (destinations.c:103) ==27361== by 0x804B5EC: do_distribute_log (center.c:149) ==27361== by 0x804B02A: do_add_source_name (sources.c:289) ==27361== by 0x804AA8C: do_handle_line (sources.c:75) ==27361== by 0x804ADA5: do_read_line (sources.c:134) ==27361== by 0x8054AF8: read_callback (in /usr/local/sbin/syslog-ng) ==27361== by 0x804A079: main_loop (main.c:253) ==27361== by 0x804A75C: main (main.c:549) io.c: Preparing fd 8 for writing io.c: connecting using fd 11 io.c: connecting using fd 11
Where's the end of it? :).
Which doesent say too much. I'm using libnet 1.1.2.1. The valgrind message only appears once - and does not appear as the memory leak contiues.
Was libnet linked statically against syslog-ng?
I'm no valgrind expert, but I'm guessing it leaks one byte for each UDP packet sent. Not sure why spoofing would cause this inside libnet.
If you need to create a packet, you'd want to use libnet, unless you've got enough spare time to code. Otherwise I don't see why libnet could be used within syslog-ng. Best regards, Roberto Nibali, ratz -- echo '[q]sa[ln0=aln256%Pln256/snlbx]sb3135071790101768542287578439snlbxq'|dc
On Wed, 2005-03-02 at 22:19 +0100, Roberto Nibali wrote:
io.c: Preparing fd 6 for writing ==27361== Invalid read of size 2
There seems to be a off-by-one error in a string. This is the result if you do something like follows:
This message is not triggered for me, but I'm going to try to use your exact configuration as well.
==27361== at 0x805A987: libnet_in_cksum (in /usr/local/sbin/syslog-ng) ==27361== Address 0x1BA764E2 is 178 bytes inside a block of size 179 alloc'd
There seems to be a wrong free, not really a missing one.
==27361== at 0x1B902E28: malloc (vg_replace_malloc.c:131) ==27361== by 0x805912D: libnet_pblock_coalesce (in /usr/local/sbin/syslog-ng) ==27361== by 0x804C063: do_handle_log (destinations.c:103) ==27361== by 0x804B5EC: do_distribute_log (center.c:149) ==27361== by 0x804B02A: do_add_source_name (sources.c:289) ==27361== by 0x804AA8C: do_handle_line (sources.c:75) ==27361== by 0x804ADA5: do_read_line (sources.c:134) ==27361== by 0x8054AF8: read_callback (in /usr/local/sbin/syslog-ng) ==27361== by 0x804A079: main_loop (main.c:253) ==27361== by 0x804A75C: main (main.c:549) io.c: Preparing fd 8 for writing io.c: connecting using fd 11 io.c: connecting using fd 11
Again, this one does not show up in my valgrind output. In fact it reports that no blocks are leaked. I'm using 1.1.2.1-2 Debian package. The libnet changelog shows some fixed leaks before 1.1.1, but as I see you also have a newer version. Isn't it possible that you linked syslog-ng to an older libnet statically and then upgraded your libnet package? The funny part is that this version of libnet seem to expect port numbers in host byte order whereas I pass it to libnet in network byte order. I'm almost confident that this used to work when I originally did the libnet support, judging the libnet changelog again, this was a change between 1.0 <-> 1.1 Is your syslog-ng sending messages to the correct port? Can you check that with tcpdump for example? Or maybe you are using a big-endian machine? This patch fixes the byte order issue, and I'm still hunting the memory leak with your configuration: diff -u -r1.25.4.6 afinet.c --- afinet.c 5 Aug 2004 11:35:12 -0000 1.25.4.6 +++ afinet.c 3 Mar 2005 12:26:47 -0000 @@ -617,11 +617,10 @@ ADDRESS2SOCKADDR(msg->saddr, sizeof(src_addr), (struct sockaddr *) &src_addr); ADDRESS2SOCKADDR(self->super.dest_addr, sizeof(dst_addr), (struct sockaddr *) &dst_addr); - libnet_clear_packet(self->lnet_ctx); - udp = libnet_build_udp(src_addr.sin_port, - dst_addr.sin_port, + udp = libnet_build_udp(ntohs(src_addr.sin_port), + ntohs(dst_addr.sin_port), LIBNET_UDP_H + msg_line->length, 0, msg_line->data,
Which doesent say too much. I'm using libnet 1.1.2.1. The valgrind message only appears once - and does not appear as the memory leak contiues.
Was libnet linked statically against syslog-ng?
Yes, libnet is linked in statically by default.
I'm no valgrind expert, but I'm guessing it leaks one byte for each UDP packet sent. Not sure why spoofing would cause this inside libnet.
If you need to create a packet, you'd want to use libnet, unless you've got enough spare time to code. Otherwise I don't see why libnet could be used within syslog-ng.
syslog-ng uses libnet for creating UDP packets sent via a raw socket. -- Bazsi
Thanks Robert, and Balasz, I understand the functioon of libnet :). I beleive it to be the source of my memory problems, and not syslog-ng necessarily.
The funny part is that this version of libnet seem to expect port numbers in host byte order whereas I pass it to libnet in network byte order. I'm almost confident that this used to work when I originally did the libnet support, judging the libnet changelog again, this was a change between 1.0 <-> 1.1
Is your syslog-ng sending messages to the correct port? Can you check that with tcpdump for example? Or maybe you are using a big-endian machine?
Balasz, spoofed UDP packets are being sent properly, as far as I can tell - the data is getting to the target properly. tcpdump shows some minor strangeness - the source address is that of the spoofed syslog host, which is to be expected, and the target host is correct, as is the target port (514/UDP). What is strange is that all the spopofed packets are all useing UDP/514 as the source. An example of a tcpdump ron on the UDP spoofer syslogmachine(syslogng1.testdomain.org): 10:10:39.4092332 IP cisco2121.testdomain.org.syslog > syslogng2.testdomain.org.syslog: UDP, length 150 I don't thing the endianness is coming into play here. Also, I verified that libnet was not installed prior to the 1.2.2 installtion, I am certain that syslog-ng was compiled against 1.2.2. The output of libnet-config --defines: -D_BSD_SOURCE -D__BSD_SOURCE -D__FAVOR_BSD -DHAVE_NET_ETHERNET_H The output of libnet-config --cflags: The output of libnet-config --libs: -lnet One more thing, the only confiure options I used when compiling syslog-ng was "--enable-spoof-source" Oh, and I reran valgrind with Robert's exact command line - the output is pretty much the same as I had prior: # valgrind --tool=memcheck --trace-children=yes --leak-check=yes syslog-ng ==9888== Memcheck, a memory error detector for x86-linux. ==9888== Copyright (C) 2002-2004, and GNU GPL'd, by Julian Seward et al. ==9888== Using valgrind-2.2.0, a program supervision framework for x86-linux. ==9888== Copyright (C) 2000-2004, and GNU GPL'd, by Julian Seward et al. ==9888== For more details, rerun with: -v ==9888== ==9888== ==9888== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 19 from 1) ==9888== malloc/free: in use at exit: 32811 bytes in 508 blocks. ==9888== malloc/free: 734 allocs, 226 frees, 40605 bytes allocated. ==9888== For counts of detected errors, rerun with: -v ==9888== searching for pointers to 508 not-freed blocks. ==9888== checked 1598824 bytes. ==9888== ==9888== LEAK SUMMARY: ==9888== definitely lost: 0 bytes in 0 blocks. ==9888== possibly lost: 0 bytes in 0 blocks. ==9888== still reachable: 32811 bytes in 508 blocks. ==9888== suppressed: 0 bytes in 0 blocks. ==9888== Reachable blocks (those to which a pointer was found) are not shown. ==9888== To see them, rerun with: --show-reachable=yes ==9890== Invalid read of size 2 ==9890== at 0x805A987: libnet_in_cksum (in /root/syslog-ng-1.6.6/src/syslog-ng) ==9890== Address 0x1BA790BA is 178 bytes inside a block of size 179 alloc'd ==9890== at 0x1B902E28: malloc (vg_replace_malloc.c:131) ==9890== by 0x805912D: libnet_pblock_coalesce (in /root/syslog-ng-1.6.6/src/syslog-ng) ==9890== by 0x804C063: do_handle_log (destinations.c:103) ==9890== by 0x804B5EC: do_distribute_log (center.c:149) Balazs Scheidler <bazsi@balabit.hu> Sent by: syslog-ng-admin@lists.balabit.hu 03/03/2005 07:27 AM Please respond to syslog-ng@lists.balabit.hu To syslog-ng@lists.balabit.hu cc Subject Re: [syslog-ng]Syslog-NG 1.6.6 memory leak when sending UDP logs On Wed, 2005-03-02 at 22:19 +0100, Roberto Nibali wrote:
io.c: Preparing fd 6 for writing ==27361== Invalid read of size 2
There seems to be a off-by-one error in a string. This is the result if you do something like follows:
This message is not triggered for me, but I'm going to try to use your exact configuration as well.
==27361== at 0x805A987: libnet_in_cksum (in /usr/local/sbin/syslog-ng) ==27361== Address 0x1BA764E2 is 178 bytes inside a block of size 179 alloc'd
There seems to be a wrong free, not really a missing one.
==27361== at 0x1B902E28: malloc (vg_replace_malloc.c:131) ==27361== by 0x805912D: libnet_pblock_coalesce (in /usr/local/sbin/syslog-ng) ==27361== by 0x804C063: do_handle_log (destinations.c:103) ==27361== by 0x804B5EC: do_distribute_log (center.c:149) ==27361== by 0x804B02A: do_add_source_name (sources.c:289) ==27361== by 0x804AA8C: do_handle_line (sources.c:75) ==27361== by 0x804ADA5: do_read_line (sources.c:134) ==27361== by 0x8054AF8: read_callback (in /usr/local/sbin/syslog-ng) ==27361== by 0x804A079: main_loop (main.c:253) ==27361== by 0x804A75C: main (main.c:549) io.c: Preparing fd 8 for writing io.c: connecting using fd 11 io.c: connecting using fd 11
Again, this one does not show up in my valgrind output. In fact it reports that no blocks are leaked. I'm using 1.1.2.1-2 Debian package. The libnet changelog shows some fixed leaks before 1.1.1, but as I see you also have a newer version. Isn't it possible that you linked syslog-ng to an older libnet statically and then upgraded your libnet package? The funny part is that this version of libnet seem to expect port numbers in host byte order whereas I pass it to libnet in network byte order. I'm almost confident that this used to work when I originally did the libnet support, judging the libnet changelog again, this was a change between 1.0 <-> 1.1 Is your syslog-ng sending messages to the correct port? Can you check that with tcpdump for example? Or maybe you are using a big-endian machine? This patch fixes the byte order issue, and I'm still hunting the memory leak with your configuration: diff -u -r1.25.4.6 afinet.c --- afinet.c 5 Aug 2004 11:35:12 -0000 1.25.4.6 +++ afinet.c 3 Mar 2005 12:26:47 -0000 @@ -617,11 +617,10 @@ ADDRESS2SOCKADDR(msg->saddr, sizeof(src_addr), (struct sockaddr *) &src_addr); ADDRESS2SOCKADDR(self->super.dest_addr, sizeof(dst_addr), (struct sockaddr *) &dst_addr); - libnet_clear_packet(self->lnet_ctx); - udp = libnet_build_udp(src_addr.sin_port, - dst_addr.sin_port, + udp = libnet_build_udp(ntohs(src_addr.sin_port), + ntohs(dst_addr.sin_port), LIBNET_UDP_H + msg_line->length, 0, msg_line->data,
Which doesent say too much. I'm using libnet 1.1.2.1. The valgrind message only appears once - and does not appear as the memory leak contiues.
Was libnet linked statically against syslog-ng?
Yes, libnet is linked in statically by default.
I'm no valgrind expert, but I'm guessing it leaks one byte for each
UDP
packet sent. Not sure why spoofing would cause this inside libnet.
If you need to create a packet, you'd want to use libnet, unless you've got enough spare time to code. Otherwise I don't see why libnet could be
used within syslog-ng.
syslog-ng uses libnet for creating UDP packets sent via a raw socket. -- Bazsi _______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
On Fri, 2005-03-04 at 11:48 -0500, henry@shoelacecity.com wrote:
The funny part is that this version of libnet seem to expect port numbers in host byte order whereas I pass it to libnet in network byte order. I'm almost confident that this used to work when I originally did the libnet support, judging the libnet changelog again, this was a change between 1.0 <-> 1.1
Is your syslog-ng sending messages to the correct port? Can you check that with tcpdump for example? Or maybe you are using a big-endian machine?
Balasz, spoofed UDP packets are being sent properly, as far as I can tell - the data is getting to the target properly.
This is suspicious, unless you are using a non-x86 machine there should be some problems with the port number as it was the case for my local installation when trying to reproduce the problem.
tcpdump shows some minor strangeness - the source address is that of the spoofed syslog host, which is to be expected, and the target host is correct, as is the target port (514/UDP). What is strange is that all the spopofed packets are all useing UDP/514 as the source.
syslog-ng spoofs the source IP and source port as well, so it uses the same port number as the originating syslog sender.
An example of a tcpdump ron on the UDP spoofer syslogmachine(syslogng1.testdomain.org):
10:10:39.4092332 IP cisco2121.testdomain.org.syslog > syslogng2.testdomain.org.syslog: UDP, length 150
I don't thing the endianness is coming into play here. Also, I verified that libnet was not installed prior to the 1.2.2 installtion, I am certain that syslog-ng was compiled against 1.2.2.
There is no version 1.2.2, the latest version is 1.1.2.1 (or 1.1.3 which is BETA) -- Bazsi
participants (4)
-
Balazs Scheidler
-
henry＠shoelacecity.com
-
Roberto Nibali
-
Roberto Nibali