[syslog-ng]Syslog-NG 1.6.6 memory leak when sending UDP logs
syslog-ng@lists.balabit.hu
syslog-ng@lists.balabit.hu
Fri, 4 Mar 2005 11:48:29 -0500
This is a multipart message in MIME format.
--=_alternative 005C70AC85256FBA_=
Content-Type: text/plain; charset="US-ASCII"
Thanks Robert, and Balasz, I understand the functioon of libnet :).
I beleive it to be the source of my memory problems, and not syslog-ng
necessarily.
>The funny part is that this version of libnet seem to expect port
>numbers in host byte order whereas I pass it to libnet in network byte
>order. I'm almost confident that this used to work when I originally did
>the libnet support, judging the libnet changelog again, this was a
>change between 1.0 <-> 1.1
>
>Is your syslog-ng sending messages to the correct port? Can you check
>that with tcpdump for example? Or maybe you are using a big-endian
>machine?
Balasz, spoofed UDP packets are being sent properly, as far as I can
tell - the data is getting to the target properly.
tcpdump shows some minor strangeness - the source address is that of the
spoofed syslog host, which is to be expected, and the target host is
correct, as is the target port (514/UDP). What is strange is that all the
spopofed packets are all useing UDP/514 as the source.
An example of a tcpdump ron on the UDP spoofer
syslogmachine(syslogng1.testdomain.org):
10:10:39.4092332 IP cisco2121.testdomain.org.syslog >
syslogng2.testdomain.org.syslog: UDP, length 150
I don't thing the endianness is coming into play here. Also, I verified
that libnet was not installed prior to the 1.2.2 installtion, I am certain
that syslog-ng was compiled against 1.2.2.
The output of libnet-config --defines:
-D_BSD_SOURCE -D__BSD_SOURCE -D__FAVOR_BSD -DHAVE_NET_ETHERNET_H
The output of libnet-config --cflags:
The output of libnet-config --libs:
-lnet
One more thing, the only confiure options I used when compiling syslog-ng
was "--enable-spoof-source"
Oh, and I reran valgrind with Robert's exact command line - the output is
pretty much the same as I had prior:
# valgrind --tool=memcheck --trace-children=yes --leak-check=yes syslog-ng
==9888== Memcheck, a memory error detector for x86-linux.
==9888== Copyright (C) 2002-2004, and GNU GPL'd, by Julian Seward et al.
==9888== Using valgrind-2.2.0, a program supervision framework for
x86-linux.
==9888== Copyright (C) 2000-2004, and GNU GPL'd, by Julian Seward et al.
==9888== For more details, rerun with: -v
==9888==
==9888==
==9888== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 19 from 1)
==9888== malloc/free: in use at exit: 32811 bytes in 508 blocks.
==9888== malloc/free: 734 allocs, 226 frees, 40605 bytes allocated.
==9888== For counts of detected errors, rerun with: -v
==9888== searching for pointers to 508 not-freed blocks.
==9888== checked 1598824 bytes.
==9888==
==9888== LEAK SUMMARY:
==9888== definitely lost: 0 bytes in 0 blocks.
==9888== possibly lost: 0 bytes in 0 blocks.
==9888== still reachable: 32811 bytes in 508 blocks.
==9888== suppressed: 0 bytes in 0 blocks.
==9888== Reachable blocks (those to which a pointer was found) are not
shown.
==9888== To see them, rerun with: --show-reachable=yes
==9890== Invalid read of size 2
==9890== at 0x805A987: libnet_in_cksum (in
/root/syslog-ng-1.6.6/src/syslog-ng)
==9890== Address 0x1BA790BA is 178 bytes inside a block of size 179
alloc'd
==9890== at 0x1B902E28: malloc (vg_replace_malloc.c:131)
==9890== by 0x805912D: libnet_pblock_coalesce (in
/root/syslog-ng-1.6.6/src/syslog-ng)
==9890== by 0x804C063: do_handle_log (destinations.c:103)
==9890== by 0x804B5EC: do_distribute_log (center.c:149)
Balazs Scheidler <bazsi@balabit.hu>
Sent by: syslog-ng-admin@lists.balabit.hu
03/03/2005 07:27 AM
Please respond to
syslog-ng@lists.balabit.hu
To
syslog-ng@lists.balabit.hu
cc
Subject
Re: [syslog-ng]Syslog-NG 1.6.6 memory leak when sending UDP logs
On Wed, 2005-03-02 at 22:19 +0100, Roberto Nibali wrote:
> > io.c: Preparing fd 6 for writing
> > ==27361== Invalid read of size 2
>
> There seems to be a off-by-one error in a string. This is the result if
> you do something like follows:
This message is not triggered for me, but I'm going to try to use your
exact configuration as well.
> > ==27361== at 0x805A987: libnet_in_cksum (in
/usr/local/sbin/syslog-ng)
> > ==27361== Address 0x1BA764E2 is 178 bytes inside a block of size 179
> > alloc'd
>
> There seems to be a wrong free, not really a missing one.
>
> > ==27361== at 0x1B902E28: malloc (vg_replace_malloc.c:131)
> > ==27361== by 0x805912D: libnet_pblock_coalesce (in
> > /usr/local/sbin/syslog-ng)
> > ==27361== by 0x804C063: do_handle_log (destinations.c:103)
> > ==27361== by 0x804B5EC: do_distribute_log (center.c:149)
> > ==27361== by 0x804B02A: do_add_source_name (sources.c:289)
> > ==27361== by 0x804AA8C: do_handle_line (sources.c:75)
> > ==27361== by 0x804ADA5: do_read_line (sources.c:134)
> > ==27361== by 0x8054AF8: read_callback (in
/usr/local/sbin/syslog-ng)
> > ==27361== by 0x804A079: main_loop (main.c:253)
> > ==27361== by 0x804A75C: main (main.c:549)
> > io.c: Preparing fd 8 for writing
> > io.c: connecting using fd 11
> > io.c: connecting using fd 11
Again, this one does not show up in my valgrind output. In fact it
reports that no blocks are leaked.
I'm using 1.1.2.1-2 Debian package. The libnet changelog shows some
fixed leaks before 1.1.1, but as I see you also have a newer version.
Isn't it possible that you linked syslog-ng to an older libnet
statically and then upgraded your libnet package?
The funny part is that this version of libnet seem to expect port
numbers in host byte order whereas I pass it to libnet in network byte
order. I'm almost confident that this used to work when I originally did
the libnet support, judging the libnet changelog again, this was a
change between 1.0 <-> 1.1
Is your syslog-ng sending messages to the correct port? Can you check
that with tcpdump for example? Or maybe you are using a big-endian
machine?
This patch fixes the byte order issue, and I'm still hunting the memory
leak with your configuration:
diff -u -r1.25.4.6 afinet.c
--- afinet.c 5 Aug 2004 11:35:12 -0000 1.25.4.6
+++ afinet.c 3 Mar 2005 12:26:47 -0000
@@ -617,11 +617,10 @@
ADDRESS2SOCKADDR(msg->saddr, sizeof(src_addr),
(struct sockaddr *) &src_addr);
ADDRESS2SOCKADDR(self->super.dest_addr,
sizeof(dst_addr), (struct sockaddr *) &dst_addr);
-
libnet_clear_packet(self->lnet_ctx);
- udp = libnet_build_udp(src_addr.sin_port,
- dst_addr.sin_port,
+ udp = libnet_build_udp(ntohs(src_addr.sin_port),
+ ntohs(dst_addr.sin_port),
LIBNET_UDP_H +
msg_line->length,
0,
msg_line->data,
>
> > Which doesent say too much. I'm using libnet 1.1.2.1. The valgrind
> > message only appears once - and does not appear as the memory leak
> > contiues.
>
> Was libnet linked statically against syslog-ng?
Yes, libnet is linked in statically by default.
>
> > I'm no valgrind expert, but I'm guessing it leaks one byte for each
UDP
> > packet sent. Not sure why spoofing would cause this inside libnet.
>
> If you need to create a packet, you'd want to use libnet, unless you've
> got enough spare time to code. Otherwise I don't see why libnet could be
> used within syslog-ng.
syslog-ng uses libnet for creating UDP packets sent via a raw socket.
--
Bazsi
_______________________________________________
syslog-ng maillist - syslog-ng@lists.balabit.hu
https://lists.balabit.hu/mailman/listinfo/syslog-ng
Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
--=_alternative 005C70AC85256FBA_=
Content-Type: text/html; charset="US-ASCII"
<br><font size=2 face="sans-serif">Thanks Robert, and Balasz, I understand
the functioon of libnet :). </font>
<br>
<br>
<br><font size=2 face="sans-serif">I beleive it to be the source of my
memory problems, and not syslog-ng necessarily.</font>
<br>
<br>
<br>
<br><font size=2><tt>>The funny part is that this version of libnet
seem to expect port<br>
>numbers in host byte order whereas I pass it to libnet in network byte<br>
>order. I'm almost confident that this used to work when I originally
did<br>
>the libnet support, judging the libnet changelog again, this was a<br>
>change between 1.0 <-> 1.1<br>
></tt></font>
<br><font size=2><tt>>Is your syslog-ng sending messages to the correct
port? Can you check<br>
>that with tcpdump for example? Or maybe you are using a big-endian<br>
>machine?<br>
</tt></font>
<br>
<br><font size=2 face="sans-serif">Balasz, spoofed UDP packets are
being sent properly, as far as I can tell - the data is getting
to the target properly.</font>
<br><font size=2 face="sans-serif">tcpdump shows some minor strangeness
- the source address is that of the spoofed syslog host, which is
to be expected, and the target host is correct, as is the target port (514/UDP).
What is strange is that all the spopofed packets are all useing UDP/514
as the source.</font>
<br>
<br><font size=2 face="sans-serif">An example of a tcpdump ron on the UDP
spoofer syslogmachine(syslogng1.testdomain.org):</font>
<br>
<br><font size=2 face="sans-serif">10:10:39.4092332 IP cisco2121.testdomain.org.syslog
> syslogng2.testdomain.org.syslog: UDP, length 150</font>
<br>
<br><font size=2 face="sans-serif">I don't thing the endianness is coming
into play here. Also, I verified that libnet was not installed prior
to the 1.2.2 installtion, I am certain that syslog-ng was compiled against
1.2.2. </font>
<br>
<br><font size=2 face="sans-serif">The output of libnet-config --defines:</font>
<br><font size=2 face="sans-serif">-D_BSD_SOURCE -D__BSD_SOURCE -D__FAVOR_BSD
-DHAVE_NET_ETHERNET_H</font>
<br>
<br><font size=2 face="sans-serif">The output of libnet-config --cflags:</font>
<br>
<br>
<br><font size=2 face="sans-serif">The output of libnet-config --libs:</font>
<br><font size=2 face="sans-serif">-lnet</font>
<br>
<br><font size=2 face="sans-serif">One more thing, the only confiure options
I used when compiling syslog-ng was "--enable-spoof-source"</font>
<br>
<br><font size=2 face="sans-serif">Oh, and I reran valgrind with Robert's
exact command line - the output is pretty much the same as I had prior:</font>
<br>
<br><font size=2 face="sans-serif"><br>
# valgrind --tool=memcheck --trace-children=yes --leak-check=yes syslog-ng</font><font size=3>
</font><font size=2 face="sans-serif"><br>
==9888== Memcheck, a memory error detector for x86-linux.</font><font size=3>
</font><font size=2 face="sans-serif"><br>
==9888== Copyright (C) 2002-2004, and GNU GPL'd, by Julian Seward et al.</font><font size=3>
</font><font size=2 face="sans-serif"><br>
==9888== Using valgrind-2.2.0, a program supervision framework for x86-linux.</font><font size=3>
</font><font size=2 face="sans-serif"><br>
==9888== Copyright (C) 2000-2004, and GNU GPL'd, by Julian Seward et al.</font><font size=3>
</font><font size=2 face="sans-serif"><br>
==9888== For more details, rerun with: -v</font><font size=3> </font><font size=2 face="sans-serif"><br>
==9888==</font><font size=3> </font><font size=2 face="sans-serif"><br>
==9888==</font><font size=3> </font><font size=2 face="sans-serif"><br>
==9888== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 19 from 1)</font><font size=3>
</font><font size=2 face="sans-serif"><br>
==9888== malloc/free: in use at exit: 32811 bytes in 508 blocks.</font><font size=3>
</font><font size=2 face="sans-serif"><br>
==9888== malloc/free: 734 allocs, 226 frees, 40605 bytes allocated.</font><font size=3>
</font><font size=2 face="sans-serif"><br>
==9888== For counts of detected errors, rerun with: -v</font><font size=3>
</font><font size=2 face="sans-serif"><br>
==9888== searching for pointers to 508 not-freed blocks.</font><font size=3>
</font><font size=2 face="sans-serif"><br>
==9888== checked 1598824 bytes.</font><font size=3> </font><font size=2 face="sans-serif"><br>
==9888==</font><font size=3> </font><font size=2 face="sans-serif"><br>
==9888== LEAK SUMMARY:</font><font size=3> </font><font size=2 face="sans-serif"><br>
==9888== definitely lost: 0 bytes in 0 blocks.</font><font size=3>
</font><font size=2 face="sans-serif"><br>
==9888== possibly lost: 0 bytes in 0 blocks.</font><font size=3>
</font><font size=2 face="sans-serif"><br>
==9888== still reachable: 32811 bytes in 508 blocks.</font><font size=3>
</font><font size=2 face="sans-serif"><br>
==9888== suppressed: 0 bytes in 0 blocks.</font><font size=3>
</font><font size=2 face="sans-serif"><br>
==9888== Reachable blocks (those to which a pointer was found) are not
shown.</font><font size=3> </font><font size=2 face="sans-serif"><br>
==9888== To see them, rerun with: --show-reachable=yes</font><font size=3>
</font><font size=2 face="sans-serif"><br>
==9890== Invalid read of size 2</font><font size=3> </font><font size=2 face="sans-serif"><br>
==9890== at 0x805A987: libnet_in_cksum (in /root/syslog-ng-1.6.6/src/syslog-ng)</font><font size=3>
</font><font size=2 face="sans-serif"><br>
==9890== Address 0x1BA790BA is 178 bytes inside a block of size 179
alloc'd</font><font size=3> </font><font size=2 face="sans-serif"><br>
==9890== at 0x1B902E28: malloc (vg_replace_malloc.c:131)</font><font size=3>
</font><font size=2 face="sans-serif"><br>
==9890== by 0x805912D: libnet_pblock_coalesce (in /root/syslog-ng-1.6.6/src/syslog-ng)</font><font size=3>
</font><font size=2 face="sans-serif"><br>
==9890== by 0x804C063: do_handle_log (destinations.c:103)</font><font size=3>
</font><font size=2 face="sans-serif"><br>
==9890== by 0x804B5EC: do_distribute_log (center.c:149)</font><font size=3>
<br>
</font>
<br>
<br>
<br>
<table width=100%>
<tr valign=top>
<td width=40%><font size=1 face="sans-serif"><b>Balazs Scheidler <bazsi@balabit.hu></b>
</font>
<br><font size=1 face="sans-serif">Sent by: syslog-ng-admin@lists.balabit.hu</font>
<p><font size=1 face="sans-serif">03/03/2005 07:27 AM</font>
<table border>
<tr valign=top>
<td bgcolor=white>
<div align=center><font size=1 face="sans-serif">Please respond to<br>
syslog-ng@lists.balabit.hu</font></div></table>
<br>
<td width=59%>
<table width=100%>
<tr>
<td>
<div align=right><font size=1 face="sans-serif">To</font></div>
<td valign=top><font size=1 face="sans-serif">syslog-ng@lists.balabit.hu</font>
<tr>
<td>
<div align=right><font size=1 face="sans-serif">cc</font></div>
<td valign=top>
<tr>
<td>
<div align=right><font size=1 face="sans-serif">Subject</font></div>
<td valign=top><font size=1 face="sans-serif">Re: [syslog-ng]Syslog-NG
1.6.6 memory leak when sending UDP logs</font></table>
<br>
<table>
<tr valign=top>
<td>
<td></table>
<br></table>
<br>
<br>
<br><font size=2><tt>On Wed, 2005-03-02 at 22:19 +0100, Roberto Nibali
wrote:<br>
<br>
> > io.c: Preparing fd 6 for writing<br>
> > ==27361== Invalid read of size 2<br>
> <br>
> There seems to be a off-by-one error in a string. This is the result
if <br>
> you do something like follows:<br>
<br>
This message is not triggered for me, but I'm going to try to use your<br>
exact configuration as well.<br>
<br>
> > ==27361== at 0x805A987: libnet_in_cksum (in /usr/local/sbin/syslog-ng)<br>
> > ==27361== Address 0x1BA764E2 is 178 bytes inside a block
of size 179 <br>
> > alloc'd<br>
> <br>
> There seems to be a wrong free, not really a missing one.<br>
> <br>
> > ==27361== at 0x1B902E28: malloc (vg_replace_malloc.c:131)<br>
> > ==27361== by 0x805912D: libnet_pblock_coalesce (in
<br>
> > /usr/local/sbin/syslog-ng)<br>
> > ==27361== by 0x804C063: do_handle_log (destinations.c:103)<br>
> > ==27361== by 0x804B5EC: do_distribute_log (center.c:149)<br>
> > ==27361== by 0x804B02A: do_add_source_name (sources.c:289)<br>
> > ==27361== by 0x804AA8C: do_handle_line (sources.c:75)<br>
> > ==27361== by 0x804ADA5: do_read_line (sources.c:134)<br>
> > ==27361== by 0x8054AF8: read_callback (in /usr/local/sbin/syslog-ng)<br>
> > ==27361== by 0x804A079: main_loop (main.c:253)<br>
> > ==27361== by 0x804A75C: main (main.c:549)<br>
> > io.c: Preparing fd 8 for writing<br>
> > io.c: connecting using fd 11<br>
> > io.c: connecting using fd 11<br>
<br>
<br>
Again, this one does not show up in my valgrind output. In fact it<br>
reports that no blocks are leaked.<br>
<br>
I'm using 1.1.2.1-2 Debian package. The libnet changelog shows some<br>
fixed leaks before 1.1.1, but as I see you also have a newer version.<br>
<br>
Isn't it possible that you linked syslog-ng to an older libnet<br>
statically and then upgraded your libnet package?<br>
<br>
The funny part is that this version of libnet seem to expect port<br>
numbers in host byte order whereas I pass it to libnet in network byte<br>
order. I'm almost confident that this used to work when I originally did<br>
the libnet support, judging the libnet changelog again, this was a<br>
change between 1.0 <-> 1.1<br>
<br>
Is your syslog-ng sending messages to the correct port? Can you check<br>
that with tcpdump for example? Or maybe you are using a big-endian<br>
machine?<br>
<br>
This patch fixes the byte order issue, and I'm still hunting the memory<br>
leak with your configuration:<br>
<br>
diff -u -r1.25.4.6 afinet.c<br>
--- afinet.c 5 Aug 2004 11:35:12 -0000
1.25.4.6<br>
+++ afinet.c 3 Mar 2005 12:26:47 -0000<br>
@@ -617,11 +617,10 @@<br>
ADDRESS2SOCKADDR(msg->saddr, sizeof(src_addr), (struct
sockaddr *) &src_addr);<br>
ADDRESS2SOCKADDR(self->super.dest_addr, sizeof(dst_addr),
(struct sockaddr *) &dst_addr);<br>
<br>
-<br>
libnet_clear_packet(self->lnet_ctx);<br>
<br>
-
udp = libnet_build_udp(src_addr.sin_port,<br>
-
dst_addr.sin_port,<br>
+
udp = libnet_build_udp(ntohs(src_addr.sin_port),<br>
+
ntohs(dst_addr.sin_port),<br>
LIBNET_UDP_H + msg_line->length,<br>
0,<br>
msg_line->data,<br>
<br>
<br>
> <br>
> > Which doesent say too much. I'm using libnet 1.1.2.1. The
valgrind <br>
> > message only appears once - and does not appear as the memory
leak <br>
> > contiues.<br>
> <br>
> Was libnet linked statically against syslog-ng?<br>
<br>
Yes, libnet is linked in statically by default.<br>
<br>
> <br>
> > I'm no valgrind expert, but I'm guessing it leaks one byte for
each UDP <br>
> > packet sent. Not sure why spoofing would cause this inside libnet.
<br>
> <br>
> If you need to create a packet, you'd want to use libnet, unless you've
<br>
> got enough spare time to code. Otherwise I don't see why libnet could
be <br>
> used within syslog-ng.<br>
<br>
syslog-ng uses libnet for creating UDP packets sent via a raw socket.<br>
<br>
-- <br>
Bazsi<br>
<br>
<br>
_______________________________________________<br>
syslog-ng maillist - syslog-ng@lists.balabit.hu<br>
https://lists.balabit.hu/mailman/listinfo/syslog-ng<br>
Frequently asked questions at http://www.campin.net/syslog-ng/faq.html<br>
<br>
</tt></font>
<br>
--=_alternative 005C70AC85256FBA_=--