spoof-source performance issues
I'm finding that with a destination of udp("10.x.x.x", port(514) spoof_source(yes)) about half of messages get lost when going from one syslog-ng host to another at a high message rate (> 3k/sec). This is on 3.1 OSE and the hosts are on the same subnet and switch, so there shouldn't be any network devices interfering. Has anyone else had this same issue? My hunch is that it's either a performance issue with the way the libnet (I'm using 1.1.2.1 on SuSE 10.2) API is implemented or it's an issue within the libnet API. Has anyone else noticed performance problems when using spoof_source?
Hi, I think it will be an udp kernel buffer problem (and not syslog problem), see the earlier thread of "[syslog-ng] Tests using loggen - not receiving all the packets" in this mail list. 2010.06.28. 20:21 keltezéssel, Martin Holste írta:
I'm finding that with a destination of udp("10.x.x.x", port(514) spoof_source(yes)) about half of messages get lost when going from one syslog-ng host to another at a high message rate (> 3k/sec). This is on 3.1 OSE and the hosts are on the same subnet and switch, so there shouldn't be any network devices interfering. Has anyone else had this same issue? My hunch is that it's either a performance issue with the way the libnet (I'm using 1.1.2.1 on SuSE 10.2) API is implemented or it's an issue within the libnet API. Has anyone else noticed performance problems when using spoof_source? ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- pzolee
Actually, I did more research on this and found that two separate people back in 2007 had this same problem on the mailing list. See threads "Lost packets; UDP Checksum (chksum) errors; forwarding - source spoofing; libnet bug" as well as "Forwarding + Spoofing = Errors & Dropped Packets?" I believe I've definitively proven the problem to be invalid UDP checksums sent by libnet 1.1.2.1 as indicated in the first thread by Marvin Nipper. Further research shows that there is a Linux kernel-level setting that can act as a workaround by setting the socket option SO_NO_CHECK, which disables checksum verifications. So, either Syslog-NG needs to incorporate a newer, fixed libnet version (it was indicated that it did not compile using 1.1.3 Beta), or a socket option for receiving needs to be set or made as an available option to set like the receive buffer. On Mon, Jun 28, 2010 at 1:40 PM, Zoltán Pallagi <pzolee@balabit.hu> wrote:
Hi,
I think it will be an udp kernel buffer problem (and not syslog problem), see the earlier thread of "[syslog-ng] Tests using loggen - not receiving all the packets" in this mail list.
2010.06.28. 20:21 keltezéssel, Martin Holste írta:
I'm finding that with a destination of udp("10.x.x.x", port(514) spoof_source(yes)) about half of messages get lost when going from one syslog-ng host to another at a high message rate (> 3k/sec). This is on 3.1 OSE and the hosts are on the same subnet and switch, so there shouldn't be any network devices interfering. Has anyone else had this same issue? My hunch is that it's either a performance issue with the way the libnet (I'm using 1.1.2.1 on SuSE 10.2) API is implemented or it's an issue within the libnet API. Has anyone else noticed performance problems when using spoof_source?
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- pzolee
Hi, Disabling checksums would be a very bad workaround. If you're using a buggy libnet version then it is up to you to fix it - or build syslog-ng against a fixed version as libnet is linked statically. Linux distributors either ship a patched libnet 1.1.2.1 version or ship 1.1.4 instead which doesn't have the checksum bug. Regards, Sandor On Tue, Jun 29, 2010 at 4:31 AM, Martin Holste <mcholste@gmail.com> wrote:
Actually, I did more research on this and found that two separate people back in 2007 had this same problem on the mailing list. See threads "Lost packets; UDP Checksum (chksum) errors; forwarding - source spoofing; libnet bug" as well as "Forwarding + Spoofing = Errors & Dropped Packets?" I believe I've definitively proven the problem to be invalid UDP checksums sent by libnet 1.1.2.1 as indicated in the first thread by Marvin Nipper. Further research shows that there is a Linux kernel-level setting that can act as a workaround by setting the socket option SO_NO_CHECK, which disables checksum verifications. So, either Syslog-NG needs to incorporate a newer, fixed libnet version (it was indicated that it did not compile using 1.1.3 Beta), or a socket option for receiving needs to be set or made as an available option to set like the receive buffer.
On Mon, Jun 28, 2010 at 1:40 PM, Zoltán Pallagi <pzolee@balabit.hu> wrote:
Hi,
I think it will be an udp kernel buffer problem (and not syslog problem), see the earlier thread of "[syslog-ng] Tests using loggen - not receiving all the packets" in this mail list.
2010.06.28. 20:21 keltezéssel, Martin Holste írta:
I'm finding that with a destination of udp("10.x.x.x", port(514) spoof_source(yes)) about half of messages get lost when going from one syslog-ng host to another at a high message rate (> 3k/sec). This is on 3.1 OSE and the hosts are on the same subnet and switch, so there shouldn't be any network devices interfering. Has anyone else had this same issue? My hunch is that it's either a performance issue with the way the libnet (I'm using 1.1.2.1 on SuSE 10.2) API is implemented or it's an issue within the libnet API. Has anyone else noticed performance problems when using spoof_source?
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- pzolee
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
I agree that is a better solution. So I should have no problems compiling against 1.1.4? I'm on SuSE 10.2 which has 1.1.2.1 (apparently unpatched), so I guess some vendors are behind. I looked at the src rpm for SuSE 11 and it is also missing the correct checksum code, so as far as I can tell, spoof_source will never work correctly on SuSE without manual patching. On Tue, Jun 29, 2010 at 4:30 AM, Sandor Geller <Sandor.Geller@morganstanley.com> wrote:
Hi,
Disabling checksums would be a very bad workaround. If you're using a buggy libnet version then it is up to you to fix it - or build syslog-ng against a fixed version as libnet is linked statically. Linux distributors either ship a patched libnet 1.1.2.1 version or ship 1.1.4 instead which doesn't have the checksum bug.
Regards,
Sandor
On Tue, Jun 29, 2010 at 4:31 AM, Martin Holste <mcholste@gmail.com> wrote:
Actually, I did more research on this and found that two separate people back in 2007 had this same problem on the mailing list. See threads "Lost packets; UDP Checksum (chksum) errors; forwarding - source spoofing; libnet bug" as well as "Forwarding + Spoofing = Errors & Dropped Packets?" I believe I've definitively proven the problem to be invalid UDP checksums sent by libnet 1.1.2.1 as indicated in the first thread by Marvin Nipper. Further research shows that there is a Linux kernel-level setting that can act as a workaround by setting the socket option SO_NO_CHECK, which disables checksum verifications. So, either Syslog-NG needs to incorporate a newer, fixed libnet version (it was indicated that it did not compile using 1.1.3 Beta), or a socket option for receiving needs to be set or made as an available option to set like the receive buffer.
On Mon, Jun 28, 2010 at 1:40 PM, Zoltán Pallagi <pzolee@balabit.hu> wrote:
Hi,
I think it will be an udp kernel buffer problem (and not syslog problem), see the earlier thread of "[syslog-ng] Tests using loggen - not receiving all the packets" in this mail list.
2010.06.28. 20:21 keltezéssel, Martin Holste írta:
I'm finding that with a destination of udp("10.x.x.x", port(514) spoof_source(yes)) about half of messages get lost when going from one syslog-ng host to another at a high message rate (> 3k/sec). This is on 3.1 OSE and the hosts are on the same subnet and switch, so there shouldn't be any network devices interfering. Has anyone else had this same issue? My hunch is that it's either a performance issue with the way the libnet (I'm using 1.1.2.1 on SuSE 10.2) API is implemented or it's an issue within the libnet API. Has anyone else noticed performance problems when using spoof_source?
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- pzolee
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Hello, 2010-06-29 16:52 keltezéssel, Martin Holste írta:
I agree that is a better solution. So I should have no problems compiling against 1.1.4? I'm on SuSE 10.2 That is EoL for more than a year...
which has 1.1.2.1 (apparently unpatched), so I guess some vendors are behind. I looked at the src rpm for SuSE 11 Which SuSE version exactly? 11.0 (also EoL soon), 11.1, 11.2 or 11.3, which is still in development. If this last one, we could still get it fixed (OK, complete freeze of factory was just announced today, but even if it is too late there, it would be a good candidate for on-line update)
and it is also missing the correct checksum code, so as far as I can tell, spoof_source will never work correctly on SuSE without manual patching.
This is the content of the latest source rpm: -rw-r--r-- 1 czanik users 913 2007 jan 16 libnet-1.1.2.1-arrray-fix.diff -rw-r--r-- 1 czanik users 437 2007 jan 16 libnet-1.1.2.1-makefile.diff -rw-r--r-- 1 czanik users 1700 2007 jan 16 libnet-1.1.2.1-strict-aliasing-fix.diff -rw-r--r-- 1 czanik users 505 2007 jan 16 libnet-1.1.2.1-uninitialized-fix.diff -rw-r--r-- 1 czanik users 1351 2007 jan 16 libnet-endianess-fix.diff -rw-r--r-- 1 czanik users 1203 2009 okt 4 libnet-shared.diff -rw-r--r-- 1 czanik users 4663 jún 10 02.53 libnet.spec -rw-r--r-- 1 czanik users 767201 2007 jan 16 libnet.tar.bz2 So, it still seems to be missing in factory (patches were not touched since previous release). Could somebody point me to the missing patch with a description what it fixes exactly? I'll pass it on to SuSE release manager and libnet maintainers as soon as I receive it. Thanks, bye, CzP
On Tue, Jun 29, 2010 at 4:30 AM, Sandor Geller <Sandor.Geller@morganstanley.com> wrote:
Hi,
Disabling checksums would be a very bad workaround. If you're using a buggy libnet version then it is up to you to fix it - or build syslog-ng against a fixed version as libnet is linked statically. Linux distributors either ship a patched libnet 1.1.2.1 version or ship 1.1.4 instead which doesn't have the checksum bug.
Regards,
Sandor
On Tue, Jun 29, 2010 at 4:31 AM, Martin Holste <mcholste@gmail.com> wrote:
Actually, I did more research on this and found that two separate people back in 2007 had this same problem on the mailing list. See threads "Lost packets; UDP Checksum (chksum) errors; forwarding - source spoofing; libnet bug" as well as "Forwarding + Spoofing = Errors & Dropped Packets?" I believe I've definitively proven the problem to be invalid UDP checksums sent by libnet 1.1.2.1 as indicated in the first thread by Marvin Nipper. Further research shows that there is a Linux kernel-level setting that can act as a workaround by setting the socket option SO_NO_CHECK, which disables checksum verifications. So, either Syslog-NG needs to incorporate a newer, fixed libnet version (it was indicated that it did not compile using 1.1.3 Beta), or a socket option for receiving needs to be set or made as an available option to set like the receive buffer.
On Mon, Jun 28, 2010 at 1:40 PM, Zoltán Pallagi <pzolee@balabit.hu> wrote:
Hi,
I think it will be an udp kernel buffer problem (and not syslog problem), see the earlier thread of "[syslog-ng] Tests using loggen - not receiving all the packets" in this mail list.
2010.06.28. 20:21 keltezéssel, Martin Holste írta:
I'm finding that with a destination of udp("10.x.x.x", port(514) spoof_source(yes)) about half of messages get lost when going from one syslog-ng host to another at a high message rate (> 3k/sec). This is on 3.1 OSE and the hosts are on the same subnet and switch, so there shouldn't be any network devices interfering. Has anyone else had this same issue? My hunch is that it's either a performance issue with the way the libnet (I'm using 1.1.2.1 on SuSE 10.2) API is implemented or it's an issue within the libnet API. Has anyone else noticed performance problems when using spoof_source?
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- pzolee
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Yep, I was looking at the latest 11 release SRPM I could find, which was still libnet-1.1.2.1-140.22. In one of the previous threads on this mailing list, a very valuable link was provided which has the exact source code needed for the patch: http://www.securityfocus.com/archive/89/384197/30/90/threaded . On Tue, Jun 29, 2010 at 11:44 AM, Peter Czanik <czanik@balabit.hu> wrote:
Hello,
2010-06-29 16:52 keltezéssel, Martin Holste írta:
I agree that is a better solution. So I should have no problems compiling against 1.1.4? I'm on SuSE 10.2 That is EoL for more than a year...
which has 1.1.2.1 (apparently unpatched), so I guess some vendors are behind. I looked at the src rpm for SuSE 11 Which SuSE version exactly? 11.0 (also EoL soon), 11.1, 11.2 or 11.3, which is still in development. If this last one, we could still get it fixed (OK, complete freeze of factory was just announced today, but even if it is too late there, it would be a good candidate for on-line update)
and it is also missing the correct checksum code, so as far as I can tell, spoof_source will never work correctly on SuSE without manual patching.
This is the content of the latest source rpm:
-rw-r--r-- 1 czanik users 913 2007 jan 16 libnet-1.1.2.1-arrray-fix.diff -rw-r--r-- 1 czanik users 437 2007 jan 16 libnet-1.1.2.1-makefile.diff -rw-r--r-- 1 czanik users 1700 2007 jan 16 libnet-1.1.2.1-strict-aliasing-fix.diff -rw-r--r-- 1 czanik users 505 2007 jan 16 libnet-1.1.2.1-uninitialized-fix.diff -rw-r--r-- 1 czanik users 1351 2007 jan 16 libnet-endianess-fix.diff -rw-r--r-- 1 czanik users 1203 2009 okt 4 libnet-shared.diff -rw-r--r-- 1 czanik users 4663 jún 10 02.53 libnet.spec -rw-r--r-- 1 czanik users 767201 2007 jan 16 libnet.tar.bz2
So, it still seems to be missing in factory (patches were not touched since previous release).
Could somebody point me to the missing patch with a description what it fixes exactly? I'll pass it on to SuSE release manager and libnet maintainers as soon as I receive it. Thanks, bye, CzP
On Tue, Jun 29, 2010 at 4:30 AM, Sandor Geller <Sandor.Geller@morganstanley.com> wrote:
Hi,
Disabling checksums would be a very bad workaround. If you're using a buggy libnet version then it is up to you to fix it - or build syslog-ng against a fixed version as libnet is linked statically. Linux distributors either ship a patched libnet 1.1.2.1 version or ship 1.1.4 instead which doesn't have the checksum bug.
Regards,
Sandor
On Tue, Jun 29, 2010 at 4:31 AM, Martin Holste <mcholste@gmail.com> wrote:
Actually, I did more research on this and found that two separate people back in 2007 had this same problem on the mailing list. See threads "Lost packets; UDP Checksum (chksum) errors; forwarding - source spoofing; libnet bug" as well as "Forwarding + Spoofing = Errors & Dropped Packets?" I believe I've definitively proven the problem to be invalid UDP checksums sent by libnet 1.1.2.1 as indicated in the first thread by Marvin Nipper. Further research shows that there is a Linux kernel-level setting that can act as a workaround by setting the socket option SO_NO_CHECK, which disables checksum verifications. So, either Syslog-NG needs to incorporate a newer, fixed libnet version (it was indicated that it did not compile using 1.1.3 Beta), or a socket option for receiving needs to be set or made as an available option to set like the receive buffer.
On Mon, Jun 28, 2010 at 1:40 PM, Zoltán Pallagi <pzolee@balabit.hu> wrote:
Hi,
I think it will be an udp kernel buffer problem (and not syslog problem), see the earlier thread of "[syslog-ng] Tests using loggen - not receiving all the packets" in this mail list.
2010.06.28. 20:21 keltezéssel, Martin Holste írta:
I'm finding that with a destination of udp("10.x.x.x", port(514) spoof_source(yes)) about half of messages get lost when going from one syslog-ng host to another at a high message rate (> 3k/sec). This is on 3.1 OSE and the hosts are on the same subnet and switch, so there shouldn't be any network devices interfering. Has anyone else had this same issue? My hunch is that it's either a performance issue with the way the libnet (I'm using 1.1.2.1 on SuSE 10.2) API is implemented or it's an issue within the libnet API. Has anyone else noticed performance problems when using spoof_source?
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- pzolee
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
2010-06-29 20:15 keltezéssel, Martin Holste írta:
Yep, I was looking at the latest 11 release SRPM I could find, which was still libnet-1.1.2.1-140.22. In one of the previous threads on this mailing list, a very valuable link was provided which has the exact source code needed for the patch: http://www.securityfocus.com/archive/89/384197/30/90/threaded .
Thanks! I just got a "go ahead" from one of the openSUSE project leads, so I'll post an updated libnet package this week to factory. You can expect a fixed libnet in openSUSE 11.3. Bye, CzP
Beautiful, thanks very much! On Wed, Jun 30, 2010 at 2:01 AM, Peter Czanik <czanik@balabit.hu> wrote:
2010-06-29 20:15 keltezéssel, Martin Holste írta:
Yep, I was looking at the latest 11 release SRPM I could find, which was still libnet-1.1.2.1-140.22. In one of the previous threads on this mailing list, a very valuable link was provided which has the exact source code needed for the patch: http://www.securityfocus.com/archive/89/384197/30/90/threaded .
Thanks! I just got a "go ahead" from one of the openSUSE project leads, so I'll post an updated libnet package this week to factory. You can expect a fixed libnet in openSUSE 11.3. Bye, CzP
Hello, 2010-06-30 14:51 keltezéssel, Martin Holste írta:
Beautiful, thanks very much!
You are welcome! Right now local package build by 'osc' does not work, so the package is not yet ready :-( BTW: I don't use UDP and this way source spoofing. Could you help me to create a test environment, so I could test it before submitting the package to factory? Bye, CzP
On Wed, Jun 30, 2010 at 2:01 AM, Peter Czanik <czanik@balabit.hu> wrote:
2010-06-29 20:15 keltezéssel, Martin Holste írta:
Yep, I was looking at the latest 11 release SRPM I could find, which was still libnet-1.1.2.1-140.22. In one of the previous threads on this mailing list, a very valuable link was provided which has the exact source code needed for the patch: http://www.securityfocus.com/archive/89/384197/30/90/threaded .
Thanks! I just got a "go ahead" from one of the openSUSE project leads, so I'll post an updated libnet package this week to factory. You can expect a fixed libnet in openSUSE 11.3. Bye, CzP
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
2010-06-29 20:15 keltezéssel, Martin Holste írta:
Yep, I was looking at the latest 11 release SRPM I could find, which was still libnet-1.1.2.1-140.22. In one of the previous threads on this mailing list, a very valuable link was provided which has the exact source code needed for the patch: http://www.securityfocus.com/archive/89/384197/30/90/threaded .
OK. Instead of just checking dates, I inspected now the SuSE patches more closely. Another patch is already applied to src/libnet_checksum.c in the patch called libnet-1.1.2.1-strict-aliasing-fix.diff which replaces the libnet_in_cksum() function with another one. The relevant part of the patch is: --- src/libnet_checksum.c +++ src/libnet_checksum.c @@ -58,6 +58,27 @@ return (sum); } +#include <assert.h> +/* same as above, just takes *u_int32_t */ +int +libnet_in_cksum32(u_int32_t *addr32, int len) +{ + int sum; + + sum = 0; + + while (len > 3) + { + sum += (*addr32) >> 16; + sum += (*addr32) & 0x0000ffff; + len -= 4; + addr32++; + } + assert(len == 0); + + return (sum); +} + int libnet_toggle_checksum(libnet_t *l, libnet_ptag_t ptag, int mode) { @@ -173,7 +194,7 @@ } else { - sum = libnet_in_cksum((u_int16_t *)&iph_p->ip_src, 8); + sum = libnet_in_cksum32((u_int32_t *)&iph_p->ip_src, 8); } sum += ntohs(IPPROTO_TCP + len); sum += libnet_in_cksum((u_int16_t *)tcph_p, len); @@ -191,7 +212,7 @@ } else { - sum = libnet_in_cksum((u_int16_t *)&iph_p->ip_src, 8); + sum = libnet_in_cksum32((u_int32_t *)&iph_p->ip_src, 8); } sum += ntohs(IPPROTO_UDP + len); sum += libnet_in_cksum((u_int16_t *)udph_p, len); With my limited C knowledge I don't know how much is this different from the one on securityfocus.com. Did you test 11.X that the problem is still there? Bye, CzP
Hello, 2010-06-30 15:51 keltezéssel, Peter Czanik írta:
2010-06-29 20:15 keltezéssel, Martin Holste írta:
Yep, I was looking at the latest 11 release SRPM I could find, which was still libnet-1.1.2.1-140.22. In one of the previous threads on this mailing list, a very valuable link was provided which has the exact source code needed for the patch: http://www.securityfocus.com/archive/89/384197/30/90/threaded .
OK. Instead of just checking dates, I inspected now the SuSE patches more closely. Another patch is already applied to src/libnet_checksum.c in the patch called libnet-1.1.2.1-strict-aliasing-fix.diff which replaces the libnet_in_cksum() function with another one. The relevant part of the patch is:
[...] With my limited C knowledge I don't know how much is this different from the one on securityfocus.com. Did you test 11.X that the problem is still there?
OK. Built a test environment with openSUSE 11.2 and Factory using source spoofing and had no UDP problems at all. So the three years old SuSE patch seems to fix the problem too. SLES 10, which is used by the reporter, was released earlier, so no wonder, that it fails. SLES 11 has the patch and does not seem to be affected either. Bye, CzP
Sorry... into this thread late, but in case someone is still interested in the checksum fix..... I found an original mention of the fix here: http://www.securityfocus.com/archive/89/384197/30/90/threaded This is text directly out of my current "how to build syslog-ng" document, in the section related to libnet (it's written in "dummy mode", in case I get run over by a truck...):
From the libnet package directory, one needs to edit the checksum module, i.e.:
vi src/libnet_checksum.c Then, you will need to locate this section of code: libnet_in_cksum(u_int16_t *addr, int len) { int sum; sum = 0; while (len > 1) { sum += *addr++; len -= 2; } if (len == 1) { sum += *(u_int16_t *)addr; } return (sum); } Now, replace that section with the following (and save/quit from the editor): libnet_in_cksum(u_int16_t *addr, int len) { int sum; u_int16_t last_byte; sum = 0; last_byte = 0; while (len > 1) { sum += *addr++; len -= 2; } if (len == 1) { *(u_int8_t*)&last_byte = *(u_int8_t*)addr; sum += last_byte; } return (sum); } -------------------------- That change should rectify the checksum issues. Hope that helps. -----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Peter Czanik Sent: Tuesday, June 29, 2010 10:45 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] spoof-source performance issues Hello, 2010-06-29 16:52 keltezéssel, Martin Holste írta:
I agree that is a better solution. So I should have no problems compiling against 1.1.4? I'm on SuSE 10.2 That is EoL for more than a year...
which has 1.1.2.1 (apparently unpatched), so I guess some vendors are behind. I looked at the src rpm for SuSE 11 Which SuSE version exactly? 11.0 (also EoL soon), 11.1, 11.2 or 11.3, which is still in development. If this last one, we could still get it fixed (OK, complete freeze of factory was just announced today, but even if it is too late there, it would be a good candidate for on-line update)
and it is also missing the correct checksum code, so as far as I can tell, spoof_source will never work correctly on SuSE without manual patching.
This is the content of the latest source rpm: -rw-r--r-- 1 czanik users 913 2007 jan 16 libnet-1.1.2.1-arrray-fix.diff -rw-r--r-- 1 czanik users 437 2007 jan 16 libnet-1.1.2.1-makefile.diff -rw-r--r-- 1 czanik users 1700 2007 jan 16 libnet-1.1.2.1-strict-aliasing-fix.diff -rw-r--r-- 1 czanik users 505 2007 jan 16 libnet-1.1.2.1-uninitialized-fix.diff -rw-r--r-- 1 czanik users 1351 2007 jan 16 libnet-endianess-fix.diff -rw-r--r-- 1 czanik users 1203 2009 okt 4 libnet-shared.diff -rw-r--r-- 1 czanik users 4663 jún 10 02.53 libnet.spec -rw-r--r-- 1 czanik users 767201 2007 jan 16 libnet.tar.bz2 So, it still seems to be missing in factory (patches were not touched since previous release). Could somebody point me to the missing patch with a description what it fixes exactly? I'll pass it on to SuSE release manager and libnet maintainers as soon as I receive it. Thanks, bye, CzP
On Tue, Jun 29, 2010 at 4:30 AM, Sandor Geller <Sandor.Geller@morganstanley.com> wrote:
Hi,
Disabling checksums would be a very bad workaround. If you're using a buggy libnet version then it is up to you to fix it - or build syslog-ng against a fixed version as libnet is linked statically. Linux distributors either ship a patched libnet 1.1.2.1 version or ship 1.1.4 instead which doesn't have the checksum bug.
Regards,
Sandor
On Tue, Jun 29, 2010 at 4:31 AM, Martin Holste <mcholste@gmail.com> wrote:
Actually, I did more research on this and found that two separate people back in 2007 had this same problem on the mailing list. See threads "Lost packets; UDP Checksum (chksum) errors; forwarding - source spoofing; libnet bug" as well as "Forwarding + Spoofing = Errors & Dropped Packets?" I believe I've definitively proven the problem to be invalid UDP checksums sent by libnet 1.1.2.1 as indicated in the first thread by Marvin Nipper. Further research shows that there is a Linux kernel-level setting that can act as a workaround by setting the socket option SO_NO_CHECK, which disables checksum verifications. So, either Syslog-NG needs to incorporate a newer, fixed libnet version (it was indicated that it did not compile using 1.1.3 Beta), or a socket option for receiving needs to be set or made as an available option to set like the receive buffer.
On Mon, Jun 28, 2010 at 1:40 PM, Zoltán Pallagi <pzolee@balabit.hu> wrote:
Hi,
I think it will be an udp kernel buffer problem (and not syslog problem), see the earlier thread of "[syslog-ng] Tests using loggen - not receiving all the packets" in this mail list.
2010.06.28. 20:21 keltezéssel, Martin Holste írta:
I'm finding that with a destination of udp("10.x.x.x", port(514) spoof_source(yes)) about half of messages get lost when going from one syslog-ng host to another at a high message rate (> 3k/sec). This is on 3.1 OSE and the hosts are on the same subnet and switch, so there shouldn't be any network devices interfering. Has anyone else had this same issue? My hunch is that it's either a performance issue with the way the libnet (I'm using 1.1.2.1 on SuSE 10.2) API is implemented or it's an issue within the libnet API. Has anyone else noticed performance problems when using spoof_source?
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- pzolee
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
participants (5)
-
Martin Holste
-
Marvin Nipper
-
Peter Czanik
-
Sandor Geller
-
Zoltán Pallagi