Spoofed source bug introduced in 3.3.5
Hi. All of this pertains to a Solaris 10 (x86) environment.... I tried, at the end of the week, to jump up to the new 3.3.5 release, mostly to get up to a version that already included all of the memory leak fixes that I had manually deployed to 3.3.4. It all compiled fine, and seemed to work fine.... Until this weekend. When my log rotation script ran this weekend, all hell broke loose. The specific issue is that the instant that syslog-ng 3.3.5 receives the HUP (from my rotation script), all of the packets that are being forwarded (via UDP, with source spoofing) instantly start being forwarded with null (0.0.0.0) destination addresses. (And, FYI, I can easily recreate, and see this behavior.) Unfortunately, instead of _only_ resulting in the destination server not receiving these packets, the effect is actually far worse, because a null destination address is (at least on Solaris) interpreted as an old form of a broadcast packet, and all of these packets actually create an exponential feedback loop. I.E. they are seen by this same syslog-ng server's UDP listener, as valid inbound packets, which (of course), then are forwarded, outbound (by syslog-ng, as part of the spoofed forwarding), which then (of course) causes them to be seen (yet again) by syslog-ng as inbound packets. And "very quickly" the whole thing piles up on itself, causing the server to pretty much collapse under the weight of an ever increasing volume of "perceived" inbound UDP syslog packets. So... I'm hoping that someone will say "ah ha", and have some idea about what code might have changed between 3.3.4 and 3.3.5, that might have (accidentally) resulted in this behavior. Certainly, I've dropped back (for now) to 3.3.4 (as it does not have the problem), but if there is some sort of debugging that you need me to do (i.e. "if it's not obvious what coding change may have introduced this bug"), I'm more than willing to deploy the 3.3.5 code in a test environment, and do whatever testing that you might need. Again, it's easily reproducible with a simple HUP. Sorry to bring you a new/different issue. Any and all input/help would be appreciated. As always, THANKS for your help!! Marvin Nipper This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
On Mon, Apr 30, 2012 at 16:57, Marvin Nipper <Marvin.Nipper@stream.com> wrote:
So… I’m hoping that someone will say “ah ha”, and have some idea about what code might have changed between 3.3.4 and 3.3.5, that might have (accidentally) resulted in this behavior.
While not necessarily an ah-ha moment, but can you try reverting this patch: http://git.madhouse-project.org/debian/syslog-ng/patch/?id=a898014482f733e9c... This is the only patch between 3.3.4 and 3.3.5 that I can imagine having an effect you describe. Hopefully my hunch is correct, and then reverting this patch will fix your issue until we find a better solution. -- |8]
And.... you win the kewpie doll, grand prize. Excellent guess. I unwound that patch, and the HUP works as it should. Sorry to throw a monkey wrench into that fix. When y'all rework it, feel free to throw it my way, and I'll gladly try out the new version (if you need me to verify that it doesn't mangle the spoofing). THANKS, as always, for the super fast response. Much appreciated!! -----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Gergely Nagy Sent: Monday, April 30, 2012 10:08 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Spoofed source bug introduced in 3.3.5 On Mon, Apr 30, 2012 at 16:57, Marvin Nipper <Marvin.Nipper@stream.com> wrote:
So… I’m hoping that someone will say “ah ha”, and have some idea about what code might have changed between 3.3.4 and 3.3.5, that might have (accidentally) resulted in this behavior.
While not necessarily an ah-ha moment, but can you try reverting this patch: http://git.madhouse-project.org/debian/syslog-ng/patch/?id=a898014482f733e9c... This is the only patch between 3.3.4 and 3.3.5 that I can imagine having an effect you describe. Hopefully my hunch is correct, and then reverting this patch will fix your issue until we find a better solution. -- |8] ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
participants (2)
-
Gergely Nagy
-
Marvin Nipper