[Bug 178] New: Spoofed source address bug introduced in 3.3.5
https://bugzilla.balabit.com/show_bug.cgi?id=178 Summary: Spoofed source address bug introduced in 3.3.5 Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Solaris Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi@balabit.hu ReportedBy: marvin.nipper@stream.com Type of the Report: regression Estimated Hours: 0.0 Hi. Sorry if this is not the right way to do this. There was an email thread regarding this on 4/30, but I figured that the right way to do this way to simply open an official bug report. As this is my 1st one for syslog-ng, apologies if I screw it up! This all pertains to 3.3.5, on Solaris 10 x86. Per my original email, changes made between 3.3.4 and 3.3.5 have caused a situation wherein forwarded UDP packets, with spoofed source addressing, all revert to null addresses, the instant that a HUP is issued against syslog-ng (e.g. during log rotation). Also, as per that email, with Gergely's excellent (and educated) guess, I backed out the patch noted here: http://git.madhouse-project.org/debian/syslog-ng/patch/?id=a898014482f733e9c..., and the result was that the null addressing issue was resolved. Obviously, as there were (no doubt) other reasons for that patch, I'm just wanting to be sure that this problem doesn't get lost in the shuffle, and that the patch gets reworked (to achieve its originally desired results, while managing to also resolve the null address problem that it caused with spoofing). Let me know if you need any further information regarding this problem. And, as always, I'm MORE than happy to be a test guinea pig, if need be. THANKS. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=178 Gergely Nagy <algernon@balabit.hu> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |algernon@balabit.hu AssignedTo|bazsi@balabit.hu |algernon@balabit.hu -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=178 Gergely Nagy <algernon@balabit.hu> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED --- Comment #1 from Gergely Nagy <algernon@balabit.hu> 2012-05-08 00:01:55 --- It's actually quite high on my TODO list, and I've been thinking about a possible fix just recently. If all goes well, I'll have something you could test by the end of the week. Thanks for the report and the reminder too! -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=178 --- Comment #2 from Gergely Nagy <algernon@balabit.hu> 2012-05-18 14:35:58 --- I've been looking at fixing this recently, but I'm having a hard time reproducing it: it doesn't happen on neither Linux nor FreeBSD (or I'm doing something wrong), and all the Solaris installs I tried (OpenIndiana and I forgot which one was the oter) installed fine in KVM and VirtualBox both, but neither could boot it, it just hung there for 6 hours after which I gave up. I'll keep trying during the weekend, but if that fails, I'd like to ask you to test a patch or two for me, which would help me determine how to proceed. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=178 --- Comment #3 from Marvin Nipper <marvin.nipper@stream.com> 2012-05-18 14:40:58 --- No problem. I'm glad to test whatever you come up with. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=178 Gergely Nagy <algernon@balabit.hu> changed: What |Removed |Added ---------------------------------------------------------------------------- Target Milestone|--- |3.3.6 -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=178 --- Comment #4 from Gergely Nagy <algernon@balabit.hu> 2012-06-08 13:31:26 --- Progress! I managed to reproduce the same issue on FreeBSD 9, so now I am able to step through the code and see where it goes wrong. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=178 --- Comment #5 from Marvin Nipper <marvin.nipper@stream.com> 2012-06-08 13:36:38 --- GREAT! Thanks, again, for the continued updates. Good luck on the fixes! -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=178 --- Comment #6 from Gergely Nagy <algernon@balabit.hu> 2012-07-03 12:55:28 --- (In reply to comment #4)
Progress! I managed to reproduce the same issue on FreeBSD 9, so now I am able to step through the code and see where it goes wrong.
Well, there goes that. I can't reproduce the problem anymore on FreeBSD 9, I have no idea why (I certainly did not fix the issue yet...). However, I did manage to install Solaris 11 in VirtualBox, and even have the latest syslog-ng git running, but spoof-source doesn't want to work at all yet: the destination does not receive the messages :| On the other hand, once I get that working, figuring out a fix should be really easy now, that I'm on the same platform too. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=178 --- Comment #7 from Marvin Nipper <marvin.nipper@stream.com> 2012-07-03 14:50:22 --- I see that I did not say this in the original submission, but I did want to point out that I have never been able to get a clean compile (in Solaris 10, x86) of the SourceForge version of libnet. It seemed to have a rat's nest of issues in Solaris. As such, I'm still using the older (original PacketFactory) version of libnet, albeit patched to fix a checksum bug. I'm not sure if that will matter at all, in your work, but I did want to make sure that I mentioned it. And as a reference, here are my internal documentation notes about the issues that I ran into with the SourceForge version of libnet. I've not even begun playing with Solaris 11, so I'm curious if you had any issues with libnet, when you built it there. My notes (just copy/pasted without any real editing)......... One problem occurs in lib/core/connhelp.c, due to a conflict with an existing system variable on Solaris (index_t). That error is easily fixed by editing the source and replacing the conflicting name (i.e. using “:1,$s/index_t/cindex_t/g”). But then an additional failure occurs in lib/drivers/internet.c, due to a missing header file (needed only for Solaris). Even that is fixable by adding “#include <sys/filio.h>” after the line with “#include "inetdefs.h"”, BUT THEN, the same module fails with errors related to an undeclared variable FIONBIO. In short, at this point in our efforts to try to make this version of the library work, it just seemed like this was headed down the rabbit hole, and not worth the effort, especially when the other libnet version worked fine. As always. THANKS for your help. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=178 --- Comment #8 from Gergely Nagy <algernon@balabit.hu> 2012-07-03 16:42:39 --- (In reply to comment #7)
I see that I did not say this in the original submission, but I did want to point out that I have never been able to get a clean compile (in Solaris 10, x86) of the SourceForge version of libnet. It seemed to have a rat's nest of issues in Solaris.
As such, I'm still using the older (original PacketFactory) version of libnet, albeit patched to fix a checksum bug. I'm not sure if that will matter at all, in your work, but I did want to make sure that I mentioned it.
And as a reference, here are my internal documentation notes about the issues that I ran into with the SourceForge version of libnet. I've not even begun playing with Solaris 11, so I'm curious if you had any issues with libnet, when you built it there.
I didn't build it. I used the http://www.opencsw.org/ binaries, they had libnet 1.1.5 prebuilt (which should have the checksum bug fixed). -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=178 --- Comment #9 from Gergely Nagy <algernon@balabit.hu> 2012-07-06 09:10:56 --- Ok, I think I understand what's happening. Pre-patch, resolve_hostname was called from afinet_dd_apply_transport(), which gets called after reload. The patch moved it to afinet_dd_setup_socket() which only gets called if there is no writer set up - but in case of reload, there is. This made it so that dest_addr never gets initialised after a reload, and stays at 0.0.0.0. The simple fix is to make sure resolve_hostname gets called sometime during reload too. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=178 --- Comment #10 from Gergely Nagy <algernon@balabit.hu> 2012-07-06 09:41:56 --- Created an attachment (id=62) --> (https://bugzilla.balabit.com/attachment.cgi?id=62) Workaround for the bug This patch works around the issue for me - can you try if it works for you aswell? I'm not merging this to master yet, because I'm not entirely happy with the solution, but it is a start, and making it pretty will be far easier than understanding what happens. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=178 --- Comment #11 from Marvin Nipper <marvin.nipper@stream.com> 2012-07-06 16:36:44 --- Hi! OK... I rebuilt 3.3.5, with only this patch, pushed it out to a server, and did a kill -HUP, and it continued to forward the spoofed packets with legitimate addressing. I'll leave this version running on that server, just to ensure that there are no other subsequent issues, but it would seem that this did the trick. If you do work up some further variation of the fix, and want me to test that, just feel free to ping me with it. As always, THANKS for your continued help! -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=178 --- Comment #12 from Marvin Nipper <marvin.nipper@stream.com> 2012-07-09 23:35:20 --- Hi. Just an "informational follow-up". I mentioned the rat's nest of problems with the version of libnet that I had previously found at SourceForge. Having seen (in your previous post) that someone else did a successful Solaris build, I checked back and saw that there was now a 1.1.6 version there, and that both the 1.1.5 and 1.1.6 versions have had about a gazillion fixes. So... I downloaded that 1.1.6 version, and gave that a try on my Solaris 10 (U10) system, and it compiled perfectly clean. I then compiled that into the patched 3.3.5 version of syslog-ng, and it worked just fine. I just wanted you to know that someone did address all of the Solaris-specific issues, and that it is no longer problematic to build that in-house, if needed. Again, just FYI. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=178 --- Comment #13 from Gergely Nagy <algernon@balabit.hu> 2012-07-09 23:49:36 --- (In reply to comment #12)
Just an "informational follow-up". I mentioned the rat's nest of problems with the version of libnet that I had previously found at SourceForge.
Having seen (in your previous post) that someone else did a successful Solaris build, I checked back and saw that there was now a 1.1.6 version there, and that both the 1.1.5 and 1.1.6 versions have had about a gazillion fixes.
So... I downloaded that 1.1.6 version, and gave that a try on my Solaris 10 (U10) system, and it compiled perfectly clean. I then compiled that into the patched 3.3.5 version of syslog-ng, and it worked just fine.
I just wanted you to know that someone did address all of the Solaris-specific issues, and that it is no longer problematic to build that in-house, if needed.
Thanks! It's certainly good to know that, I'll see if we can put that info into a FAQ or similar somewhere accessible. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=178 --- Comment #14 from Gergely Nagy <algernon@balabit.hu> 2012-07-11 11:49:10 --- I was about to merge this, and release 3.3.6, but found a regression: if a host cannot be resolved at reload time, the whole whole reload will fail and fall back to the old, which is not what we want. I can ignore the return value of resolve_hostname() during afinet_dd_init, since we're using it only for side-effects anyway, but that will mean that if a host is not resolvable at reload time, then it won't be retried, and the same bug will surface - just less often, and under rare circumstances. I'm not entirely sure how to fix this new regression.. the best would be if the resolve_hostname() could be moved out of afinet_dd_init, so it wouldn't trigger during reload, but it would trigger before trying to send anything to the host. That would not only fix the regression introduced by the fix for the other regression, it would also look a lot better, and less hackish. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=178 --- Comment #15 from Marvin Nipper <marvin.nipper@stream.com> 2012-07-11 15:48:30 ---
but that will mean that if a host is not resolvable at reload time, then it won't be retried, and the same bug will surface - just less often, and under rare circumstances.
Well, that would obviously "not be desirable" either, as it basically means "random" failures, and loss of data (that should have been forwarded), which will then not be easy to parse out and (so to speak) "re-forward" to the intended target. I'm guessing that you really weren't asking me for anything anyway(??), but were simply lamenting the conundrum that has resulted from the original patch, that introduced this spoofing problem. (And I've certainly lamented over my own coding nightmares in the past, so I sympathize!) Let me know if you need anything from me. I'm obviously more than happy to test some other variation that you might come up with. Apologizes for not being familiar enough with the syslog-ng coding to dive in and provide any help. Thanks again for all of your efforts! -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=178 --- Comment #16 from Gergely Nagy <algernon@balabit.hu> 2012-07-11 15:55:32 --- (In reply to comment #15)
I'm guessing that you really weren't asking me for anything anyway(??), but were simply lamenting the conundrum that has resulted from the original patch, that introduced this spoofing problem. (And I've certainly lamented over my own coding nightmares in the past, so I sympathize!)
Indeed, I was just pouring my heart out, so to say. And giving an update on why the posted patch has not been merged yet.
Let me know if you need anything from me. I'm obviously more than happy to test some other variation that you might come up with.
Once I have something, I'll ask for another test, thanks for the offer!
Apologizes for not being familiar enough with the syslog-ng coding to dive in and provide any help.
There's really no need to apologize, you already helped a lot by finding the problem, and continuing to test the patches I come up with. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=178 Gergely Nagy <algernon@balabit.hu> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #62 is|0 |1 obsolete| | --- Comment #17 from Gergely Nagy <algernon@balabit.hu> 2012-08-06 16:10:57 --- Created an attachment (id=63) --> (https://bugzilla.balabit.com/attachment.cgi?id=63) Another (better) fix We've sat down with Bazsi earlier, went through the code, and Bazsi came up with the attached patch, that we believe should fix the issue, without undesirable side effects. I did not have the time to test it throughly yet, but we spent quite some time exploring all possible options, and I'm convinced this'll work. I'll test it later today & merge it if all goes well, and 3.3.6 will follow soon after. Any help with testing on Solaris is greatly appreciated :) -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=178 --- Comment #18 from Marvin Nipper <marvin.nipper@stream.com> 2012-08-07 15:09:53 --- OK. I put this fix in, in place of the previous fix, and it also seems to survive a HUP, without mangling the forwarded packets. So, if this works for the other OS environments as well, then I think that it should be good for 3.3.6. As always... THANKS for your efforts to come up with this permanent fix. I really do appreciate it! -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=178 Gergely Nagy <algernon@balabit.hu> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution| |FIXED Status|ASSIGNED |RESOLVED --- Comment #19 from Gergely Nagy <algernon@balabit.hu> 2012-08-10 09:08:48 --- Thanks for the testing! Picked it over to master, marking it fixed. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
participants (1)
-
bugzilla＠bugzilla.balabit.com