Fwd: [syslog-ng] spoof_source not working
Ok, I have tried everything you have given men and I am still coming up with nothing... Here is the output of my ldd: ldd /usr/local/sbin/syslog-ng libpthread.so.1 => /usr/lib/libpthread.so.1 libnsl.so.1 => /usr/lib/libnsl.so.1 libsocket.so.1 => /usr/lib/libsocket.so.1 libdoor.so.1 => /usr/lib/libdoor.so.1 libresolv.so.2 => /usr/lib/libresolv.so.2 libxnet.so.1 => /usr/lib/libxnet.so.1 libc.so.1 => /usr/lib/libc.so.1 libdl.so.1 => /usr/lib/libdl.so.1 libmp.so.2 => /usr/lib/libmp.so.2 libthread.so.1 => /usr/lib/libthread.so.1 /usr/platform/SUNW,Sun-Fire-V210/lib/libc_psr.so.1 Notice libxnet. Should I be using it? Can it be causing the problem? How do I tell syslog-ng at compile time to use libnet.a or libnet.so? On 10/14/05, Nate Campi <nate@campin.net> wrote:
Ellis,
I didn't mention in my email to you that you'll want the libnet.so file in /tmp/foo - but first make sure you're dynamically liked using the ldd command:
-0-[root@duo:masterfiles]# ldd /sbin/syslog-ng libnsl.so.1 => /lib/libnsl.so.1 (0x40028000) libresolv.so.2 => /lib/libresolv.so.2 (0x4003c000) libc.so.6 => /lib/libc.so.6 (0x4004e000) /lib/ld-linux.so.2 (0x40000000)
You should see libnet.so in there somewhere if you're dynamically linked. If you're statically linked then you have to narrow it down with truss as Bazsi has said.
Good luck, Nate
On Fri, Oct 14, 2005 at 05:35:13PM +0200, Balazs Scheidler wrote:
On Thu, 2005-10-13 at 15:57 -0400, Chance Ellis wrote:
Ok,
I copied the the Solaris 8 libnet-config file to the /tmp/foo folder on Solaris 9. I then ran:
LD_LIBRARY_PATH=/tmp/foo:$LD_LIBRARY_PATH truss /usr/local/sbin/syslog-ng -f /usr/local/etc/syslog-ng.conf -F &
I get the same result... Whenever I apply the spoof_source(yes) to the config I do not get any messages forwarded to the destination. If I remove the spoof_source(yes) messages flow but with the source IP address from the syslog-ng server...
The truss output is quite huge! Is there any piece of the truss output that would help me to troubleshoot this? Is libnet-config the only thing I need or do I need something in addition to libnet-config?
libnet-config is not used at all while running syslog-ng, it is a build-time script that is invoked to query header location and linking information about the libnet library.
libnet is usually linked statically, so there's no point in setting LD_LIBRARY_PATH either, you can check which one syslog-ng is using by using "ldd" and/or checking if you have a libnet.a file (static), or libnet.so file (dynamic), or both. If you have both, the linking parameters determine which one is used by syslog-ng, in this case ldd should tell you the truth.
To analyze the truss path, you should look for the pattern of the message that should be sent out with a spoofed source address, and then you should see system calls like recvmsg() when the message is received and either send() or write() when it is sent. libnet uses raw sockets to send messages, syslog-ng is probably opening AF_INET, SOCK_RAW sockets in that case.
Although Solaris uses socket emulation and thus the actual system calls you see in the truss output might not be the same as in Linux, this is pretty straightforward. By the way, it might be easier to analyze the truss output if you are running a syslog-ng instance which does not actually deliver a lot of messages, but only a single one, this way the truss output will not be so large.
-- Nate
"Often, when I am reading a good book, I stop and thank my teacher. That is, I used to, until she got an unlisted number." - Unknown 15-year-old
All, I work with Chance Ellis and have been trying to tackle the spoof_source issue with him. I thought I would chime in with some additional documentation and a brief description of our current dilema. For those new to this thread and wanting the Cliffs Notes, we have been unable to get the 'spoof_source' feature working under Solaris 9. In spite of syslog-ng being compiled with the --enable-spoof-source option and the corresponding dependencies satisfied to the best of my knowledge, when 'spoof_source(yes)' is present in a destination like: destination d_foo { udp("<ip>" port(514) spoof_source(yes)); }; with a corresponding log statement like: log { source(net); destination(d_foo); }; the destination syslog server does not receive a message at all. If 'spoof_source(yes)' is removed from the configuration the syslog message is received - sans source spoofing as expected. To make things even more interesting, we have a Solaris 8 server running the same versions of software with the exact syslog-ng configuration that works flawlessly. Both boxen are running the following: - syslog-ng 1.6.8 - libol 0.3.16 - libnet 1.1.2.1 I captured the truss output of 'syslog-ng -dv' for a single message on both the Solaris 8 server (operational) and the Solaris 9 server (soon to become a boat anchor) with spoof_source enabled. I've made these text files available at the following URL's: http://benfield.org/lists/syslog-ng/truss_1message_sol8_20051018-01.txt http://benfield.org/lists/syslog-ng/truss_1message_sol9_20051018-01.txt I've also uploaded the truss output from the Solaris 9 server prior to the syslog message being received. http://benfield.org/lists/syslog-ng/truss_initial_sol9_20051018-01.txt Please let me know if any other information would be helpful. I appreciate everyone's help! Thanks, Jay -- Japheth Benfield http://benfield.org/ On 10/17/05, Chance Ellis <chance.ellis@gmail.com> wrote:
Ok,
I have tried everything you have given men and I am still coming up with nothing... Here is the output of my ldd: ldd /usr/local/sbin/syslog-ng libpthread.so.1 => /usr/lib/libpthread.so.1 libnsl.so.1 => /usr/lib/libnsl.so.1 libsocket.so.1 => /usr/lib/libsocket.so.1 libdoor.so.1 => /usr/lib/libdoor.so.1 libresolv.so.2 => /usr/lib/libresolv.so.2 libxnet.so.1 => /usr/lib/libxnet.so.1 libc.so.1 => /usr/lib/libc.so.1 libdl.so.1 => /usr/lib/libdl.so.1 libmp.so.2 => /usr/lib/libmp.so.2 libthread.so.1 => /usr/lib/libthread.so.1 /usr/platform/SUNW,Sun-Fire-V210/lib/libc_psr.so.1
Notice libxnet. Should I be using it? Can it be causing the problem? How do I tell syslog-ng at compile time to use libnet.a or libnet.so?
On 10/14/05, Nate Campi <nate@campin.net> wrote:
Ellis,
I didn't mention in my email to you that you'll want the libnet.so file in /tmp/foo - but first make sure you're dynamically liked using the ldd command:
-0-[root@duo:masterfiles]# ldd /sbin/syslog-ng libnsl.so.1 => /lib/libnsl.so.1 (0x40028000) libresolv.so.2 => /lib/libresolv.so.2 (0x4003c000) libc.so.6 => /lib/libc.so.6 (0x4004e000) /lib/ld-linux.so.2 (0x40000000)
You should see libnet.so in there somewhere if you're dynamically linked. If you're statically linked then you have to narrow it down with truss as Bazsi has said.
Good luck, Nate
On Fri, Oct 14, 2005 at 05:35:13PM +0200, Balazs Scheidler wrote:
On Thu, 2005-10-13 at 15:57 -0400, Chance Ellis wrote:
Ok,
I copied the the Solaris 8 libnet-config file to the /tmp/foo folder on Solaris 9. I then ran:
LD_LIBRARY_PATH=/tmp/foo:$LD_LIBRARY_PATH truss /usr/local/sbin/syslog-ng -f /usr/local/etc/syslog-ng.conf -F &
I get the same result... Whenever I apply the spoof_source(yes) to the config I do not get any messages forwarded to the destination. If I remove the spoof_source(yes) messages flow but with the source IP address from the syslog-ng server...
The truss output is quite huge! Is there any piece of the truss output that would help me to troubleshoot this? Is libnet-config the only thing I need or do I need something in addition to libnet-config?
libnet-config is not used at all while running syslog-ng, it is a build-time script that is invoked to query header location and linking information about the libnet library.
libnet is usually linked statically, so there's no point in setting LD_LIBRARY_PATH either, you can check which one syslog-ng is using by using "ldd" and/or checking if you have a libnet.a file (static), or libnet.so file (dynamic), or both. If you have both, the linking parameters determine which one is used by syslog-ng, in this case ldd should tell you the truth.
To analyze the truss path, you should look for the pattern of the message that should be sent out with a spoofed source address, and then you should see system calls like recvmsg() when the message is received and either send() or write() when it is sent. libnet uses raw sockets to send messages, syslog-ng is probably opening AF_INET, SOCK_RAW sockets in that case.
Although Solaris uses socket emulation and thus the actual system calls you see in the truss output might not be the same as in Linux, this is pretty straightforward. By the way, it might be easier to analyze the truss output if you are running a syslog-ng instance which does not actually deliver a lot of messages, but only a single one, this way the truss output will not be so large.
-- Nate
"Often, when I am reading a good book, I stop and thank my teacher. That is, I used to, until she got an unlisted number." - Unknown 15-year-old
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
On Tue, 2005-10-18 at 14:28 -0400, Jay Benfield wrote:
For those new to this thread and wanting the Cliffs Notes, we have been unable to get the 'spoof_source' feature working under Solaris 9. In spite of syslog-ng being compiled with the --enable-spoof-source option and the corresponding dependencies satisfied to the best of my knowledge, when 'spoof_source(yes)' is present in a destination like:
destination d_foo { udp("<ip>" port(514) spoof_source(yes)); };
with a corresponding log statement like:
log { source(net); destination(d_foo); };
the destination syslog server does not receive a message at all. If 'spoof_source(yes)' is removed from the configuration the syslog message is received - sans source spoofing as expected.
To make things even more interesting, we have a Solaris 8 server running the same versions of software with the exact syslog-ng configuration that works flawlessly. Both boxen are running the following:
- syslog-ng 1.6.8 - libol 0.3.16 - libnet 1.1.2.1
I captured the truss output of 'syslog-ng -dv' for a single message on both the Solaris 8 server (operational) and the Solaris 9 server (soon to become a boat anchor) with spoof_source enabled. I've made these text files available at the following URL's:
http://benfield.org/lists/syslog-ng/truss_1message_sol8_20051018-01.txt http://benfield.org/lists/syslog-ng/truss_1message_sol9_20051018-01.txt
I've also uploaded the truss output from the Solaris 9 server prior to the syslog message being received.
http://benfield.org/lists/syslog-ng/truss_initial_sol9_20051018-01.txt
Please let me know if any other information would be helpful. I appreciate everyone's help!
As I see from the truss output, the raw socket is opened (it becomes fd number 8), but no message is written to this fd afterwards for some reason. so_socket(PF_INET, SOCK_RAW, IPPROTO_RAW, "", 1) = 8 setsockopt(8, ip, 2, 0xFFBFF864, 4, 1) = 0 getsockopt(8, SOL_SOCKET, SO_SNDBUF, 0xFFBFF864, 0xFFBFF860, 1) = 0 setsockopt(8, SOL_SOCKET, SO_SNDBUF, 0xFFBFF864, 4, 1) = 0 I can also see that instead of fd 8, syslog-ng writes the formatted UDP packet to fd 10, which is probably opened during initialization but was stripped from the truss output: setsockopt(8, SOL_SOCKET, SO_SNDBUF, 0xFFBFF864, 4, 1) = 0 // MESSAGE ABOVE REPEATED ELEVENTY-BILLION TIMES setsockopt(10, SOL_SOCKET, SO_SNDBUF, 0xFFBFF864, 4, 1) Err#132 ENOBUFS setsockopt(10, SOL_SOCKET, SO_BROADCAST, 0xFFBFF864, 4, 1) = 0 so syslog-ng definitely sends off a packet using fd 10, but I don't know what fd 10 is. Could you check your complete truss output to check whether it is there? Another question whether you one or two spoof_source enabled destination? As it seems you have at least two, because two similar fds are opened. Nevertheless syslog-ng (more precisely libnet) sends off the packet here: sendto(10, " E10\0 Z\0\0\0\0 @11 dED".., 90, 0, 0xFFBFF3C0, 16) = 90 basically it is the same as on Solaris 8: sendto(8, " E10\0 Z\0\0\0\0 @11 dED".., 90, 0, 0xFFBEF2F8, 16) = 90 (the only difference is the fd number used, but it does not really matter) My suspicion is that there's some packet filter/OS issue that prevents RAW sockets to work properly. Are you sure you cannot see the packet if you snoop the outgoing interface? Do you have proper outgoing routes? -- Bazsi
Balazs, I think we've figured it out. After doing a number of packet captures and so forth we determined that the load balancer that the Solaris 9 syslog-ng servers sit behind are dropping outbound spoof_source messages but are permitting messages that are not spoofed. Our Solaris 8 test server, that has always worked with spoof_source forwarding, does not sit behind a load balancer. This is consistent with the configuration of our load balancers and was an oversight (read: duh) on our part. I think that once we get this straightened out we'll be in good shape. I appreciate everyone's help! Thanks, Jay -- Japheth Benfield http://benfield.org/ On 10/20/05, Balazs Scheidler <bazsi@balabit.hu> wrote:
On Tue, 2005-10-18 at 14:28 -0400, Jay Benfield wrote:
For those new to this thread and wanting the Cliffs Notes, we have been unable to get the 'spoof_source' feature working under Solaris 9. In spite of syslog-ng being compiled with the --enable-spoof-source option and the corresponding dependencies satisfied to the best of my knowledge, when 'spoof_source(yes)' is present in a destination like:
destination d_foo { udp("<ip>" port(514) spoof_source(yes)); };
with a corresponding log statement like:
log { source(net); destination(d_foo); };
the destination syslog server does not receive a message at all. If 'spoof_source(yes)' is removed from the configuration the syslog message is received - sans source spoofing as expected.
To make things even more interesting, we have a Solaris 8 server running the same versions of software with the exact syslog-ng configuration that works flawlessly. Both boxen are running the following:
- syslog-ng 1.6.8 - libol 0.3.16 - libnet 1.1.2.1
I captured the truss output of 'syslog-ng -dv' for a single message on both the Solaris 8 server (operational) and the Solaris 9 server (soon to become a boat anchor) with spoof_source enabled. I've made these text files available at the following URL's:
http://benfield.org/lists/syslog-ng/truss_1message_sol8_20051018-01.txt http://benfield.org/lists/syslog-ng/truss_1message_sol9_20051018-01.txt
I've also uploaded the truss output from the Solaris 9 server prior to the syslog message being received.
http://benfield.org/lists/syslog-ng/truss_initial_sol9_20051018-01.txt
Please let me know if any other information would be helpful. I appreciate everyone's help!
As I see from the truss output, the raw socket is opened (it becomes fd number 8), but no message is written to this fd afterwards for some reason.
so_socket(PF_INET, SOCK_RAW, IPPROTO_RAW, "", 1) = 8 setsockopt(8, ip, 2, 0xFFBFF864, 4, 1) = 0 getsockopt(8, SOL_SOCKET, SO_SNDBUF, 0xFFBFF864, 0xFFBFF860, 1) = 0 setsockopt(8, SOL_SOCKET, SO_SNDBUF, 0xFFBFF864, 4, 1) = 0
I can also see that instead of fd 8, syslog-ng writes the formatted UDP packet to fd 10, which is probably opened during initialization but was stripped from the truss output:
setsockopt(8, SOL_SOCKET, SO_SNDBUF, 0xFFBFF864, 4, 1) = 0 // MESSAGE ABOVE REPEATED ELEVENTY-BILLION TIMES
setsockopt(10, SOL_SOCKET, SO_SNDBUF, 0xFFBFF864, 4, 1) Err#132 ENOBUFS setsockopt(10, SOL_SOCKET, SO_BROADCAST, 0xFFBFF864, 4, 1) = 0
so syslog-ng definitely sends off a packet using fd 10, but I don't know what fd 10 is. Could you check your complete truss output to check whether it is there?
Another question whether you one or two spoof_source enabled destination? As it seems you have at least two, because two similar fds are opened.
Nevertheless syslog-ng (more precisely libnet) sends off the packet here:
sendto(10, " E10\0 Z\0\0\0\0 @11 dED".., 90, 0, 0xFFBFF3C0, 16) = 90
basically it is the same as on Solaris 8:
sendto(8, " E10\0 Z\0\0\0\0 @11 dED".., 90, 0, 0xFFBEF2F8, 16) = 90
(the only difference is the fd number used, but it does not really matter)
My suspicion is that there's some packet filter/OS issue that prevents RAW sockets to work properly. Are you sure you cannot see the packet if you snoop the outgoing interface? Do you have proper outgoing routes?
-- Bazsi
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
All, Just to close out this thread, spoof_source is now working flawlessly on our Solaris 9 boxen after adjusting our load balancer configuration. Thanks again! - Jay -- Japheth Benfield http://benfield.org/ On 10/20/05, Jay Benfield <jabenfield@gmail.com> wrote:
Balazs,
I think we've figured it out. After doing a number of packet captures and so forth we determined that the load balancer that the Solaris 9 syslog-ng servers sit behind are dropping outbound spoof_source messages but are permitting messages that are not spoofed. Our Solaris 8 test server, that has always worked with spoof_source forwarding, does not sit behind a load balancer.
This is consistent with the configuration of our load balancers and was an oversight (read: duh) on our part. I think that once we get this straightened out we'll be in good shape.
I appreciate everyone's help!
Thanks, Jay
-- Japheth Benfield http://benfield.org/
On 10/20/05, Balazs Scheidler <bazsi@balabit.hu> wrote:
On Tue, 2005-10-18 at 14:28 -0400, Jay Benfield wrote:
For those new to this thread and wanting the Cliffs Notes, we have been unable to get the 'spoof_source' feature working under Solaris 9. In spite of syslog-ng being compiled with the --enable-spoof-source option and the corresponding dependencies satisfied to the best of my knowledge, when 'spoof_source(yes)' is present in a destination like:
destination d_foo { udp("<ip>" port(514) spoof_source(yes)); };
with a corresponding log statement like:
log { source(net); destination(d_foo); };
the destination syslog server does not receive a message at all. If 'spoof_source(yes)' is removed from the configuration the syslog message is received - sans source spoofing as expected.
To make things even more interesting, we have a Solaris 8 server running the same versions of software with the exact syslog-ng configuration that works flawlessly. Both boxen are running the following:
- syslog-ng 1.6.8 - libol 0.3.16 - libnet 1.1.2.1
I captured the truss output of 'syslog-ng -dv' for a single message on both the Solaris 8 server (operational) and the Solaris 9 server (soon to become a boat anchor) with spoof_source enabled. I've made these text files available at the following URL's:
http://benfield.org/lists/syslog-ng/truss_1message_sol8_20051018-01.txt http://benfield.org/lists/syslog-ng/truss_1message_sol9_20051018-01.txt
I've also uploaded the truss output from the Solaris 9 server prior to the syslog message being received.
http://benfield.org/lists/syslog-ng/truss_initial_sol9_20051018-01.txt
Please let me know if any other information would be helpful. I appreciate everyone's help!
As I see from the truss output, the raw socket is opened (it becomes fd number 8), but no message is written to this fd afterwards for some reason.
so_socket(PF_INET, SOCK_RAW, IPPROTO_RAW, "", 1) = 8 setsockopt(8, ip, 2, 0xFFBFF864, 4, 1) = 0 getsockopt(8, SOL_SOCKET, SO_SNDBUF, 0xFFBFF864, 0xFFBFF860, 1) = 0 setsockopt(8, SOL_SOCKET, SO_SNDBUF, 0xFFBFF864, 4, 1) = 0
I can also see that instead of fd 8, syslog-ng writes the formatted UDP packet to fd 10, which is probably opened during initialization but was stripped from the truss output:
setsockopt(8, SOL_SOCKET, SO_SNDBUF, 0xFFBFF864, 4, 1) = 0 // MESSAGE ABOVE REPEATED ELEVENTY-BILLION TIMES
setsockopt(10, SOL_SOCKET, SO_SNDBUF, 0xFFBFF864, 4, 1) Err#132 ENOBUFS setsockopt(10, SOL_SOCKET, SO_BROADCAST, 0xFFBFF864, 4, 1) = 0
so syslog-ng definitely sends off a packet using fd 10, but I don't know what fd 10 is. Could you check your complete truss output to check whether it is there?
Another question whether you one or two spoof_source enabled destination? As it seems you have at least two, because two similar fds are opened.
Nevertheless syslog-ng (more precisely libnet) sends off the packet here:
sendto(10, " E10\0 Z\0\0\0\0 @11 dED".., 90, 0, 0xFFBFF3C0, 16) = 90
basically it is the same as on Solaris 8:
sendto(8, " E10\0 Z\0\0\0\0 @11 dED".., 90, 0, 0xFFBEF2F8, 16) = 90
(the only difference is the fd number used, but it does not really matter)
My suspicion is that there's some packet filter/OS issue that prevents RAW sockets to work properly. Are you sure you cannot see the packet if you snoop the outgoing interface? Do you have proper outgoing routes?
-- Bazsi
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
participants (3)
-
Balazs Scheidler
-
Chance Ellis
-
Jay Benfield