Need your input regarding some issue with tproxy (may not be related to tproxy)
Hello, We have a http proxy server which is in full transparent mode using tproxy 2.0.6 + kernel 2.6.20.15. The iptables rule to redirect port 80 traffic from clients is: 574K 34M REDIRECT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpts:80:82 redir ports 8001 8001 is our proxy port. We use tproxy to send connections to web servers using the clients IP. Using setsockopt(TPROXY_ASSIGN) ... The server is at around 240req/s. Most of the time its working fine i.e. I see a lot of traffic on port 80 going out using the clients IP, but I see quite a few pkts going out using src port 8001 (trace below). The rate is not as high port 80 traffic, but still high enough to be a concern. The interesting thing is I don't see any pkts destined to port 8001. I have been monitoring this server for an entire day. Hence, I am guessing that at times for some reason tproxy is not able to assign the foreign address and just uses the local address:port ... Any ideas or suggestions on how I should go about debugging this. Or whether I should be looking somewhere other than the tproxy module. Its a production server, so I can't get the client side traces, but I can get more info from the server. Let me know if you need any other information. I would really appreciate any help I can get. Thanks -- Pranav Here is a small part of the trace using (tcpdump -vvv -nn -i bond1 port 8001) =========================================================== 03:15:19.915737 IP (tos 0x0, ttl 64, id 28888, offset 0, flags [DF], proto 6, length: 40) 10.10.224.6.8001 > 10.1.1.8.4849: F [tcp sum ok] 4133498992:4133498992(0) ack 3495097709 win 7290 03:15:22.180160 IP (tos 0x0, ttl 64, id 12306, offset 0, flags [DF], proto 6, length: 40) 10.10.224.6.8001 > 10.1.1.8.4826: F [tcp sum ok] 4117981558:4117981558(0) ack 2998309917 win 9310 03:15:24.578292 IP (tos 0x0, ttl 64, id 65515, offset 0, flags [DF], proto 6, length: 40) 10.10.224.6.8001 > 10.1.2.47.1106: F [tcp sum ok] 4086941566:4086941566(0) ack 3480965117 win 6432 03:15:25.064570 IP (tos 0x0, ttl 64, id 38848, offset 0, flags [DF], proto 6, length: 40) 10.10.224.6.8001 > 10.1.3.131.1044: F [tcp sum ok] 4055165713:4055165713(0) ack 3713243767 win 9648 03:15:27.232857 IP (tos 0x0, ttl 64, id 17142, offset 0, flags [DF], proto 6, length: 40) 10.10.224.6.8001 > 10.1.1.8.4888: F [tcp sum ok] 4134874226:4134874226(0) ack 2610855931 win 7380 03:15:30.079850 IP (tos 0x0, ttl 64, id 13364, offset 0, flags [DF], proto 6, length: 1185) 10.10.224.6.8001 > 10.1.4.49.46316: P 22595600:22596733(1133) ack 805897755 win 1746 <nop,nop,timestamp 24258316 3379834444> 03:15:31.089784 IP (tos 0x0, ttl 64, id 37000, offset 0, flags [DF], proto 6, length: 40) 10.10.224.6.8001 > 10.1.3.131.1045: F [tcp sum ok] 4053960032:4053960032(0) ack 1538769655 win 6432 03:15:36.686007 IP (tos 0x0, ttl 64, id 3070, offset 0, flags [DF], proto 6, length: 40) 10.10.224.6.8001 > 10.1.1.8.4824: F [tcp sum ok] 4120796002:4120796002(0) ack 2440799003 win 10992 03:15:37.229926 IP (tos 0x0, ttl 64, id 60997, offset 0, flags [DF], proto 6, length: 52) 10.10.224.6.8001 > 10.1.1.86.2288: F [tcp sum ok] 4164262937:4164262937(0) ack 3891303954 win 7497 <nop,nop,timestamp 24260104 27180085> 03:15:37.802145 IP (tos 0x0, ttl 64, id 60433, offset 0, flags [DF], proto 6, length: 40) 10.10.224.6.8001 > 10.1.1.8.4816: F [tcp sum ok] 4105151875:4105151875(0) ack 189658004 win 6432
On Wed, Oct 15, 2008 at 5:29 PM, Pranav Desai <pranavadesai@gmail.com> wrote:
Hello,
We have a http proxy server which is in full transparent mode using tproxy 2.0.6 + kernel 2.6.20.15.
The iptables rule to redirect port 80 traffic from clients is:
574K 34M REDIRECT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpts:80:82 redir ports 8001
8001 is our proxy port. We use tproxy to send connections to web servers using the clients IP. Using setsockopt(TPROXY_ASSIGN) ...
The server is at around 240req/s. Most of the time its working fine i.e. I see a lot of traffic on port 80 going out using the clients IP, but I see quite a few pkts going out using src port 8001 (trace below). The rate is not as high port 80 traffic, but still high enough to be a concern.
The interesting thing is I don't see any pkts destined to port 8001. I have been monitoring this server for an entire day. Hence, I am guessing that at times for some reason tproxy is not able to assign the foreign address and just uses the local address:port ...
Any ideas or suggestions on how I should go about debugging this. Or whether I should be looking somewhere other than the tproxy module.
Its a production server, so I can't get the client side traces, but I can get more info from the server. Let me know if you need any other information.
I would really appreciate any help I can get.
Thanks -- Pranav
The 10.10.224.6 is the server IP running tproxy. The 10.1.x.x addresses are the client address. So the return traffic or the response seems to be having the problem where sometimes the src is 10.10.224.6:8001 instead of the origin server IP:PORT.
Here is a small part of the trace using (tcpdump -vvv -nn -i bond1 port 8001) =========================================================== 03:15:19.915737 IP (tos 0x0, ttl 64, id 28888, offset 0, flags [DF], proto 6, length: 40) 10.10.224.6.8001 > 10.1.1.8.4849: F [tcp sum ok] 4133498992:4133498992(0) ack 3495097709 win 7290 03:15:22.180160 IP (tos 0x0, ttl 64, id 12306, offset 0, flags [DF], proto 6, length: 40) 10.10.224.6.8001 > 10.1.1.8.4826: F [tcp sum ok] 4117981558:4117981558(0) ack 2998309917 win 9310 03:15:24.578292 IP (tos 0x0, ttl 64, id 65515, offset 0, flags [DF], proto 6, length: 40) 10.10.224.6.8001 > 10.1.2.47.1106: F [tcp sum ok] 4086941566:4086941566(0) ack 3480965117 win 6432 03:15:25.064570 IP (tos 0x0, ttl 64, id 38848, offset 0, flags [DF], proto 6, length: 40) 10.10.224.6.8001 > 10.1.3.131.1044: F [tcp sum ok] 4055165713:4055165713(0) ack 3713243767 win 9648 03:15:27.232857 IP (tos 0x0, ttl 64, id 17142, offset 0, flags [DF], proto 6, length: 40) 10.10.224.6.8001 > 10.1.1.8.4888: F [tcp sum ok] 4134874226:4134874226(0) ack 2610855931 win 7380 03:15:30.079850 IP (tos 0x0, ttl 64, id 13364, offset 0, flags [DF], proto 6, length: 1185) 10.10.224.6.8001 > 10.1.4.49.46316: P 22595600:22596733(1133) ack 805897755 win 1746 <nop,nop,timestamp 24258316 3379834444> 03:15:31.089784 IP (tos 0x0, ttl 64, id 37000, offset 0, flags [DF], proto 6, length: 40) 10.10.224.6.8001 > 10.1.3.131.1045: F [tcp sum ok] 4053960032:4053960032(0) ack 1538769655 win 6432 03:15:36.686007 IP (tos 0x0, ttl 64, id 3070, offset 0, flags [DF], proto 6, length: 40) 10.10.224.6.8001 > 10.1.1.8.4824: F [tcp sum ok] 4120796002:4120796002(0) ack 2440799003 win 10992 03:15:37.229926 IP (tos 0x0, ttl 64, id 60997, offset 0, flags [DF], proto 6, length: 52) 10.10.224.6.8001 > 10.1.1.86.2288: F [tcp sum ok] 4164262937:4164262937(0) ack 3891303954 win 7497 <nop,nop,timestamp 24260104 27180085> 03:15:37.802145 IP (tos 0x0, ttl 64, id 60433, offset 0, flags [DF], proto 6, length: 40) 10.10.224.6.8001 > 10.1.1.8.4816: F [tcp sum ok] 4105151875:4105151875(0) ack 189658004 win 6432
On Wed, 2008-10-15 at 23:22 -0700, Pranav Desai wrote:
On Wed, Oct 15, 2008 at 5:29 PM, Pranav Desai <pranavadesai@gmail.com> wrote:
Hello,
We have a http proxy server which is in full transparent mode using tproxy 2.0.6 + kernel 2.6.20.15.
The iptables rule to redirect port 80 traffic from clients is:
574K 34M REDIRECT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpts:80:82 redir ports 8001
8001 is our proxy port. We use tproxy to send connections to web servers using the clients IP. Using setsockopt(TPROXY_ASSIGN) ...
The server is at around 240req/s. Most of the time its working fine i.e. I see a lot of traffic on port 80 going out using the clients IP, but I see quite a few pkts going out using src port 8001 (trace below). The rate is not as high port 80 traffic, but still high enough to be a concern.
The interesting thing is I don't see any pkts destined to port 8001. I have been monitoring this server for an entire day. Hence, I am guessing that at times for some reason tproxy is not able to assign the foreign address and just uses the local address:port ...
Any ideas or suggestions on how I should go about debugging this. Or whether I should be looking somewhere other than the tproxy module.
Its a production server, so I can't get the client side traces, but I can get more info from the server. Let me know if you need any other information.
I would really appreciate any help I can get.
Thanks -- Pranav
The 10.10.224.6 is the server IP running tproxy. The 10.1.x.x addresses are the client address.
So the return traffic or the response seems to be having the problem where sometimes the src is 10.10.224.6:8001 instead of the origin server IP:PORT.
Are you using bridging as well? Do you have CONFIG_NETFILTER_BRIDGE enabled in your kernel? because in that case tcpdump will see the unnated traffic, bridging plays some nasty games with netfilter. On the client side of the proxy, the TPROXY_ASSIGN stuff does not really matter, only when going to the server. -- Bazsi
On Thu, Oct 16, 2008 at 5:37 AM, Balazs Scheidler <bazsi@balabit.hu> wrote:
On Wed, 2008-10-15 at 23:22 -0700, Pranav Desai wrote:
On Wed, Oct 15, 2008 at 5:29 PM, Pranav Desai <pranavadesai@gmail.com> wrote:
Hello,
We have a http proxy server which is in full transparent mode using tproxy 2.0.6 + kernel 2.6.20.15.
The iptables rule to redirect port 80 traffic from clients is:
574K 34M REDIRECT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpts:80:82 redir ports 8001
8001 is our proxy port. We use tproxy to send connections to web servers using the clients IP. Using setsockopt(TPROXY_ASSIGN) ...
The server is at around 240req/s. Most of the time its working fine i.e. I see a lot of traffic on port 80 going out using the clients IP, but I see quite a few pkts going out using src port 8001 (trace below). The rate is not as high port 80 traffic, but still high enough to be a concern.
The interesting thing is I don't see any pkts destined to port 8001. I have been monitoring this server for an entire day. Hence, I am guessing that at times for some reason tproxy is not able to assign the foreign address and just uses the local address:port ...
Any ideas or suggestions on how I should go about debugging this. Or whether I should be looking somewhere other than the tproxy module.
Its a production server, so I can't get the client side traces, but I can get more info from the server. Let me know if you need any other information.
I would really appreciate any help I can get.
Thanks -- Pranav
The 10.10.224.6 is the server IP running tproxy. The 10.1.x.x addresses are the client address.
So the return traffic or the response seems to be having the problem where sometimes the src is 10.10.224.6:8001 instead of the origin server IP:PORT.
Are you using bridging as well? Do you have CONFIG_NETFILTER_BRIDGE enabled in your kernel?
There is no bridging, only bonding.
because in that case tcpdump will see the unnated traffic, bridging plays some nasty games with netfilter.
we thought that tcpdump might be screwing up somewhere, but even on external network elements (load balancer) we don't see anything going to the tproxy server on port 8001.
On the client side of the proxy, the TPROXY_ASSIGN stuff does not really matter, only when going to the server.
Hmm. so does the netfilter nat rules control the client-side, and could that be screwing up somehow. Can the conntrack table be of some assistance here ? Thanks -- Pranav
-- Bazsi
On Thu, 2008-10-16 at 10:33 -0700, Pranav Desai wrote:
On Thu, Oct 16, 2008 at 5:37 AM, Balazs Scheidler <bazsi@balabit.hu> wrote:
On Wed, 2008-10-15 at 23:22 -0700, Pranav Desai wrote:
On Wed, Oct 15, 2008 at 5:29 PM, Pranav Desai <pranavadesai@gmail.com> wrote:
Hello,
We have a http proxy server which is in full transparent mode using tproxy 2.0.6 + kernel 2.6.20.15.
The iptables rule to redirect port 80 traffic from clients is:
574K 34M REDIRECT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpts:80:82 redir ports 8001
8001 is our proxy port. We use tproxy to send connections to web servers using the clients IP. Using setsockopt(TPROXY_ASSIGN) ...
The server is at around 240req/s. Most of the time its working fine i.e. I see a lot of traffic on port 80 going out using the clients IP, but I see quite a few pkts going out using src port 8001 (trace below). The rate is not as high port 80 traffic, but still high enough to be a concern.
The interesting thing is I don't see any pkts destined to port 8001. I have been monitoring this server for an entire day. Hence, I am guessing that at times for some reason tproxy is not able to assign the foreign address and just uses the local address:port ...
Any ideas or suggestions on how I should go about debugging this. Or whether I should be looking somewhere other than the tproxy module.
Its a production server, so I can't get the client side traces, but I can get more info from the server. Let me know if you need any other information.
I would really appreciate any help I can get.
Thanks -- Pranav
The 10.10.224.6 is the server IP running tproxy. The 10.1.x.x addresses are the client address.
So the return traffic or the response seems to be having the problem where sometimes the src is 10.10.224.6:8001 instead of the origin server IP:PORT.
Are you using bridging as well? Do you have CONFIG_NETFILTER_BRIDGE enabled in your kernel?
There is no bridging, only bonding.
because in that case tcpdump will see the unnated traffic, bridging plays some nasty games with netfilter.
we thought that tcpdump might be screwing up somewhere, but even on external network elements (load balancer) we don't see anything going to the tproxy server on port 8001.
On the client side of the proxy, the TPROXY_ASSIGN stuff does not really matter, only when going to the server.
Hmm. so does the netfilter nat rules control the client-side, and could that be screwing up somehow. Can the conntrack table be of some assistance here ?
Yes, it'd be useful to see the conntrack entry of the invalid packets. -- Bazsi
Does anybody have a working version of the patch for 2.6.27, I've tried to rediff manually the last working version for 2.6.26, but that didn't go anyware. Thanks. Pablo.
participants (3)
-
Balazs Scheidler
-
Pablo Sebastián Greco
-
Pranav Desai