Re: [tproxy] proxy app and server app on the same host
Scheidler, Balázs wrote, On 01/18/2015 07:49 AM:
The problem is that the synack in response to the syn goes out to the internet towards the faked IP address, and not back to the loopback interface.
The problem is that the same return traffic is used by the client side of the proxy as well.
So if you have a means to differentiate the two (match on the process somehow for example), then use that to mark server side packets, and then have routing tables that enforfes these packets to be delivered locally would work.
Yes, that makes sense and at first look it seems to be easy to do. I'll have to experiment, but if you, or anybody else, have already done this successfully I would be happy to see your solution. But I think a more general and user-friendly solution would be if that fix could be implemented into the kernel where the routing decision is made, isn't it? Thx Uenal
On Jan 18, 2015 3:23 AM, "U.Mutlu" <for-forums@mutluit.com> wrote:
Hi John,
John Lauro wrote, On 01/18/2015 01:59 AM:
I have done used it with single interface on the box running tproxy and the server on the same subnet also with a single interface. (Technically they have extra interfaces with policy based routing for handling outgoing connections differently, but probably not strictly required unless you are doing ANYCAST).
yes, my current working solution is similar, but now I need to run both the server app and the proxy app on the same box.
I would think that at a minimum, if you run on the same box and want to reuse the client's ip on the application, you will have to use a secondary IP on the same interface, or possibly a loopback interface. You might have to setup policy based routing from that secondary IP so it stays local, just a guess. Is it ok to bind the app to listen on 127.0.0.1 or some other secondary IP?
Have you in practice used such a single-box solution? If yes, I would very much appreciate if you could post the relevant steps (iptables, ip route etc.).
As said. I already tried this by binding the server app to a secondary IP, but I guess my other stuff (iptables, rules, routes) wasn't right.
Easier would be if you could loose the client IP info on the app (track or tag it with the proxy) and just recreate a new session... but I assume that is not an option...
Hmm. I try to avoid this in favour of keeping the tproxying independent of the capabilities of the server app(s).
Thx Uenal
----- Original Message -----
From: "U.Mutlu" <for-forums@mutluit.com> To: tproxy@lists.balabit.hu Sent: Saturday, January 17, 2015 6:56:57 PM Subject: [tproxy] proxy app and server app on the same host
Hello,
TPROXY works fine if the server application is on a different host, and if that host has the proxy host defined as its gateway.
I have this special tproxy requirement: A transparent proxy program needs to sit between client and server program, ie. like a MITM, and both (proxy pgm and server pgm) must be on the same host. The host has only one physical interface (eth0).
So, I have to move the server app to the proxy host. But then TPROXYing no more functions. I'm sure it's a routing problem, but I couldn't find a solution yet. I know some advanced routing (policy routing), ie. using different routine tables and assigning rules for them etc., but if everything is on the same host then somewhere it hangs; the SYN-ACK of the server app doesn't receive the proxy pgm, it seems to go out to the internet.
I also tried with alias IPs, virtual subnets, and also with tap devices, unfortunately the results so far are negative :-(
The author of haproxy writes at their web site
http://blog.loadbalancer.org/configure-haproxy-with-tproxy-kernel-for-full-t...
"We also need to ensure that we have the correct architecture for the TPROXY trick to work. Using the normal HAProxy you can have real servers anywhere on the internet because the source address always points back at the HAProxy units IP address. However if the clients source IP address is going to be used then the HAProxy server MUST BE IN THE PATH of the return traffic. The easiest way to do this is to put the backend servers in a different subnet to the front end clients and make sure that the default gateway points back at the HAProxy load balancer. NB. With clever routing this should be possible on the same subnet but I haven’t tried that yet! "
So, I'm looking for the above said "clever routing" solution.
Could someone help me please?
My environment: recent Linux kernel (3.16), Debian 8, x86_64
Thx U.Mutlu
I did some experimenting: Flow-Analysis of SYN+ACK in iptables tables SYN+ACK (sent by server app as step 2 of the 3-way-handshake): 1) Connect by remote client to proxy server, server app is on other host in LAN: STATUS: OK mangle : PREROUTING INPUT filter : INPUT nat : raw : PREROUTING security: INPUT 2) Connect by local client to proxy server, server app is on other host in LAN: STATUS: OK mangle : PREROUTING INPUT filter : INPUT nat : raw : PREROUTING security: INPUT 3) Connect by remote client to proxy server, server app is on same host (ip same or different): STATUS: FAIL mangle : OUTPUT POSTROUTING filter : OUTPUT nat : raw : OUTPUT security: OUTPUT 4) connect by local client to proxy server, server app is on same host (ip same or different): STATUS: OK mangle : PREROUTING INPUT OUTPUT POSTROUTING filter : INPUT OUTPUT nat : raw : PREROUTING OUTPUT security: INPUT OUTPUT It is the 3rd case which doesn't work ("FAIL"). My interpretation for the cause of the error is as follows: the routing decision seems to be done only in the PREROUTING chain, but not in POSTROUTING, because otherwise it would work. To me it seems impossible to fix this via some clever iptables rules, I think this can be fixed only by patching the kernel. Any thoughts or further ideas for fixing the 3rd case above? Regards, U.Mutlu U.Mutlu wrote, On 01/18/2015 06:44 PM:
Scheidler, Balázs wrote, On 01/18/2015 07:49 AM:
The problem is that the synack in response to the syn goes out to the internet towards the faked IP address, and not back to the loopback interface.
The problem is that the same return traffic is used by the client side of the proxy as well.
So if you have a means to differentiate the two (match on the process somehow for example), then use that to mark server side packets, and then have routing tables that enforfes these packets to be delivered locally would work.
Yes, that makes sense and at first look it seems to be easy to do. I'll have to experiment, but if you, or anybody else, have already done this successfully I would be happy to see your solution.
But I think a more general and user-friendly solution would be if that fix could be implemented into the kernel where the routing decision is made, isn't it?
Thx Uenal
Problem solved: I finally managed to get both the proxy-app and server-app run on the same host by placing the server app into a linux container (LXC) on the same host and using a bridge (br0) for the container(s). The server-app (currently a test app) sends this msg: hello to client 5.199.129.129:57456 from server 10.10.20.2:8847 via proxy 37.139.71.2:8846 Previously the host had the IP 192.168.68.22, then I experimented with public IPs behind NAT :-), so since now the 37.139.71.2 works I'll let it so. This IP belongs to someone else in Australia, but it IMHO doesn't matter because I'm using it inhouse behind NAT... :-) for testing only. The 5.199.129.129 is one of my own public IPs (mutluit.net). The test client sits there. If I find time I'll repeat the test with the 192.168 adress, but not now... :-) Phew.. this was a prohibitly hard & expensive job... took me about a week to finally come to this f..ing solution... :-) cu Uenal
I am unsure how expensive it was but the idea is ORIGINAL!! Eliezer On 21/01/2015 18:19, U.Mutlu wrote:
If I find time I'll repeat the test with the 192.168 adress, but not now...:-) Phew.. this was a prohibitly hard & expensive job... took me about a week to finally come to this f..ing solution...:-)
cu Uenal
Eliezer Croitoru wrote, On 01/21/2015 08:40 PM:
I am unsure how expensive it was but the idea is ORIGINAL!!
Maybe I'm the first one using such a setup, ie. using public IPs within NAT :-) my (paranoid) setup: internet <--> NATrouter(192.168.1.1) <--> (192.168.1.2)2ndNATrouter(37.139.71.1) <--> (37.139.71.2)host(br0 10.10.20.1) and LXC container on the same host at 10.10.20.2) The 2ndNATrouter w/ open source firmware is just for safety against my very possibly government controlled ISP... :-) BTW, I don't think there was a neccessity for IPv6 at all b/c IPv4+port gives about 2^64*2^16 = 2^80 maximal connection partners, and that is a 24-digit number in base10; that's enough for the whole universe any beyond... cu Uenal
Eliezer
On 21/01/2015 18:19, U.Mutlu wrote:
If I find time I'll repeat the test with the 192.168 adress, but not now...:-) Phew.. this was a prohibitly hard & expensive job... took me about a week to finally come to this f..ing solution...:-)
cu Uenal
U.Mutlu wrote, On 01/22/2015 09:56 AM:
BTW, I don't think there was a neccessity for IPv6 at all b/c IPv4+port gives about 2^64*2^16 = 2^80 maximal connection partners, and that is a 24-digit number in base10; that's enough for the whole universe any beyond...
Opps. sorry a bug in the numbers. Here a fixed statement: BTW, I don't think there was a neccessity for IPv6 at all b/c IPv4+port gives about 2^32*2^16 = 2^48 maximal connection partners, and that is a 15-digit number in base10; that's IMO enough for the whole universe any beyond...
Important addendum: The below described tproxy-solution on single host works only if the bridge is created with the brctl tool and brought up with the tool ifconfig (for example in /etc/rc.local): brctl addbr br0 ifconfig br0 10.10.20.1/24 up It does not work if instead the bridge is created in /etc/network/interfaces ! I don't know whom to blame for this issue, is it an issue of netfilter/iptables or the kernel? cu Uenal U.Mutlu wrote, On 01/21/2015 05:19 PM:
Problem solved:
I finally managed to get both the proxy-app and server-app run on the same host by placing the server app into a linux container (LXC) on the same host and using a bridge (br0) for the container(s).
participants (2)
-
Eliezer Croitoru
-
U.Mutlu