November 2009 - tproxy - lists.balabit.hu

[HELP] Tproxy server Can't receive any client packet
by 박제호 02 Dec '09

02 Dec '09

Hello everyone i have a problem in my transparent proxy test, i recently made up the testbed as below to run the tproxy patched apache proxy [mod_proxy], and i applied all iptables and routing rules with referencing the readme file [http://www.balabit.com/downloads/files/tproxy/README.txt, http://www.mjmwired.net/kernel/Documentation/networking/tproxy.txt] the proxy server listening the port 3128 and i checked there were no problem. but when the client tried to connect the web server, the packets reached to the box and i found the usage counts of filter rules in the mangle table incresed but my tproxy server could not receive any corresponding packet from the socket I want to know why my proxy server can't receive any packet through the socket, Do i need some more DNAT rules ? ---------------- testbed --------------- [client ] <-----> [tproxy patched apache mod_proxy] <-----> [web server] --------------------------------------------------- proxybox [Environment of the tproxy server] 1. kernel kernel 2.6.24 + tproxy-kernel-2.6.24-20080602-165651-1212418611.tar.bz2 [http://www.balabit.com/downloads/files/tproxy/tproxy-kernel-2.6.24-20080602…] 2. iptables iptables 4.0 + tproxy-iptables-1.4.0-20080521-113954-1211362794.patc [http://www.balabit.com/downloads/files/tproxy/tproxy-iptables-1.4.0-2008052…] 3. apache apache 2.2.9 + httpd-2.2.9-tproxy.patch [i got this patch from google and ported it to this version ] here are the config files of the kernel and apache proxy

3 3

Re: [tproxy] [tproxy,regression] tproxy broken in 2.6.32
by KOVACS Krisztian 30 Nov '09

30 Nov '09

Hi, On Mon, 2009-11-30 at 07:15 -0500, jamal wrote: > On Sun, 2009-11-29 at 21:35 +0100, KOVACS Krisztian wrote: > > > The story is that we really do want to deliver these packets locally, as > > if the destination IP address was locally configured on the host. The only > > way I know of to get the packet to ip_local_deliver() is by using a local > > route. > > Aha, now i understand where both you and Patrick are coming from. So > you literally have to hit the main(or default) table in the reverse > source validation. How does the workaround (that you suggested) work > then? i.e you are going to fail the RTN_UNICAST test no matter what. No, because by narrowing the rule to specific ingress interfaces the lookup done in fib_validate_source() won't match the rule(s) (because the flow used will have iif set to the loopback device), and thus it will look up the main table and select a unicast route. > Dave, give me some short time to mull this over. I am not sure i like > the sysctl approach - we may have to just revert the whole thing > instead. I don't think it would be unreasonable to add a sysctl but disable the feature by default. It's up to you, of course. Cheers, Krisztian

1 0

Re: [tproxy] [tproxy,regression] tproxy broken in 2.6.32
by KOVACS Krisztian 29 Nov '09

29 Nov '09

Hi, On szo, nov 28, 2009 at 02:44:02 -0500, jamal wrote: > > The source address *is* unicast. > > Sorry - I meant the route type is unicast. The fact that an address is > unicast or not is already dealt with by the time you get to source > address validation (in ip_input()) > > > The problem is that the routing setup is > > asymmetrical, as Patrick has already mentioned: we're using a mark to > > force certain packets (those that have matching sockets on the host) being > > delivered locally. > > > > In the other direction, reply packets won't be marked by the iptables > > rules and thus will be routed on egress just fine. > > In that case i dont understand the reluctance to use unicast routes. > Maybe you can explain and put me at ease because i see youve put extra > effort to use local addresses. The short answer is: it doesn't work with unicast routes. The story is that we really do want to deliver these packets locally, as if the destination IP address was locally configured on the host. The only way I know of to get the packet to ip_local_deliver() is by using a local route. (Now that you mentioned this I've actually gave it a try. Changing 'local' in the route to 'unicast' doesn't work at all: incoming packets get dropped because forwarding is not enabled on the ingress interface.) -- KOVACS Krisztian

1 0

Re: [tproxy] [tproxy,regression] tproxy broken in 2.6.32
by KOVACS Krisztian 28 Nov '09

28 Nov '09

Hi, On szo, nov 28, 2009 at 12:36:14 -0500, jamal wrote: > On Sat, 2009-11-28 at 18:07 +0100, Patrick McHardy wrote: > > > Right, its source validation. But the setup is valid, its asking for > > specifically marked packets to be delivered locally for transparent > > proxying. There's no requirement that rules using marks must resolve > > to RTN_UNICAST. > > True, but that requirement is needed for source validation;-> > i.e it is source address validation imposing the requirement > that we must have a RTN_UNICAST route. The tproxy iproute setup entered > a route that was not RTN_UNICAST. I think that the packet deserves to be > beaten with a club then dropped hard into an abyss (Feel free to come up > with something more medievial to do to it Patrick;-> ) > > It doesnt make sense to have a source address that is not unicast > belonging to a host or pretending to belong to a host. The source address *is* unicast. The problem is that the routing setup is asymmetrical, as Patrick has already mentioned: we're using a mark to force certain packets (those that have matching sockets on the host) being delivered locally. In the other direction, reply packets won't be marked by the iptables rules and thus will be routed on egress just fine. Your modification has the assumption that routing is symmetrical, and that reply packets will have the same mark. That assumption is not necessarily right, and I think it's not entirely unreasonable to think that not only tproxy setups will be broken by the change. > So i didnt introduce that logic thats causing this pain. Well, it depends whether or not you consider the initial setup valid. > If it worked before it was hack or fluke imo ;-> If we think that > source address validation needs to check for something else > additionally, i think thats a separate topic (but doesnt > seem worth a change) My only concern is that this definitely breaks current setups, and while we do have a workaround we don't have a way to let all users know what needs to be done... -- KOVACS Krisztian

1 0

Re: [tproxy] [tproxy,regression] tproxy broken in 2.6.32
by KOVACS Krisztian 28 Nov '09

28 Nov '09

Hi, On szo, nov 28, 2009 at 10:45:57 -0500, jamal wrote: > > However, with your > > change, and because of the ip rule above not being specific enough now > > it's returning with type RTN_LOCAL, and that's considered invalid and thus > > the skb is dropped. > > Well, since we are validating a source address - only unicast routes > are legitimate imo. i.e it was wrong to allow local before. > > > > > The workaround is using more specific ip rules that include the ingress > > interface name: > > > > # ip rule add dev eth0 fwmark 1 lookup 100 > > > > Or adding routes into table 100 with type "unicast" would do it as > well. Well, the only route we're interested in is the following (see Documentation/networking/tproxy.txt for the details): ip route add local 0.0.0.0/0 dev lo table 100 Adding a unicast route is not really an option, so I'd say the only workaround is modifying rules to include the ingress device names. -- KOVACS Krisztian

1 0

Re: [tproxy] [tproxy,regression] tproxy broken in 2.6.32
by KOVACS Krisztian 28 Nov '09

28 Nov '09

Hi, On p, nov 27, 2009 at 11:05:32 -0500, jamal wrote: > On Fri, 2009-11-27 at 09:26 +0100, KOVACS Krisztian wrote: > > Hi, > > > > On Thu, 2009-11-26 at 18:19 +0100, Andreas Schultz wrote: > > > Hi, > > > > > > git bisect shows that TPROXY has been broken by commit > > > f7c6fd2465d8e6f4f89c5d1262da10b4a6d499d0, [PATCH] net: Fix RPF to work > > > with policy routing > > > > > > I had a look at the patch, and it seems logical that this would break TPROXY. > > > > Indeed, that's a good catch. If this is indeed the problem you should be > > able to work it around by disabling rpfilter on the ingress interface. > > Does it work that way? > > Not familiar with tproxy, but I suspect the system doesnt see the mark > before policy routing happens. So probably the wrong route cache gets > created. Easy to validate by dumping the route cache. > If thats so, you have to set the mark in pre-route hook if it uses > iptables. It's already on prerouting, so that's not the problem. The problem is that for tproxy to work we've used to have a rule like this: # ip rule add fwmark 1 lookup 100 plus a few iptables rules setting mark values. The issue is that previously fib_validate_source ignored the mark set on the skb, and thus when fib_validate_source() did a FIB lookup, it all went fine, because it found a result of type RTN_UNICAST. However, with your change, and because of the ip rule above not being specific enough now it's returning with type RTN_LOCAL, and that's considered invalid and thus the skb is dropped. The workaround is using more specific ip rules that include the ingress interface name: # ip rule add dev eth0 fwmark 1 lookup 100 (repeat the above for each interface except lo.) Andreas, would you mind giving it a try on your system? -- KOVACS Krisztian

1 0

tproxy broken in 2.6.32
by Andreas Schultz 27 Nov '09

27 Nov '09

Hi, I was trying to replace a setup based on a 2.6.27.14 kernel with a 2.6.32-rc8 kernel and found that TPROXY is no longer working. The 2.6.27.14 kernel had the last stable tproxy patch plus some additional fixes (TIME_WAIT, inet_sk_flowi_flags). Since 2.6.32 is supposed to have working tproxy support, i dropped all patches. Now, connections to the local tproxy port no longer arrive at that port. >From the kernel log: Nov 23 12:32:31 scg01-wiwob user.debug kernel: tproxy socket lookup: proto 6 ac19c4df:49175 -> c0a80208:80, lookup type: 2, sock (null) Nov 23 12:32:31 scg01-wiwob user.debug kernel: tproxy socket lookup: proto 6 ac19c4df:49175 -> c0a80208:3128, lookup type: 1, sock debae040 Nov 23 12:32:31 scg01-wiwob user.debug kernel: redirecting: proto 6 c0a80208:80 -> 00000000:3128, mark: 880400a0 The redirecting message is the last indication of the packet. tcpdump shows that no answer for the initial packet goes out and the listening socket it not notified either. Andreas

2 4

[HELP] Tproxy server Can't receive any client packet
by JeHo Park 23 Nov '09

23 Nov '09

Hello everyone ~ i have a problem in my transparent proxy test, i recently made up the testbed as below to run the tproxy patched apache proxy [mod_proxy], and i applied all iptables and routing rules with referencing the readme file [http://www.balabit.com/downloads/files/tproxy/README.txt, http://www.mjmwired.net/kernel/Documentation/networking/tproxy.txt] the proxy server listening the port 3128 and i checked there were no problem. but when the client tried to connect the web server, the packets reached to the box and i found the usage counts of filter rules in the mangle table incresed but my tproxy server could not receive any corresponding packet from the socket I want to know why my proxy server can't receive any packet through the socket, Do i need some more DNAT rules ? ---------------- testbed --------------- [client ] <-----> [tproxy patched apache mod_proxy] <-----> [web server] --------------------------------------------------- proxybox [Environment of the tproxy server] 1. kernel kernel 2.6.24 + tproxy-kernel-2.6.24-20080602-165651-1212418611.tar.bz2 [http://www.balabit.com/downloads/files/tproxy/tproxy-kernel-2.6.24-20080602…] 2. iptables iptables 4.0 + tproxy-iptables-1.4.0-20080521-113954-1211362794.patch [http://www.balabit.com/downloads/files/tproxy/tproxy-iptables-1.4.0-2008052…] 3. apache apache 2.2.9 + httpd-2.2.9-tproxy.patch [i got this patch from google and ported it to this version ] 4. iptables and routing rules #!/bin/sh IPTABLES=/usr/local/sbin/iptables ${IPTABLES} -t nat -A PREROUTING -p tcp --dport 80 -j DNAT --to 127.0.0.1:3128 ${IPTABLES} -t mangle -N DIVERT ${IPTABLES} -t mangle -A DIVERT -j MARK --set-mark 1 ${IPTABLES} -t mangle -A DIVERT -j ACCEPT ${IPTABLES} -t mangle -A PREROUTING -p tcp -m socket -j DIVERT ${IPTABLES} -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY --tproxy-mark 0x1/0x1 --on-port 3128 ip rule add fwmark 0x1/0x1 lookup 100 ip route add local 0.0.0.0/0 dev lo table 100 sysctl net.ipv4.ip_nonlocal_bind=1 sysctl net.ipv4.ip_forward=1 here are the config files of the kernel and apache proxy previously thanks~

1 0

Can't receive any client packet in the tproxy server
by 박제호 19 Nov '09

19 Nov '09

Hello everyone and Balazs Scheidler ~ i have one problem, i recently made up the testbed like below to run the tproxy patched apache proxy, so i applied all iptables and routing rules with reading the readme file [http://www.balabit.com/downloads/files/tproxy/README.txt] but when the client tried to connect the web server, the packets reached to the box but my tproxy server could not receive any corresponding packet from clients i want to know why my proxy server can't receive any packet do i need some DNAT rules ? testbed: [client ] <---------> [tproxy patched apache mod_proxy] <--------> [web server] --------------------------- proxybox [proxybox] 1. OS: linux 2.6.31.6 vanilla kernel 2. iptables: 1.4.5 , no tproxy patched 3. proxy: tproxy patched [ please refer to the httpd-2.2.9-tproxy.patch ] APACHE 2.2.9 4. iptables and routing rules iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY --on-port 3128 --tproxy-mark 0x1/0x1 ip rule add fwmark 1 lookup 100 ip route add local 0.0.0.0/0 dev lo table 100 iptables -t mangle -N DIVERT iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT iptables -t mangle -A DIVERT -j MARK --set-xmark 0x1/0xffffffff iptables -t mangle -A DIVERT -j ACCEPT

2 1

Tproxy4 2.6.27.7-octeon-customized-by-cavium-kernel
by Marko Kotar 11 Nov '09

11 Nov '09

Hi I am trying to get TPROXY4 work on mentioned kernel. I've tryed to patch it with patch for 2.6.25 kernel. there were some conflicts but i think ive solved them. But tproxy didn't work. So i ve tryed another patch from wt6 kernel found here: http://linux.1wt.eu/alix/kernel-src/2.6.27-wt6/patches-2.6.27-wt6.tar.bz2. But it still doesn't work. What I have found out: If i create a bridge between interfaces packets go through only if process is listening on tproxy port. They don't get through or get to listening process. If I modify packet flow with ebtables to go through by routing doesn't come out(borute target-redirect accept or drop) or to listening process. If i use interfaces to be routed also packets don't get through or to listening process. Nothing happens. I was suspecting route.c is not working as it should so i've used it from next version of kernel where tproxy4 is working.But no difference at all. Also TPROXY4 is working fine with connect(). I set device according to: http://wiki.squid-cache.org/Features/Tproxy4(ebtables without -i option and also tried with ACCEPT option) All settings were working on 2.6.28 Ubuntu-x86_64 kernel. What could be wrong? Any clues? Where in the kernel code packets are going to tcp? Thanks to anyone who has any idea about that. Marko

1 0