- tproxy - lists.balabit.hu

Re: [tproxy] [tproxy,regression] tproxy broken in 2.6.32
by KOVACS Krisztian 28 Nov '09

28 Nov '09

Hi, On szo, nov 28, 2009 at 12:36:14 -0500, jamal wrote: > On Sat, 2009-11-28 at 18:07 +0100, Patrick McHardy wrote: > > > Right, its source validation. But the setup is valid, its asking for > > specifically marked packets to be delivered locally for transparent > > proxying. There's no requirement that rules using marks must resolve > > to RTN_UNICAST. > > True, but that requirement is needed for source validation;-> > i.e it is source address validation imposing the requirement > that we must have a RTN_UNICAST route. The tproxy iproute setup entered > a route that was not RTN_UNICAST. I think that the packet deserves to be > beaten with a club then dropped hard into an abyss (Feel free to come up > with something more medievial to do to it Patrick;-> ) > > It doesnt make sense to have a source address that is not unicast > belonging to a host or pretending to belong to a host. The source address *is* unicast. The problem is that the routing setup is asymmetrical, as Patrick has already mentioned: we're using a mark to force certain packets (those that have matching sockets on the host) being delivered locally. In the other direction, reply packets won't be marked by the iptables rules and thus will be routed on egress just fine. Your modification has the assumption that routing is symmetrical, and that reply packets will have the same mark. That assumption is not necessarily right, and I think it's not entirely unreasonable to think that not only tproxy setups will be broken by the change. > So i didnt introduce that logic thats causing this pain. Well, it depends whether or not you consider the initial setup valid. > If it worked before it was hack or fluke imo ;-> If we think that > source address validation needs to check for something else > additionally, i think thats a separate topic (but doesnt > seem worth a change) My only concern is that this definitely breaks current setups, and while we do have a workaround we don't have a way to let all users know what needs to be done... -- KOVACS Krisztian

1 0

Re: [tproxy] [tproxy,regression] tproxy broken in 2.6.32
by KOVACS Krisztian 28 Nov '09

28 Nov '09

Hi, On szo, nov 28, 2009 at 10:45:57 -0500, jamal wrote: > > However, with your > > change, and because of the ip rule above not being specific enough now > > it's returning with type RTN_LOCAL, and that's considered invalid and thus > > the skb is dropped. > > Well, since we are validating a source address - only unicast routes > are legitimate imo. i.e it was wrong to allow local before. > > > > > The workaround is using more specific ip rules that include the ingress > > interface name: > > > > # ip rule add dev eth0 fwmark 1 lookup 100 > > > > Or adding routes into table 100 with type "unicast" would do it as > well. Well, the only route we're interested in is the following (see Documentation/networking/tproxy.txt for the details): ip route add local 0.0.0.0/0 dev lo table 100 Adding a unicast route is not really an option, so I'd say the only workaround is modifying rules to include the ingress device names. -- KOVACS Krisztian

1 0

Re: [tproxy] [tproxy,regression] tproxy broken in 2.6.32
by KOVACS Krisztian 28 Nov '09

28 Nov '09

Hi, On p, nov 27, 2009 at 11:05:32 -0500, jamal wrote: > On Fri, 2009-11-27 at 09:26 +0100, KOVACS Krisztian wrote: > > Hi, > > > > On Thu, 2009-11-26 at 18:19 +0100, Andreas Schultz wrote: > > > Hi, > > > > > > git bisect shows that TPROXY has been broken by commit > > > f7c6fd2465d8e6f4f89c5d1262da10b4a6d499d0, [PATCH] net: Fix RPF to work > > > with policy routing > > > > > > I had a look at the patch, and it seems logical that this would break TPROXY. > > > > Indeed, that's a good catch. If this is indeed the problem you should be > > able to work it around by disabling rpfilter on the ingress interface. > > Does it work that way? > > Not familiar with tproxy, but I suspect the system doesnt see the mark > before policy routing happens. So probably the wrong route cache gets > created. Easy to validate by dumping the route cache. > If thats so, you have to set the mark in pre-route hook if it uses > iptables. It's already on prerouting, so that's not the problem. The problem is that for tproxy to work we've used to have a rule like this: # ip rule add fwmark 1 lookup 100 plus a few iptables rules setting mark values. The issue is that previously fib_validate_source ignored the mark set on the skb, and thus when fib_validate_source() did a FIB lookup, it all went fine, because it found a result of type RTN_UNICAST. However, with your change, and because of the ip rule above not being specific enough now it's returning with type RTN_LOCAL, and that's considered invalid and thus the skb is dropped. The workaround is using more specific ip rules that include the ingress interface name: # ip rule add dev eth0 fwmark 1 lookup 100 (repeat the above for each interface except lo.) Andreas, would you mind giving it a try on your system? -- KOVACS Krisztian

1 0

tproxy broken in 2.6.32
by Andreas Schultz 27 Nov '09

27 Nov '09

Hi, I was trying to replace a setup based on a 2.6.27.14 kernel with a 2.6.32-rc8 kernel and found that TPROXY is no longer working. The 2.6.27.14 kernel had the last stable tproxy patch plus some additional fixes (TIME_WAIT, inet_sk_flowi_flags). Since 2.6.32 is supposed to have working tproxy support, i dropped all patches. Now, connections to the local tproxy port no longer arrive at that port. >From the kernel log: Nov 23 12:32:31 scg01-wiwob user.debug kernel: tproxy socket lookup: proto 6 ac19c4df:49175 -> c0a80208:80, lookup type: 2, sock (null) Nov 23 12:32:31 scg01-wiwob user.debug kernel: tproxy socket lookup: proto 6 ac19c4df:49175 -> c0a80208:3128, lookup type: 1, sock debae040 Nov 23 12:32:31 scg01-wiwob user.debug kernel: redirecting: proto 6 c0a80208:80 -> 00000000:3128, mark: 880400a0 The redirecting message is the last indication of the packet. tcpdump shows that no answer for the initial packet goes out and the listening socket it not notified either. Andreas

2 4

[HELP] Tproxy server Can't receive any client packet
by JeHo Park 23 Nov '09

23 Nov '09

Hello everyone ~ i have a problem in my transparent proxy test, i recently made up the testbed as below to run the tproxy patched apache proxy [mod_proxy], and i applied all iptables and routing rules with referencing the readme file [http://www.balabit.com/downloads/files/tproxy/README.txt, http://www.mjmwired.net/kernel/Documentation/networking/tproxy.txt] the proxy server listening the port 3128 and i checked there were no problem. but when the client tried to connect the web server, the packets reached to the box and i found the usage counts of filter rules in the mangle table incresed but my tproxy server could not receive any corresponding packet from the socket I want to know why my proxy server can't receive any packet through the socket, Do i need some more DNAT rules ? ---------------- testbed --------------- [client ] <-----> [tproxy patched apache mod_proxy] <-----> [web server] --------------------------------------------------- proxybox [Environment of the tproxy server] 1. kernel kernel 2.6.24 + tproxy-kernel-2.6.24-20080602-165651-1212418611.tar.bz2 [http://www.balabit.com/downloads/files/tproxy/tproxy-kernel-2.6.24-20080602…] 2. iptables iptables 4.0 + tproxy-iptables-1.4.0-20080521-113954-1211362794.patch [http://www.balabit.com/downloads/files/tproxy/tproxy-iptables-1.4.0-2008052…] 3. apache apache 2.2.9 + httpd-2.2.9-tproxy.patch [i got this patch from google and ported it to this version ] 4. iptables and routing rules #!/bin/sh IPTABLES=/usr/local/sbin/iptables ${IPTABLES} -t nat -A PREROUTING -p tcp --dport 80 -j DNAT --to 127.0.0.1:3128 ${IPTABLES} -t mangle -N DIVERT ${IPTABLES} -t mangle -A DIVERT -j MARK --set-mark 1 ${IPTABLES} -t mangle -A DIVERT -j ACCEPT ${IPTABLES} -t mangle -A PREROUTING -p tcp -m socket -j DIVERT ${IPTABLES} -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY --tproxy-mark 0x1/0x1 --on-port 3128 ip rule add fwmark 0x1/0x1 lookup 100 ip route add local 0.0.0.0/0 dev lo table 100 sysctl net.ipv4.ip_nonlocal_bind=1 sysctl net.ipv4.ip_forward=1 here are the config files of the kernel and apache proxy previously thanks~

1 0

Can't receive any client packet in the tproxy server
by 박제호 19 Nov '09

19 Nov '09

Hello everyone and Balazs Scheidler ~ i have one problem, i recently made up the testbed like below to run the tproxy patched apache proxy, so i applied all iptables and routing rules with reading the readme file [http://www.balabit.com/downloads/files/tproxy/README.txt] but when the client tried to connect the web server, the packets reached to the box but my tproxy server could not receive any corresponding packet from clients i want to know why my proxy server can't receive any packet do i need some DNAT rules ? testbed: [client ] <---------> [tproxy patched apache mod_proxy] <--------> [web server] --------------------------- proxybox [proxybox] 1. OS: linux 2.6.31.6 vanilla kernel 2. iptables: 1.4.5 , no tproxy patched 3. proxy: tproxy patched [ please refer to the httpd-2.2.9-tproxy.patch ] APACHE 2.2.9 4. iptables and routing rules iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY --on-port 3128 --tproxy-mark 0x1/0x1 ip rule add fwmark 1 lookup 100 ip route add local 0.0.0.0/0 dev lo table 100 iptables -t mangle -N DIVERT iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT iptables -t mangle -A DIVERT -j MARK --set-xmark 0x1/0xffffffff iptables -t mangle -A DIVERT -j ACCEPT

2 1

Tproxy4 2.6.27.7-octeon-customized-by-cavium-kernel
by Marko Kotar 11 Nov '09

11 Nov '09

Hi I am trying to get TPROXY4 work on mentioned kernel. I've tryed to patch it with patch for 2.6.25 kernel. there were some conflicts but i think ive solved them. But tproxy didn't work. So i ve tryed another patch from wt6 kernel found here: http://linux.1wt.eu/alix/kernel-src/2.6.27-wt6/patches-2.6.27-wt6.tar.bz2. But it still doesn't work. What I have found out: If i create a bridge between interfaces packets go through only if process is listening on tproxy port. They don't get through or get to listening process. If I modify packet flow with ebtables to go through by routing doesn't come out(borute target-redirect accept or drop) or to listening process. If i use interfaces to be routed also packets don't get through or to listening process. Nothing happens. I was suspecting route.c is not working as it should so i've used it from next version of kernel where tproxy4 is working.But no difference at all. Also TPROXY4 is working fine with connect(). I set device according to: http://wiki.squid-cache.org/Features/Tproxy4(ebtables without -i option and also tried with ACCEPT option) All settings were working on 2.6.28 Ubuntu-x86_64 kernel. What could be wrong? Any clues? Where in the kernel code packets are going to tcp? Thanks to anyone who has any idea about that. Marko

1 0

One way tproxy problem
by Tim Price 03 Nov '09

03 Nov '09

Hi All, I'm trying to setup a Debain + Squid + tproxy setup using kernel 2.6.31.4, Squid 3.1.0.14 (--enable-linux-netfilter) and Iptables 1.4.3 I suspect you've heard this all before, my story is nearly identical with one exception: It kind of works. I've got a server with two interfaces in a bridge, the bridge interface has the server IP address of 218.185.224.15/26. The eth0 interface faces the internet The eth1 interface faces the test client (218.185.224.20) I have the following ebtables rules setup: ----------------------------------------------- Bridge chain: BROUTING, entries: 2, policy: ACCEPT -p IPv4 -i eth1 --ip-proto tcp --ip-dport 80 -j redirect --redirect-target DROP -p IPv4 -i eth0 --ip-proto tcp --ip-sport 80 -j redirect --redirect-target DROP I have the following Iptables rules setup: ----------------------------------------------- -A PREROUTING -p tcp -m tcp -m socket -j DIVERT -A PREROUTING -p tcp -m tcp --dport 80 -j TPROXY --on-port 3129 --tproxy-mark 1 -A DIVERT -j MARK --set-xmark 1 -A DIVERT -j ACCEPT General routing config: ----------------------------------------------- set net.ipv4.forwarding = 1 ip rule add fwmark 1 lookup 100 ip route add local 0.0.0.0/0 dev lo table 100 Squid Config: ----------------------------------------------- acl manager proto cache_object acl localhost src 127.0.0.1/32 acl to_localhost dst 127.0.0.0/8 acl TPN src 218.185.224.0/21 acl localnet src 10.0.0.0/8 # RFC1918 possible internal network acl localnet src 172.16.0.0/12 # RFC1918 possible internal network acl localnet src 192.168.0.0/16 # RFC1918 possible internal network acl SSL_ports port 443 acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl CONNECT method CONNECT acl TPN src 125.238.251.203 http_access allow manager localhost http_access deny manager http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access allow localnet http_access allow TPN http_access deny all icp_access allow localnet icp_access deny all htcp_access allow localnet htcp_access deny all http_port 3128 http_port 3129 tproxy cache_mem 2000 MB maximum_object_size_in_memory 80 KB cache_dir ufs /var/spool/squid3 45000 16 256 maximum_object_size 1 GB access_log /usr/local/squid/var/logs/access.log squid pid_filename /var/run/squid.pid debug_options ALL,1 89,6 url_rewrite_host_header off refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern . 30 20% 4320 override-expire reload-into-ims cache_effective_user nobody httpd_suppress_version_string on icp_port 3130 memory_pools_limit 0 forwarded_for off coredump_dir /usr/local/squid/var/cache cache_effective_group nogroup The testing appears to work ok: ----------------------------------------- The client makes a http request which get redirected as expected to squid and I get a line in my access.log something along the lines of: 1257212804.993 0 218.185.224.20 TCP_IMS_HIT/304 301 GET http://pagead2.googlesyndication.com/pagead/sma8.js - NONE/- text/javascript I also get something like this in my cache.log: 2009/11/03 14:46:44.446| IpIntercept.cc(381) NatLookup: address BEGIN: me= 66.102.7.156:80, client= 66.102.7.156:80, dst= 218.185.224.20:5054, peer= 218.185.224.20:5054 Heres where it gets interesting: If I go to whatismyip.com (or similar) I get told that I'm behind a squid proxy and my IP address is the IP address of the squid proxy... If I do a tcpdump on the client facing interface (eth1) I see what basically amounts to a conversation between the client and the remote server, checking the link level headers I see that the 'remote server' mac address is actually the mac address of the bridge interface of the squid server (which I would have expected). So it's spoofing the server's IP address to the client. However doing a tcpdump on the internet facing interface I see a conversation between the squid server and the remote server. At this point I'm a bit lost. I did try turning on full debug in squid at some point but it was a bit of information overload and didn't appear to contain any of the common problems documented on the squid wiki etc. It should be working from all the documentation I've read but it simply refuses to, and even more inexplicably it works in one direction. Does anyone have any ideas? Regards, Tim Price.

1 0

TPROXY understanding for transparent proxy to Virtual Machines
by Daniel Huhardeaux 21 Oct '09

21 Oct '09

Hi all, I setup a server with 2 virtual machines (kvm) using libvirt. Public interface is eth0, virbr1 being the VM interface with IP 10.1.70.1. My goal is to send external traffic smtp and http to one VM in tcp (10.1.70.13). The other VM (10.1.70.11) -Asterisk server- should receive IAX, SIP and RTP in udp. I can ping VMs from the host. With iptables and DNAT everything is working like I want, the only problem is that source IP is the one from my vibr1 10.1.70.1 and not the origine source one. People from netfilter told me I should give a try to tproxy, that's what I'm trying to setup. Context: Debian Lenny with kernel 2.6.30 and iptables 1.4.4 from backports. tproxy modules are loaded. Applied rules is sudo iptables -t mangle -A PREROUTING -p tcp -d <public IP> --dport 25 -j TPROXY --on-port 25 --on-ip 10.1.70.13 --tproxy-mark 254 The mark 254 is lookup main 0: from all lookup local 32759: from all to 10.1.0.0/16 lookup main 32760: from all to 10.100.0.0/16 lookup main 32761: from all to 10.99.4.0/16 lookup main 32762: from all fwmark 0xca lookup isp2 32763: from all fwmark 0xc9 lookup isp1 32764: from 91.137.30.70 lookup isp1 32765: from all to 0.0.0.0 lookup main 32766: from all lookup main 32767: from all lookup default From the host I can telnet port 25 of VM $ telnet 10.1.70.13 25 Trying 10.1.70.13... Connected to 10.1.70.13. Escape character is '^]'. 220 xxx.xxx.xxx.xxx ESMTP Postfix (Debian/GNU) ^] telnet> quit Connection closed. but I can't telnet the public IP from the host or from outside :-( A tshark running on the VM shows no incoming traffic, problem is on the host. The following iptables rules are the one I use for DNAT and which are working: $IPTABLES -t nat -A PREROUTING -p tcp -i eth0 -d <public IP> --dport 25 -j DNAT --to 10.1.70.13 $IPTABLES -A FORWARD -p tcp -m tcp --dport 25 -j ACCEPT What am I missing? Can tproxy do what I want? Thanks for any hint. Regards -- Daniel

1 0

TPROXY4 and TPROXY2
by Zhuang Yuyao 21 Oct '09

21 Oct '09

Hi, I am writing a proxy program using tproxy features to provide transparent proxy on linux platform. the previous version of the proxy is running on kernel 2.6.17 + tproxy2, the new version is running on kernel 2.6.28 + tproxy4. the simple net topo looks like this: 192.168.0.1(client) -> 192.168.0.2:80(eth0)-192.168.10.1(eth1) -> 192.168.10.1(server running apache2) with tproxy2, all i have to do is to add the following code: * modprobe iptable_tproxy * modprobe ipt_tproxy 1) accept client connection 2) determine which address to bind. in the case above, it is 192.168.10.1 3) assign client address and set connect flag on socket /* create socket connectsockfd */ .... /* bind connectsockfd to 192.168.10.1 */ .... /* assign client address */ itp.op = TPROXY_ASSIGN; itp.v.addr.faddr = client_addr.in.sin_addr; itp.v.addr.fport = 0; if (setsockopt(connectsockfd, SOL_IP, IP_TPROXY, &itp, sizeof(itp)) == -1) { s_log(LOG_NOTICE, "error assigning foreign address: %s", inet_ntoa(client_addr.in.sin_addr)); sockerror("setsockopt(SOL_IP, IP_TPROXY, TPROXY_ASSIGN)"); longjmp(c->err, 1); } s_log(LOG_NOTICE, "TPROXY: assigned foreign address: %s", inet_ntoa(client_addr.in.sin_addr)); /* set connect flag on socket */ itp.op = TPROXY_FLAGS; itp.v.flags = ITP_CONNECT|ITP_ONCE; if (setsockopt(connectsockfd, SOL_IP, IP_TPROXY, &itp, sizeof(itp)) == -1) { sockerror("setsockopt(SOL_IP, IP_TPROXY, TPROXY_FLAGS)"); longjmp(c->err, 1); } // then connect to server as usual ** no special iptables rules need to be set, the transparent proxy just works. ** with tproxy4, i tried the following code: * modprobe xt_TPROXY (module nf_tproxy_core is automatically loadded) * create listen socket listensockfd * setsockopt(listensockfd) to make it transparent (IP_TRANSPARENT) * bind(listensockfd) to 192.168.0.2:80 * listen(listensockfd) as usual accept() as usual, then for the outbound socket: * create socket connectsockfd * setsockopt(connectsockfd) to make it transparent (IP_TRANSPARENT) * bind(connectsockfd) to 192.168.10.1:0 * connect(connectsockfd) as usual but the transparent proxy doesn't work. the server access.log still shows that the connections are from 192.168.10.1. my question is: Are those special iptables rules (http://wiki.squid-cache.org/Features/Tproxy4#iptables_1.4.3_Configuration) must be set while using tproxy4? I really do not want my user to setup any special iptables rules for my proxy to work as transparent. thanks very much!

1 0