July 2009 - tproxy - lists.balabit.hu

Configuration for multiple listening sockets, none of them on 'lo'
by Ashwani Wason 31 Jul '09

31 Jul '09

Hi All, I have a transparent proxy, which I want to listen on four VLAN interfaces on 2.6.30 kernel. Various reasons, but the proxy cannot listen on 'lo' or on INADDR_ANY. The interfaces are eth0.31 through eth0.34 and the proxy opens one listening socket per interface. All listening sockets are bound to the same port, 2345. I am trying to set up rules similar to what has been documented so far. This is what I have - the idea is to mark all packets identically (with mark 1) based on destination port 80 but using IP rules have them lookup a different routing table based on the incoming interface. The routing table would then deliver the packet to the corresponding physical interface. The problem is that the proxy never sees these connections (netstat –ant also does not show them, which means that IP is not delivering them). > "iptables -t mangle -N DIVERT" > "iptables -t mangle -A DIVERT -j MARK --set-mark 1" > "iptables -t mangle -A DIVERT -j ACCEPT" > "iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT" > "iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY --tproxy-mark 0x1/0x1 --on-port 2345" > > "ip rule add fwmark 1 iif eth0.31 lookup 101" > "ip route add local 0.0.0.0/0 dev eth0.31 table 101" > > "ip rule add fwmark 1 iif eth0.32 lookup 102" > "ip route add local 0.0.0.0/0 dev eth0.32 table 102" > > "ip rule add fwmark 1 iif eth0.33 lookup 103" > "ip route add local 0.0.0.0/0 dev eth0.33 table 103" > > "ip rule add fwmark 1 iif eth0.34 lookup 104" > "ip route add local 0.0.0.0/0 dev eth0.34 table 104" There is not much in the man page about the 'dev' argument of 'ip route add' other than that it is the output device name. Perhaps my interpretation is incorrect. The IP tables rules are working okay, as the "hit count" of pkts and bytes keeps going up in the output of 'iptables -t mangle -L -nv'. I am not sure what is happening thereafter. The packets are seemingly going into a black hole because 'netstat -sw' does not show them discarded, delivered, or forwarded. [root@]# netstat -sw Ip: 1859156 total packets received 0 forwarded 18 with unknown protocol 0 incoming packets discarded 3917 incoming packets delivered 2692 requests sent out Can you please help identify what is going on? If this has already been posted elsewhere then please let me know. Thanks, - Ashwani

2 2

Squid 3.1 + Tproxy 4.1 in Bridge Mode
by Tiago Resende 31 Jul '09

31 Jul '09

Hi, I tried to run squid 3.1 with tproxy in a bridge mode, but I just receive a Connection timed out. In access.log shows this entries: 1249010814.562 11319 187.0.xxx.xxx TCP_MISS/000 0 GET http://teklimbu.wordpress.com/2007/10/11/running-a-transparent-linux-squid-… DIRECT/ teklimbu.wordpress.com 1249010999.257 182407 187.0.xxx.xxx TCP_MISS/504 4146 GET http://teklimbu.wordpress.com/2007/10/11/running-a-transparent-linux-squid-… DIRECT/ 72.233.2.59 text/html These entries were transparently redirected to Squid, but after a long time the Squid returns a error page (110 connection timed out). I tried many ways, and nothing! The actual configuration is: eth0(Internet) eth1(intranet) br0 - bridge with eth0 and eth1 kernel 2.6.30.3 ebtables -t broute -A BROUTING -i eth0 -p ipv4 --ip-proto tcp --ip-dport 80 -j redirect --redirect-target DROP iptables -t mangle -N DIVERT iptables -t mangle -A DIVERT -j MARK --set-mark 1 iptables -t mangle -A DIVERT -j ACCEPT iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY --tproxy-mark 0x1/0x1 --on-port 3129 ip rule add fwmark 1 lookup 100 ip route add local 0.0.0.0/0 dev lo table 100 I already tried: cd /proc/sys/net/bridge for i in *; do echo 0 > $i; done Someone has an idea? Thanks, Tiago Resende

1 0

Tproxy NAT reservations.
by Saikiran Madugula 20 Jul '09

20 Jul '09

Is there some place I can find NAT reservations patch for 2.6.27 kernel ? I was able to find one for 2.6.17 kernel. Thanks, Kiran. Apologies if the mail is received more than once by the list. It did not show up in the Tproxy archives, so resend.

2 1

Backport to 2.6.24
by Harry Mason 13 Jul '09

13 Jul '09

Has anyone backported TPROXY to 2.6.24, the kernel used in Ubuntu Hardy? Any reason why it'd be difficult or impossible? -- Harry Mason Developer SmoothWall Limited 1 John Charles Way Leeds LS12 6QA United Kingdom 1 800 959 3760 (USA, Canada and North America) 0870 1 999 500 (United Kingdom) +44 870 1 999 500 (all other countries) SmoothWall is registered in England: 4298247 This email and any attachments transmitted with it are confidential to the intended recipient(s) and may not be communicated to any other person or published by any means without the permission of SmoothWall Limited. Any opinions stated in this message are solely those of the author. See: http://smoothwall.net/company/email.php for the full text of this notice.

3 2

Re: [tproxy] netcat for tproxy (and additional noob questions)
by KOVACS Krisztian 08 Jul '09

08 Jul '09

Hi, On Tue, Jul 07, 2009 at 07:49:17AM -0400, John Lauro wrote: > > I have a very simplistic patch for netcat here: > > > > http://home.sch.bme.hu/~piglet/netfilter/tproxy/netcat-ip_transparent- > > support.patch > > I think I came across that patch before. The problem is I can not find that > version (1.10) of netcat anywhere. Found some places that mention being > pointers to it, but dead links... It appears there a new GNU version that > has lower version number, and also a BSD version, but it is way different > too (or at least different enough that the patch doesn't apply). I think it was based on Debian netcat, so you could give the Debian version a try: ftp://ftp.debian.org/debian/pool/main/n/netcat/ > > Reusing the original port is usually a bad idea. A notable example of > > things breaking is Netfilter connection tracking, which gets confused > > if > > you reuse the exact same endpoints for a different connection. > > Technically they are not the exact same if you include the interface. If it > doesn't consider the interface then they would appear the same. Netfilter conntrack is interface agnostic -- and you're right that it's exactly that what's causing the problem here. -- KOVACS Krisztian

2 2

Re: [tproxy] netcat for tproxy (and additional noob questions)
by Adrian Chadd 07 Jul '09

07 Jul '09

2009/7/7 John Lauro <john.lauro(a)covenanteyes.com>: >> Reusing the original port is usually a bad idea. A notable example of >> things breaking is Netfilter connection tracking, which gets confused >> if >> you reuse the exact same endpoints for a different connection. > > Technically they are not the exact same if you include the interface. If it > doesn't consider the interface then they would appear the same. Well, you don't always have >1 interface. Consider WCCPv2 "cache on a stick" with only one upstream interface to the router redirecting it both the client and server traffic. Adrian

1 0

Re: [tproxy] netcat for tproxy (and additional noob questions)
by KOVACS Krisztian 07 Jul '09

07 Jul '09

Hi, On Sun, Jul 05, 2009 at 03:16:47PM -0400, John Lauro wrote: > > * create listen socket > > * setsockopt to make it transparent > > * bind() to 0.0.0.0, relevant port that you've redirected traffic to > > * listen() as usual > > > > Then, you accept() as usual, use getsockname() to determine the > > original destination. Then for the outbound socket: > > Getsockname is returning back the source/client IP & port instead of the > original destination. I will need that, but that's only half... I'm > possibly just not passing the correct structure(?) Switching a couple of > things around, instead it returns back the generic 0.0.0.0 address I bound > as, which is not any better. It might not help that I am tested it from a > slightly modified netcat instead of a more simple test and I am probably > overlooking something simple. Perhaps next I'll try a more simple code test > from scratch instead of patching netcat. I have a very simplistic patch for netcat here: http://home.sch.bme.hu/~piglet/netfilter/tproxy/netcat-ip_transparent-suppo… May or may not work, since it was a long time ago I wrote that. In theory, getsockname() for the fd returned by accept() should be the original destination address, while getpeername() should return the client (source) address. For UDP, you should be able to get the original destination address via an ancillary message returned by recvmsg() (You can find the git commit introducing that feature here: http://tinyurl.com/n4yjlt) > > * create socket > > * setsockopt to make it transparent > > * bind() to relevant IP (and port, or leave it 0 to let the stack > > choose a port IIRC) > Should I be able to reuse the original source port, or will that fail/cause > problems as it would already be in the connection table (except with > local/foreign reversed)? Reusing the original port is usually a bad idea. A notable example of things breaking is Netfilter connection tracking, which gets confused if you reuse the exact same endpoints for a different connection. -- KOVACS Krisztian

3 3

Re: [tproxy] netcat for tproxy (and additional noob questions)
by Adrian Chadd 07 Jul '09

07 Jul '09

2009/7/5 John Lauro <john.lauro(a)covenanteyes.com>: > > Most of the examples I found are for more specific cases, such as for use > with squid on just port 80. I am not certain I understand fully the > interaction between the rules for iptables and the interaction with tproxy. > Is there someplace this is all documented, along with all the details from a > programmer perspective such as the use of setsockopt? (I think I just need > updated man pages?) Ive been looking to put some better documentation together for what Lusca does for TPROXY4 (I'm guessing you're talking about that and not TPROXY2.) In any case, the process is somewhat like this: * iptables/routing table setup - http://wiki.squid-cache.org/Features/Tproxy4 * create listen socket * setsockopt to make it transparent * bind() to 0.0.0.0, relevant port that you've redirected traffic to * listen() as usual Then, you accept() as usual, use getsockname() to determine the original destination. Then for the outbound socket: * create socket * setsockopt to make it transparent * bind() to relevant IP (and port, or leave it 0 to let the stack choose a port IIRC) * connect() as usual I haven't yet really looked into the TPROXY4 kernel implementation to know precisely what is going on. I can't find some clear, concise documentation either explaining the "how" or "why" behind the iptables rules and why packets need to be punted via the loopback interface. HTH, Adrian

3 2

netcat for tproxy (and additional noob questions)
by John Lauro 05 Jul '09

05 Jul '09

Hello, I am new to the list, and couldn't find this discussed in the archives: Does anyone have a working version of nc/netcat that supports tproxy? There appears to be several implementations of netcat, but only found patches for a version that appears not to be available any longer. I made a patch to netcat, but it's not working and so I am not certain if it's a problem with my patch or my ip tables setup (probably the later). Specific links to versions of netcat and patch for it would be appreciated. I do have kernel 2.6.29.3 and latest (or at least not more than a couple months old) iptables. A quick sanity check that the following is possible: (99.9% sure based on other comments on the list) 1. Want to have a box with two NICs between the internet and local machines, basically a router. (Although a bridge might be even better). 2. Have that machine intercept all connections from local machines -> internet with tproxy on all ports, and create matching connections using src ip of original machines -> internet. (Interception of connections the other way, internet -> local machines is not required, but will not hurt, but obviously they have to be able to at least forward through ok). 3. Do some custom stuff on the streams mainly for monitoring, and also a little manipulation (mainly http redirects). Assuming the above is possible, will it be possible to keep the same source port number on the outgoing connections? (Which would mean two connections with the same src/dst pairs, but communicating on different interfaces). What should the ip tables rules look like for this to function? Most of the examples I found are for more specific cases, such as for use with squid on just port 80. I am not certain I understand fully the interaction between the rules for iptables and the interaction with tproxy. Is there someplace this is all documented, along with all the details from a programmer perspective such as the use of setsockopt? (I think I just need updated man pages?) Thanks in advance for any help/advice.

1 0