Re: [tproxy] tproxy4, kernel 2.6.22 and squid-2.6.stable13
From: "Ming-Ching Tiew" <mingching.tiew@redtone.com>
Sorry to contradict my ownself. It seems my testing was not quite conclusive. I will report again as soon as I have conclusion. Please ignore my
previous
posts.
OK this is my observation after modifying squid-2.6.STABLE.13 to use IP_FREEBIND :- 1. Transparent tproxy is working without NAT. 2. When SNAT is done in the nat table POSTROUTING chain, packets goes out and comes back using public IP ( tcpdump confirms it ) however, squid don't seem to be able to get the return packet. In the tproxy2 case, the packet goes out using spoofed private IP and hence unable to route back but there is a patch created by Arun which fixes this problem. In the case of tproxy4 using IP_FREEBIND, I wonder if there is someone who can work on an equivalent patch. Regards. -------------------------------------------- Important Warning! *************************** This electronic communication (including any attached files) may contain confidential and/or legally privileged information and is only intended for the use of the person to whom it is addressed. If you are not the intended recipient, you do not have permission to read, use, disseminate, distribute, copy or retain any part of this communication or its attachments in any form. If this e-mail was sent to you by mistake, please take the time to notify the sender so that they can identify the problem and avoid any more mistakes in sending e-mail to you. The unauthorised use of information contained in this communication or its attachments may result in legal action against any person who uses it.
Hi, On Mon, Nov 26, 2007 at 02:36:20PM +0800, Ming-Ching Tiew wrote:
From: "Ming-Ching Tiew" <mingching.tiew@redtone.com>
Sorry to contradict my ownself. It seems my testing was not quite conclusive. I will report again as soon as I have conclusion. Please ignore my
previous
posts.
OK this is my observation after modifying squid-2.6.STABLE.13 to use IP_FREEBIND :-
1. Transparent tproxy is working without NAT.
2. When SNAT is done in the nat table POSTROUTING chain, packets goes out and comes back using public IP ( tcpdump confirms it ) however, squid don't seem to be able to get the return packet.
In the tproxy2 case, the packet goes out using spoofed private IP and hence unable to route back but there is a patch created by Arun which fixes this problem.
In the case of tproxy4 using IP_FREEBIND, I wonder if there is someone who can work on an equivalent patch.
First of all, thanks for the nice analysis. It's really helpful. And yes, a modification of tproxy4 to support NAT is in the way -- it's just that I did not have time to work on it in the last few weeks. But it's certainly possible to implement NAT compatibility with tproxy4. (In a way which is much cleaner than the modifications necessary for tproxy2.) -- KOVACS Krisztian
From: "KOVACS Krisztian" <hidden@sch.bme.hu>
In the case of tproxy4 using IP_FREEBIND, I wonder if there is someone who can work on an equivalent patch.
First of all, thanks for the nice analysis. It's really helpful.
And yes, a modification of tproxy4 to support NAT is in the way -- it's just that I did not have time to work on it in the last few weeks. But it's certainly possible to implement NAT compatibility with tproxy4. (In a way which is much cleaner than the modifications necessary for tproxy2.)
First of all, I am quite a newbie with regards to kernel socket/netfilter programming. But if I check on the reply packets in the tproxy table prerouting chain and given the socket buffer, is there is a way for me to query the connnect tracking to find out the original source IP before SNAT ? My idea is that perhaps I could use the code in the tproxy4 patch to lookup the IP_FREEBIND socket so that the reply traffic can be diverted locally too using tproxy :- see fragments of tproxy-4.0.3 patch :- + sk = ipt_tproxy_get_sock(protocol, iph->saddr, iph->daddr, + hp->source, hp->dest, in); + if (sk) { + /* mark skb */ + inet = inet_sk(sk); + if (inet == NULL) + goto out; + + if (tproxy_any || inet->freebind) { + skb->ip_tproxy = 1; + + indev = in_dev_get(in); + if (indev == NULL) + goto out; + + ip_divert_local(skb, indev, sk); + in_dev_put(indev); + Is this a workable approach ? Any comments ?
From: "Ming-Ching Tiew" <mingching.tiew@redtone.com>
My idea is that perhaps I could use the code in the tproxy4 patch to lookup the IP_FREEBIND socket so that the reply traffic can be diverted locally too using tproxy :-
I hope I have not bored you guys to death with my solo show. Anyway include please find a patch which I created, to be applied on top of tprox4.0.3, which based on my own testing, it seems to work. This patch is weird, as it modifies the ip header data in the prerouting chain and I don't have the slightest idea what will be the implication. In any case, the purpose is not to show that it is a working solution, but rather, is to invite comments from the gurus here. Regards.
Hi, On Wed, Dec 05, 2007 at 03:47:57PM +0800, Ming-Ching Tiew wrote:
From: "Ming-Ching Tiew" <mingching.tiew@redtone.com>
My idea is that perhaps I could use the code in the tproxy4 patch to lookup the IP_FREEBIND socket so that the reply traffic can be diverted locally too using tproxy :-
I hope I have not bored you guys to death with my solo show.
Hey, not at all! It's just that we all have other things to work on and because of this we're usually not that quick replying. (Yeah, I know this sucks. Sorry.)
Anyway include please find a patch which I created, to be applied on top of tprox4.0.3, which based on my own testing, it seems to work. This patch is weird, as it modifies the ip header data in the prerouting chain and I don't have the slightest idea what will be the implication.
In any case, the purpose is not to show that it is a working solution, but rather, is to invite comments from the gurus here.
Hmm, I don't really get why you want to modify the header here. I understand the first chunk (although I guess you got it wrong: you'd have to use the IP_CT_DIR_REPLY tuple's source as the destination address here). However, I don't think that if you have found a socket this way then why you'd need to modify the packet header. The whole idea of tproxy4 is doing a socket lookup and then pre-assigning a dst entry with that socket reference so that the packet gets delivered locally to that socket. Could you try if applying the attached patch on top of 4.0.3 helps you with SNAT? (The patch is completely untested but at the moment I can't do any testing.) -- KOVACS Krisztian
From: "KOVACS Krisztian" <hidden@sch.bme.hu>
Could you try if applying the attached patch on top of 4.0.3 helps you with SNAT? (The patch is completely untested but at the moment I can't do any testing.)
I have got more conclusive testing results now after doing further isolation of the problem :- 1. The packet path for SNAT works now. 2. The packet path without SNAT has problem working together with 'mangle' table OUTPUT chain ( maybe also with other chains in the mangle table as well). It happens that I have iptables command which mark the packets on the OUTPUT chain, then squid will fail to work. If I flush the entire OUTPUT chain in the mangle table, then squid will work. However I am doing policy routing, I hope to use the fwmark to route the packets accordingly. I guess it is because tproxy is sharing the mark values with all other packet mark and as soon as something else is making a mark, it will mess up tproxy ? Regards
Ming-Ching Tiew írta:
From: "KOVACS Krisztian" <hidden@sch.bme.hu>
Could you try if applying the attached patch on top of 4.0.3 helps you with SNAT? (The patch is completely untested but at the moment I can't do any testing.)
I have got more conclusive testing results now after doing further isolation of the problem :-
1. The packet path for SNAT works now.
2. The packet path without SNAT has problem working together with 'mangle' table OUTPUT chain ( maybe also with other chains in the mangle table as well).
It happens that I have iptables command which mark the packets on the OUTPUT chain, then squid will fail to work. If I flush the entire OUTPUT chain in the mangle table, then squid will work.
However I am doing policy routing, I hope to use the fwmark to route the packets accordingly.
I guess it is because tproxy is sharing the mark values with all other packet mark and as soon as something else is making a mark, it will mess up tproxy ?
In 4.0.3 the fwmark is not used by tproxy, another value is used for it: sk_buff.tproxy With the tproxy match it should work, currently I don't see what can be the problem. The policy/fwmark usage is propably independent from it. -- Panther
From: "Laszlo Attila Toth" <panther@balabit.hu>
In 4.0.3 the fwmark is not used by tproxy, another value is used for it: sk_buff.tproxy
With the tproxy match it should work, currently I don't see what can be the problem. The policy/fwmark usage is propably independent from it.
You are right, it might not be 'tproxy' at all. Because when squid makes a outbound connection, the modified squid will use IP_FREEBIND to spoof an outgoing IP. There is no tproxy redirect involved at all. It's purely an outbound connection binded to a foreign IP using IP_FREEBIND. I have a little program which I used to simulate this behaviour ( which I also used it to verify the SNAT problem as well ). To invoke the program :- # ./spoof 192.168.1.5 72.14.235.99 where 192.168.1.5 is the IP I want to spoof, and 72.14.235.99 is the IP where I have web services available to verify the return path. I have used www.google.com in this example. If I flush the mangle OUTPUT chain, I could spoof the IP and get a reply from google. If I have something which MARKs the outgoing packet, the program 'spoof' will hang. The (arbitrary ) iptables command I used :- iptables -t mangle -A OUTPUT -j MARK --set-mark 5 But it can be any other MARKs as well. Regards.
From: "Ming-Ching Tiew" <mingching.tiew@redtone.com>
You are right, it might not be 'tproxy' at all. Because when squid makes a outbound connection, the modified squid will use IP_FREEBIND to spoof an outgoing IP. There is no tproxy redirect involved at all. It's purely an outbound connection binded to a foreign IP using IP_FREEBIND.
To put things in perspective, this is my conclusion now :- 1. IP_FREEBIND has problem working with mangle table MARK in the OUTPUT chain. This problem exists before Kovac's patch to fix SNAT. So I believe all current users of tproxy 4.0.3 on kernel 2.6.22 will see the same problem. 2. When there is an iptables rule which marks the mangle OUTPUT chain, the packets cannot leave the computer at all if the source IP is a foreign IP. This is not about unable to get replies. This has been confirmed using tcpdump. 3. In case (2), if the source IP is a local IP, then there is no problem, even if it is an ip_freebind socket and there is FWMARK in the socket buffer. So it is not so much about FWMARK or freebind socket per se. It's more too do with FWMARK packets seems to be taking a different route compared to without, and that route has problem with foreign IP in the source. 4. My very humble guess is that it's all to do with routing of FWMARK packets with foreign IP in the source. But I am not knowledgible enough to debug further. This is as far as I can go. Anyone who finds it otherwise please let me know. :-) I will appreciate further pointers to further debug the problem. Regards.
On Mon, 2007-12-10 at 17:27 +0800, Ming-Ching Tiew wrote:
From: "Ming-Ching Tiew" <mingching.tiew@redtone.com>
You are right, it might not be 'tproxy' at all. Because when squid makes a outbound connection, the modified squid will use IP_FREEBIND to spoof an outgoing IP. There is no tproxy redirect involved at all. It's purely an outbound connection binded to a foreign IP using IP_FREEBIND.
To put things in perspective, this is my conclusion now :-
1. IP_FREEBIND has problem working with mangle table MARK in the OUTPUT chain.
This problem exists before Kovac's patch to fix SNAT. So I believe all current users of tproxy 4.0.3 on kernel 2.6.22 will see the same problem.
2. When there is an iptables rule which marks the mangle OUTPUT chain, the packets cannot leave the computer at all if the source IP is a foreign IP. This is not about unable to get replies.
This has been confirmed using tcpdump.
3. In case (2), if the source IP is a local IP, then there is no problem, even if it is an ip_freebind socket and there is FWMARK in the socket buffer.
So it is not so much about FWMARK or freebind socket per se. It's more too do with FWMARK packets seems to be taking a different route compared to without, and that route has problem with foreign IP in the source.
4. My very humble guess is that it's all to do with routing of FWMARK packets with foreign IP in the source. But I am not knowledgible enough to debug further.
Probably this is the case, if the packet generated by a tproxied connection is visible on the OUTPUT chain, it means that tproxy did its job. You can check advanced routing rules by listing: ip rule ls You'll probably see lines like this: 32765: from all fwmark 0x64 lookup 100 This means that all packets with the specified fwmark will use a potentially different routing table than the ones without this mark. If the referenced routing table does not have a route to the specified subnet (or has a blackhole route), then the packet will not leave the box. You can list the referenced routing table by issuing: ip route ls table 100 where 100 is the name/id of the routing table in question. -- Bazsi
From: "Balazs Scheidler" <bazsi@balabit.hu>
Probably this is the case, if the packet generated by a tproxied connection is visible on the OUTPUT chain, it means that tproxy did its job.
You can check advanced routing rules by listing:
ip rule ls
You'll probably see lines like this: 32765: from all fwmark 0x64 lookup 100
This means that all packets with the specified fwmark will use a potentially different routing table than the ones without this mark. If the referenced routing table does not have a route to the specified subnet (or has a blackhole route), then the packet will not leave the box.
You can list the referenced routing table by issuing:
ip route ls table 100
where 100 is the name/id of the routing table in question.
I should have mentioned in my previous post these points :- The problem persists irregardless of whether I set up policy routing or not and it is independent of what mark value I used. I have tested many times the simplest case where I do not have any policy routing and only one default route. So certainly this is not a case of the user-level routing problem. Regards.
From: "Balazs Scheidler" <bazsi@balabit.hu>
Probably this is the case, if the packet generated by a tproxied connection is visible on the OUTPUT chain, it means that tproxy did its job.
You can check advanced routing rules by listing:
ip rule ls
You'll probably see lines like this: 32765: from all fwmark 0x64 lookup 100
This means that all packets with the specified fwmark will use a potentially different routing table than the ones without this mark. If the referenced routing table does not have a route to the specified subnet (or has a blackhole route), then the packet will not leave the box.
You can list the referenced routing table by issuing:
ip route ls table 100
where 100 is the name/id of the routing table in question.
I should have mentioned in my previous post these points :-
The problem persists irregardless of whether I set up policy routing or not and it is independent of what mark value I used.
I have tested many times the simplest case where I do not have any policy routing and only one default route.
So certainly this is not a case of the user-level routing problem.
Regards.
I'm kinda confused... Which version exactly are you discussing? In balabit site[1], tproxy needs iptable_tproxy and hacks route code, but in KOVACS Krisztian's webpage[2], the tproxy patch use policy route to make non-local sockets work without both NAT and iptable_tproxy. PS: I hope to see a tproxy-4.0.4 patchset before tproxy being merged into kernel 2.6.25. ;) [1]http://www.balabit.com/downloads/files/tproxy/ [2]http://people.netfilter.org/hidden/tproxy/ Regards Daniel 2007-12-11
From: "Daniel" <tooldcas@163.com>
I'm kinda confused...
Which version exactly are you discussing?
In balabit site[1], tproxy needs iptable_tproxy and hacks route code, but
in KOVACS Krisztian's webpage[2],
the tproxy patch use policy route to make non-local sockets work without both NAT and iptable_tproxy.
It's not that confusing if you have attempted to use them. All my post, there is mentioned of IP_FREEBIND and that will mean this patch :-
The other ones on here :-
I could not find a suitable matching iptables counterpart and the kernel versions are too new for me to use.
PS: I hope to see a tproxy-4.0.4 patchset before tproxy being merged into kernel 2.6.25. ;) Regards
But before that perhaps you could at least try to use version 4.0.3 and see if you could repeat my problem ? I don't want the case where I am giving the wrong feedback. Strange enough none is noticing this. And I think the developers are skeptical about my feedback too ! :-) And I don't want it's the case of me making a systematic error in my testings and therefore making wrong conclusion ! To repeat this is what I suggest :- 1. compile a kernel 2.6.22 with the patch on the balabit website. 2. compile iptables 1.3.8 using the iptables patch on the balabit website. 3. configure the system with a default route without SNAT so that it can access internet. No policy routing needed. 4. download this small program which I posted earlier :- https://lists.balabit.hu/pipermail/tproxy/2007-December/000618.html And compile it as 'spoof'. 5. No need to set up bridge, ebtables or even tproxy targets (but you could set it up too it does not matter ). The objective to the test is to check if packets could get out of the box, so we are not worried about return path. Use 'tcpdump' to check the outgoing. 6. Invoked the program this way :- # ./spoof 192.168.1.5 72.14.235.99 Where 192.168.1.5 could be a local IP or a foreign IP and 72.14.235.99 is any website's IP address. Without MARK in mangle OUTPUT chain, whether it's local IP or foreign IP, packets could get out of the box ( check it using tcpdump ). With MARK, only local IP could have packets going out of the box. To mark outgoing packets, do this :- iptables -t mangle -A OUTPUT -j MARK --set-mark 5 Looking forward to your testing results. Regards.
Hi, On Wed, Dec 12, 2007 at 09:20:41AM +0800, Ming-Ching Tiew wrote:
To repeat this is what I suggest :-
1. compile a kernel 2.6.22 with the patch on the balabit website. 2. compile iptables 1.3.8 using the iptables patch on the balabit website. 3. configure the system with a default route without SNAT so that it can access internet. No policy routing needed. 4. download this small program which I posted earlier :-
https://lists.balabit.hu/pipermail/tproxy/2007-December/000618.html
And compile it as 'spoof'.
5. No need to set up bridge, ebtables or even tproxy targets (but you could set it up too it does not matter ). The objective to the test is to check if packets could get out of the box, so we are not worried about return path. Use 'tcpdump' to check the outgoing.
6. Invoked the program this way :-
# ./spoof 192.168.1.5 72.14.235.99
Where 192.168.1.5 could be a local IP or a foreign IP and 72.14.235.99 is any website's IP address.
Without MARK in mangle OUTPUT chain, whether it's local IP or foreign IP, packets could get out of the box ( check it using tcpdump ).
With MARK, only local IP could have packets going out of the box.
To mark outgoing packets, do this :-
iptables -t mangle -A OUTPUT -j MARK --set-mark 5
Looking forward to your testing results.
Ah, the penny dropped. So you're stating that a simple MARK rule on the OUTPUT chain ruins your egress path? I guess that's because chaning the mark in the mangle table causes Netfilter to try and re-route the packet. There's some magic in ip_route_me_harder() that for some reason does an ip_route_input() on your packet ruining everything... I think this was originally meant as a way of overcoming the source address restriction in ip_route_output() but in the current version of the tproxy patches from Balabit those checks are commented out anyway. Could you try what happens if you change ip_route_me_harder() in net/ipv4/netfilter.c so that it does the same ip_route_output() call even if addr_type != RTN_LOCAL? -- KOVACS Krisztian
From: "KOVACS Krisztian" <hidden@sch.bme.hu>
Could you try what happens if you change ip_route_me_harder() in net/ipv4/netfilter.c so that it does the same ip_route_output() call even if addr_type != RTN_LOCAL?
You hit the nail on the head ! I am able to get the packets out and replies now. I am checking my mods to see if there is any side effects. Thank you very much and best regards.
Hi there, I just realized a new challenge, and this seems potentially more difficult than the older ones. It has to do with the fundamental difference between tproxy4 and tproxy2. Let's for convenient sake call the spoofing mechanism of tproxy4 as early spoofing, ie spoofing is decided from the start, the code do a bind from the start ! And in tproxy2 I call it as late spoofing, ie spoofing occurs later after OS routing has been made ( correct me if I am wrong, that's what I noticed from the behaviour of the system ). But how does it affects squid then ? In squid, there is this thing call 'tcp_outgoing_address', basically this is a mechanism where squid could select an outgoing (source ) address, base on elaborate ACL mechanism available in squid. For example, one can select an outgoing address based on the URL he wants to visit. With tproxy2, they both work together, the packets starts off with the selected tcp_outgoing_address, which allows the operating system to make a policy routing based on the source address, and then subsquently, the tproxy takes over to spoof using the original http request IP. However, in tproxy4 this feature cannot coexist with early spoofing. Either I use squid's tcp_outgoing_address or I used the spoof-ed sender address. And for tproxy users, spoofing sender address is more important, and therefore there is no way to use squid's ACL to influence the outgoing path. All outgoing path decisions will have to be made external to squid then. Regards.
From: "Ming-Ching Tiew" <mingching.tiew@redtone.com>
However, in tproxy4 this feature cannot coexist with early spoofing. Either I use squid's tcp_outgoing_address or I used the spoof-ed sender address. And for tproxy users, spoofing sender address is more important, and therefore there is no way to use squid's ACL to influence the outgoing path. All outgoing path decisions will have to be made external to squid then.
I think I will work around this problem using TOS then, since squid support TOS setting, ie all tcp_outgoing_address settings will be ignored and squid will incluence the external route using TOS setttings. Cheers.
Hi, On Thu, Dec 13, 2007 at 02:24:53PM +0800, Ming-Ching Tiew wrote:
From: "Ming-Ching Tiew" <mingching.tiew@redtone.com>
However, in tproxy4 this feature cannot coexist with early spoofing. Either I use squid's tcp_outgoing_address or I used the spoof-ed sender address. And for tproxy users, spoofing sender address is more important, and therefore there is no way to use squid's ACL to influence the outgoing path. All outgoing path decisions will have to be made external to squid then.
I think I will work around this problem using TOS then, since squid support TOS setting, ie all tcp_outgoing_address settings will be ignored and squid will incluence the external route using TOS setttings.
Yes, this is clearly the way to go. -- KOVACS Krisztian
participants (5)
-
Balazs Scheidler
-
Daniel
-
KOVACS Krisztian
-
Laszlo Attila Toth
-
Ming-Ching Tiew