tproxy can't work with ip_conntrack
Hi,all When I use a transparent proxy application using tproxy, I find there is something wrong when I load ip_conntrack module Here is a sample: network topology(router mode): Client[192.168.2.2]--------[192.168.2.1]TPROXY Server[192.168.1.1]--------[192.168.1.2]HTTP Server 1. Client try to connect to HTTP Server, and TPROXY Server redirect the traffic to itself. Now the conntrack is: ORIG: 192.168.2.2:12345->192.168.1.2:80 REPLY:192.168.1.2:80->192.168.2.2 After 3 way handshake the connection between Client and TPROXY server should be in ESTABLISHED state 2. TPROXY Server connect to HTTP Server using fake src ip and src port. Now the SYN packet should be 192.168.2.2:12345->192.168.1.2:80. But kernel find there is an old ip_conntrack entry for this SYN packet whose state is ESTABLISHED So, kernel will drop the SYN packet, and delete the old entry 3. SYN packet will be retransmitted after 1 sec on TPROXY Server. And then kernel will allocate an new entry for this SYN packet. 4. After 3 way handshake between TPROXY Server and HTTP Server, this ip_conntrack entry should be in ESTABLISHED state I think that we can add some new fields to identify the two HTTP connections in ip_conntrack structure(one is Client->TPROXY Server, and the other is TPROXY Server->HTTP Server). I hope we can get the right conntrack when we just know the tuple containing src_ip,src_port,dst_ip,dst_port. Because there is lots of kernel code like this: 1. get the tuple 4 fields(src_ip,src_port,dst_ip,dst_port) from skb 2. call: nf_conntrack_find_get(tuple) to find the corresponding ip_conntrack. So in this case we can only know the 4 fields, we can't get any more message from skb. Assuming there are 2 ip_conntrack: [1]: 192.168.2.2:12345->192.168.1.2:80(client->TPROXY Server) [2]: 192.168.2.2:12345->192.168.1.2:80(TPROXY Server->HTTP Server) When we process TCP packets between client and TPROXY Server, we find the conntrack for 192.168.2.2:123456->192.168.1.2:80, the result should be [1], and if the TCP packets belong to TPROXY Server and HTTP Server connection, the result should be [2] Does anyone have good idea about the requirement mentioned above? -- Thanks BR. Wei Dong
Hi, On sze, okt 22, 2008 at 09:50:29 +0800, Dong Wei wrote:
When I use a transparent proxy application using tproxy, I find there is something wrong when I load ip_conntrack module Here is a sample: network topology(router mode): Client[192.168.2.2]--------[192.168.2.1]TPROXY Server[192.168.1.1]--------[192.168.1.2]HTTP Server
1. Client try to connect to HTTP Server, and TPROXY Server redirect the traffic to itself. Now the conntrack is: ORIG: 192.168.2.2:12345->192.168.1.2:80 REPLY:192.168.1.2:80->192.168.2.2 After 3 way handshake the connection between Client and TPROXY server should be in ESTABLISHED state
2. TPROXY Server connect to HTTP Server using fake src ip and src port. Now the SYN packet should be 192.168.2.2:12345->192.168.1.2:80. But kernel find there is an old ip_conntrack entry for this SYN packet whose state is ESTABLISHED So, kernel will drop the SYN packet, and delete the old entry
3. SYN packet will be retransmitted after 1 sec on TPROXY Server. And then kernel will allocate an new entry for this SYN packet.
4. After 3 way handshake between TPROXY Server and HTTP Server, this ip_conntrack entry should be in ESTABLISHED state
This is a well-known conflic between transparent proxying and connection tracking. The problem is, just as you point out below, that conntrack identifies the connection based solely on the endpoint addresses so that you cannot have two connections with exactly the same endpoints. Unfortunately, with transparent proxying you'd want to do exactly the same: have a fake connection that completely matches the original one.
I think that we can add some new fields to identify the two HTTP connections in ip_conntrack structure(one is Client->TPROXY Server, and the other is TPROXY Server->HTTP Server). I hope we can get the right conntrack when we just know the tuple containing src_ip,src_port,dst_ip,dst_port. Because there is lots of kernel code like this:
1. get the tuple 4 fields(src_ip,src_port,dst_ip,dst_port) from skb 2. call: nf_conntrack_find_get(tuple) to find the corresponding ip_conntrack. So in this case we can only know the 4 fields, we can't get any more message from skb.
Assuming there are 2 ip_conntrack: [1]: 192.168.2.2:12345->192.168.1.2:80(client->TPROXY Server) [2]: 192.168.2.2:12345->192.168.1.2:80(TPROXY Server->HTTP Server) When we process TCP packets between client and TPROXY Server, we find the conntrack for 192.168.2.2:123456->192.168.1.2:80, the result should be [1], and if the TCP packets belong to TPROXY Server and HTTP Server connection, the result should be [2]
Does anyone have good idea about the requirement mentioned above?
I think most users work around the problem: you usually don't really need the source port to be preserved exactly and by choosing a different source port the problem goes away. (If you don't care about the source port then bind the socket to port 0 and the kernel will choose an unused port.) -- KOVACS Krisztian
Hi,
I think that we can add some new fields to identify the two HTTP connections in ip_conntrack structure(one is Client->TPROXY Server, and the other is TPROXY Server->HTTP Server). I hope we can get the right conntrack when we just know the tuple containing src_ip,src_port,dst_ip,dst_port. Because there is lots of kernel code like this:
1. get the tuple 4 fields(src_ip,src_port,dst_ip,dst_port) from skb 2. call: nf_conntrack_find_get(tuple) to find the corresponding ip_conntrack. So in this case we can only know the 4 fields, we can't get any more message from skb.
Assuming there are 2 ip_conntrack: [1]: 192.168.2.2:12345->192.168.1.2:80(client->TPROXY Server) [2]: 192.168.2.2:12345->192.168.1.2:80(TPROXY Server->HTTP Server) When we process TCP packets between client and TPROXY Server, we find the conntrack for 192.168.2.2:123456->192.168.1.2:80, the result should be [1], and if the TCP packets belong to TPROXY Server and HTTP Server connection, the result should be [2]
Does anyone have good idea about the requirement mentioned above?
I think most users work around the problem: you usually don't really need the source port to be preserved exactly and by choosing a different source port the problem goes away.
(If you don't care about the source port then bind the socket to port 0 and the kernel will choose an unused port.)
Thanks for your reply. But, I think maybe this solution still have problems. Here is a sample: Client -> TPROXY Server (192.168.2.2:12345 ->192.168.1.2:80) TPROXY Server -> HTTP Server(192.168.2.2:54321->192.168.1.2:80) Now the two connections have different ip_conntrack. TPROXY Server use port 54321 instead of 12345 as src port Next time, the client connect to the HTTP Server Client -> HTTP Server(192.168.2.2:54321 -> 192.168.1.2:80). Now Client use the port 54321 connecting to 192.168.1.2:80(This is common for a NAT server connecting to the same HTTP Server using different src port) When TPROXY receive this packet, and find ip_conntrack. It will match the old one - TPROXY Server -> HTTP Server. This still exist the ip_conntrack conflict.
-- KOVACS Krisztian
-- Thanks BR. Wei Dong
Hi, On sze, okt 22, 2008 at 04:11:09 +0800, Dong Wei wrote:
I think that we can add some new fields to identify the two HTTP connections in ip_conntrack structure(one is Client->TPROXY Server, and the other is TPROXY Server->HTTP Server). I hope we can get the right conntrack when we just know the tuple containing src_ip,src_port,dst_ip,dst_port. Because there is lots of kernel code like this:
1. get the tuple 4 fields(src_ip,src_port,dst_ip,dst_port) from skb 2. call: nf_conntrack_find_get(tuple) to find the corresponding ip_conntrack. So in this case we can only know the 4 fields, we can't get any more message from skb.
Assuming there are 2 ip_conntrack: [1]: 192.168.2.2:12345->192.168.1.2:80(client->TPROXY Server) [2]: 192.168.2.2:12345->192.168.1.2:80(TPROXY Server->HTTP Server) When we process TCP packets between client and TPROXY Server, we find the conntrack for 192.168.2.2:123456->192.168.1.2:80, the result should be [1], and if the TCP packets belong to TPROXY Server and HTTP Server connection, the result should be [2]
Does anyone have good idea about the requirement mentioned above?
I think most users work around the problem: you usually don't really need the source port to be preserved exactly and by choosing a different source port the problem goes away.
(If you don't care about the source port then bind the socket to port 0 and the kernel will choose an unused port.)
Thanks for your reply. But, I think maybe this solution still have problems.
Here is a sample: Client -> TPROXY Server (192.168.2.2:12345 ->192.168.1.2:80) TPROXY Server -> HTTP Server(192.168.2.2:54321->192.168.1.2:80) Now the two connections have different ip_conntrack. TPROXY Server use port 54321 instead of 12345 as src port
Next time, the client connect to the HTTP Server Client -> HTTP Server(192.168.2.2:54321 -> 192.168.1.2:80). Now Client use the port 54321 connecting to 192.168.1.2:80(This is common for a NAT server connecting to the same HTTP Server using different src port) When TPROXY receive this packet, and find ip_conntrack. It will match the old one - TPROXY Server -> HTTP Server.
This still exist the ip_conntrack conflict.
Yes, and unfortunately you can't do much about that if you're using connection tracking. (Except for implementing some method of differentiating between the two conntracks in ip_conntrack/nf_conntrack, of course.) -- KOVACS Krisztian
On Wed, 2008-10-22 at 16:59 +0200, KOVACS Krisztian wrote:
Hi,
On sze, okt 22, 2008 at 04:11:09 +0800, Dong Wei wrote:
I think that we can add some new fields to identify the two HTTP connections in ip_conntrack structure(one is Client->TPROXY Server, and the other is TPROXY Server->HTTP Server). I hope we can get the right conntrack when we just know the tuple containing src_ip,src_port,dst_ip,dst_port. Because there is lots of kernel code like this:
1. get the tuple 4 fields(src_ip,src_port,dst_ip,dst_port) from skb 2. call: nf_conntrack_find_get(tuple) to find the corresponding ip_conntrack. So in this case we can only know the 4 fields, we can't get any more message from skb.
Assuming there are 2 ip_conntrack: [1]: 192.168.2.2:12345->192.168.1.2:80(client->TPROXY Server) [2]: 192.168.2.2:12345->192.168.1.2:80(TPROXY Server->HTTP Server) When we process TCP packets between client and TPROXY Server, we find the conntrack for 192.168.2.2:123456->192.168.1.2:80, the result should be [1], and if the TCP packets belong to TPROXY Server and HTTP Server connection, the result should be [2]
Does anyone have good idea about the requirement mentioned above?
I think most users work around the problem: you usually don't really need the source port to be preserved exactly and by choosing a different source port the problem goes away.
(If you don't care about the source port then bind the socket to port 0 and the kernel will choose an unused port.)
Thanks for your reply. But, I think maybe this solution still have problems.
Here is a sample: Client -> TPROXY Server (192.168.2.2:12345 ->192.168.1.2:80) TPROXY Server -> HTTP Server(192.168.2.2:54321->192.168.1.2:80) Now the two connections have different ip_conntrack. TPROXY Server use port 54321 instead of 12345 as src port
Next time, the client connect to the HTTP Server Client -> HTTP Server(192.168.2.2:54321 -> 192.168.1.2:80). Now Client use the port 54321 connecting to 192.168.1.2:80(This is common for a NAT server connecting to the same HTTP Server using different src port) When TPROXY receive this packet, and find ip_conntrack. It will match the old one - TPROXY Server -> HTTP Server.
This still exist the ip_conntrack conflict.
Yes, and unfortunately you can't do much about that if you're using connection tracking. (Except for implementing some method of differentiating between the two conntracks in ip_conntrack/nf_conntrack, of course.)
Or using the NOTRACK target to exempt tproxied connections from conntrack. -- Bazsi
I thing there is no need to track connction from client to TPROXY server. And what about to move -m socket match and other stuff to the raw conntrack table and then use a -j NOTRACK target to selectively do not conntrack connections from client to tproxy server, while other connections, include from tproxy server to the world , will be still conntracked ? please execuse my wrong english # ------------ Původní zpráva ------------ # Od: KOVACS Krisztian <hidden@sch.bme.hu> # Předmět: Re: [tproxy] tproxy can't work with ip_conntrack # Datum: 22.10.2008 17:30:04 # ---------------------------------------- # Hi, # # On sze, okt 22, 2008 at 04:11:09 +0800, Dong Wei wrote: # > >> I think that we can add some new fields to identify the two HTTP # > >> connections in ip_conntrack structure(one is Client->TPROXY Server, # > >> and the other is TPROXY Server->HTTP Server). # > >> I hope we can get the right conntrack when we just know the tuple # > >> containing src_ip,src_port,dst_ip,dst_port. Because there is lots of # > >> kernel code like this: # > >> # > >> 1. get the tuple 4 fields(src_ip,src_port,dst_ip,dst_port) from skb # > >> 2. call: nf_conntrack_find_get(tuple) to find the corresponding # ip_conntrack. # > >> So in this case we can only know the 4 fields, we can't get any more # > >> message from skb. # > >> # > >> Assuming there are 2 ip_conntrack: # > >> [1]: 192.168.2.2:12345->192.168.1.2:80(client->TPROXY Server) # > >> [2]: 192.168.2.2:12345->192.168.1.2:80(TPROXY Server->HTTP Server) # > >> When we process TCP packets between client and TPROXY Server, we find # > >> the conntrack for 192.168.2.2:123456->192.168.1.2:80, the result # > >> should # > >> be [1], and if the TCP packets belong to TPROXY Server and HTTP Server # > >> connection, the result should be [2] # > >> # > >> Does anyone have good idea about the requirement mentioned above? # > > # > > I think most users work around the problem: you usually don't really # > > need the source port to be preserved exactly and by choosing a different # > > source port the problem goes away. # > > # > > (If you don't care about the source port then bind the socket to port 0 # > > and the kernel will choose an unused port.) # > # > Thanks for your reply. But, I think maybe this solution still have problems. # > # > Here is a sample: # > Client -> TPROXY Server (192.168.2.2:12345 ->192.168.1.2:80) # > TPROXY Server -> HTTP Server(192.168.2.2:54321->192.168.1.2:80) # > Now the two connections have different ip_conntrack. TPROXY Server use # > port 54321 instead of 12345 as src port # > # > Next time, the client connect to the HTTP Server # > Client -> HTTP Server(192.168.2.2:54321 -> 192.168.1.2:80). # > Now Client use the port 54321 connecting to 192.168.1.2:80(This is # > common for a NAT server connecting to the same HTTP Server using # > different src port) # > When TPROXY receive this packet, and find ip_conntrack. It will match # > the old one - TPROXY Server -> HTTP Server. # > # > This still exist the ip_conntrack conflict. # # Yes, and unfortunately you can't do much about that if you're using # connection tracking. (Except for implementing some method of # differentiating between the two conntracks in ip_conntrack/nf_conntrack, # of course.) # # -- # KOVACS Krisztian # _______________________________________________ # tproxy mailing list # tproxy@lists.balabit.hu # https://lists.balabit.hu/mailman/listinfo/tproxy # # #
Hi, 2008/10/23 NTPT <NTPT@seznam.cz>:
I thing there is no need to track connction from client to TPROXY server.
And what about to move -m socket match and other stuff to the raw conntrack table and then use a -j NOTRACK target to selectively do not conntrack connections from client to tproxy server, while other connections, include from tproxy server to the world , will be still conntracked ?
please execuse my wrong english
I think it's hard for us to check whether a TCP packet belongs to a tproxy socket. When we receive a SYN from the client, we can't make sure this packet belongs to a tproxy socket before the packet travel to TPROXY target in mangle table PREROUTING chain. Here is an sample: iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY \ --tproxy-mark 0x1/0x1 --on-port 50080 When TPROXY target handle the SYN packet, find the dport is 80, then it will look up the tproxy socket listening on 50080. But in raw table we don't know which listening port should we take when we find the SYN packet dport is 80. I think maybe we need write a match like TPROXY, all the matched packet do NOTRACK
# ------------ Původní zpráva ------------ # Od: KOVACS Krisztian <hidden@sch.bme.hu> # Předmět: Re: [tproxy] tproxy can't work with ip_conntrack # Datum: 22.10.2008 17:30:04 # ---------------------------------------- # Hi, # # On sze, okt 22, 2008 at 04:11:09 +0800, Dong Wei wrote: # > >> I think that we can add some new fields to identify the two HTTP # > >> connections in ip_conntrack structure(one is Client->TPROXY Server, # > >> and the other is TPROXY Server->HTTP Server). # > >> I hope we can get the right conntrack when we just know the tuple # > >> containing src_ip,src_port,dst_ip,dst_port. Because there is lots of # > >> kernel code like this: # > >> # > >> 1. get the tuple 4 fields(src_ip,src_port,dst_ip,dst_port) from skb # > >> 2. call: nf_conntrack_find_get(tuple) to find the corresponding # ip_conntrack. # > >> So in this case we can only know the 4 fields, we can't get any more # > >> message from skb. # > >> # > >> Assuming there are 2 ip_conntrack: # > >> [1]: 192.168.2.2:12345->192.168.1.2:80(client->TPROXY Server) # > >> [2]: 192.168.2.2:12345->192.168.1.2:80(TPROXY Server->HTTP Server) # > >> When we process TCP packets between client and TPROXY Server, we find # > >> the conntrack for 192.168.2.2:123456->192.168.1.2:80, the result # > >> should # > >> be [1], and if the TCP packets belong to TPROXY Server and HTTP Server # > >> connection, the result should be [2] # > >> # > >> Does anyone have good idea about the requirement mentioned above? # > > # > > I think most users work around the problem: you usually don't really # > > need the source port to be preserved exactly and by choosing a different # > > source port the problem goes away. # > > # > > (If you don't care about the source port then bind the socket to port 0 # > > and the kernel will choose an unused port.) # > # > Thanks for your reply. But, I think maybe this solution still have problems. # > # > Here is a sample: # > Client -> TPROXY Server (192.168.2.2:12345 ->192.168.1.2:80) # > TPROXY Server -> HTTP Server(192.168.2.2:54321->192.168.1.2:80) # > Now the two connections have different ip_conntrack. TPROXY Server use # > port 54321 instead of 12345 as src port # > # > Next time, the client connect to the HTTP Server # > Client -> HTTP Server(192.168.2.2:54321 -> 192.168.1.2:80). # > Now Client use the port 54321 connecting to 192.168.1.2:80(This is # > common for a NAT server connecting to the same HTTP Server using # > different src port) # > When TPROXY receive this packet, and find ip_conntrack. It will match # > the old one - TPROXY Server -> HTTP Server. # > # > This still exist the ip_conntrack conflict. # # Yes, and unfortunately you can't do much about that if you're using # connection tracking. (Except for implementing some method of # differentiating between the two conntracks in ip_conntrack/nf_conntrack, # of course.) # # -- # KOVACS Krisztian # _______________________________________________ # tproxy mailing list # tproxy@lists.balabit.hu # https://lists.balabit.hu/mailman/listinfo/tproxy # # #
-- Thanks BR. Wei Dong
On Thu, 2008-10-23 at 10:24 +0800, Dong Wei wrote:
Hi,
2008/10/23 NTPT <NTPT@seznam.cz>:
I thing there is no need to track connction from client to TPROXY server.
And what about to move -m socket match and other stuff to the raw conntrack table and then use a -j NOTRACK target to selectively do not conntrack connections from client to tproxy server, while other connections, include from tproxy server to the world , will be still conntracked ?
please execuse my wrong english
I think it's hard for us to check whether a TCP packet belongs to a tproxy socket. When we receive a SYN from the client, we can't make sure this packet belongs to a tproxy socket before the packet travel to TPROXY target in mangle table PREROUTING chain. Here is an sample: iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY \ --tproxy-mark 0x1/0x1 --on-port 50080 When TPROXY target handle the SYN packet, find the dport is 80, then it will look up the tproxy socket listening on 50080. But in raw table we don't know which listening port should we take when we find the SYN packet dport is 80.
I think maybe we need write a match like TPROXY, all the matched packet do NOTRACK
Well, tproxy could be made working in the raw table as well, technically there are no differences between mangle and raw, apart from their netfilter priorities. But also, I think it is quite easy to add a rule to raw/OUTPUT which applies NOTRACK on all tproxied connections, something like: iptables -t raw -A OUTPUT -m addrtype ! --src-type LOCAL -j NOTRACK The only question is how to match the reverse direction, but you probably could match that by interface or something. -- Bazsi
participants (4)
-
Balazs Scheidler
-
Dong Wei
-
KOVACS Krisztian
-
NTPT