I'm having a little trouble binding a tproxy and I might do something wrong.
Hello, I am writing a small proxy and while trying to use tproxy I am getting a problem while I am not sure I am doing it right. The server accepts the connection and identify the two ends. But when I try to connect using the client IP and PORT I am getting error "Connection timed out - connect(2)" while on netstat I am getting two sockets: tcp 0 1 192.168.10.100:51573 81.218.79.155:80 SYN_SENT tcp 287 0 81.218.79.155:80 192.168.10.100:51573 ESTABLISHED The upper one is the server "connect" which seems to stuck like that with no response. This problem accrues only when I try to bind the same ip and port of the client. When I use other random port as src port for the forged connection I can connect and everything is fine. I might be doing something wrong but since I have no clue. I am using ruby and the basic code: #!/usr/bin/ruby require 'socket' server = Socket.new(Socket::AF_INET,Socket::SOCK_STREAM) server.setsockopt(Socket::SOL_SOCKET, Socket::SO_REUSEADDR, true) server.setsockopt(Socket::SOL_SOCKET, Socket::SO_BROADCAST, true) server.setsockopt(Socket::SOL_IP, 19, 1) server_port = Socket.pack_sockaddr_in( 1111, '0.0.0.0') result = server.bind(server_port) server.listen(10) puts "server started with code: #{result}" while (connection = server.accept) Thread.new(connection) do |conn| puts "new connection" port, host = Socket.unpack_sockaddr_in conn[1] client = "#{host}:#{port}" puts "#{client} is connected" local_address = (Socket.unpack_sockaddr_in conn[0].local_address) + [conn[0].local_address.ipv4?] remote_address = (Socket.unpack_sockaddr_in conn[0].remote_address) + [conn[0].remote_address.ipv4?] puts "local_address: #{local_address}" puts "remote_address: #{remote_address}" fake_local = Socket.pack_sockaddr_in(0,remote_address[1]) if local_address[2] remote_connection = Socket.new(Socket::AF_INET,Socket::SOCK_STREAM,0) remote_connection.setsockopt(Socket::SOL_IP, Socket::IP_TRANSPARENT, 1) result = remote_connection.bind(conn[0].remote_address) puts "Bind result: #{result}" else remote_connection = Socket.new(Socket::AF_INET6,Socket::SOCK_STREAM,0) remote_connection.setsockopt(Socket::SOL_IP, Socket::IP_TRANSPARENT, 1) result = remote_connection.bind(conn[0].remote_address) puts "Bind result: #{result}" end begin puts "Server Connect result: #{remote_connection.connect(conn[0].local_address)}" rescue => e puts e.exception puts e.message end end end ##end of file The output from the server is: server started with code: 0 new connection 192.168.10.100:51573 is connected local_address: [80, "81.218.79.155", true] remote_address: [51573, "192.168.10.100", true] Bind result: 0 Connection timed out - connect(2) Connection timed out - connect(2) Thanks, Eliezer
Hi, On 12/20/2012 06:54 PM, Eliezer Croitoru wrote:
Hello,
I am writing a small proxy and while trying to use tproxy I am getting a problem while I am not sure I am doing it right.
The server accepts the connection and identify the two ends. But when I try to connect using the client IP and PORT I am getting error "Connection timed out - connect(2)" while on netstat I am getting two sockets: tcp 0 1 192.168.10.100:51573 81.218.79.155:80 SYN_SENT tcp 287 0 81.218.79.155:80 192.168.10.100:51573 ESTABLISHED
The upper one is the server "connect" which seems to stuck like that with no response.
This problem accrues only when I try to bind the same ip and port of the client. When I use other random port as src port for the forged connection I can connect and everything is fine.
I might be doing something wrong but since I have no clue.
I think this might be the limitation of Netfilter's connection tracking code. Do you have connection tracking enabled? If so, the problem is that while you have two distinct TCP connections in the TCP stack, the connection tracking code is using exclusively the connection endpoint addresses as the key for its own state table and thus tries to use only a single conntrack entry for your two TCP connections. I have seen exactly this problem before and you have a few options: * do not use connection tracking (not always an option) * use connection tracking but use connection tracking zones to have the two separate conntrack state tables (https://lwn.net/Articles/370152/) * you can combine connection tracking zones with a custom hack (involving introducing a new socket flag for sockets on which connect() has been called and extending the socket match so that you can select a conntrack zone based on that flag) Unfortunately not using the same source port is not an ultimate solution, either: if you use a random source port you still have a chance that it will clash with the endpoint of another existing TCP connection. -- KOVACS Krisztian
On 2/4/2013 2:02 PM, KOVACS Krisztian wrote:
Unfortunately not using the same source port is not an ultimate solution, either: if you use a random source port you still have a chance that it will clash with the endpoint of another existing TCP connection. Most likely to not since it's a pair of ip+port to ip+port. Your basic assumption is that there are two devices that controls the same ip and port assignment. on a machine the OS tries to avoid using the same port for the same dst as a basic rule. on a nat machine it depends on the nat type but linux from box don't do this kind of nat that will make such thing happen.
-- Eliezer Croitoru http://www1.ngtech.co.il
Hi, On Mon 04 Feb 2013 01:19:10 PM CET, Eliezer Croitoru wrote:
On 2/4/2013 2:02 PM, KOVACS Krisztian wrote:
Unfortunately not using the same source port is not an ultimate solution, either: if you use a random source port you still have a chance that it will clash with the endpoint of another existing TCP connection. Most likely to not since it's a pair of ip+port to ip+port. Your basic assumption is that there are two devices that controls the same ip and port assignment. on a machine the OS tries to avoid using the same port for the same dst as a basic rule.
Yes, but only for local sockets. However, in this case the endpoint address is first chosen by the client's TCP stack and then on the proxy's TCP stack. The latter does not have a socket bound to the address yet, so it will be happy to choose the exact same port.
on a nat machine it depends on the nat type but linux from box don't do this kind of nat that will make such thing happen.
Yep, that's true, the NAT code avoids conntrack duplicates at all costs. (Even if that means an extra implicit translation.) -- KOVACS Krisztian
On 2/4/2013 5:46 PM, KOVACS Krisztian wrote:
Yes, but only for local sockets. However, in this case the endpoint address is first chosen by the client's TCP stack and then on the proxy's TCP stack. The latter does not have a socket bound to the address yet, so it will be happy to choose the exact same port. From the proxy point of view it's a connection and he can use a random port which the OS will make sure that is ok since it actually pairs the src IP to the dst IP when binding. I wanted the same as you. This adds a bit complexity to the kernel and by the way the tproxy socket is a local socket from OS eyes but have another non local IP. You can try it in the real world and see that unless you are working with specific network protocols and you need to know things about the src side you wont have any troubles with TPROXY and TCP.
There is too much experience with it that makes it a fact the it works. From any application the TPROXY outgoing socket is another FD so the dst and src are only important for loging. What are you working on? Regards, Eliezer -- Eliezer Croitoru http://www1.ngtech.co.il IT consulting for Nonprofit organizations eliezer <at> ngtech.co.il
On 2/4/2013 5:46 PM, KOVACS Krisztian wrote:
Hi,
On Mon 04 Feb 2013 01:19:10 PM CET, Eliezer Croitoru wrote:
On 2/4/2013 2:02 PM, KOVACS Krisztian wrote:
Unfortunately not using the same source port is not an ultimate solution, either: if you use a random source port you still have a chance that it will clash with the endpoint of another existing TCP connection. Most likely to not since it's a pair of ip+port to ip+port. Your basic assumption is that there are two devices that controls the same ip and port assignment. on a machine the OS tries to avoid using the same port for the same dst as a basic rule.
Yes, but only for local sockets. However, in this case the endpoint address is first chosen by the client's TCP stack and then on the proxy's TCP stack. The latter does not have a socket bound to the address yet, so it will be happy to choose the exact same port.
on a nat machine it depends on the nat type but linux from box don't do this kind of nat that will make such thing happen.
Yep, that's true, the NAT code avoids conntrack duplicates at all costs. (Even if that means an extra implicit translation.)
Sorry I havn't seen the context of the mail and it seems like I got my answers while you were trying to help them. Thanks, Eliezer
-- KOVACS Krisztian
-- Eliezer Croitoru http://www1.ngtech.co.il IT consulting for Nonprofit organizations eliezer <at> ngtech.co.il
participants (2)
-
Eliezer Croitoru
-
KOVACS Krisztian