[tproxy] TProxy and REUSEADDR

Thu Apr 15 10:31:58 CEST 2010

On Tue, 2010-03-02 at 16:23 +0100, nicolas normand wrote:
> Hi
> 
> It is possible for a proxy to transparently and simultaneously connect
> to several backends (with different IP) with the same client IP/Port ?
> I have tried to SO_REUSEADDR before binding but I have a strange behavior:
> 
> * the concerned program is a tproxy apache patch, which works
> perfectly for simple and single request, but doesn't handle
> subrequests (and probably some other twisted cases).
> * the patch is based on this incomplete one:
> http://miscfiles.googlecode.com/svn/trunk/tproxy.patch , which I just
> ported it to apache 2.2.14, made the listening socket transparent and
> made the options to work. You can found it included this mail.
> * Code in proxy/module/proxy_utils.c now look likes:
>         if (conf->tproxy) {
>             if ((rv = apr_socket_opt_set(newsock, APR_IP_TRANSPARENT,
> 1)) != APR_SUCCESS) {
>                 ap_log_error(APLOG_MARK, APLOG_ERR, rv, s,
>                              "apr_socket_opt_set(APR_IP_TRANSPARENT):
> Failed to set");
>             }
> 
>             if ((rv = apr_socket_opt_set(newsock, APR_SO_REUSEADDR,
> 1)) != APR_SUCCESS) {
>                 ap_log_error(APLOG_MARK, APLOG_ERR, rv, s,
>                              "apr_socket_opt_set(APR_SO_REUSE_ADDR):
> Failed to set");
>             }
> 
>             if ((rv = apr_socket_bind(newsock, remote_addr)) != APR_SUCCESS) {
>                 ap_log_error(APLOG_MARK, APLOG_ERR, rv, s,
>                 "proxy: error binding to socket %pI", remote_addr);
>                 return HTTP_INTERNAL_SERVER_ERROR;
>             }
>         }
> 
> When connecting the second backend (which has a different IP address),
> I got this (192.168.100.12 is the CLIENT, 192.168.100.127 is the
> PROXY, 192.168.100.238 and 192.168.100.229 are the BACKENDS):
> 
> // Connecting to first backend, all is okay
> 15:49:32.037788 IP 192.168.100.12.57009 > 192.168.100.238.80: S
> 1617296302:1617296302(0) win 5840 <mss 1460,sackOK,timestamp 21314512
> 0,nop,wscale 6>
> 15:49:32.037827 IP 192.168.100.238.80 > 192.168.100.12.57009: S
> 1072906513:1072906513(0) ack 1617296303 win 5792 <mss
> 1460,sackOK,timestamp 107453675 21314512,nop,wscale 4>
> 15:49:32.038268 IP 192.168.100.12.57009 > 192.168.100.238.80: . ack 1
> win 92 <nop,nop,timestamp 21314512 107453675>
> 15:49:32.038855 IP 192.168.100.12.57009 > 192.168.100.238.80: P
> 1:386(385) ack 1 win 92 <nop,nop,timestamp 21314512 107453675>
> 15:49:32.038877 IP 192.168.100.238.80 > 192.168.100.12.57009: . ack
> 386 win 429 <nop,nop,timestamp 107453676 21314512>
> 15:49:32.040221 IP 192.168.100.238.80 > 192.168.100.12.57009: .
> 1:1449(1448) ack 386 win 429 <nop,nop,timestamp 107453676 21314512>
> 15:49:32.040377 IP 192.168.100.238.80 > 192.168.100.12.57009: P
> 1449:1795(346) ack 386 win 429 <nop,nop,timestamp 107453676 21314512>
> 15:49:32.040485 IP 192.168.100.12.57009 > 192.168.100.238.80: . ack
> 1449 win 137 <nop,nop,timestamp 21314512 107453676>
> 15:49:32.040608 IP 192.168.100.12.57009 > 192.168.100.238.80: . ack
> 1795 win 182 <nop,nop,timestamp 21314513 107453676>
> 
> // And now try connecting the second backend... Hell, the client sent
> the Reset flag...
> 15:49:32.067711 IP 192.168.100.12.57009 > 192.168.100.229.80: S
> 1618236269:1618236269(0) win 5840 <mss 1460,sackOK,timestamp 21314519
> 0,nop,wscale 6>
> 15:49:32.067861 IP 192.168.100.229.80 > 192.168.100.12.57009: S
> 840737942:840737942(0) ack 1618236270 win 5792 <mss
> 1460,sackOK,timestamp 107448908 21314519,nop,wscale 4>
> 15:49:32.068837 IP 192.168.100.12.57009 > 192.168.100.229.80: R
> 1618236270:1618236270(0) win 0
> 15:49:35.069466 IP 192.168.100.12.57009 > 192.168.100.229.80: S
> 1618236269:1618236269(0) win 5840 <mss 1460,sackOK,timestamp 21315270
> 0,nop,wscale 6>
> 15:49:35.069483 IP 192.168.100.229.80 > 192.168.100.12.57009: S
> 887635567:887635567(0) ack 1618236270 win 5792 <mss
> 1460,sackOK,timestamp 107449658 21315270,nop,wscale 4>
> 15:49:35.072257 IP 192.168.100.12.57009 > 192.168.100.229.80: R
> 1618236270:1618236270(0) win 0
> 15:49:41.070224 IP 192.168.100.12.57009 > 192.168.100.229.80: S
> 1618236269:1618236269(0) win 5840 <mss 1460,sackOK,timestamp 21316770
> 0,nop,wscale 6>
> 15:49:41.070363 IP 192.168.100.229.80 > 192.168.100.12.57009: S
> 981396815:981396815(0) ack 1618236270 win 5792 <mss
> 1460,sackOK,timestamp 107451158 21316770,nop,wscale 4>
> 15:49:41.071604 IP 192.168.100.12.57009 > 192.168.100.229.80: R
> 1618236270:1618236270(0) win 0
> 
> 
> I have also tried to make a client to simultaneously connect to two
> different backends while both connecting sockets binded to the same ip
> (no proxy, no transparency) with no problems... I don't understand
> what I have missed

Hm.. it should work. I've just tried it with a local virtual machine and
it does indeed work.

I simulated what you did with two netcat instances, using the same local
port and a different destination port (instead of different IPs I've
used different ports)

$ nc -T -s 10.30.1.33 -p 10000 10.30.1.1 2000
$ nc -T -s 10.30.1.33 -p 10000 10.30.1.1 2001

I've also confirmed that the netcat instances indeed use the same local
endpoint, here are the SYN packets:

1) one connection
10:30:20.356370 08:00:27:44:3b:03 > 0a:00:27:00:00:00, ethertype IPv4
(0x0800), length 74: 10.30.1.33.10000 > 10.30.1.1.2001: Flags [S], seq
917488010, win 5840, options [mss 1460,sackOK,TS val 117738 ecr
0,nop,wscale 5], length 0

2) the other connection
10:30:23.589674 08:00:27:44:3b:03 > 0a:00:27:00:00:00, ethertype IPv4
(0x0800), length 74: 10.30.1.33.10000 > 10.30.1.1.2000: Flags [S], seq
960239781, win 5840, options [mss 1460,sackOK,TS val 118542 ecr
0,nop,wscale 5], length 0

So indeed this should work and I guess there's something in your
environment that prevents it from working.

-- 
Bazsi