[zorp] Zorp and TPROXY can't bind or loops infinitely

Sheldon Hearn zorp@lists.balabit.hu
Mon, 12 Apr 2004 01:14:42 +0200


Hi folks,

Just before I finish pulling out all my hair in frustration, I thought
I'd ask for help here. :-)

Basically, I think I've done everything correctly, but can't get zorp to
establish the outbound connection to the real server.  So obviously I
haven't done everything correctly. :-)

Client:		172.16.100.2

Server:		10.0.0.101

Zorp01:
	Linux 2.6.3 (Gentoo-r1 w/ Balabit TPROXY patch)
	zorp-2.0.8 + libzorpll-2.0.26.24
		OR
	zorp-2.1.5.5 + libzorpll-2.1.12.2-1

	eth0	1.2.3.4	<public address obscured>
		192.168.0.1
	eth1	192.168.1.1
	dummy0	192.168.4.1

Routing is correct:

Destination     Gateway         Genmask         Iface
1.2.3.4         0.0.0.0         255.255.255.240 eth0
172.16.100.0    192.168.0.254   255.255.255.0   eth0
10.0.0.0        192.168.1.254   255.255.255.0   eth1
192.168.1.0     0.0.0.0         255.255.255.0   eth1
192.168.0.0     0.0.0.0         255.255.255.0   eth0
...

The client routes to the server through the zorp01 box via
192.168.0.154, which is an LVS-DR/fwmarks load-balancer and which
delivers the HTTP traffic to 192.168.0.1's MAC address (eth0), but still
with 10.0.0.101 as the destination address.

Similarly, 192.168.1.154 will gateway traffic between the zorp box and
the server, although I haven't bothered to set it up, since zorp still
isn't pushing traffic at it (and because I'm still not sure how I'm
gonna get the reverse load balancing working, assuming paired SH/DH
doesn't work).

Here's the tproxy iptable:

*tproxy
:PREROUTING ACCEPT
:OUTPUT ACCEPT
:PRinter -
-A PREROUTING -i eth0 -j PRinter
-A PRinter -p tcp --dport 80 -j LOG \
   --log-prefix "NF: PRinter TPROXY 50080: "
// Using --on-ip didn't change results
-A PRinter -p tcp --dport 80 -j TPROXY --on-port 50080
COMMIT

Here's instances.conf:

zorp_http --verbose=5  --policy /etc/zorp/policy.py --autobind-ip 192.168.4.1

Here's the policy.py:

-----------
Zorp.firewall_name = 'zorp@zorp01'
 
InetZone("internet", "0.0.0.0/0",
	inbound_services=["intra_http"],
	outbound_services=["intra_http"])
 
# We get different results with and without snat=...; see description
# to follow.
def zorp_http():
	Service("intra_http", HttpProxy, snat=ForgeClientSourceNAT())
	Listener(SockAddrInet("0.0.0.0", 50080), "intra_http")
-----------

With "snat=ForgeClientSourceNAT()" shown above, we get

	bind() failed; error='Cannot assign requested address'

as shown in this log extract:

-----------
00:05:08 zorp_http[14429]: (zorp@zorp01/intra_http): Starting service; name='intra_http'
00:05:08 zorp_http[14429]: (zorp@zorp01/intra_http:0): Starting proxy instance; client_fd='15', client_address='AF_INET(172.16.100.2:32929)', client_zone='Zone(internet, 0.0.0.0/0)', client_local='AF_INET(1.2.3.4:50080)'
00:05:08 zorp_http[14429]: (zorp@zorp01/intra_http:0/http): Proxy starting; class='HttpProxy', module='http'
00:05:08 zorp_http[14429]: (zorp_http@zorp@zorp01/nosession): accept count; accepts='1'
00:21:40 zorp01 zorp_http[14848]: (zorp@zorp01/intra_http:0/http): bind() failed; error='Cannot assign requested address'
00:21:40 zorp01 zorp_http[14848]: (zorp@zorp01/intra_http:0/http): Server connection failure; server_address='AF_INET(1.2.3.4:50080)', server_zone='Zone(internet, 0.0.0.0/0)', server_local='None'
00:21:40 zorp01 zorp_http[14848]: (zorp@zorp01/intra_http:0/http): Proxy ending; class='HttpProxy', module='http'
00:21:40 zorp01 zorp_http[14848]: (zorp@zorp01/intra_http:0): Ending proxy instance;
00:21:40 zorp01 zorp_http[14848]: (zorp@zorp01/intra_http:0/http/client): accounting info; type='stream', duration='2', sent='840', received='43'
-----------

Without snat=ForgeClientSourceNAT(), zorp connects to itself, causing
the new instance to connect to itself, etc. until we run out of threads.
Tcpdump on the loopback interface confirms that zorp is connecting to
itself (but see NetFilter log extract below):

-----------
23:43:40 zorp01 zorp_http[13705]: (zorp@zorp01/intra_http): Starting service; name='intra_http'
23:43:40 zorp01 zorp_http[13705]: (zorp@zorp01/intra_http:2): Starting proxy instance; client_fd='15', client_address='AF_INET(172.16.100.2:32926)', client_zone='Zone(internet, 0.0.0.0/0)', client_local='AF_INET(1.2.3.4:50080)'
23:43:40 zorp01 zorp_http[13705]: (zorp@zorp01/intra_http:2/http): Proxy starting; class='HttpProxy', module='http'
23:43:40 zorp01 zorp_http[13705]: (zorp_http@zorp@zorp01/nosession): accept count; accepts='1'
23:43:49 zorp01 zorp_http[13724]: (zorp@zorp01/intra_http:2/http): Server connection established; server_fd='16', server_address='AF_INET(1.2.3.4:50080)', server_zone='Zone(internet, 0.0.0.0/0)', server_local='AF_INET(1.2.3.4:34216)'
23:43:49 zorp01 zorp_http[13705]: (zorp@zorp01/intra_http): Starting service; name='intra_http'
23:43:49 zorp01 zorp_http[13705]: (zorp@zorp01/intra_http:3): Starting proxy instance; client_fd='17', client_address='AF_INET(1.2.3.4:34216)', client_zone='Zone(internet, 0.0.0.0/0)', client_local='AF_INET(1.2.3.4:50080)'
23:43:49 zorp01 zorp_http[13705]: (zorp@zorp01/intra_http:3/http): Proxy starting; class='HttpProxy', module='http'
23:43:49 zorp01 zorp_http[13705]: (zorp_http@zorp@zorp01/nosession): accept count; accepts='1'
23:43:49 zorp01 zorp_http[13705]: (zorp@zorp01/intra_http): Starting service; name='intra_http'
23:43:49 zorp01 zorp_http[13725]: (zorp@zorp01/intra_http:3/http): Server connection established; server_fd='18', server_address='AF_INET(1.2.3.4:50080)', server_zone='Zone(internet, 0.0.0.0/0)', server_local='AF_INET(1.2.3.4:34217)'
23:43:49 zorp01 zorp_http[13705]: (zorp@zorp01/intra_http:4): Starting proxy instance; client_fd='19', client_address='AF_INET(1.2.3.4:34217)', client_zone='Zone(internet, 0.0.0.0/0)', client_local='AF_INET(1.2.3.4:50080)'
23:43:49 zorp01 zorp_http[13705]: (zorp@zorp01/intra_http:4/http): Proxy starting; class='HttpProxy', module='http'
23:43:49 zorp01 zorp_http[13705]: (zorp_http@zorp@zorp01/nosession): accept count; accepts='1'
...
23:43:49 zorp01 zorp_http[13821]: (zorp@zorp01/intra_http:99/http): Server connection established; server_fd='210', server_address='AF_INET(1.2.3.4:50080)', server_zone='Zone(internet, 0.0.0.0/0)', server_local='AF_INET(1.2.3.4:34313)'
23:43:49 zorp01 zorp_http[13705]: (zorp@zorp01/intra_http:100): Starting proxy instance; client_fd='211', client_address='AF_INET(1.2.3.4:34313)', client_zone='Zone(internet, 0.0.0.0/0)', client_local='AF_INET(1.2.3.4:50080)'
23:43:49 zorp01 zorp_http[13705]: (zorp@zorp01/intra_http:100/http): Proxy starting; class='HttpProxy', module='http'
23:43:49 zorp01 zorp_http[13705]: (zorp_http@zorp@zorp01/nosession): Too many running threads, waiting for one to become free; num_threads='100', max_threads='100'
23:43:49 zorp01 zorp_http[13705]: (zorp_http@zorp@zorp01/nosession): accept count; accepts='1'
-----------

The NetFilter logs show zorp connecting to the local host, but note that
that this infinite self-connection problem is not caused by the iptables
rules (remember we only jump to TPROXY for traffic inbound on eth0),
because then we'd see a lot more 'PRinter TPROXY 50080' entries:

-----------
23:43:40 zorp01 NF: PRinter TPROXY 50080: IN=eth0 OUT= MAC=00:0c:6e:d2:b0:6d:00:0e:a6:2f:df:76:08:00 SRC=172.16.100.2 DST=10.0.0.101 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=16898 DF PROTO=TCP SPT=32926 DPT=80 WINDOW=5840 RES=0x00 SYN URGP=0
23:43:40 zorp01 NF: INPUT tproxy: IN=eth0 OUT= MAC=00:0c:6e:d2:b0:6d:00:0e:a6:2f:df:76:08:00 SRC=172.16.100.2 DST=1.2.3.4 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=16898 DF PROTO=TCP SPT=32926 DPT=50080 WINDOW=5840 RES=0x00 SYN URGP=0
23:43:40 zorp01 NF: INPUT tproxy: IN=eth0 OUT= MAC=00:0c:6e:d2:b0:6d:00:0e:a6:2f:df:76:08:00 SRC=172.16.100.2 DST=1.2.3.4 LEN=52 TOS=0x10 PREC=0x00 TTL=64 ID=16899 DF PROTO=TCP SPT=32926 DPT=50080 WINDOW=5840 RES=0x00 ACK URGP=0
23:43:49 zorp01 NF: INPUT tproxy: IN=eth0 OUT= MAC=00:0c:6e:d2:b0:6d:00:0e:a6:2f:df:76:08:00 SRC=172.16.100.2 DST=1.2.3.4 LEN=54 TOS=0x10 PREC=0x00 TTL=64 ID=16902 DF PROTO=TCP SPT=32926 DPT=50080 WINDOW=5840 RES=0x00 ACK PSH URGP=0
-----------

It took us a while to get LVS-DR/fwmarks to route the packets to the
Zorp box correctly, and when we got it working I thought the hard work
was over.  But now I'm completely stuck on the Zorp setup.  I can't
figure out why Zorp is behaving this way, but I'm pretty sure it must be
my fault.

Any help would be very much appreciated.

Thanks,
Sheldon.