[tproxy] One way tproxy problem

[Tim Price]

Hi All,

I'm trying to setup a Debain + Squid + tproxy setup using kernel 2.6.31.4, Squid 3.1.0.14 (--enable-linux-netfilter) and Iptables 1.4.3
I suspect you've heard this all before, my story is nearly identical with one exception:  It kind of works.

I've got a server with two interfaces in a bridge, the bridge interface has the server IP address of 218.185.224.15/26.
The eth0 interface faces the internet
The eth1 interface faces the test client (218.185.224.20)

I have the following ebtables rules setup:
-----------------------------------------------
Bridge chain: BROUTING, entries: 2, policy: ACCEPT
-p IPv4 -i eth1 --ip-proto tcp --ip-dport 80 -j redirect  --redirect-target DROP
-p IPv4 -i eth0 --ip-proto tcp --ip-sport 80 -j redirect  --redirect-target DROP

I have the following Iptables rules setup:
-----------------------------------------------
-A PREROUTING -p tcp -m tcp -m socket -j DIVERT
-A PREROUTING -p tcp -m tcp --dport 80 -j TPROXY  --on-port 3129 --tproxy-mark 1
-A DIVERT -j MARK  --set-xmark 1
-A DIVERT -j ACCEPT

General routing config:
-----------------------------------------------
set net.ipv4.forwarding = 1
ip rule add fwmark 1 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100

Squid Config:
-----------------------------------------------
acl manager proto cache_object
acl localhost src 127.0.0.1/32
acl to_localhost dst 127.0.0.0/8
acl TPN src 218.185.224.0/21
acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
acl TPN src 125.238.251.203
http_access allow manager localhost
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localnet
http_access allow TPN
http_access deny all
icp_access allow localnet
icp_access deny all
htcp_access allow localnet
htcp_access deny all
http_port 3128
http_port 3129 tproxy
cache_mem 2000 MB
maximum_object_size_in_memory 80 KB
cache_dir ufs /var/spool/squid3 45000 16 256
maximum_object_size 1 GB
access_log /usr/local/squid/var/logs/access.log squid
pid_filename /var/run/squid.pid
debug_options ALL,1 89,6
url_rewrite_host_header off
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern . 30 20% 4320 override-expire reload-into-ims
cache_effective_user nobody
httpd_suppress_version_string on
icp_port 3130
memory_pools_limit 0
forwarded_for off
coredump_dir /usr/local/squid/var/cache
cache_effective_group nogroup

The testing appears to work ok:
-----------------------------------------
The client makes a http request which get redirected as expected to squid and I get a line in my access.log something along the lines of:  
1257212804.993      0 218.185.224.20 TCP_IMS_HIT/304 301 GET http://pagead2.googlesyndication.com/pagead/sma8.js - NONE/- text/javascript

I also get something like this in my cache.log:
2009/11/03 14:46:44.446| IpIntercept.cc(381) NatLookup: address BEGIN: me= 66.102.7.156:80, client= 66.102.7.156:80, dst= 218.185.224.20:5054, peer= 218.185.224.20:5054

Heres where it gets interesting:
If I go to whatismyip.com (or similar) I get told that I'm behind a squid proxy and my IP address is the IP address of the squid proxy...

If I do a tcpdump on the client facing interface (eth1) I see what basically amounts to a conversation between the client and the remote server, checking the link level headers I see that the 'remote server' mac address is actually the mac address of the bridge interface of the squid server (which I would have expected).  So it's spoofing the server's IP address to the client.

However doing a tcpdump on the internet facing interface I see a conversation between the squid server and the remote server.

At this point I'm a bit lost.  I did try turning on full debug in squid at some point but it was a bit of information overload and didn't appear to contain any of the common problems documented on the squid wiki etc.

It should be working from all the documentation I've read but it simply refuses to, and even more inexplicably it works in one direction.

Does anyone have any ideas?


Regards,

Tim Price.