[tproxy] TPROXY but without bridging?
Wickus Botha - MWEB
WBotha at mweb.com
Wed Apr 9 08:25:30 CEST 2008
System Layout:
Cisco 7200 with wccpv2
Dell 2950 with 1 ethernet interface
Kernel:
Linux cache 2.6.25-rc6 #1 SMP Tue Apr 8 17:19:10 SAST 2008 i686 i686
i386 GNU/Linux
Patch
tproxy-kernel-2.6.25-rc6.20080402-130957-1207134597.patch
had to fix a few rej.
Squid:
Squid Cache version 3.HEAD-CVS from cvs
Patch
http://www.balabit.com/downloads/files/tproxy/tproxy-squid-3_20080408.pa
tch
Few rej but they were resolved.
Configs:
echo 1 > /proc/sys/net/ipv4/ip_forward
echo 1 > /proc/sys/net/ipv4/ip_nonlocal_bind
echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter
iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY --on-port
3128 --tproxy-mark 0x1/0x1
iptables -t mangle -N DIVERT
iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
iptables -t mangle -A DIVERT -p tcp -j MARK --set-mark 1
iptables -t mangle -A DIVERT -j ACCEPT
ip rule add fwmark 1 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100
[root at cache com]# iptables -L -v -t mangle
Chain PREROUTING (policy ACCEPT 34018 packets, 3155K bytes)
pkts bytes target prot opt in out source
destination
13 676 TPROXY tcp -- any any anywhere
anywhere tcp dpt:http TPROXY redirect 0.0.0.0:3128 mark
0x1/0x1
285 20188 DIVERT tcp -- any any anywhere
anywhere socket
Chain INPUT (policy ACCEPT 110K packets, 8097K bytes)
pkts bytes target prot opt in out source
destination
Chain FORWARD (policy ACCEPT 10466 packets, 553K bytes)
pkts bytes target prot opt in out source
destination
Chain OUTPUT (policy ACCEPT 144K packets, 28M bytes)
pkts bytes target prot opt in out source
destination
Chain POSTROUTING (policy ACCEPT 151K packets, 28M bytes)
pkts bytes target prot opt in out source
destination
Chain DIVERT (1 references)
pkts bytes target prot opt in out source
destination
285 20188 MARK tcp -- any any anywhere
anywhere MARK set 0x1
285 20188 ACCEPT all -- any any anywhere
anywhere
[root at cache com]# ip rule list
0: from all lookup local
32765: from all fwmark 0x1 lookup 100
32766: from all lookup main
32767: from all lookup default
[root at cache com]# ip route list table 100
local default dev lo scope host
[root at cache com]#
Squid Startup:
squid -f /etc/squid.conf -d 9 -N
2008/04/09 07:32:46| Starting Squid Cache version 3.HEAD-CVS for
i686-pc-linux-gnu...
2008/04/09 07:32:46| Process ID 11754
2008/04/09 07:32:46| With 1024 file descriptors available
2008/04/09 07:32:46| Performing DNS Tests...
2008/04/09 07:32:47| Successful DNS name lookup tests...
2008/04/09 07:32:47| DNS Socket created at 0.0.0.0, FD 5
2008/04/09 07:32:47| Adding nameserver 196.22.160.63 from squid.conf
2008/04/09 07:32:47| Unlinkd pipe opened on FD 10
2008/04/09 07:32:47| Store logging disabled
2008/04/09 07:32:47| Swap maxSize 512000000 KB, estimated 39384615
objects
2008/04/09 07:32:47| Target number of buckets: 1969230
2008/04/09 07:32:47| Using 2097152 Store buckets
2008/04/09 07:32:47| Max Mem size: 614400 KB
2008/04/09 07:32:47| Max Swap size: 512000000 KB
2008/04/09 07:32:47| Version 1 of swap file with LFS support detected...
2008/04/09 07:32:47| Rebuilding storage in /CACHE1 (CLEAN)
2008/04/09 07:32:47| Version 1 of swap file with LFS support detected...
2008/04/09 07:32:47| Rebuilding storage in /CACHE2 (CLEAN)
2008/04/09 07:32:47| Version 1 of swap file with LFS support detected...
2008/04/09 07:32:47| Rebuilding storage in /CACHE3 (CLEAN)
2008/04/09 07:32:47| Version 1 of swap file with LFS support detected...
2008/04/09 07:32:47| Rebuilding storage in /CACHE4 (CLEAN)
2008/04/09 07:32:47| Version 1 of swap file with LFS support detected...
2008/04/09 07:32:47| Rebuilding storage in /CACHE5 (CLEAN)
2008/04/09 07:32:47| Using Least Load store dir selection
2008/04/09 07:32:47| Set Current Directory to /usr/local/squid/var/cache
2008/04/09 07:32:47| Loaded Icons.
2008/04/09 07:32:47| Accepting transparently proxied HTTP connections at
0.0.0.0:3128, FD 21.
2008/04/09 07:32:47| Accepting ICP messages at 0.0.0.0:3130, FD 22.
2008/04/09 07:32:47| HTCP Disabled.
2008/04/09 07:32:47| Accepting WCCPv2 messages on port 2048, FD 23.
2008/04/09 07:32:47| Initialising all WCCPv2 lists
2008/04/09 07:32:47| ICMPSquid.cc(252) Open: Pinger socket opened on FD
25
2008/04/09 07:32:47| ICMPSquid.cc(125) SendEcho: Wrote 33 of 33 bytes
2008/04/09 07:32:47| Ready to serve requests.
2008/04/09 07:32:47| Store rebuilding is 8.08% complete
2008/04/09 07:32:49| Done reading /CACHE3 swaplog (48442 entries)
2008/04/09 07:32:49| Done reading /CACHE4 swaplog (48580 entries)
2008/04/09 07:32:49| Done reading /CACHE1 swaplog (50685 entries)
2008/04/09 07:32:49| Done reading /CACHE2 swaplog (56017 entries)
2008/04/09 07:32:49| Done reading /CACHE5 swaplog (59238 entries)
2008/04/09 07:32:49| Finished rebuilding storage from disk.
2008/04/09 07:32:49| 262962 Entries scanned
2008/04/09 07:32:49| 0 Invalid entries.
2008/04/09 07:32:49| 0 With invalid flags.
2008/04/09 07:32:49| 262962 Objects loaded.
2008/04/09 07:32:49| 0 Objects expired.
2008/04/09 07:32:49| 0 Objects cancelled.
2008/04/09 07:32:49| 0 Duplicate URLs purged.
2008/04/09 07:32:49| 0 Swapfile clashes avoided.
2008/04/09 07:32:49| Took 1.75 seconds (150574.61 objects/sec).
2008/04/09 07:32:49| Beginning Validation Procedure
2008/04/09 07:32:49| 262144 Entries Validated so far.
2008/04/09 07:32:49| Completed Validation Procedure
2008/04/09 07:32:49| Validated 525949 Entries
2008/04/09 07:32:49| store_swap_size = 9901896
2008/04/09 07:32:49| storeLateRelease: released 0 objects
WCCPV2 Setup
interface GigabitEthernet0/1.777
ip wccp 80 redirect out
ip wccp 90 redirect in
!
Gre Tunnel setup
ip tunnel add wccp mode gre remote 196.28.112.1 local 196.28.38.73 dev
eth0
ifconfig wccp 196.28.38.73 netmask 255.255.255.240 up
Squid config:
acl 196.22.160.0-19 src 196.22.160.0/19
acl TEST src 196.28.112.0/20
acl manager proto cache_object
acl localhost src 127.0.0.1/32
acl to_localhost dst 127.0.0.0/8
acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
http_access allow manager localhost
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localnet
http_access allow 196.22.160.0-19
http_access allow TEST
http_access deny all
icp_access allow localnet
icp_access deny all
htcp_access allow localnet
htcp_access deny all
http_port 3128 tproxy transparent
tcp_outgoing_address 196.28.38.73
hierarchy_stoplist cgi-bin ?
cache_mem 600 MB
maximum_object_size_in_memory 128 KB
cache_dir aufs /CACHE1 100000 16 256
cache_dir aufs /CACHE2 100000 16 256
cache_dir aufs /CACHE3 100000 16 256
cache_dir aufs /CACHE4 100000 16 256
cache_dir aufs /CACHE5 100000 16 256
minimum_object_size 0 KB
maximum_object_size 2 GB
cache_swap_low 70
cache_swap_high 80
access_log /var/log/squid/access.log squid
cache_log /var/log/squid/cache.log
cache_store_log none
emulate_httpd_log off
log_ip_on_direct on
debug_options ALL,1
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern (cgi-bin|\?) 0 0% 0
refresh_pattern . 0 20% 4320
read_ahead_gap 60 KB
positive_dns_ttl 24 hours
negative_dns_ttl 30 seconds
via off
ie_refresh on
connect_timeout 30 seconds
request_timeout 60 seconds
half_closed_clients off
shutdown_lifetime 5 seconds
visible_hostname cache.*.*
wccp2_router 196.22.185.38
wccp2_service dynamic 80 password=ci5co
wccp2_service_info 80 protocol=tcp flags=src_ip_hash priority=240
ports=80
wccp2_service dynamic 90 password=ci5co
wccp2_service_info 90 protocol=tcp flags=dst_ip_hash,ports_source
priority=240 ports=80
icp_port 3130
always_direct allow all
dns_nameservers 196.22.160.63
ipcache_size 51200
fqdncache_size 51200
memory_pools on
forwarded_for off
coredump_dir /usr/local/squid/var/cache
pipeline_prefetch on
With the default iptable rules above I don't see any packets on the
destination machine.
Ive then started playing around with the settings and add the rules
below.
iptables -t mangle -A PREROUTING -p tcp --dport 80 -j LOG --log-level 6
--log-prefix "TPROXY : "
iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY --on-port
3128 --tproxy-mark 0x1/0x1
iptables -t mangle -A PREROUTING -p tcp --sport 80 -j LOG --log-level 6
--log-prefix "TPROXY_SPORT : "
iptables -t mangle -A PREROUTING -p tcp --sport 80 -j TPROXY --on-port
3128 --tproxy-mark 0x2/0x2
iptables -t mangle -A POSTROUTING -p tcp --dport 80 -j LOG --log-level 6
--log-prefix "MANGLE_POSTROUTING1 : "
iptables -t mangle -A POSTROUTING -p tcp --dport 80 -j ACCEPT
These will log to syslog.
This is the output I'm seeing on the squid server.
Apr 9 08:14:13 cache kernel: MANGLE_PREROUTING : IN=wccp OUT=
MAC=45:00:00:58:00:4a:00:00:fd:2f:9e:a9:c4:1c:70:01:c4:1c:26:49:00:00:88
:3e:01:50:00:b1:45:00:00:3c:07:ce:40:00:3f:06 SRC=196.28.113.24
DST=196.22.160.68 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=1998 DF PROTO=TCP
SPT=53604 DPT=80 WINDOW=5840 RES=0x00 SYN URGP=0
Apr 9 08:14:13 cache kernel: MANGLE_POSTROUTING1 : IN= OUT=eth0
SRC=196.28.113.24 DST=196.22.160.68 LEN=60 TOS=0x00 PREC=0x00 TTL=62
ID=1998 DF PROTO=TCP SPT=53604 DPT=80 WINDOW=5840 RES=0x00 SYN URGP=0
Apr 9 08:14:13 cache kernel: TPROXY_SPORT : IN=wccp OUT=
MAC=45:00:00:50:07:0b:00:00:fd:2f:97:f0:c4:1c:70:01:c4:1c:26:49:00:00:88
:3e:01:5a:00:b1:45:00:00:34:00:00:40:00:3e:06 SRC=196.22.160.68
DST=196.28.113.24 LEN=52 TOS=0x00 PREC=0x00 TTL=62 ID=0 DF PROTO=TCP
SPT=80 DPT=53604 WINDOW=5840 RES=0x00 ACK SYN URGP=0
Apr 9 08:14:13 cache kernel: MANGLE_POSTROUTING1 : IN= OUT=eth0
SRC=196.28.113.24 DST=196.22.160.68 LEN=40 TOS=0x00 PREC=0x00 TTL=64
ID=0 DF PROTO=TCP SPT=53604 DPT=80 WINDOW=0 RES=0x00 RST URGP=0
This is a tcpdump on the destination machine
08:15:30.278021 196.28.113.24.53604 > 196.22.160.68.80: S
3782047292:3782047292(0) win 5840 <mss 1452,sackOK,timestamp 817000946
0,nop,wscale 4> (DF)
08:15:30.278079 196.22.160.68.80 > 196.28.113.24.53604: S
718132722:718132722(0) ack 3782047293 win 5840 <mss
1460,nop,nop,sackOK,nop,wscale 0> (DF)
08:15:30.278775 196.28.113.24.53604 > 196.22.160.68.80: R
3782047293:3782047293(0) win 0 (DF)
My squid servers ip address is 196.28.38.73.
Client ip is 196.28.113.24
Destination ip is 196.22.160.68.
Can somebody please tell me if I'm right with the following statements.
Packet comes in on interface wccp gre tunnel from router. With the src
and dst address.
Apr 9 08:14:13 cache kernel: MANGLE_PREROUTING : IN=wccp OUT=
MAC=45:00:00:58:00:4a:00:00:fd:2f:9e:a9:c4:1c:70:01:c4:1c:26:49:00:00:88
:3e:01:50:00:b1:45:00:00:3c:07:ce:40:00:3f:06 SRC=196.28.113.24
DST=196.22.160.68 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=1998 DF PROTO=TCP
SPT=53604 DPT=80 WINDOW=5840 RES=0x00 SYN URGP=0
Packet leaves on interface eth0 with the clients ip as source address.
Apr 9 08:14:13 cache kernel: MANGLE_PREROUTING : IN=wccp OUT=
MAC=45:00:00:58:00:4a:00:00:fd:2f:9e:a9:c4:1c:70:01:c4:1c:26:49:00:00:88
:3e:01:50:00:b1:45:00:00:3c:07:ce:40:00:3f:06 SRC=196.28.113.24
DST=196.22.160.68 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=1998 DF PROTO=TCP
SPT=53604 DPT=80 WINDOW=5840 RES=0x00 SYN URGP=0
Packet comes back via the wccp interface. This time with the destination
pc as src and the clients ip as destination.
Apr 9 08:14:13 cache kernel: TPROXY_SPORT : IN=wccp OUT=
MAC=45:00:00:50:07:0b:00:00:fd:2f:97:f0:c4:1c:70:01:c4:1c:26:49:00:00:88
:3e:01:5a:00:b1:45:00:00:34:00:00:40:00:3e:06 SRC=196.22.160.68
DST=196.28.113.24 LEN=52 TOS=0x00 PREC=0x00 TTL=62 ID=0 DF PROTO=TCP
SPT=80 DPT=53604 WINDOW=5840 RES=0x00 ACK SYN URGP=0
I'm pretty sure this is where everything goes dead as the request is
sent to the destination server again.
If there is anything that I missed or can try to resolve it please let
me know.
In the mean time I will continue to play with the settings.
Keep up the good work.
Thanks
Wickus
More information about the tproxy
mailing list