System Layout: Cisco 7200 with wccpv2 Dell 2950 with 1 ethernet interface Kernel: Linux cache 2.6.25-rc6 #1 SMP Tue Apr 8 17:19:10 SAST 2008 i686 i686 i386 GNU/Linux Patch tproxy-kernel-2.6.25-rc6.20080402-130957-1207134597.patch had to fix a few rej. Squid: Squid Cache version 3.HEAD-CVS from cvs Patch http://www.balabit.com/downloads/files/tproxy/tproxy-squid-3_20080408.pa tch Few rej but they were resolved. Configs: echo 1 > /proc/sys/net/ipv4/ip_forward echo 1 > /proc/sys/net/ipv4/ip_nonlocal_bind echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY --on-port 3128 --tproxy-mark 0x1/0x1 iptables -t mangle -N DIVERT iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT iptables -t mangle -A DIVERT -p tcp -j MARK --set-mark 1 iptables -t mangle -A DIVERT -j ACCEPT ip rule add fwmark 1 lookup 100 ip route add local 0.0.0.0/0 dev lo table 100 [root@cache com]# iptables -L -v -t mangle Chain PREROUTING (policy ACCEPT 34018 packets, 3155K bytes) pkts bytes target prot opt in out source destination 13 676 TPROXY tcp -- any any anywhere anywhere tcp dpt:http TPROXY redirect 0.0.0.0:3128 mark 0x1/0x1 285 20188 DIVERT tcp -- any any anywhere anywhere socket Chain INPUT (policy ACCEPT 110K packets, 8097K bytes) pkts bytes target prot opt in out source destination Chain FORWARD (policy ACCEPT 10466 packets, 553K bytes) pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 144K packets, 28M bytes) pkts bytes target prot opt in out source destination Chain POSTROUTING (policy ACCEPT 151K packets, 28M bytes) pkts bytes target prot opt in out source destination Chain DIVERT (1 references) pkts bytes target prot opt in out source destination 285 20188 MARK tcp -- any any anywhere anywhere MARK set 0x1 285 20188 ACCEPT all -- any any anywhere anywhere [root@cache com]# ip rule list 0: from all lookup local 32765: from all fwmark 0x1 lookup 100 32766: from all lookup main 32767: from all lookup default [root@cache com]# ip route list table 100 local default dev lo scope host [root@cache com]# Squid Startup: squid -f /etc/squid.conf -d 9 -N 2008/04/09 07:32:46| Starting Squid Cache version 3.HEAD-CVS for i686-pc-linux-gnu... 2008/04/09 07:32:46| Process ID 11754 2008/04/09 07:32:46| With 1024 file descriptors available 2008/04/09 07:32:46| Performing DNS Tests... 2008/04/09 07:32:47| Successful DNS name lookup tests... 2008/04/09 07:32:47| DNS Socket created at 0.0.0.0, FD 5 2008/04/09 07:32:47| Adding nameserver 196.22.160.63 from squid.conf 2008/04/09 07:32:47| Unlinkd pipe opened on FD 10 2008/04/09 07:32:47| Store logging disabled 2008/04/09 07:32:47| Swap maxSize 512000000 KB, estimated 39384615 objects 2008/04/09 07:32:47| Target number of buckets: 1969230 2008/04/09 07:32:47| Using 2097152 Store buckets 2008/04/09 07:32:47| Max Mem size: 614400 KB 2008/04/09 07:32:47| Max Swap size: 512000000 KB 2008/04/09 07:32:47| Version 1 of swap file with LFS support detected... 2008/04/09 07:32:47| Rebuilding storage in /CACHE1 (CLEAN) 2008/04/09 07:32:47| Version 1 of swap file with LFS support detected... 2008/04/09 07:32:47| Rebuilding storage in /CACHE2 (CLEAN) 2008/04/09 07:32:47| Version 1 of swap file with LFS support detected... 2008/04/09 07:32:47| Rebuilding storage in /CACHE3 (CLEAN) 2008/04/09 07:32:47| Version 1 of swap file with LFS support detected... 2008/04/09 07:32:47| Rebuilding storage in /CACHE4 (CLEAN) 2008/04/09 07:32:47| Version 1 of swap file with LFS support detected... 2008/04/09 07:32:47| Rebuilding storage in /CACHE5 (CLEAN) 2008/04/09 07:32:47| Using Least Load store dir selection 2008/04/09 07:32:47| Set Current Directory to /usr/local/squid/var/cache 2008/04/09 07:32:47| Loaded Icons. 2008/04/09 07:32:47| Accepting transparently proxied HTTP connections at 0.0.0.0:3128, FD 21. 2008/04/09 07:32:47| Accepting ICP messages at 0.0.0.0:3130, FD 22. 2008/04/09 07:32:47| HTCP Disabled. 2008/04/09 07:32:47| Accepting WCCPv2 messages on port 2048, FD 23. 2008/04/09 07:32:47| Initialising all WCCPv2 lists 2008/04/09 07:32:47| ICMPSquid.cc(252) Open: Pinger socket opened on FD 25 2008/04/09 07:32:47| ICMPSquid.cc(125) SendEcho: Wrote 33 of 33 bytes 2008/04/09 07:32:47| Ready to serve requests. 2008/04/09 07:32:47| Store rebuilding is 8.08% complete 2008/04/09 07:32:49| Done reading /CACHE3 swaplog (48442 entries) 2008/04/09 07:32:49| Done reading /CACHE4 swaplog (48580 entries) 2008/04/09 07:32:49| Done reading /CACHE1 swaplog (50685 entries) 2008/04/09 07:32:49| Done reading /CACHE2 swaplog (56017 entries) 2008/04/09 07:32:49| Done reading /CACHE5 swaplog (59238 entries) 2008/04/09 07:32:49| Finished rebuilding storage from disk. 2008/04/09 07:32:49| 262962 Entries scanned 2008/04/09 07:32:49| 0 Invalid entries. 2008/04/09 07:32:49| 0 With invalid flags. 2008/04/09 07:32:49| 262962 Objects loaded. 2008/04/09 07:32:49| 0 Objects expired. 2008/04/09 07:32:49| 0 Objects cancelled. 2008/04/09 07:32:49| 0 Duplicate URLs purged. 2008/04/09 07:32:49| 0 Swapfile clashes avoided. 2008/04/09 07:32:49| Took 1.75 seconds (150574.61 objects/sec). 2008/04/09 07:32:49| Beginning Validation Procedure 2008/04/09 07:32:49| 262144 Entries Validated so far. 2008/04/09 07:32:49| Completed Validation Procedure 2008/04/09 07:32:49| Validated 525949 Entries 2008/04/09 07:32:49| store_swap_size = 9901896 2008/04/09 07:32:49| storeLateRelease: released 0 objects WCCPV2 Setup interface GigabitEthernet0/1.777 ip wccp 80 redirect out ip wccp 90 redirect in ! Gre Tunnel setup ip tunnel add wccp mode gre remote 196.28.112.1 local 196.28.38.73 dev eth0 ifconfig wccp 196.28.38.73 netmask 255.255.255.240 up Squid config: acl 196.22.160.0-19 src 196.22.160.0/19 acl TEST src 196.28.112.0/20 acl manager proto cache_object acl localhost src 127.0.0.1/32 acl to_localhost dst 127.0.0.0/8 acl localnet src 10.0.0.0/8 # RFC1918 possible internal network acl localnet src 172.16.0.0/12 # RFC1918 possible internal network acl localnet src 192.168.0.0/16 # RFC1918 possible internal network acl SSL_ports port 443 acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl CONNECT method CONNECT http_access allow manager localhost http_access deny manager http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access allow localnet http_access allow 196.22.160.0-19 http_access allow TEST http_access deny all icp_access allow localnet icp_access deny all htcp_access allow localnet htcp_access deny all http_port 3128 tproxy transparent tcp_outgoing_address 196.28.38.73 hierarchy_stoplist cgi-bin ? cache_mem 600 MB maximum_object_size_in_memory 128 KB cache_dir aufs /CACHE1 100000 16 256 cache_dir aufs /CACHE2 100000 16 256 cache_dir aufs /CACHE3 100000 16 256 cache_dir aufs /CACHE4 100000 16 256 cache_dir aufs /CACHE5 100000 16 256 minimum_object_size 0 KB maximum_object_size 2 GB cache_swap_low 70 cache_swap_high 80 access_log /var/log/squid/access.log squid cache_log /var/log/squid/cache.log cache_store_log none emulate_httpd_log off log_ip_on_direct on debug_options ALL,1 refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern (cgi-bin|\?) 0 0% 0 refresh_pattern . 0 20% 4320 read_ahead_gap 60 KB positive_dns_ttl 24 hours negative_dns_ttl 30 seconds via off ie_refresh on connect_timeout 30 seconds request_timeout 60 seconds half_closed_clients off shutdown_lifetime 5 seconds visible_hostname cache.*.* wccp2_router 196.22.185.38 wccp2_service dynamic 80 password=ci5co wccp2_service_info 80 protocol=tcp flags=src_ip_hash priority=240 ports=80 wccp2_service dynamic 90 password=ci5co wccp2_service_info 90 protocol=tcp flags=dst_ip_hash,ports_source priority=240 ports=80 icp_port 3130 always_direct allow all dns_nameservers 196.22.160.63 ipcache_size 51200 fqdncache_size 51200 memory_pools on forwarded_for off coredump_dir /usr/local/squid/var/cache pipeline_prefetch on With the default iptable rules above I don't see any packets on the destination machine. Ive then started playing around with the settings and add the rules below. iptables -t mangle -A PREROUTING -p tcp --dport 80 -j LOG --log-level 6 --log-prefix "TPROXY : " iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY --on-port 3128 --tproxy-mark 0x1/0x1 iptables -t mangle -A PREROUTING -p tcp --sport 80 -j LOG --log-level 6 --log-prefix "TPROXY_SPORT : " iptables -t mangle -A PREROUTING -p tcp --sport 80 -j TPROXY --on-port 3128 --tproxy-mark 0x2/0x2 iptables -t mangle -A POSTROUTING -p tcp --dport 80 -j LOG --log-level 6 --log-prefix "MANGLE_POSTROUTING1 : " iptables -t mangle -A POSTROUTING -p tcp --dport 80 -j ACCEPT These will log to syslog. This is the output I'm seeing on the squid server. Apr 9 08:14:13 cache kernel: MANGLE_PREROUTING : IN=wccp OUT= MAC=45:00:00:58:00:4a:00:00:fd:2f:9e:a9:c4:1c:70:01:c4:1c:26:49:00:00:88 :3e:01:50:00:b1:45:00:00:3c:07:ce:40:00:3f:06 SRC=196.28.113.24 DST=196.22.160.68 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=1998 DF PROTO=TCP SPT=53604 DPT=80 WINDOW=5840 RES=0x00 SYN URGP=0 Apr 9 08:14:13 cache kernel: MANGLE_POSTROUTING1 : IN= OUT=eth0 SRC=196.28.113.24 DST=196.22.160.68 LEN=60 TOS=0x00 PREC=0x00 TTL=62 ID=1998 DF PROTO=TCP SPT=53604 DPT=80 WINDOW=5840 RES=0x00 SYN URGP=0 Apr 9 08:14:13 cache kernel: TPROXY_SPORT : IN=wccp OUT= MAC=45:00:00:50:07:0b:00:00:fd:2f:97:f0:c4:1c:70:01:c4:1c:26:49:00:00:88 :3e:01:5a:00:b1:45:00:00:34:00:00:40:00:3e:06 SRC=196.22.160.68 DST=196.28.113.24 LEN=52 TOS=0x00 PREC=0x00 TTL=62 ID=0 DF PROTO=TCP SPT=80 DPT=53604 WINDOW=5840 RES=0x00 ACK SYN URGP=0 Apr 9 08:14:13 cache kernel: MANGLE_POSTROUTING1 : IN= OUT=eth0 SRC=196.28.113.24 DST=196.22.160.68 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=53604 DPT=80 WINDOW=0 RES=0x00 RST URGP=0 This is a tcpdump on the destination machine 08:15:30.278021 196.28.113.24.53604 > 196.22.160.68.80: S 3782047292:3782047292(0) win 5840 <mss 1452,sackOK,timestamp 817000946 0,nop,wscale 4> (DF) 08:15:30.278079 196.22.160.68.80 > 196.28.113.24.53604: S 718132722:718132722(0) ack 3782047293 win 5840 <mss 1460,nop,nop,sackOK,nop,wscale 0> (DF) 08:15:30.278775 196.28.113.24.53604 > 196.22.160.68.80: R 3782047293:3782047293(0) win 0 (DF) My squid servers ip address is 196.28.38.73. Client ip is 196.28.113.24 Destination ip is 196.22.160.68. Can somebody please tell me if I'm right with the following statements. Packet comes in on interface wccp gre tunnel from router. With the src and dst address. Apr 9 08:14:13 cache kernel: MANGLE_PREROUTING : IN=wccp OUT= MAC=45:00:00:58:00:4a:00:00:fd:2f:9e:a9:c4:1c:70:01:c4:1c:26:49:00:00:88 :3e:01:50:00:b1:45:00:00:3c:07:ce:40:00:3f:06 SRC=196.28.113.24 DST=196.22.160.68 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=1998 DF PROTO=TCP SPT=53604 DPT=80 WINDOW=5840 RES=0x00 SYN URGP=0 Packet leaves on interface eth0 with the clients ip as source address. Apr 9 08:14:13 cache kernel: MANGLE_PREROUTING : IN=wccp OUT= MAC=45:00:00:58:00:4a:00:00:fd:2f:9e:a9:c4:1c:70:01:c4:1c:26:49:00:00:88 :3e:01:50:00:b1:45:00:00:3c:07:ce:40:00:3f:06 SRC=196.28.113.24 DST=196.22.160.68 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=1998 DF PROTO=TCP SPT=53604 DPT=80 WINDOW=5840 RES=0x00 SYN URGP=0 Packet comes back via the wccp interface. This time with the destination pc as src and the clients ip as destination. Apr 9 08:14:13 cache kernel: TPROXY_SPORT : IN=wccp OUT= MAC=45:00:00:50:07:0b:00:00:fd:2f:97:f0:c4:1c:70:01:c4:1c:26:49:00:00:88 :3e:01:5a:00:b1:45:00:00:34:00:00:40:00:3e:06 SRC=196.22.160.68 DST=196.28.113.24 LEN=52 TOS=0x00 PREC=0x00 TTL=62 ID=0 DF PROTO=TCP SPT=80 DPT=53604 WINDOW=5840 RES=0x00 ACK SYN URGP=0 I'm pretty sure this is where everything goes dead as the request is sent to the destination server again. If there is anything that I missed or can try to resolve it please let me know. In the mean time I will continue to play with the settings. Keep up the good work. Thanks Wickus