April 2010 - tproxy - lists.balabit.hu

help me!plz!
by 王琳 16 Apr '10

16 Apr '10

Hello all I try to implement haproxy in transparent mode like in http://blog.loadbalancer.org/configure-haproxy-with-tproxy-kernel-for-full-… I add rules in iptables, but when I try to access to the service, the result is “503 Service Unavailable”. Without the line “source 0.0.0.0 usesrc clientip” all works correctly, obviously without transparency. Have you got any ideas? Where I’m wrong? Thanks!

1 0

EADDRNOTAVAIL from connect, but only sometimes
by Ron Parker 15 Apr '10

15 Apr '10

Hi, We are using the tproxy patch for Linux 2.6.24 (Ubuntu 8.0.4). When placing outgoing connections, we use the original socket address (4-tuple) in the bind and set SO_REUSEADDR on the socket. The sequence we are having difficulty with is: * Client connects to transparent proxy * Transparent proxy connects to remote server * Normal data transfer... * Remote server closes the connection (but client connection is maintained) * Transparent proxy attempts to connect again to remote server using the original 4-tuple (again) o Bind succeeds o Connect fails with EADDRNOTAVAIL The original socket is probably in TIME_WAIT at this point. I thought the SO_REUSEADDR would take care of the problem. What am I missing here? Thanks. Ron

3 4

Problems with the Tproxy and Zorp.
by Lupi The Loop 15 Apr '10

15 Apr '10

Hello, I have configured a transparent HTTP proxy using Iptables and Zorp but It does not work. *SYSTEM INFO:* *loop@santeles:~$ uname -a Linux santeles 2.6.31-19-server #56-Ubuntu SMP Thu Jan 28 02:39:34 UTC 2010 x86_64 GNU/Linux loop@santeles:~$ iptables -V iptables v1.4.4 root@santeles:/home/loop# zorpctl version Zorp 3.0.8 Revision: Compile-Date: May 4 2009 04:17:42 Config-Date: 2009/05/04 Trace: off Debug: off IPOptions: off IPFilter-Tproxy: off Netfilter-Tproxy: on Netfilter-Linux22-Fallback: on Linux22-Tproxy: off Conntrack: on Zorplib 3.0.6.4.2 Revision: devel(a)balabit.hu--zorp-1/zorp-lib--mainline--3.0--patch-116 Compile-Date: Nov 9 2009 09:50:26 Trace: off MemTrace: off Caps: on Debug: off StackDump: off **SYSTEM CONFIG: root@santeles:/home/loop# ifconfig -a** dummy0 Link encap:Ethernet HWaddr 00:21:9b:ee:61:14 inet addr:1.2.3.4 Bcast:1.255.255.255 Mask:255.255.255.255 inet6 addr: fe80::24c4:26ff:fec7:914/64 Scope:Link UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:3 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:210 (210.0 KB) wlan0 Link encap:Ethernet HWaddr 00:1f:3b:6d:30:9b inet addr:10.1.1.2 Bcast:10.1.1.255 Mask:255.255.255.0 inet6 addr: fe80::226:18ff:fef2:31bc/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:14 errors:0 dropped:0 overruns:0 frame:0 TX packets:2910 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:900 (900.0 B) TX bytes:1692 (1.6 KB) Interrupt:27 Base address:0x8000 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:4 errors:0 dropped:0 overruns:0 frame:0 TX packets:4 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:368 (368.0 B) TX bytes:368 (368.0 B) root@santeles:/home/loop# cat /etc/iproute2/rt_tables # # reserved values # 255 local 254 main 253 default 200 proxy 0 unspec # # local # #1 inr.ruhep root@santeles:/home/loop# ip rule list 0: from all lookup local 32765: from all fwmark 0x01 lookup proxy 32766: from all lookup main 32767: from all lookup default root@santeles:/home/loop# ip route show table proxy local default dev dummy0 scope host * * root@santeles:/home/loop# cat /etc/zorp/instances.conf secret -v10 -p /etc/zorp/policy.py --autobind-ip 1.2.3.4 --tproxy netfilter root@DPP3-GREC:/home/evalues# cat /etc/zorp/policy.py from Zorp.Core import * from Zorp.Plug import * from Zorp.Http import * Zorp.firewall_name = 'DPP3-GREC' InetZone("secret-net", "0.0.0.0/0", outbound_services=["*"], inbound_services=["*"]) def secret(): Service("serv", HttpProxy ) Listener(SockAddrInet("1.2.3.4",50080), "serv") * * iptables rules* * iptables -t mangle -P PREROUTING ACCEPT iptables -t mangle -N DIVERT iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT iptables -t mangle -A DIVERT -j MARK --set-mark 1 iptables -t mangle -A DIVERT -j ACCEPT iptables -t mangle -A PREROUTING -p tcp --dport 80 -j LOG --log-prefix "Passing request to proxy" --log-level debug iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY --on-ip 1.2.3.4 --tproxy-mark 1 --on-port 50080 iptables -t mangle -P OUTPUT ACCEPT iptables -P INPUT ACCEPT iptables -P FORWARD ACCEPT iptables -P OUTPUT ACCEPT * *IPTABLES IS LOADED CORRECTLY * *root@DPP3-GREC:/home/evalues# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination LOG all -- anywhere anywhere LOG level debug prefix `Input' Chain FORWARD (policy ACCEPT) target prot opt source destination LOG all -- anywhere anywhere LOG level debug prefix `Forward' Chain OUTPUT (policy ACCEPT) target prot opt source destination LOG all -- anywhere anywhere LOG level debug prefix `Output' root@DPP3-GREC:/home/evalues# iptables -t mangle -L Chain PREROUTING (policy ACCEPT) target prot opt source destination DIVERT tcp -- anywhere anywhere socket LOG tcp -- anywhere anywhere tcp dpt:www LOG level debug prefix `Passing request to proxy' TPROXY tcp -- anywhere anywhere tcp dpt:www TPROXY redirect 1.2.3.4:50080 mark 0x1/0xffffffff Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain POSTROUTING (policy ACCEPT) target prot opt source destination Chain DIVERT (1 references) target prot opt source destination MARK all -- anywhere anywhere MARK xset 0x1/0xffffffff ACCEPT all -- anywhere anywhere **ZORP STARTS WITHOUT PROBLEM* *root@DPP3-GREC:/home/evalues# zorpctl start Starting Zorp Firewall Suite: secret root@DPP3-GREC:/home/evalues# netstat -a -p | grep zorp tcp 0 0 1.2.3.4:50080 *:* LISTEN 1700/zorp unix 2 [ ACC ] STREAM LISTENING 8096 1700/zorp /var/run/zorp/zorpctl.secret unix 2 [ ] DGRAM 8094 1700/zorp unix 2 [ ] DGRAM 8090 1699/zorpctl superv root@DPP3-GREC:/home/evalues# tail -n 18 /var/log/syslog Mar 1 17:10:42 DPP3-GREC secret[1734]: (noname/nosession): Starting up; verbose_level='10', version='3.0.8' Mar 1 17:10:42 DPP3-GREC secret[1734]: (noname/nosession): System dependant init; sysdep_tproxy='tproxy12' Mar 1 17:10:42 DPP3-GREC secret[1734]: (szig): thread starting; Mar 1 17:10:42 DPP3-GREC secret[1734]: (szig): Start to listen; fd='8', address='AF_UNIX(/var/run/zorp/zorpctl.secret)' Mar 1 17:10:42 DPP3-GREC secret[1734]: (Log thread): thread starting; Mar 1 17:10:42 DPP3-GREC secret[1734]: (conntrack/thread): thread starting; Mar 1 17:10:42 DPP3-GREC secret[1734]: (Log thread): /usr/lib/python2.4/whrandom.py:38: DeprecationWarning: the whrandom module is deprecated; please use the random module#012 Mar 1 17:10:42 DPP3-GREC secret[1734]: (Log thread): DeprecationWarning)#012 Mar 1 17:10:42 DPP3-GREC secret[1734]: (noname/nosession): Outbound service; zone='secret-net', service='*' Mar 1 17:10:42 DPP3-GREC secret[1734]: (noname/nosession): Inbound service; zone='secret-net', service='*' Mar 1 17:10:42 DPP3-GREC secret[1734]: (secret@DPP3-GREC): Dispatcher on address; proto='1', local='AF_INET(1.2.3.4:50080)', prio='100' Mar 1 17:10:42 DPP3-GREC secret[1734]: (secret@DPP3-GREC): Start to listen; fd='14', address='AF_INET(1.2.3.4:50080)' Mar 1 17:10:42 DPP3-GREC secret[1734]: (secret@DPP3-GREC/nosession): read blob systems default attributes; tmpdir='/var/lib/zorp/tmp/', max_disk_usage='1073741824', max_mem_usage='268435456', lowat='100663296', hiwat='134217728', noswap_max='16384' Mar 1 17:10:42 DPP3-GREC secret[1734]: (secret@DPP3-GREC/nosession): creating blob management thread; Mar 1 17:10:42 DPP3-GREC secret[1734]: (secret@DPP3-GREC/nosession): blob management thread starting; Mar 1 17:10:42 DPP3-GREC secret[1734]: (secret@DPP3-GREC/nosession): blob management thread signalling back to constructor; Mar 1 17:10:42 DPP3-GREC secret[1734]: (secret@DPP3-GREC/nosession): waiting for the queue; Mar 1 17:10:42 DPP3-GREC secret[1734]: (secret@DPP3-GREC/nosession): blob management thread up and running; * *HOWEVER, IPTABLES FORWARDS THE HTTP PACKETs TO THE PROXY, BUT THE PROXY DOES NOT RECEIVE ANYTHING* *root@DPP3-GREC:/home/evalues# tail -n 10 /var/log/syslog Mar 1 17:10:42 DPP3-GREC secret[1734]: (secret@DPP3-GREC): Start to listen; fd='14', address='AF_INET(1.2.3.4:50080)' Mar 1 17:10:42 DPP3-GREC secret[1734]: (secret@DPP3-GREC/nosession): read blob systems default attributes; tmpdir='/var/lib/zorp/tmp/', max_disk_usage='1073741824', max_mem_usage='268435456', lowat='100663296', hiwat='134217728', noswap_max='16384' Mar 1 17:10:42 DPP3-GREC secret[1734]: (secret@DPP3-GREC/nosession): creating blob management thread; Mar 1 17:10:42 DPP3-GREC secret[1734]: (secret@DPP3-GREC/nosession): blob management thread starting; Mar 1 17:10:42 DPP3-GREC secret[1734]: (secret@DPP3-GREC/nosession): blob management thread signalling back to constructor; Mar 1 17:10:42 DPP3-GREC secret[1734]: (secret@DPP3-GREC/nosession): waiting for the queue; Mar 1 17:10:42 DPP3-GREC secret[1734]: (secret@DPP3-GREC/nosession): blob management thread up and running; Mar 1 17:14:08 DPP3-GREC kernel: [ 5253.789761] Passing request to proxyIN=eth0 OUT= MAC=00:26:18:f2:31:bc:00:60:97:ba:c0:53:08:00 SRC=10.0.1.2 DST=10.0.2.5 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=55686 DF PROTO=TCP SPT=44711 DPT=80 WINDOW=5840 RES=0x00 SYN URGP=0 Mar 1 17:14:11 DPP3-GREC kernel: [ 5256.789037] Passing request to proxyIN=eth0 OUT= MAC=00:26:18:f2:31:bc:00:60:97:ba:c0:53:08:00 SRC=10.0.1.2 DST=10.0.2.5 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=55687 DF PROTO=TCP SPT=44711 DPT=80 WINDOW=5840 RES=0x00 SYN URGP=0 Mar 1 17:14:17 DPP3-GREC kernel: [ 5262.786612] Passing request to proxyIN=eth0 OUT= MAC=00:26:18:f2:31:bc:00:60:97:ba:c0:53:08:00 SRC=10.0.1.2 DST=10.0.2.5 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=55688 DF PROTO=TCP SPT=44711 DPT=80 WINDOW=5840 RES=0x00 SYN URGP=0 * Any idea of what can be happening? Thanks in advance

2 2

transparent smtp
by Paulo Roncon 15 Apr '10

15 Apr '10

Hello, can I use the TPROXY to use as a transparent smtp proxy? I want to use as: serv1 --------------->(serX)--------->serv2 where: -servX receives the smtp traffic from serv1 and relay it to serv2 -serv2 receives the header information (includind IP information) as being sent from serv1 -servX is not known to serv1 nor serv2 As of now I tried to follow the posts of the TPROXY home no no avail. Tried the PROXSMTP and managed to be transparent at the SMTP level, but the servX IP was visible... Please help, thanks! Paulo

2 1

TProxy and REUSEADDR
by nicolas normand 15 Apr '10

15 Apr '10

Hi It is possible for a proxy to transparently and simultaneously connect to several backends (with different IP) with the same client IP/Port ? I have tried to SO_REUSEADDR before binding but I have a strange behavior: * the concerned program is a tproxy apache patch, which works perfectly for simple and single request, but doesn't handle subrequests (and probably some other twisted cases). * the patch is based on this incomplete one: http://miscfiles.googlecode.com/svn/trunk/tproxy.patch , which I just ported it to apache 2.2.14, made the listening socket transparent and made the options to work. You can found it included this mail. * Code in proxy/module/proxy_utils.c now look likes: if (conf->tproxy) { if ((rv = apr_socket_opt_set(newsock, APR_IP_TRANSPARENT, 1)) != APR_SUCCESS) { ap_log_error(APLOG_MARK, APLOG_ERR, rv, s, "apr_socket_opt_set(APR_IP_TRANSPARENT): Failed to set"); } if ((rv = apr_socket_opt_set(newsock, APR_SO_REUSEADDR, 1)) != APR_SUCCESS) { ap_log_error(APLOG_MARK, APLOG_ERR, rv, s, "apr_socket_opt_set(APR_SO_REUSE_ADDR): Failed to set"); } if ((rv = apr_socket_bind(newsock, remote_addr)) != APR_SUCCESS) { ap_log_error(APLOG_MARK, APLOG_ERR, rv, s, "proxy: error binding to socket %pI", remote_addr); return HTTP_INTERNAL_SERVER_ERROR; } } When connecting the second backend (which has a different IP address), I got this (192.168.100.12 is the CLIENT, 192.168.100.127 is the PROXY, 192.168.100.238 and 192.168.100.229 are the BACKENDS): // Connecting to first backend, all is okay 15:49:32.037788 IP 192.168.100.12.57009 > 192.168.100.238.80: S 1617296302:1617296302(0) win 5840 <mss 1460,sackOK,timestamp 21314512 0,nop,wscale 6> 15:49:32.037827 IP 192.168.100.238.80 > 192.168.100.12.57009: S 1072906513:1072906513(0) ack 1617296303 win 5792 <mss 1460,sackOK,timestamp 107453675 21314512,nop,wscale 4> 15:49:32.038268 IP 192.168.100.12.57009 > 192.168.100.238.80: . ack 1 win 92 <nop,nop,timestamp 21314512 107453675> 15:49:32.038855 IP 192.168.100.12.57009 > 192.168.100.238.80: P 1:386(385) ack 1 win 92 <nop,nop,timestamp 21314512 107453675> 15:49:32.038877 IP 192.168.100.238.80 > 192.168.100.12.57009: . ack 386 win 429 <nop,nop,timestamp 107453676 21314512> 15:49:32.040221 IP 192.168.100.238.80 > 192.168.100.12.57009: . 1:1449(1448) ack 386 win 429 <nop,nop,timestamp 107453676 21314512> 15:49:32.040377 IP 192.168.100.238.80 > 192.168.100.12.57009: P 1449:1795(346) ack 386 win 429 <nop,nop,timestamp 107453676 21314512> 15:49:32.040485 IP 192.168.100.12.57009 > 192.168.100.238.80: . ack 1449 win 137 <nop,nop,timestamp 21314512 107453676> 15:49:32.040608 IP 192.168.100.12.57009 > 192.168.100.238.80: . ack 1795 win 182 <nop,nop,timestamp 21314513 107453676> // And now try connecting the second backend... Hell, the client sent the Reset flag... 15:49:32.067711 IP 192.168.100.12.57009 > 192.168.100.229.80: S 1618236269:1618236269(0) win 5840 <mss 1460,sackOK,timestamp 21314519 0,nop,wscale 6> 15:49:32.067861 IP 192.168.100.229.80 > 192.168.100.12.57009: S 840737942:840737942(0) ack 1618236270 win 5792 <mss 1460,sackOK,timestamp 107448908 21314519,nop,wscale 4> 15:49:32.068837 IP 192.168.100.12.57009 > 192.168.100.229.80: R 1618236270:1618236270(0) win 0 15:49:35.069466 IP 192.168.100.12.57009 > 192.168.100.229.80: S 1618236269:1618236269(0) win 5840 <mss 1460,sackOK,timestamp 21315270 0,nop,wscale 6> 15:49:35.069483 IP 192.168.100.229.80 > 192.168.100.12.57009: S 887635567:887635567(0) ack 1618236270 win 5792 <mss 1460,sackOK,timestamp 107449658 21315270,nop,wscale 4> 15:49:35.072257 IP 192.168.100.12.57009 > 192.168.100.229.80: R 1618236270:1618236270(0) win 0 15:49:41.070224 IP 192.168.100.12.57009 > 192.168.100.229.80: S 1618236269:1618236269(0) win 5840 <mss 1460,sackOK,timestamp 21316770 0,nop,wscale 6> 15:49:41.070363 IP 192.168.100.229.80 > 192.168.100.12.57009: S 981396815:981396815(0) ack 1618236270 win 5792 <mss 1460,sackOK,timestamp 107451158 21316770,nop,wscale 4> 15:49:41.071604 IP 192.168.100.12.57009 > 192.168.100.229.80: R 1618236270:1618236270(0) win 0 I have also tried to make a client to simultaneously connect to two different backends while both connecting sockets binded to the same ip (no proxy, no transparency) with no problems... I don't understand what I have missed Thanks, Nicolas

2 1

TPROXY behavior when there is no listening socket at the target port
by Ashwani Wason 15 Apr '10

15 Apr '10

Hi All, If one has a TPROXY rule that redirects the packets to a local socket on which there is no one listening then the SYN packets keep getting dropped silently. Is this expected? Without knowing that this would happen my natural expectation was that a RST would be sent by local TCP. Is this by design or a bug? Thanks, - Ashwani

2 1

How does 'dev lo' in tproxy routing rule really work?
by Ashwani Wason 15 Apr '10

15 Apr '10

I want to understand the tproxy rules that have been kindly written for us in http://www.balabit.com/downloads/files/tproxy/README.txt. I understand every aspect of those rules, iptables and iproute2, except this one small highlighted bit in the last rule. ip route add local 0.0.0.0/0 *dev lo* table 100 # ip route show table 100 local default dev lo scope host So packets with fwmark 1 get selected for this routing table, which assumes that all destinations are assigned to this host and hence delivers the packets locally. But the output device is set to 'lo', which is the local loopback device. However even if the proxy is not listening on the IP of lo (say it only has the socket for eth0 and the packet came in on eth0), it still receives the packet. So is the 'lo' here just a way to indicate local delivery to the associated physical interface and does not really represent the actual output interface? Can someone please shed some light on this that is more than a simple yes/no answer? Thank you, - Ashwani

2 1