November 2008 - tproxy - lists.balabit.hu

Squid doesn't seem to spoof client ip address.
by Przemysław Kudyba 04 Dec '08

04 Dec '08

Hello. I have set up fully transpatent http proxy, my problem is: squid sends requests with ip:port of te box running squid instead of clients ip. Here's my config: kernel patch: tproxy4-2.6.26-200809262032 iptables patch: tproxy-iptables-1.4.0-20080521-113954-1211362794 squid: squid-3.HEAD-20081127 iptables & iproute rules: > > iptables -t mangle -N DIVERT > iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT > > > iptables -t mangle -A DIVERT -j MARK --set-mark 1 > iptables -t mangle -A DIVERT -j ACCEPT > > iptables -t mangle -A PREROUTING -i eth0 -p tcp --dport 80 -j TPROXY > --on-port 3128 --tproxy-mark 0x1/0x1 > > ip rule add fwmark 1 lookup 100 > ip route add local 0.0.0.0/0 dev lo table 100 > squid conf: > http_port 192.168.250.2:3128 tproxy 217.97.174.18 - my laptop 212.77.100.101 - some www page tcpdump : 19:06:01.736342 IP 217.97.174.18.53401 > 192.168.250.2.80: S 2658655945:2658655945(0) win 5840 <mss 1440,sackOK,timestamp 6961615 0,nop,wscale 5> 19:06:01.736597 IP 192.168.250.2.80 > 217.97.174.18.53401: S 2655745023:2655745023(0) ack 2658655946 win 5792 <mss 1460,sackOK,timestamp 2617146 6961615,nop,wscale 7> 19:06:01.745935 IP 217.97.174.18.53401 > 192.168.250.2.80: . ack 1 win 183 <nop,nop,timestamp 6961619 2617146> 19:06:15.648614 IP 217.97.174.18.53401 > 192.168.250.2.80: P 1:29(28) ack 1 win 183 <nop,nop,timestamp 6975524 2617146> 19:06:15.648681 IP 192.168.250.2.80 > 217.97.174.18.53401: . ack 29 win 46 <nop,nop,timestamp 2620624 6975524> 19:06:17.130355 IP 217.97.174.18.53401 > 192.168.250.2.80: P 29:31(2) ack 1 win 183 <nop,nop,timestamp 6977004 2620624> 19:06:17.130447 IP 192.168.250.2.80 > 217.97.174.18.53401: . ack 31 win 46 <nop,nop,timestamp 2620994 6977004> 19:06:17.131289 IP 192.168.250.2.59447 > 212.77.100.101.80: S 2887325147:2887325147(0) win 5840 <mss 1460,sackOK,timestamp 2620994 0,nop,wscale 7> 19:06:17.353255 IP 217.97.174.18.34317 > 192.168.250.2.80: S 2917413960:2917413960(0) win 5840 <mss 1440,sackOK,timestamp 6977231 0,nop,wscale 5> 19:06:17.353338 IP 192.168.250.2.80 > 217.97.174.18.34317: S 2895521199:2895521199(0) ack 2917413961 win 5792 <mss 1460,sackOK,timestamp 2621050 6977231,nop,wscale 7> 19:06:17.357848 IP 217.97.174.18.34317 > 192.168.250.2.80: . ack 1 win 183 <nop,nop,timestamp 6977235 2621050> 19:06:17.358077 IP 217.97.174.18.34317 > 192.168.250.2.80: P 1:360(359) ack 1 win 183 <nop,nop,timestamp 6977235 2621050> 19:06:17.358133 IP 192.168.250.2.80 > 217.97.174.18.34317: . ack 360 win 54 <nop,nop,timestamp 2621051 6977235> 19:06:17.358230 IP 192.168.250.2.39336 > 217.97.173.21.80: S 2890323424:2890323424(0) win 5840 <mss 1460,sackOK,timestamp 2621051 0,nop,wscale 7> 19:06:17.358467 IP 217.97.173.21.80 > 192.168.250.2.39336: S 3189038941:3189038941(0) ack 2890323425 win 5792 <mss 1460,sackOK,timestamp 117140974 2621051,nop,wscale 7> 19:06:17.358530 IP 192.168.250.2.39336 > 217.97.173.21.80: . ack 1 win 46 <nop,nop,timestamp 2621051 117140974> 19:06:17.358671 IP 192.168.250.2.39336 > 217.97.173.21.80: P 1:462(461) ack 1 win 46 <nop,nop,timestamp 2621051 117140974> 19:06:17.358958 IP 217.97.173.21.80 > 192.168.250.2.39336: . ack 462 win 54 <nop,nop,timestamp 117140975 2621051> 19:06:17.427512 IP 217.97.173.21.80 > 192.168.250.2.39336: . 1:1449(1448) ack 462 win 54 <nop,nop,timestamp 117140992 2621051>

2 2

Latest tproxy patch for kernel, iptables and squid
by Eduardo Schoedler 04 Dec '08

04 Dec '08

Hi list. I'm installing a box with linux with squid and I'm a little bit confuse. First of all, sorry my bad english... I'm brazilian. =) I need to now where I can found the latest version for: - kernel 2.6.27 (or can I use kernel 2.8.x ?); - iptables 1.4.2; - squid. I have some questions: - Which kernel is better to use in a production box? 2.6.27 or 2.6.28 ? - Which squid is better to use in a production box? 2.7 or 3.0 ? - Where I can found a documentation to install and configure tproxy patches ? Thanks in advance! Regards, Eduardo Schoedler.

3 5

example of using udp on tproxy4
by elyasaf 30 Nov '08

30 Nov '08

> While I can see example of using udp on tproxy2 onto the > redirect-udp-recv.c > file, I can't find equivalent on tproxy4. > > Here are my iptables\ip route redirection lines: > ${iptables} -t mangle -N DIVERT > ${iptables} -t mangle -A PREROUTING -p udp -m socket -j DIVERT > ${iptables} -t mangle -A DIVERT -j MARK --set-mark 1 > ${iptables} -t mangle -A DIVERT -j ACCEPT > ${iptables} -t mangle -A PREROUTING -p udp --dport 1500 -j TPROXY > --tproxy-mark 1 --on-port 3127 > > ip rule add fwmark 1 lookup 100 > ip route add local 0.0.0.0/0 dev lo table 100 #include <netdb.h> #include <stdio.h> #include <unistd.h> #include <string.h> #include <time.h> #define LOCAL_PROXY_PORT 3127 #define MAX_MSG 100 #ifndef IP_TRANSPARENT #define IP_TRANSPARENT 19 #endif #ifndef IP_ORIGADDRS #define IP_ORIGADDRS 20 #endif #ifndef IP_RECVORIGADDRS #define IP_RECVORIGADDRS IP_ORIGADDRS #endif #define CTL_BUF_SIZE 64 int main(int argc, char *argv[]) { int fd, rc, n, cliLen, flags; struct sockaddr_in sourceAddr, destinationAddr , proxyAddr; char buf[MAX_MSG]; struct in_origaddrs *ioa; /* socket creation */ fd=socket(AF_INET, SOCK_DGRAM, 0); if(fd<0) { printf("%s: cannot open socket \n",argv[0]); exit(1); } flags=1; if (setsockopt(fd, SOL_IP, IP_TRANSPARENT, &flags, sizeof(int)) == -1){ perror("proxy-setsockopt() IP_TRANSPARENT error!"); exit(1); }else printf("proxy-setsockopt IP_TRANSPARENT is OK...\n"); setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &flags, sizeof(flags)); setsockopt(fd, SOL_SOCKET, SO_BROADCAST, &flags, sizeof(flags)); if ( setsockopt(fd, SOL_IP, IP_ORIGADDRS , &flags, sizeof(flags))){ perror("proxy-setsockopt() IP_ORIGADDRS error"); }else{ printf("proxy-setsockopt IP_ORIGADDRS is OK...\n"); } /* bind local server port */ proxyAddr.sin_family = AF_INET; proxyAddr.sin_addr.s_addr = htonl(INADDR_ANY); proxyAddr.sin_port = htons(LOCAL_PROXY_PORT); rc = bind (fd, (struct sockaddr *) &proxyAddr,sizeof(proxyAddr)); if(rc<0) { printf("%s: cannot bind port number %d \n",argv[0], LOCAL_PROXY_PORT); exit(1); } printf("%s: waiting for data on port UDP %u\n",argv[0],LOCAL_PROXY_PORT); /* proxy infinite loop */ while(1) { struct msghdr msg; struct cmsghdr *cmsg; struct iovec iov; char ctl_buf[CTL_BUF_SIZE]; /* init buffer */ memset(buf,0,MAX_MSG); memset(&msg, 0, sizeof(msg)); msg.msg_name = &sourceAddr; msg.msg_namelen = sizeof(sourceAddr); msg.msg_controllen = CTL_BUF_SIZE; msg.msg_control = ctl_buf; msg.msg_iovlen = 1; msg.msg_iov = &iov; iov.iov_base = buf; iov.iov_len = MAX_MSG; n = recvmsg(fd, &msg, 0); if(n<0) { printf("%s: cannot receive data \n",argv[0]); perror("recvmsg"); continue; } printf("%s: source %s:UDP%u : %s \n", argv[0],inet_ntoa(sourceAddr.sin_addr), ntohs(sourceAddr.sin_port),buf); memset(&destinationAddr,0,sizeof(struct sockaddr_in)); for (cmsg = CMSG_FIRSTHDR(&msg); cmsg; cmsg = CMSG_NXTHDR(&msg,cmsg)){ if (cmsg->cmsg_level == SOL_IP && cmsg->cmsg_type == IP_RECVORIGADDRS){ memcpy(&destinationAddr,(struct sockaddr_in *)CMSG_DATA(cmsg),sizeof(struct sockaddr_in)); destinationAddr.sin_family = AF_INET; printf("%s: destination %s:UDP%u\n", argv[0],inet_ntoa(destinationAddr.sin_addr), ntohs(destinationAddr.sin_port)); send_to_transparenty(buf,n,&sourceAddr,&destinationAddr); //sendto(fd,buf,n,&sourceAddr,&destinationAddr); } } }/* end of proxy infinite loop */ return 0; } int send_to_transparenty(char *buf,int len,struct sockaddr_in *source,struct sockaddr_in *destination){ int fd,flags,i; fd=socket(AF_INET, SOCK_DGRAM, 0); flags=1; if (setsockopt(fd, SOL_IP, IP_TRANSPARENT, &flags, sizeof(int)) == -1){ perror("destination-setsockopt() IP_TRANSPARENT error!"); return; }else{ //printf("destination-setsockopt IP_TRANSPARENT is OK...\n"); } setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &flags, sizeof(flags)); setsockopt(fd, SOL_SOCKET, SO_BROADCAST, &flags, sizeof(flags)); if(bind (fd, (struct sockaddr *) source,sizeof(*source))==-1){ perror("source-bind() error!"); return; } flags=0; if(sendto(fd,buf,len,flags,(struct sockaddr *)destination,sizeof(*destination))==-1){ perror("destination-sendto() error!"); return 0; }else{ printf("send to other side\n"); } close(fd); return 1; }

1 0

Re: [tproxy] [PATCH] [TPROXY] implementedIP_RECVORIGDSTADDRsocketoption
by Balazs Scheidler 18 Nov '08

18 Nov '08

Hi, You were right, sendto() wasn't working properly, but not because of my last patch, it never really worked in IP_TRANSPARENT mode. I've attached two patches I've just submitted to netdev, which should solve your problem. I'm Ccing the tproxy list as well. On Mon, 2008-11-17 at 21:21 +0200, elyasaf wrote: > You are right that in my isolate example was a bug > Now I add transparenty and its continue to be "Invalid argument" (but > without transparenty it work fine) > Its some how really work before the patch, and I cant see any reason why, > the patch seen clean and not related... > > int main(int argc, char *argv[]) { > char buf[5]="aaaa"; > struct sockaddr_in source,destination; > source.sin_family = AF_INET; > source.sin_addr.s_addr = inet_addr("10.0.0.26"); > source.sin_port = htons(INADDR_ANY); > > destination.sin_family = AF_INET; > destination.sin_port = htons(1500); > destination.sin_addr.s_addr = inet_addr("10.0.0.2"); > send_to_transparenty(buf,4,&source,&destination); > } > > int send_to_transparenty(char *buf,int len,struct sockaddr_in *source,struct > sockaddr_in *destination){ > int fd=socket(AF_INET, SOCK_DGRAM, 0); > int flags=1; > if (setsockopt(fd, SOL_IP, IP_TRANSPARENT, &flags, sizeof(int)) == > -1){ > perror("destination-setsockopt() IP_TRANSPARENT error!"); > return; > }else{ > printf("destination-setsockopt IP_TRANSPARENT is OK...\n"); > } > setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &flags, sizeof(flags)); > setsockopt(fd, SOL_SOCKET, SO_BROADCAST, &flags, sizeof(flags)); > if(bind (fd, (struct sockaddr *) source,sizeof(*source))==-1){ > perror("destination-bind() error!"); > return; > } > flags=0; > if(sendto(fd,buf,len,flags,(struct sockaddr > *)destination,sizeof(*destination))==-1){ > perror("destination-sendto() error!"); > return; > } > close(fd); > } > > execve("./udpserver", ["./udpserver"], [/* 31 vars */]) = 0 > brk(0) = 0x602000 -- Bazsi

1 0

Re: [tproxy] [PATCH] [TPROXY] implemented IP_RECVORIGDSTADDR socket option
by elyasaf 16 Nov '08

16 Nov '08

Last week spoofing UDP work fine (before I try UDP patch - https://lists.balabit.hu/pipermail/tproxy/2008-November/000999.html) But now I get: " destination-sendto() error:Invalid argument" (I can achieve the source and destination IP for "UDP server") I use tproxy 4.1 kernel: 2.6.25 The code (that work fine before the above patch): int fd=socket(AF_INET, SOCK_DGRAM, 0); int flags=1; if (setsockopt(fd, SOL_IP, IP_TRANSPARENT, &flags, sizeof(int)) == -1){ perror("destination-setsockopt() IP_TRANSPARENT error!"); return; }else{ printf("destination-setsockopt IP_TRANSPARENT is OK...\n"); } setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &flags, sizeof(flags)); setsockopt(fd, SOL_SOCKET, SO_BROADCAST, &flags, sizeof(flags)); if(bind (fd, (struct sockaddr *) source,sizeof(*source))==-1){ perror("destination-bind() error!"); return; } flags=0; if(sendto(fd,buf,len,flags, (struct sockaddr *)destination, sizeof(*destination))==-1){ perror("destination-sendto() error"); return; } close(fd); Any Advice?

2 1

Re: [tproxy] [PATCH] [TPROXY] implemented IP_RECVORIGDSTADDR socket option
by Balazs Scheidler 14 Nov '08

14 Nov '08

On Thu, 2008-11-13 at 12:21 -0800, David Stevens wrote: > I know it's not part of your patch, but what about turning that into > an array of function pointers and a loop, as code cleanup? > > +-DLS > > I could do that if I get a positive feedback from DaveM, that the sockopt itself is ok. (functionally it was already tested independently from me) -- Bazsi

1 0

[PATCH] [TPROXY] implemented IP_RECVORIGDSTADDR socket option
by Balazs Scheidler 13 Nov '08

13 Nov '08

In case UDP traffic is redirected to a local UDP socket, the originally addressed destination address/port cannot be recovered with the in-kernel tproxy. This patch adds an IP_RECVORIGDSTADDR sockopt that enables a IP_ORIGDSTADDR ancillary message in recvmsg(). This ancillary message contains the original destination address/port of the packet being received. Please apply. Signed-off-by: Balazs Scheidler <bazsi(a)balabit.hu> --- include/linux/in.h | 4 ++++ net/ipv4/ip_sockglue.c | 40 +++++++++++++++++++++++++++++++++++++++- 2 files changed, 43 insertions(+), 1 deletions(-) diff --git a/include/linux/in.h b/include/linux/in.h index db458be..d60122a 100644 --- a/include/linux/in.h +++ b/include/linux/in.h @@ -80,6 +80,10 @@ struct in_addr { /* BSD compatibility */ #define IP_RECVRETOPTS IP_RETOPTS +/* TProxy original addresses */ +#define IP_ORIGDSTADDR 20 +#define IP_RECVORIGDSTADDR IP_ORIGDSTADDR + /* IP_MTU_DISCOVER values */ #define IP_PMTUDISC_DONT 0 /* Never send DF frames */ #define IP_PMTUDISC_WANT 1 /* Use per route hints */ diff --git a/net/ipv4/ip_sockglue.c b/net/ipv4/ip_sockglue.c index 465abf0..1e70488 100644 --- a/net/ipv4/ip_sockglue.c +++ b/net/ipv4/ip_sockglue.c @@ -48,6 +48,7 @@ #define IP_CMSG_RECVOPTS 8 #define IP_CMSG_RETOPTS 16 #define IP_CMSG_PASSSEC 32 +#define IP_CMSG_ORIGDSTADDR 64 /* * SOL_IP control messages. @@ -126,6 +127,27 @@ static void ip_cmsg_recv_security(struct msghdr *msg, struct sk_buff *skb) security_release_secctx(secdata, seclen); } +void ip_cmsg_recv_dstaddr(struct msghdr *msg, struct sk_buff *skb) +{ + struct sockaddr_in sin; + struct iphdr *iph = ip_hdr(skb); + u16 *ports = (u16 *) skb_transport_header(skb); + + if (skb_transport_offset(skb) + 4 > skb->len) + return; + + /* All current transport protocols have the port numbers in the + * first four bytes of the transport header and this function is + * written with this assumption in mind. + */ + + sin.sin_family = AF_INET; + sin.sin_addr.s_addr = iph->daddr; + sin.sin_port = ports[1]; + memset(sin.sin_zero, 0, sizeof(sin.sin_zero)); + + put_cmsg(msg, SOL_IP, IP_ORIGDSTADDR, sizeof(sin), &sin); +} void ip_cmsg_recv(struct msghdr *msg, struct sk_buff *skb) { @@ -160,6 +182,12 @@ void ip_cmsg_recv(struct msghdr *msg, struct sk_buff *skb) if (flags & 1) ip_cmsg_recv_security(msg, skb); + + if ((flags>>=1) == 0) + return; + if (flags & 1) + ip_cmsg_recv_dstaddr(msg, skb); + } int ip_cmsg_send(struct net *net, struct msghdr *msg, struct ipcm_cookie *ipc) @@ -421,7 +449,8 @@ static int do_ip_setsockopt(struct sock *sk, int level, (1<<IP_ROUTER_ALERT) | (1<<IP_FREEBIND) | (1<<IP_PASSSEC) | (1<<IP_TRANSPARENT))) || optname == IP_MULTICAST_TTL || - optname == IP_MULTICAST_LOOP) { + optname == IP_MULTICAST_LOOP || + optname == IP_RECVORIGDSTADDR) { if (optlen >= sizeof(int)) { if (get_user(val, (int __user *) optval)) return -EFAULT; @@ -509,6 +538,12 @@ static int do_ip_setsockopt(struct sock *sk, int level, else inet->cmsg_flags &= ~IP_CMSG_PASSSEC; break; + case IP_RECVORIGDSTADDR: + if (val) + inet->cmsg_flags |= IP_CMSG_ORIGDSTADDR; + else + inet->cmsg_flags &= ~IP_CMSG_ORIGDSTADDR; + break; case IP_TOS: /* This sets both TOS and Precedence */ if (sk->sk_type == SOCK_STREAM) { val &= ~3; @@ -1022,6 +1057,9 @@ static int do_ip_getsockopt(struct sock *sk, int level, int optname, case IP_PASSSEC: val = (inet->cmsg_flags & IP_CMSG_PASSSEC) != 0; break; + case IP_RECVORIGDSTADDR: + val = (inet->cmsg_flags & IP_CMSG_ORIGDSTADDR) != 0; + break; case IP_TOS: val = inet->tos; break; -- 1.5.4.3 -- Bazsi

2 1

Re: [tproxy] udp + tproxy
by Balazs Scheidler 13 Nov '08

13 Nov '08

On Thu, 2008-11-13 at 08:17 +0100, Jan Engelhardt wrote: > On Wednesday 2008-11-12 20:27, Balazs Scheidler wrote: > >> > >> Since tproxy 4 (unlike tproxy 2) doesn't modify the incoming packets in > >> any way you should be able to get the correct destination address by > >> simply calling recvfrom() and using the source address returned by the > >> kernel. > > > >This is not true, recvfrom() returns the client address and does not > >return the original destination. > > Mh, perhaps getsockname() could do something - well, at least > if used with accept() and as such, mostly for TCP only. > Well, here's my other mail on the subject on the tproxy list, it basically details that we do have an implementation of accept() for UDP which we use in production. But I'm not sure that could be integrated to mainline. -<- quote ->- Well, for supporting UDP with tproxy4 we use a different approach: udp_accept(), you can find it in the BalaBit kernel patch tree at http://www.balabit.com/downloads/files/kernel-patches/ The way udp_accept() works is as follows: * the userspace proxy opens a "listening" udp socket, binds it to the listening address, performs no connect() calls * tproxy redirects packets to this socket just like in the case for TCP * whenever the socket becomes readable from the userspace proxy, accept() is invoked on the UDP socket, this accept() is implemented by the patch mentioned above * in kernel space: the first packet is consulted from the socket buffer of the listening socket * the kernel opens a new UDP socket, just like the accept() call for TCP does * this new UDP socket is bound this way: * local address is the same as the destination address of the incoming packet * destination address is the same as the source address of the incoming packet * the socket buffer of the listening socket is traversed, and each packet having the same IP addresses as the first packet is moved to the newly opened socket; this traversal is happening in an atomic context, thus no new packets can arrive * a reference to the newly opened socket is returned to userspace * when a new packet comes in, the socket lookup will prefer the completely bound socket over the listening socket Since at the end of the accept() call you get a completely bound socket, you can simply call getsockname() to query the target address, just like with TCP. Hope this helps. PS: I was talking to Patrick McHardy whether to send the udp_accept() patch for kernel inclusion, he said it might be worth trying, however he was not completely sure it'd be integrated. So I didn't push it so far. -- Bazsi

1 0

udp + tproxy
by Andrey Luzgin 12 Nov '08

12 Nov '08

Hello, While I can see example of using udp on tproxy2 onto the redirect-udp-recv.c file, I can't find equivalent on tproxy4. For getting the original destination IP, I just use setsockopt IP_PKTINFO: setsockopt(sd, SOL_IP, IP_PKTINFO , &flags, sizeof(flags)); But I don't know how to get the original destination port: a) I manually defined IP_RECVORIGADDRS to be 11273 as I find on tproxy2: setsockopt(sd, SOL_IP, IP_RECVORIGADDRS , &flags, sizeof(flags)); but the setsockopt failed. b) the getsockname give me the server listening port. Here are my iptables\ip route redirection lines: ${iptables} -t mangle -N DIVERT ${iptables} -t mangle -A PREROUTING -p udp -m socket -j DIVERT ${iptables} -t mangle -A DIVERT -j MARK --set-mark 1 ${iptables} -t mangle -A DIVERT -j ACCEPT ${iptables} -t mangle -A PREROUTING -p udp --dport 1500 -j TPROXY --tproxy-mark 1 --on-port 3127 ip rule add fwmark 1 lookup 100 ip route add local 0.0.0.0/0 dev lo table 100 What is missed? Best, Moshe

3 3

tproxy for kernel 2.6.18.6
by Victoria Muntean 12 Nov '08

12 Nov '08

Hello, What is latest version of tproxy that I can apply to kernel 2.6.18.6 ? Thanks Viki

2 1