- tproxy - lists.balabit.hu

tproxy race condition? [RESEND]
by jim＠minter.demon.co.uk 21 Dec '04

21 Dec '04

Hi Krisztian and Balazs, First: a big thank-you for your work on the tproxy code! I'm trying to use tproxy to implement a fully transparent layer 7 proxy as follows: TCP connections arrive and are REDIRECTed to a single local port. A userspace process listen()s on that port, and makes ongoing (transparent) connections on new TCP sockets by calling bind(), tproxy setsockopt()s and connect(). In general it works well, but I'm having a few issues which I think are possibly SMP-related. I believe I've reduced some of these to a simple test case, sources for which are attached. I'm using Linux kernel 2.4.27 and all four patches in cttproxy-2.4.27-2.0.0 patch. To run the test case, you need two machines; I think the 'client' must be SMP. The 'server', 10.0.3.2, listens on a single TCP port and has a simple loop which accept()s and close()s TCP connections that it receives. The SMP 'client', 10.0.3.3, has two processes each connecting to the server. The clients loop through a port range 32768-49152. They bind() on 10.0.3.3, receiving some port from the kernel. They then assign a transparent port in the loop port range on unregistered IP 10.0.3.253, and connect() to the server. (The server has a route set up so that it knows to return traffic on 10.0.3.253 to the client box). The problem: once in a while, one of the client processes takes 3s to connect() to the server. Then, the resulting TCP connection is NOT TRANSPARENT (i.e. 10.0.3.3 is used, not 10.0.3.253). This can be seen by running "tcpdump host 10.0.3.3" on either box. However, none of the client process system calls fail at any point. In the case that CONFIG_IP_NF_NAT_NRES is set, at the same time this happens, the _other process_ has a -EINVAL failure in ip_tproxy_setsockopt_flags(), with corresponding "failed to register NAT reservation" error in dmesg. When CONFIG_IP_NF_NAT_NRES is unset, this failure doesn't happen. But either way, on the _original process_, the non-transparent TCP connection happens. What I believe is happening is as follows: There is evidence in dmesg that the first SYN packet of the connect() passes through the LOCAL_OUT iptables hooks (I see "ip_tproxy_fn(): new connection, hook=3" and "ip_tproxy_fn(): new connection, hook=4", but for some reason the packet never actually makes it onto the wire. I can't see where it goes missing. But anyway, connect() waits 3s and resends the SYN. This time, as the second packet goes through the iptables, for some reason it's not translated. It makes it onto the wire and the rest of the connection proceeds untranslated. I haven't been able to progress much further debugging this, and wondered if you had any ideas? My principal concern is that the userspace processes don't receive an error and have no proper way of telling that the connection is going untransparent. Am I making a stupid mistake somewhere? One other curious thing here: MUST_BE_READ_LOCKED(&ip_tproxy_lock) in ip_tproxy_relatedct_add() fails. Could this be related in any way? Finally, what is the purpose of the new CONFIG_IP_NF_NAT_NRES option? Thank-you for reading this, and for any advice you have! Jim Minter <jim(a)minter.demon.co.uk> == 8< == CLIENT CODE == 8< == #include <arpa/inet.h> #include <linux/netfilter_ipv4/ip_tproxy2.h> // the v2.0 header #include <sys/socket.h> #include <sys/time.h> #include <time.h> #include <unistd.h> #include <cerrno> #include <cstdarg> #include <cstdio> #include <cstdlib> void error(const char *c) { perror(c); exit(1); } void log(const char *fmt, ...) { char buf[10]; struct timeval timeval; gettimeofday(&timeval, NULL); strftime(buf, sizeof(buf), "%T", localtime(&timeval.tv_sec)); printf("%s.%06u ", buf, timeval.tv_usec); va_list args; va_start(args, fmt); vprintf(fmt, args); va_end(args); } int main(int argc, char **argv) { int lo = 32768; int hi = 49152; setbuf(stderr, NULL); setlinebuf(stdout); while(1) for(int port = lo; port < hi; port++) { fprintf(stderr, "."); int s = socket(PF_INET, SOCK_STREAM, 0); if(s == -1) error("socket"); { // seems to be necessary...? int param = 1; if(setsockopt(s, SOL_SOCKET, SO_REUSEADDR, &param, sizeof(param))) error("setsockopt SO_REUSEADDR"); } { // bind to our local IP, and output the port the kernel gave us struct sockaddr_in sockaddr_in; sockaddr_in.sin_family = AF_INET; sockaddr_in.sin_port = 0; sockaddr_in.sin_addr.s_addr = inet_addr("10.0.3.3"); if(bind(s, (struct sockaddr *)&sockaddr_in, sizeof(sockaddr_in))) error("bind"); socklen_t sl = sizeof(sockaddr_in); if(getsockname(s, (struct sockaddr *)&sockaddr_in, &sl)) error("getsockname"); log("%u\n", ntohs(sockaddr_in.sin_port)); } { // now get ourselves a looped port on unregistered IP 10.0.3.253. struct in_tproxy in_tproxy; in_tproxy.op = TPROXY_ASSIGN; in_tproxy.v.addr.faddr.s_addr = inet_addr("10.0.3.253"); in_tproxy.v.addr.fport = htons(port); if(setsockopt(s, SOL_IP, IP_TPROXY, &in_tproxy, sizeof(in_tproxy))) error("setsockopt TPROXY_ASSIGN"); log("a\n"); in_tproxy.op = TPROXY_CONNECT; in_tproxy.v.addr.faddr.s_addr = inet_addr("10.0.3.2"); in_tproxy.v.addr.fport = htons(7000); if(setsockopt(s, SOL_IP, IP_TPROXY, &in_tproxy, sizeof(in_tproxy))) error("setsockopt TPROXY_CONNECT"); log("b\n"); in_tproxy.op = TPROXY_FLAGS; in_tproxy.v.flags = ITP_CONNECT | ITP_ONCE; if(setsockopt(s, SOL_IP, IP_TPROXY, &in_tproxy, sizeof(in_tproxy))) { perror("setsockopt TPROXY_FLAGS"); close(s); continue; } log("c\n"); } { // now connect struct sockaddr_in sockaddr_in; sockaddr_in.sin_family = AF_INET; sockaddr_in.sin_port = htons(7000); sockaddr_in.sin_addr.s_addr = inet_addr("10.0.3.2"); if(connect(s, (struct sockaddr *)&sockaddr_in, sizeof(sockaddr_in))) error("connect"); log("d\n"); } { // wait for other side to close char buf; if(read(s, &buf, 1) != 0) error("read"); } close(s); log("e\n"); } return 0; } == 8< == CLIENT CODE ENDS == 8< == == 8< == SERVER CODE == 8< == #include <netinet/in.h> #include <sys/socket.h> #include <unistd.h> #include <cerrno> #include <cstdio> #include <cstdlib> void error(const char *c) { perror(c); exit(1); } int main() { int s = socket(PF_INET, SOCK_STREAM, 0); if(s == -1) error("socket"); { int param = 1; if(setsockopt(s, SOL_SOCKET, SO_REUSEADDR, &param, sizeof(param))) error("setsockopt SO_REUSEADDR"); } { struct sockaddr_in sockaddr_in; sockaddr_in.sin_family = AF_INET; sockaddr_in.sin_port = htons(7000); sockaddr_in.sin_addr.s_addr = INADDR_ANY; if(bind(s, (struct sockaddr *)&sockaddr_in, sizeof(sockaddr_in))) error("bind"); } if(listen(s, SOMAXCONN)) error("listen"); while(1) { int fd = accept(s, NULL, 0); if(fd == -1) error("accept"); close(fd); } } == 8< == SERVER CODE ENDS == 8< ==

3 10

Re: [tproxy] tproxy race condition? [RESEND]
by jim＠minter.demon.co.uk 21 Dec '04

21 Dec '04

Hi! > > The above tarball also has a log of a run without the ITP_ONCE flag. > > It's encouraging that you've been able to reproduce the problem at > > your end -- was it on an SMP box? > > Yes, it was. Is your test box SMP as well? Yes, it is. > > By "failed to apply NAT mapping" error, I assume you mean the > > "IP_TPROXY: error applying NAT mapping" error? Just to confirm, I'm > > not getting any of these error messages at all (perhaps because I've > > configured NAT reservations off?) > > Yes, I meant that error. Of course you're right, this might be because > of NAT reservations. (But is interesting anyway.) Do let me know what you find, and if I can get you any more information about the issue(s) from this end. Cheers, Jim

1 0

[PATCH] Tproxy 2.0.0 patch for 2.6.9
by Igmar Palsenberg 02 Dec '04

02 Dec '04

Hi, Is there any official patch for tproxy 2.0.x for 2.6.9 ? I've merged basically everything, but someone who is more familiar with the whole stuff needs to have a look at it. My version is at http://www.jdimedia.nl/igmar/tproxy-2.0.0-2.6.9.patch Can someone have a look at this ? regards, Igmar

2 1

ICMP and tproxy
by Javier Govea 25 Nov '04

25 Nov '04

Hi, Thank you for your prompt response. I was planning to use raw sockets from the begining. But I got a little problem. This is my setup: Client ----------- Proxy --------- Server If I run a raw socket server in my Proxy box and I ping the Proxy from my Client, then my raw socket server can get all the pings. But if I ping the Server from my Client, then my my raw server in the Proxy does not capture anything, because (I'm guessing) it is not inteded for the Proxy box. So, somehow I need to pass that ping to my raw server. I cannot use tproxy and the iptables REDIRECT target because REDIRECT requires a port number, and there are no ports for ICMP packets. So, is there any alternative solution? Am I missing something? Below is the code I am using for my raw server. Thanks in advance. Xavier int main() { int sockfd; if ((sockfd = socket(AF_INET, SOCK_RAW, IPPROTO_ICMP)) < 0) { perror("Socket:"); return -1; } struct sockaddr_in srcaddr; socklen_t srclen = sizeof(struct sockaddr_in); unsigned char buffer [2000]; int len = 2000; int bytes; struct icmphdr *icmphdr; while(1) { printf ("Waiting for pings...\n"); if ( (bytes = recvfrom(sockfd, buffer, len, 0, (struct sockaddr *)&srcaddr, &srclen)) < 0) perror("rcvfrom"); else { icmphdr = (struct icmphdr *)(buffer + sizeof(struct iphdr)); printf ("Received %d bytes from %s:%d ICMP header: type %d code %d seq %d id %d\n", bytes, inet_ntoa(srcaddr.sin_addr), ntohs(srcaddr.sin_port), icmphdr->type, icmphdr->code, ntohs(icmphdr->un.echo.sequence), ntohs(icmphdr->un.echo.id) ); } } return 1; } ICMP is not currently supported by TProxy directly. However you can use raw sockets to achieve the same result. On Wed, 2004-11-24 at 23:53, Javier Govea wrote: > Hi, > > I am writing an proxy for different appliacations. So, far it's working with UDP and TCP. > All packets arriving at my proxy machine are redirected, with iptables, to the the proxy > server, which extracts the final destination of each packet. For the UDP case I am using a > tproxy rule to forward all traffic to the proxy server. The proxy server is using > something similar to redirect-udp-recv.c to exctract final destinations. > > Now, I need my proxy to support pings and thus ICMP packets. Can use the tproxy to forward > ICMP packets to my proxy??? and to extract final destination similar to > redirect-udp-recv.c??? If this is not possible, is there any alternative? > > Thanks in advance > Xavier > > > _______________________________________________ > tproxy mailing list > tproxy(a)lists.balabit.hu > https://lists.balabit.hu/mailman/listinfo/tproxy -- Bazsi

1 0

ICMP and tproxy
by Javier Govea 25 Nov '04

25 Nov '04

Hi, I am writing an proxy for different appliacations. So, far it's working with UDP and TCP. All packets arriving at my proxy machine are redirected, with iptables, to the the proxy server, which extracts the final destination of each packet. For the UDP case I am using a tproxy rule to forward all traffic to the proxy server. The proxy server is using something similar to redirect-udp-recv.c to exctract final destinations. Now, I need my proxy to support pings and thus ICMP packets. Can use the tproxy to forward ICMP packets to my proxy??? and to extract final destination similar to redirect-udp-recv.c??? If this is not possible, is there any alternative? Thanks in advance Xavier

2 1

NAT Reservation
by Tim Burress 10 Nov '04

10 Nov '04

Hello! We're using TPROXY 2.0.0 for 2.4.27 and are running into a strange issue which seems related to NAT reservation. As usual, we have a client TCP connection coming in from a given <saddr,sport> going to a particular <daddr,dport>. Netfilter rules REDIRECT this to our proxy code, which then uses TPROXY to connect out to <daddr,dport> using <saddr,sport> as the apparent source. We use the same operation sequence that appears in the test directory: first bind the socket to a local address, then do a TPROXY ASSIGN assigning <saddr,sport> to the socket, then TPROXY FLAGS to set ITP_CONNECT|ITP_ONCE, and then the actual connect(). With earlier versions of TPROXY (though I can't guarantee that kernel options haven't changed) this worked fine. Now, though, we see two problems. First, on when we make the call to ASSIGN, we get an error: IP_TPROXY: ip_tproxy_nat_reserve proto 6 foreign 10.0.0.2:32772 peer 0.0.0.0:0 IP_TPROXY: IP_TPROXY_ASSIGN cannot register NAT reservation 0200000a:0480 Apparently the NAT reservation is failing because when the initial TCP connection came in, conntrack set up a record expecting a reply to <saddr,sport>. The only way I could see to get around this was to set SO_REUSEADDR on the socket before the call to ASSIGN. However the second problem is that, if I do this, I get an error in the FLAGS call: IP_TPROXY: ip_tproxy_nat_reserve proto 6 foreign 10.0.0.2:32775 peer 0.0.0.0:0 IP_TPROXY: IP_TPROXY_FLAGS sr c660bf9c: failed to register NAT reservation I was able to work around these problems by first setting SO_REUSEADDR on the socket before the ASSIGN than clearing it before the FLAGS call, but that doesn't seem ideal. Is this how it's supposed to be done, or do I just have some basic misunderstanding about how TPROXY should be used? I have to admit I'm not clear on the purpose of NAT reservations. Thanks! Tim __________________________________ Do you Yahoo!? Check out the new Yahoo! Front Page. www.yahoo.com

2 2

Re: TPROXY and NAT
by KOVACS Krisztian 08 Nov '04

08 Nov '04

Hi, 2004-11-06, szo keltezéssel 10:18-kor Sumit Pandya ezt írta: > I'd like to know does the limitation described on following thread has been > overcame after release of 2.0.0 version? > https://lists.balabit.hu/pipermail/tproxy/2003-November/000053.html Basically it's a bad interaction between Netfilter DNAT/MASQUERADE and tproxy. Unfortunately to work around this limitation you have to modify the NAT core code. For more information see https://lists.balabit.hu/pipermail/tproxy/2003-November/000053.html -- Regards, Krisztian KOVACS

1 0

RE: [tproxy] tproxy on bridge
by eric_li＠trend.com.tw 04 Nov '04

04 Nov '04

Oh?! I didn't try it in bridge mode. When I saw the README file, the command is iptables -t tproxy -A PREROUTING -j TPROXY --on-port <proxyport> iptables won't see packets in bridge mode, unless you enable CONFIG_BRIDGE_NETFILTER option, right? So, for tproxy to work in bridge, we must enable this option? Regards, Eric Li -----Original Message----- From: Lennert Buytenhek [mailto:buytenh@wantstofly.org] Sent: Thursday, November 04, 2004 2:27 AM To: Eric Li (RD-TW) Cc: tproxy(a)lists.balabit.hu Subject: Re: [tproxy] tproxy on bridge On Thu, Nov 04, 2004 at 12:05:04AM +0800, eric_li(a)trend.com.tw wrote: > Hi, Hello, > It seems to me that tproxy works on router (IP layer). > Is there any plan to support bridge mode? It works for me quite nicely in bridge mode. If you can describe what problems you are seeing, perhaps we can fix them. TREND MICRO EMAIL NOTICE The information contained in this email and any attachments is confidential and may be subject to copyright or other intellectual property protection. If you are not the intended recipient, you are not authorized to use or disclose this information, and we request that you notify us by reply mail or telephone and delete the original message from your mail system.

1 0

tproxy on bridge
by eric_li＠trend.com.tw 03 Nov '04

03 Nov '04

Hi, It seems to me that tproxy works on router (IP layer). Is there any plan to support bridge mode? Or do there exist some similar buddy of tproxy in ebtables ?? Regards, Eric Li TREND MICRO EMAIL NOTICE The information contained in this email and any attachments is confidential and may be subject to copyright or other intellectual property protection. If you are not the intended recipient, you are not authorized to use or disclose this information, and we request that you notify us by reply mail or telephone and delete the original message from your mail system.

2 1

Interaction with IPVS
by Tim Burress 19 Oct '04

19 Oct '04

Hello! We're running into another potential interaction, this time between tproxy and IPVS http://www.linuxvirtualserver.org on Linux 2.4.27. What we see with TPPROXY 2.0, and going at least as far back as 1.9.6, is that incoming connections that go through IPVS for load balancing are properly mapped to a real server, but when the reply packets come back from the real server, they aren't recognized. Conntrack considers the connection [UNREPLIED] even though tcpdump shows that the reply packets are arriving with the expected source and destination address/ports. It seems like the reply packets are being dropped. The only evidence suggesting that TPROXY is involved at this point is the observation that the problem doesn't appear when we use TPROXY 1.2. One other data point is that, when we use TPROXY 2.0, the problem appears even when the module is not loaded, so it seems to be a side effect of patches to other modules. I realize this is vague, and we're trying to track it down, but I thought I would post and just see if anyone else has observed similar behavior, or if there are any ideas for locating the source of the problem. Thanks! Tim _______________________________ Do you Yahoo!? Declare Yourself - Register online to vote today! http://vote.yahoo.com

2 4