December 2006 - tproxy - lists.balabit.hu

TPROXY 2.0.5 with NAT reservations deadlocks on 2.4.33.3
by Lior Dotan 02 Jan '07

02 Jan '07

Hi, I'm running TPROXY 2.0.5 compiled with the NAT reservations patch on 2.4.33.3. >From time to time the machine freezes. I was able to capture the following stack trace: Stack traceback for pid 0 0x403b2000 0 0 1 0 R 0x403b2370 *swapper EBP EIP Function (args) 0x403b3bd8 0x40107e23 __write_lock_failed+0xf (0x60ad9300, 0x140, 0x58e55214, 0x 58b74a68, 0x58b74a68) kernel .text 0x40100000 0x40107e14 0x40107e34 0x608c0e03 [iptable_nat].text.lock.ip_nat_core+0x6d iptable_nat .text 0x608be060 0x608c0d96 0x608c0eb 0 0x608bf5d1 [iptable_nat]ip_nat_reserved_unregister_all+0x31 (0x58e552 14) iptable_nat .text 0x608be060 0x608bf5a0 0x608bf67 0 0x403b3be4 0x608bf681 [iptable_nat]ip_nat_reserved_cleanup_expect+0x11 (0x58e552 14, 0x58e55214) iptable_nat .text 0x608be060 0x608bf670 0x608bf69 0 0x403b3bf4 0x608a7bb9 [ip_conntrack]destroy_expect+0x29 (0x58e55214, 0x58e55214) ip_conntrack .text 0x608a7060 0x608a7b90 0x608a7b c0 0x403b3c04 0x608a7dc8 [ip_conntrack]__unexpect_related+0x68 (0x58e55214, 0x58e55 214) ip_conntrack .text 0x608a7060 0x608a7d60 0x608a7d d0 0x403b3c14 0x608a7e00 [ip_conntrack]unexpect_related+0x30 (0x58e55214, 0x58b74a1 0, 0x403b3d60, 0x0) ip_conntrack .text 0x608a7060 0x608a7dd0 0x608a7e 10 0x403b3c2c 0x608a7e7e [ip_conntrack]remove_expectations+0x6e (0x58b74a10, 0x1, 0 x58b74a10) ip_conntrack .text 0x608a7060 0x608a7e10 0x608a7e 80 0x403b3c40 0x608a7eea [ip_conntrack]clean_from_lists+0x6a (0x58b74a10, 0x58b74a3 4) ip_conntrack .text 0x608a7060 0x608a7e80 0x608a7f 00 0x403b3c50 0x608a81d5 [ip_conntrack]death_by_timeout+0x55 (0x58b74a10, 0x5a7805b 4, 0x1d6002c, 0x1500, 0x1d6002d) ip_conntrack .text 0x608a7060 0x608a8180 0x608a82 30 0x403b3c74 0x608bf73a [iptable_nat]ip_nat_used_tuple+0xaa (0x403b3d60, 0x5a7805b 4, 0x1, 0x1, 0x608c4a00) iptable_nat .text 0x608be060 0x608bf690 0x608bf74 0 0x403b3cbc 0x608bfd5c [iptable_nat]get_unique_tuple+0xcc (0x403b3d60, 0x403b3d30 , 0x403b3d9c, 0x5a7805b4, 0x0) iptable_nat .text 0x608be060 0x608bfc90 0x608bfeb 0 0x403b3d7c 0x608bff11 [iptable_nat]ip_nat_setup_info+0x61 (0x5a7805b4, 0x403b3d9 c, 0x0, 0x0, 0x608c48a0) iptable_nat .text 0x608be060 0x608bfeb0 0x608c023 0 0x403b3dc4 0x608beacb [iptable_nat]ip_nat_rule_find+0xab (0x403b3ec8, 0x0, 0x60a d9300, 0x0, 0x5a7805b4) iptable_nat .text 0x608be060 0x608bea20 0x608beae 0 0x403b3df8 0x608be1a5 [iptable_nat]ip_nat_fn+0x145 (0x0, 0x403b3ec8, 0x60ad9300, 0x0, 0x4025a270) iptable_nat .text 0x608be060 0x608be060 0x608be2f 0 0x403b3e20 0x608be331 [iptable_nat]ip_nat_in+0x41 (0x0, 0x403b3ec8, 0x60ad9300, 0x0, 0x4025a270) iptable_nat .text 0x608be060 0x608be2f0 0x608be37 0 0x403b3e48 0x4024c175 nf_iterate+0x65 (0x404e35c0, 0x403b3ec8, 0x0, 0x60ad9300, 0x0) kernel .text 0x40100000 0x4024c110 0x4024c190 0x403b3e84 0x4024c552 nf_hook_slow+0xc2 (0x2, 0x0, 0x403b3ec8, 0x60ad9300, 0x0) kernel .text 0x40100000 0x4024c490 0x4024c690 0x403b3ec0 0x4025a02c ip_rcv+0x45c (0x5ee7e4c4, 0x60ad9300, 0x403ab390, 0x60ad93 00, 0x4040ee2c) kernel .text 0x40100000 0x40259bd0 0x4025a0e0 0x403b3ee0 0x402429a8 netif_receive_skb+0x148 (0x5ee7e4c4, 0x1a5220, 0x4040ee20, 0x40, 0x4040ef10) kernel .text 0x40100000 0x40242860 0x40242a80 0x403b3f04 0x40242b0a process_backlog+0x8a (0x4040ee4c, 0x403b3f20, 0x1a5220, 0x 4040ee20, 0x0) kernel .text 0x40100000 0x40242a80 0x40242bb0 0x403b3f30 0x40242c7a net_rx_action+0xca (0x4040e8f0, 0x46, 0x0, 0x403b3f44, 0x4 040a980) kernel .text 0x40100000 0x40242bb0 0x40242d30 0x403b3f54 0x40120339 do_softirq+0x109 (0xc, 0x403b3f84, 0x5eee0424, 0x180, 0x0) kernel .text 0x40100000 0x40120230 0x40120370 0x403b3f7c 0x4010b396 do_IRQ+0x106 (0x40107210, 0x5b06e000, 0x403b2000, 0x403b20 00, 0x403b2000) kernel .text 0x40100000 0x4010b290 0x4010b3b0 0x403b3fb8 0x4010de68 call_do_IRQ+0x5 (0x1, 0xa0600, 0x40105000) kernel .text 0x40100000 0x4010de63 0x4010de70 0x401072d2 cpu_idle+0x52 (0x40105090, 0x0, 0x10e00) kernel .text 0x40100000 0x40107280 0x401072f0 0x403b3fe0 0x40105044 stext+0x44 (0x402bd6c0, 0x40413440, 0x0, 0x40413340) kernel .text 0x40100000 0x40105000 0x40105060 0x403b3ff8 0x403b49ab start_kernel+0x15b kernel .text.init 0x403b4000 0x403b4850 0x403b4a10 Is this a known issue with TPROXY 2.0.5?

4 4

New TProxy snapshots
by KOVACS Krisztian 12 Dec '06

12 Dec '06

Hi, In preparation for the upcoming 2.0.6 release I've created new snapshots of the TProxy kernel patches. Some of the problems fixed: * an SMP race condition possibly leading to memory corruption in the nat_delete patch (reported and fixed by torinet.co.kr user wckwon) * a serious problem related to the TCP time-wait socket unassign callback leading to referencing invalid memory addresses There's also a patch for Linux 2.6.19 now, although it's compile-tested only. Downloads: http://people.balabit.hu/hidden/tproxy2-2.6.16_20061211.tar.bz2 http://people.balabit.hu/hidden/tproxy2-2.6.16_20061211.tar.bz2.asc http://people.balabit.hu/hidden/tproxy2-2.6.17_20061211.tar.bz2 http://people.balabit.hu/hidden/tproxy2-2.6.17_20061211.tar.bz2.asc http://people.balabit.hu/hidden/tproxy2-2.6.18_20061211.tar.bz2 http://people.balabit.hu/hidden/tproxy2-2.6.18_20061211.tar.bz2.asc http://people.balabit.hu/hidden/tproxy2-2.6.19_20061211.tar.bz2 http://people.balabit.hu/hidden/tproxy2-2.6.19_20061211.tar.bz2.asc -- Regards, Krisztian Kovacs

1 1

connection go to CLOSE_WAIT after sending FIN
by Eyal Rundstein 11 Dec '06

11 Dec '06

Hi, I am using kernel 2.4.32 with tproxy version 2.0.2. My client is a transparent proxy. My client opens a transparent connection to the server, sends a message and then closes the connection with FIN. The server replies with an ACK, WITHOUT sending a FIN. Now I see that the connection stays in the ip_conntrack table in CLOSE_WAIT state. During that time I can not reuse the connection. (SYNs to the same dest are not sent). 1) Isn't the correct behavior for that connection is to go to FIN_WAIT_2 state? Is it a bug? 2) The CLOSE_WAIT timeout is 500 seconds. Is there a way I can still open a new connection to the same destination? - When I use an older kernel (2.4.18) with old tproxy (version 23) I don't see this problem. Help is appreciated. Thanks in advance, E.R

3 4

RE: [tproxy] connection go to CLOSE_WAIT after sending FIN
by Eyal Rundstein 11 Dec '06

11 Dec '06

EXACTLY! The client SHOULD go to FIN_WAIT_2 but what I see is that it goes to CLOSE_WAIT That's the bug! -----Original Message----- From: Jan Engelhardt [mailto:jengelh@linux01.gwdg.de] Sent: Monday, December 11, 2006 4:09 PM To: Eyal Rundstein Cc: tproxy(a)lists.balabit.hu Subject: RE: [tproxy] connection go to CLOSE_WAIT after sending FIN >>>My client opens a transparent connection to the server, sends a >>>message and then closes the connection with FIN. The server replies >>>with an ACK, WITHOUT sending a FIN. >>>Now I see that the connection stays in the ip_conntrack table in >>>CLOSE_WAIT state. During that time I can not reuse the connection. >>>(SYNs to the same dest are not sent). >> >>This is correct, as per RFC 793 page 21 and page 39. >> >>>1) Isn't the correct behavior for that connection is to go to >>>FIN_WAIT_2 state? Is it a bug? >>>2) The CLOSE_WAIT timeout is 500 seconds. Is there a way I can still >>>open a new connection to the same destination? >> >>setsockopt SO_REUSEADDR may help. > >As far as I understand, the RFC 793 states that the correct behavior is >to go to state FIN_WAIT_2. >We go to CLOSE_WAIT only if we receive FIN and send ACK. >Page 21: >CLOSE-WAIT - represents waiting for a connection termination request > from the LOCAL user. >FIN-WAIT-2 - represents waiting for a connection termination request > from the REMOTE TCP. If the client sends a FIN (optionally with ACK), the server responds with ACK and goes into CLOSE-WAIT. The client goes to FIN-WAIT-2. The server may continue to send data. -`J' --

2 1

RE: [tproxy] connection go to CLOSE_WAIT after sending FIN
by Eyal Rundstein 11 Dec '06

11 Dec '06

As far as I understand, the RFC 793 states that the correct behavior is to go to state FIN_WAIT_2. We go to CLOSE_WAIT only if we receive FIN and send ACK. Page 21: CLOSE-WAIT - represents waiting for a connection termination request from the LOCAL user. FIN-WAIT-2 - represents waiting for a connection termination request from the REMOTE TCP. E.R -----Original Message----- From: Jan Engelhardt [mailto:jengelh@linux01.gwdg.de] Sent: Monday, December 11, 2006 2:38 PM To: Eyal Rundstein Cc: tproxy(a)lists.balabit.hu Subject: Re: [tproxy] connection go to CLOSE_WAIT after sending FIN >Hi, >I am using kernel 2.4.32 with tproxy version 2.0.2. >My client is a transparent proxy. >My client opens a transparent connection to the server, sends a message >and then closes the connection with FIN. The server replies with an >ACK, WITHOUT sending a FIN. >Now I see that the connection stays in the ip_conntrack table in >CLOSE_WAIT state. During that time I can not reuse the connection. >(SYNs to the same dest are not sent). This is correct, as per RFC 793 page 21 and page 39. >1) Isn't the correct behavior for that connection is to go to >FIN_WAIT_2 state? Is it a bug? >2) The CLOSE_WAIT timeout is 500 seconds. Is there a way I can still >open a new connection to the same destination? setsockopt SO_REUSEADDR may help. >- When I use an older kernel (2.4.18) with old tproxy (version 23) I >don't see this problem. > >Help is appreciated. >Thanks in advance, >E.R > > -`J' --

2 1

Using tproxy with squid on another box
by queronemsaber＠flash.tv.br 05 Dec '06

05 Dec '06

Hello! Is it possible to run tproxy with the squid server in another box? I have this situation: [Router]-----[FIREWALL]------SWITCH | +------ PROXY SERVER (internal IP address) The firewall and proxy servers are Linux. The tproxy works perfectly when squid runs on the same server as the firewall, but I couldn't figure out how to run it when squid is on a separate box. I installed everything and tried the iptables rule on firewall: iptables -t tproxy -A PREROUTING -i eth3 -p tcp -m tcp --dport 80 -j TPROXY --on-port 3128 --on-ip INTERNAL_IP_OF_THE_PROXY The connection works just fine, but I got several: Dec 5 11:25:17 cache squid[217]: tproxy ip=xxx.xxx.xxx.xxx,0xcc78e7c8,port=0 ERROR ASSIGN I have tcp_outgoing_address on squid.conf set to the internal IP of the proxy server. Anyone could help me? Thanks! ---------------------------------------------------------- Este e-mail foi enviado pelo Flash Mail. http://www.flash.tv.br

3 2

New Linux 2.6 tproxy snapshots
by KOVACS Krisztian 05 Dec '06

05 Dec '06

Hi, Yesterday we've finally found the cause of a serious problem in TProxy, which occasionally led to connections being NAT-ted to bogus addresses on SMP hardware. It was a really nasty problem with a dead-simple solution: the removal of a single 'static' keyword... Yeah, it was a typical bang-your-head-against-the-wall problem. :) Because of this and a few other (less significant) changes made in the last months I'm goint to publish a new "official" release next week. For the impatient, I've created new tproxy snapshot tarballs: http://people.balabit.hu/hidden/tproxy2-2.6.16_20060908.tar.bz2 http://people.balabit.hu/hidden/tproxy2-2.6.17_20060908.tar.bz2 http://people.balabit.hu/hidden/tproxy2-2.6.18_20060908.tar.bz2 -- Regards, Krisztian Kovacs

3 6

Unprotected critical section in tproxy-patched ip_nat_core.c
by wckwon 01 Dec '06

01 Dec '06

While I load-test for MAX-OPEN-SESSION, I found a bug that some critical section was not protected by lock. ---------------------------------------------------------------------------- ----------------------- Some Histories. I've used TPROXY for changing source IP address. To increase MAX-OPEN-SESSION, I assign several IP addresses to interface and I manage port number pools per one IP. whenever it bind() before 'setsockopt(IPT_ASSIGN)', it assigns unused IP:PORT pair from pool. And then, I can connect to one server many session that exceed the number of "/proc/sys/net/ipv4/ip_local_port_range". I could succeed to make 200000 connections to one server. But when I close all the session at the same moment, I found the kernel BUG message. ---------------------------------------------------------------------------- ----------------------- TEST Environment CPU : XEON3.0 x 2 (64bit) OS : kernel-2.6.18-1.2679.fc6.src.rpm + cttproxy-2.6.18-2.0.5.tar.gz ETC : used Bridge interface. ---------------------------------------------------------------------------- ----------------------- Kernel BUG message Kernel BUG at include/linux/list.h:167 invalid opcode: 0000 [3] SMP last sysfs file: /class/net/br0/bridge/topology_change_detected CPU 2 Modules linked in: ipt_REDIRECT(U) xt_tcpudp(U) iptable_nat(U) iptable_filter(U) iptable_tproxy(U) ip_nat(U) ip_tables(U) ip_conntrack(U) nfnetlink(U) ipt_TPROXY(U) x _tables(U) ehci_hcd(U) piix(U) usbcore(U) Pid: 1802, comm: heimdall Not tainted 2.6.15-prep #3 RIP: 0010:[<ffffffff8804e1fd>] <ffffffff8804e1fd>{:ip_nat:ip_nat_used_tuple+110} RSP: 0018:ffff81012ba09808 EFLAGS: 00010206 RAX: 000000000000159e RBX: ffff8101153b9aa0 RCX: ffff8101153b9b60 RDX: ffff81011a492ca8 RSI: ffff8101153b9ba8 RDI: ffffffff88045e00 RBP: 0000000000000000 R08: 000000000001a63a R09: 000000003da86da6 R10: 0000000080000000 R11: ffffffff8803ac88 R12: ffff8101153b9be8 R13: ffffffff88059a00 R14: ffff81012ba098bc R15: 0000000000000000 FS: 0000000048294950(0063) GS:ffff81013fc6f940(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00002aab38305020 CR3: 0000000132497000 CR4: 00000000000006e0 Process heimdall (pid: 1802, threadinfo ffff81012ba08000, task ffff81013c9ce1c0) Stack: ffff50002902a8c0 010626126b00a8c0 0000000000005000 0000000000000001 0000000000001226 ffffffff8804f5f8 ffff81010eebf4f8 ffff8101153b9be8 ffff81012ba098b8 ffffffff88052680 Call Trace: <ffffffff8804f5f8>{:ip_nat:tcp_unique_tuple+247} <ffffffff8804e71d>{:ip_nat:ip_nat_setup_info+796} <ffffffff8805587c>{:iptable_tproxy:ip_tproxy_setup_nat+223} <ffffffff8804e1af>{:ip_nat:ip_nat_used_tuple+32} <ffffffff88055183>{:iptable_tproxy:ip_tproxy_sockref_find_local+39} <ffffffff88055cf0>{:iptable_tproxy:ip_tproxy_fn+575} <ffffffff8804a7e6>{:ip_tables:ipt_do_table+751} <ffffffff8032fe51>{nf_iterate+65} <ffffffff80339b4d>{ip_finish_output+0} <ffffffff803300a1>{nf_hook_slow+88} <ffffffff80339b4d>{ip_finish_output+0} <ffffffff8033b1b1>{ip_output+159} <ffffffff8033aa4d>{ip_queue_xmit+1127} <ffffffff803329cc>{__ip_route_output_key+2134} <ffffffff801488ec>{__alloc_pages+87} <ffffffff80369349>{xfrm_lookup+60} <ffffffff80348e23>{tcp_transmit_skb+1552} <ffffffff8034af19>{tcp_connect+699} <ffffffff8034e291>{tcp_v4_connect+1343} <ffffffff8031a6e7>{lock_sock+175} <ffffffff80358bc9>{inet_stream_connect+148} <ffffffff8033d152>{inet_bind_bucket_create+21} <ffffffff8033ee68>{inet_csk_get_port+492} <ffffffff8031a5a1>{release_sock+19} <ffffffff80319a7e>{sys_connect+118} <ffffffff8033fd43>{tcp_setsockopt+29} <ffffffff80318e29>{sockfd_lookup+12} <ffffffff8031922e>{sys_setsockopt+149} <ffffffff8010a816>{system_call+126} Nov 21 20:29:21 is4 kernel: Code: 0f 0b 68 a2 f9 04 88 c2 a7 00 48 8b 46 b8 48 39 48 08 74 0a rip <ffffffff8804e1fd>{:ip_nat:ip_nat_used_tuple+110} RSP <ffff81012ba09808> ---------------------------------------------------------------------------- ----------------------- I think it happens while kernel is about to use conntrack node that is in the TIME_WAIT state. I found something bad in ip_nat_core.c so I added another lock like this. write_lock_bh(&__ip_nat_lock2); /*** ADD ****/ h = ip_conntrack_tuple_taken(&reply, ignored_conntrack); #if defined(CONFIG_IP_NF_TPROXY) || defined (CONFIG_IP_NF_TPROXY_MODULE) /* check if that conntrack is marked MAY_DELETE, if so, get rid of it... */ if ((h != NULL) && (ctrack = tuplehash_to_ctrack(h)) && test_bit(IPS_MAY_DELETE_BIT, &ctrack->status)) { DEBUGP("Deleting old conntrack entry for NAT\n"); __ip_nat_cleanup_conntrack(ctrack); ctrack->status &= ~IPS_NAT_DONE_MASK; if (del_timer(&ctrack->timeout)) { if (ctrack->timeout.function) { ctrack->timeout.function((unsigned long)ctrack); } } h = NULL; } write_unlock_bh(&__ip_nat_lock2); /*** ADD ****/ #endif And I also added __ip_nat_lock2 in other part that uses "__ip_nat_cleanup_conntrack(ctrack)". It works well until now. ---------------------------------------------------------------------------- ----------------------- Question : Do you have any other TIPs to increase MAX-OPEN-SESSION?

2 1