<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:st1="urn:schemas-microsoft-com:office:smarttags" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 11 (filtered medium)">
<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="time"/>
<!--[if !mso]>
<style>
st1\:*{behavior:url(#default#ieooui) }
</style>
<![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
        {font-family:바탕;
        panose-1:2 3 6 0 0 1 1 1 1 1;}
@font-face
        {font-family:굴림;
        panose-1:2 11 6 0 0 1 1 1 1 1;}
@font-face
        {font-family:"\@굴림";
        panose-1:2 11 6 0 0 1 1 1 1 1;}
@font-face
        {font-family:"\@바탕";
        panose-1:2 3 6 0 0 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
        {margin:0cm;
        margin-bottom:.0001pt;
        text-align:justify;
        text-justify:inter-ideograph;
        text-autospace:none;
        word-break:break-hangul;
        font-size:10.0pt;
        font-family:바탕;}
a:link, span.MsoHyperlink
        {color:blue;
        text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
        {color:purple;
        text-decoration:underline;}
span.EmailStyle17
        {mso-style-type:personal-compose;
        font-family:굴림;
        color:windowtext;}
/* Page Definitions */
@page Section1
        {size:595.3pt 841.9pt;
        margin:99.25pt 3.0cm 3.0cm 3.0cm;
        layout-grid:18.0pt;}
div.Section1
        {page:Section1;}
-->
</style>
</head>
<body lang=KO link=blue vlink=purple>
<div class=Section1 style='layout-grid:18.0pt'>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>While I load-test for MAX-OPEN-SESSION, <o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>I found a bug that some critical section was not protected by
lock.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>---------------------------------------------------------------------------------------------------<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>Some Histories.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>I've used TPROXY for changing source IP address.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>To increase MAX-OPEN-SESSION, I assign several IP addresses to
interface and I manage <o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>port number pools per one IP.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>whenever it bind() before 'setsockopt(IPT_ASSIGN)', it assigns
unused IP:PORT pair from pool.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>And then, I can connect to one server many session that exceed
the number of <o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>"/proc/sys/net/ipv4/ip_local_port_range". I could
succeed to make 200000 connections to one server.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>But when I close all the session at the same moment, I found
the kernel BUG message.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>---------------------------------------------------------------------------------------------------<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>TEST Environment<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>CPU : XEON3.0 x 2 (64bit)<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>OS : kernel-2.6.18-1.2679.fc6.src.rpm +
cttproxy-2.6.18-2.0.5.tar.gz<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>ETC : used Bridge interface.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>---------------------------------------------------------------------------------------------------<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>Kernel BUG message<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>Kernel BUG at include/linux/list.h:167<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>invalid opcode: 0000 [3] SMP<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>last sysfs file: /class/net/br0/bridge/topology_change_detected<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>CPU 2<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>Modules linked in: ipt_REDIRECT(U) xt_tcpudp(U) iptable_nat(U)
iptable_filter(U) iptable_tproxy(U) ip_nat(U) ip_tables(U) ip_conntrack(U)
nfnetlink(U) ipt_TPROXY(U) x<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>_tables(U) ehci_hcd(U) piix(U) usbcore(U)<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>Pid: 1802, comm: heimdall Not tainted 2.6.15-prep #3<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>RIP: 0010:[<ffffffff8804e1fd>]
<ffffffff8804e1fd>{:ip_nat:ip_nat_used_tuple+110}<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>RSP: 0018:ffff81012ba09808 EFLAGS: 00010206<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>RAX: 000000000000159e RBX: ffff8101153b9aa0 RCX:
ffff8101153b9b60<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>RDX: ffff81011a492ca8 RSI: ffff8101153b9ba8 RDI: ffffffff88045e00<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>RBP: 0000000000000000 R08: 000000000001a63a R09:
000000003da86da6<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>R10: 0000000080000000 R11: ffffffff8803ac88 R12:
ffff8101153b9be8<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>R13: ffffffff88059a00 R14: ffff81012ba098bc R15:
0000000000000000<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>FS: 0000000048294950(0063) GS:ffff81013fc6f940(0000)
knlGS:0000000000000000<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>CR2: 00002aab38305020 CR3: 0000000132497000 CR4:
00000000000006e0<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>Process heimdall (pid: 1802, threadinfo ffff81012ba08000, task ffff81013c9ce1c0)<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>Stack: ffff50002902a8c0 010626126b00a8c0 0000000000005000
0000000000000001<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'> 0000000000001226
ffffffff8804f5f8 ffff81010eebf4f8 ffff8101153b9be8<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'> ffff81012ba098b8
ffffffff88052680<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>Call Trace: <ffffffff8804f5f8>{:ip_nat:tcp_unique_tuple+247}<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>
<ffffffff8804e71d>{:ip_nat:ip_nat_setup_info+796}
<ffffffff8805587c>{:iptable_tproxy:ip_tproxy_setup_nat+223}<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>
<ffffffff8804e1af>{:ip_nat:ip_nat_used_tuple+32}
<ffffffff88055183>{:iptable_tproxy:ip_tproxy_sockref_find_local+39}<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>
<ffffffff88055cf0>{:iptable_tproxy:ip_tproxy_fn+575}<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>
<ffffffff8804a7e6>{:ip_tables:ipt_do_table+751}
<ffffffff8032fe51>{nf_iterate+65}<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>
<ffffffff80339b4d>{ip_finish_output+0}
<ffffffff803300a1>{nf_hook_slow+88}<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'> <ffffffff80339b4d>{ip_finish_output+0}
<ffffffff8033b1b1>{ip_output+159}<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>
<ffffffff8033aa4d>{ip_queue_xmit+1127}
<ffffffff803329cc>{__ip_route_output_key+2134}<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>
<ffffffff801488ec>{__alloc_pages+87}
<ffffffff80369349>{xfrm_lookup+60}<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>
<ffffffff80348e23>{tcp_transmit_skb+1552}
<ffffffff8034af19>{tcp_connect+699}<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>
<ffffffff8034e291>{tcp_v4_connect+1343}
<ffffffff8031a6e7>{lock_sock+175}<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>
<ffffffff80358bc9>{inet_stream_connect+148}
<ffffffff8033d152>{inet_bind_bucket_create+21}<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'> <ffffffff8033ee68>{inet_csk_get_port+492}
<ffffffff8031a5a1>{release_sock+19}<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>
<ffffffff80319a7e>{sys_connect+118}
<ffffffff8033fd43>{tcp_setsockopt+29}<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>
<ffffffff80318e29>{sockfd_lookup+12}
<ffffffff8031922e>{sys_setsockopt+149}<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>
<ffffffff8010a816>{system_call+126}<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>Nov 21 <st1:time o:ls="trans" Hour="20" Minute="29" w:st="on">20:29:21</st1:time>
is4 kernel:<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>Code: 0f 0b 68 a2 f9 04 88 c2 a7 00 48 8b 46 b8 48 39 48 08 74
0a<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>rip <ffffffff8804e1fd>{:ip_nat:ip_nat_used_tuple+110} RSP
<ffff81012ba09808><o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>---------------------------------------------------------------------------------------------------<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>I think it happens while kernel is about to use conntrack node
that is in the TIME_WAIT state.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>I found something bad in ip_nat_core.c so I added another lock
like this.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>
write_lock_bh(&__ip_nat_lock2); /*** ADD ****/<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'> h =
ip_conntrack_tuple_taken(&reply, ignored_conntrack);<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'> <o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>#if defined(CONFIG_IP_NF_TPROXY) || defined
(CONFIG_IP_NF_TPROXY_MODULE)<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'> /* check if that
conntrack is marked MAY_DELETE, if so, get rid of it... */<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'> if ((h != NULL)
&&<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'> (ctrack
= tuplehash_to_ctrack(h)) &&<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>
test_bit(IPS_MAY_DELETE_BIT, &ctrack->status)) {<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>
DEBUGP("Deleting old conntrack entry for NAT\n");<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'> <o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>
__ip_nat_cleanup_conntrack(ctrack);<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'> <o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>
ctrack->status &= ~IPS_NAT_DONE_MASK;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>
if (del_timer(&ctrack->timeout)) {<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>
if (ctrack->timeout.function) {<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>
ctrack->timeout.function((unsigned long)ctrack);<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>
}<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'> }<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>
h = NULL;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'> }<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'> write_unlock_bh(&__ip_nat_lock2);
/*** ADD ****/<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>#endif<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>And I also added __ip_nat_lock2 in other part that uses
"__ip_nat_cleanup_conntrack(ctrack)".<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>It works well until now.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>---------------------------------------------------------------------------------------------------<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'>Question : Do you have any other TIPs to increase
MAX-OPEN-SESSION?<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=굴림><span lang=EN-US style='font-size:10.0pt;
font-family:굴림'><o:p> </o:p></span></font></p>
</div>
</body>
</html>