tproxy bind failed - limit of TPROXY bind
The tproxy bind is limit to only about 28219 socket I use TPROXY-4 (with the latest patch including latest udp patch too) Linux version 2.6.28.9 This is some sample test case: #define IP_TRANSPARENT 19 int main(){ int i,fd[100000],flag=1; struct sockaddr_in local_addr; bzero(&local_addr,sizeof(local_addr)); local_addr.sin_family = AF_INET; local_addr.sin_addr.s_addr=0xd0000000; for(i=0;i<100000;i++){ if((fd[i]=socket(AF_INET,SOCK_STREAM,0))<0){ printf("creating socket %d failed\n",i); exit(0); } if(setsockopt(fd[i],SOL_IP,IP_TRANSPARENT,&flag,sizeof(int))<0){ printf("set transparent to the %d socket (fd=%d) failed\n",i,fd[i]); exit(0); } local_addr.sin_port=0; if(bind(fd[i],(struct sockaddr *)&local_addr,sizeof(local_addr))<0){ perror("bind - error"); printf("bind for the %d socket (fd=%d) failed\nipport - %s:%d\n",i,fd[i],inet_ntoa(local_addr.sin_addr),htons(local_addr.sin_port)); exit(0); } local_addr.sin_addr.s_addr++; } } The result: bind - error: Address already in use bind for the 28223 socket (fd=28226) failed ipport - 63.110.0.208:0 Why this low limit? Can I configure something to by-pass it? If not, have chance to fix this bug?
Hi, On sze, máj 06, 2009 at 02:41:38 +0300, elyasaf wrote:
The result:
bind - error: Address already in use
bind for the 28223 socket (fd=28226) failed
ipport - 63.110.0.208:0
Why this low limit? Can I configure something to by-pass it? If not, have chance to fix this bug?
I don't think this has anything to do with tproxy. Each and every bind requires a free port -- so the 100000 iterations you're doing in your test will *never* gonna work. Furthermore, automatically selected ports are selected from a limited subset of ports, which is tunable with a sysctl setting. http://href.hu/x/8y75 As you can see, the default setting for the local port range is 32768 - 61000, so I'm fairly sure this is the limit you're reaching. But again, I doubt this has anything to do with tproxy. I suggest consulting a good introductory book on TCP/IP and Linux networking. -- KOVACS Krisztian
Thank you for your response I have more than 2Gbits (all tcp and udp via tproxy) on my networks, with more than 20k different active ips I am not using the same ip twice (local_addr.sin_addr.s_addr++;) Than I expected tproxy dont mix the bind, and treat them like different interface And my sample code, I think approve it -----Original Message----- From: KOVACS Krisztian [mailto:hidden@sch.bme.hu] Sent: Wednesday, May 06, 2009 6:43 PM To: elyasaf Cc: tproxy@lists.balabit.hu Subject: Re: [tproxy] tproxy bind failed - limit of TPROXY bind Hi, On sze, máj 06, 2009 at 02:41:38 +0300, elyasaf wrote:
The result:
bind - error: Address already in use
bind for the 28223 socket (fd=28226) failed
ipport - 63.110.0.208:0
Why this low limit? Can I configure something to by-pass it? If not, have chance to fix this bug?
I don't think this has anything to do with tproxy. Each and every bind requires a free port -- so the 100000 iterations you're doing in your test will *never* gonna work. Furthermore, automatically selected ports are selected from a limited subset of ports, which is tunable with a sysctl setting. http://href.hu/x/8y75 As you can see, the default setting for the local port range is 32768 - 61000, so I'm fairly sure this is the limit you're reaching. But again, I doubt this has anything to do with tproxy. I suggest consulting a good introductory book on TCP/IP and Linux networking. -- KOVACS Krisztian
Hi, On Wed, May 06, 2009 at 08:35:19PM +0300, elyasaf wrote:
Thank you for your response
I have more than 2Gbits (all tcp and udp via tproxy) on my networks, with more than 20k different active ips
I am not using the same ip twice (local_addr.sin_addr.s_addr++;) Than I expected tproxy dont mix the bind, and treat them like different interface And my sample code, I think approve it
Indeed, sorry. Have you tried removing the setsockopt(IP_TRANSPARENT) call from you program? I'm asking because tproxy should not make a difference in how bind() behaves (none of the tproxy patches changed that part of the kernel). -- KOVACS Krisztian
You are right, its seems like linux bug I see a patch that fixes it to free bsd, I cant find one to linux too This limit the traffic to only 400mbits for each machine, it is a pity... -----Original Message----- From: KOVACS Krisztian [mailto:hidden@sch.bme.hu] Sent: Thursday, May 07, 2009 12:14 PM To: elyasaf Cc: 'KOVACS Krisztian'; tproxy@lists.balabit.hu Subject: Re: [tproxy] tproxy bind failed - limit of TPROXY bind Hi, On Wed, May 06, 2009 at 08:35:19PM +0300, elyasaf wrote:
Thank you for your response
I have more than 2Gbits (all tcp and udp via tproxy) on my networks, with more than 20k different active ips
I am not using the same ip twice (local_addr.sin_addr.s_addr++;) Than I expected tproxy don't mix the bind, and treat them like different interface And my sample code, I think approve it
Indeed, sorry. Have you tried removing the setsockopt(IP_TRANSPARENT) call from you program? I'm asking because tproxy should not make a difference in how bind() behaves (none of the tproxy patches changed that part of the kernel). -- KOVACS Krisztian
Hi, On cs, máj 07, 2009 at 07:16:44 +0300, elyasaf wrote:
You are right, its seems like linux bug I see a patch that fixes it to free bsd, I cant find one to linux too This limit the traffic to only 400mbits for each machine, it is a pity...
Can you post the output of netstat -antp taken when the problem occurs (but the fds are still open)? -- KOVACS Krisztian
On Fri, 2009-05-08 at 10:59 +0200, KOVACS Krisztian wrote:
Hi,
On cs, máj 07, 2009 at 07:16:44 +0300, elyasaf wrote:
You are right, its seems like linux bug I see a patch that fixes it to free bsd, I cant find one to linux too This limit the traffic to only 400mbits for each machine, it is a pity...
Can you post the output of netstat -antp taken when the problem occurs (but the fds are still open)?
the problem is that autobinding in the linux kernel does not take the IP address into account, thus if a given port is used on _any_ of the IP addresses, it's not going to be used if you call bind() with port=0. The only solution I came up with was to find the first available port from userspace. And/or possibly forwardport this patch: http://www.ioremap.net/node/104 -- Bazsi
participants (3)
-
Balazs Scheidler
-
elyasaf
-
KOVACS Krisztian