Hi I am experiencing an intermittent but quite frequent problem when load testing a transparent proxy (squid) (with 100's of connections). I would welcome any advice on what I might be doing wrong. The end result is that there are lot of: * server-side connections left in TIME_WAIT state * client-side connections left in LAST_ACK state I am using Iptables to: * redirect/tproxy incoming port 80 traffic to port 3129 * set a mark on outgoing port 80 traffic (for routing) I also use the TRACE target in the raw table to log the packet paths. The problem seems to arise when the server initiates the close of the connection. In that case, the trace output shows: 1. the FIN from the server passing through the mangle:OUTPUT, filter:OUTPUT and filter:POSTROUTING tables 2. a FIN/ACK from the client arriving and passing through mangle:PREROUTING, mangle:INPUT and filter:INPUT tables 3. a final ACK from the server passing through the mangle:OUTPUT tables but getting no further. Steps 2 & 3 are repeated as the client resends its unacknowledged FIN. I am attaching the packet trace for one instance of this problem, extracted from /var/log/messages. I have used tcpdump on both client and server and confirmed that the final ACK never leaves the server. I have enabled logging of invalid packets, but am not seeing any reports. This problem is happening on Fedora 14 - 2.6.35.14-103.fc14.x86_64, with iptables v1.4.9. I don't see the problem with non-transparent connections. I am also attaching iptables rules for the mangle table and tcp-related sysctl settings. Regards Simon
Hi, On Mon 14 Nov 2011 12:57:45 PM CET, Simon James wrote:
The problem seems to arise when the server initiates the close of the connection. In that case, the trace output shows:
1. the FIN from the server passing through the mangle:OUTPUT, filter:OUTPUT and filter:POSTROUTING tables 2. a FIN/ACK from the client arriving and passing through mangle:PREROUTING, mangle:INPUT and filter:INPUT tables 3. a final ACK from the server passing through the mangle:OUTPUT tables but getting no further.
This might be related to a problem we've fixed about a month ago in the upstream kernel: https://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h... The fix made it into Linux 3.1. Can you somehow give it a try? (Backporting to your F14 kernel should be fairly trivial, since it's a one-line change in tcp_minisocks.c.) -- KOVACS Krisztian
Hi Thanks for the quick response! I've applied the patch but I'm still getting the problem. I'm progressing the diagnosis with printk() etc, and will update this post if and when I get any further. Thanks again. Simon On 15/11/2011 09:11, "KOVACS Krisztian" <hidden@balabit.hu> wrote: Hi, On Mon 14 Nov 2011 12:57:45 PM CET, Simon James wrote:
The problem seems to arise when the server initiates the close of the connection. In that case, the trace output shows:
1. the FIN from the server passing through the mangle:OUTPUT, filter:OUTPUT and filter:POSTROUTING tables 2. a FIN/ACK from the client arriving and passing through mangle:PREROUTING, mangle:INPUT and filter:INPUT tables 3. a final ACK from the server passing through the mangle:OUTPUT tables but getting no further.
This might be related to a problem we've fixed about a month ago in the upstream kernel: https://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h... The fix made it into Linux 3.1. Can you somehow give it a try? (Backporting to your F14 kernel should be fairly trivial, since it's a one-line change in tcp_minisocks.c.) -- KOVACS Krisztian
Just an interim update on this. I have established that the problem arises with the following packet sequence: 1. Tproxy-Server -> Client FIN 2. Client->Tproxy-Server ACK (of the server's FIN) 3. Client->Tproxy-Server FIN 4. Tproxy-Server-Client ACK (of the client's FIN - this never leaves the server) Steps 3 & 4 are repeated. The problem does NOT arise with the sequence: 1. Tproxy-Server -> Client FIN 2. Client->Tproxy-Server ACK (of the server's FIN) / FIN 3. Tproxy-Server-Client ACK (of the client's FIN) The search continues... On 15/11/2011 15:20, "Simon James" <sjames@btisystems.com> wrote: Hi Thanks for the quick response! I've applied the patch but I'm still getting the problem. I'm progressing the diagnosis with printk() etc, and will update this post if and when I get any further. Thanks again. Simon On 15/11/2011 09:11, "KOVACS Krisztian" <hidden@balabit.hu> wrote: Hi, On Mon 14 Nov 2011 12:57:45 PM CET, Simon James wrote:
The problem seems to arise when the server initiates the close of the connection. In that case, the trace output shows:
1. the FIN from the server passing through the mangle:OUTPUT, filter:OUTPUT and filter:POSTROUTING tables 2. a FIN/ACK from the client arriving and passing through mangle:PREROUTING, mangle:INPUT and filter:INPUT tables 3. a final ACK from the server passing through the mangle:OUTPUT tables but getting no further.
This might be related to a problem we've fixed about a month ago in the upstream kernel: https://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h... The fix made it into Linux 3.1. Can you somehow give it a try? (Backporting to your F14 kernel should be fairly trivial, since it's a one-line change in tcp_minisocks.c.) -- KOVACS Krisztian
participants (2)
-
KOVACS Krisztian
-
Simon James