[tproxy] TIME_WAIT / LAST_ACK Problem

[Simon James]

Hi

I am experiencing an intermittent but quite frequent problem when load testing a transparent proxy (squid) (with 100's of connections).
I would welcome any advice on what I might be doing wrong.

The end result is that there are lot of:

 *   server-side connections left in TIME_WAIT state
 *   client-side connections left in LAST_ACK state

I am using Iptables to:

 *   redirect/tproxy incoming port 80 traffic to port 3129
 *   set a mark on outgoing port 80 traffic (for routing)

I also use the TRACE target in the raw table to log the packet paths.

The problem seems to arise when the server initiates the close of the connection.
In that case, the trace output shows:

 1.  the FIN from the server passing through the mangle:OUTPUT, filter:OUTPUT and filter:POSTROUTING tables
 2.  a FIN/ACK from the client arriving and passing through mangle:PREROUTING, mangle:INPUT and filter:INPUT tables
 3.  a final ACK from the server passing through the mangle:OUTPUT tables but getting no further.

Steps 2 & 3 are repeated as the client resends its unacknowledged FIN.
I am attaching the packet trace for one instance of this problem, extracted from /var/log/messages.

I have used tcpdump on both client and server and confirmed that the final ACK never leaves the server.

I have enabled logging of invalid packets, but am not seeing any reports.

This problem is happening on Fedora 14 - 2.6.35.14-103.fc14.x86_64, with iptables v1.4.9.

I don't see the problem with non-transparent connections.

I am also attaching iptables rules for the mangle table and tcp-related sysctl settings.

Regards

Simon


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/tproxy/attachments/20111114/8d358d34/attachment-0001.htm 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: packet_trace
Type: application/octet-stream
Size: 22041 bytes
Desc: packet_trace
Url : http://lists.balabit.hu/pipermail/tproxy/attachments/20111114/8d358d34/attachment-0003.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: iptables_rules
Type: application/octet-stream
Size: 1202 bytes
Desc: iptables_rules
Url : http://lists.balabit.hu/pipermail/tproxy/attachments/20111114/8d358d34/attachment-0004.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tcp_settings
Type: application/octet-stream
Size: 2283 bytes
Desc: tcp_settings
Url : http://lists.balabit.hu/pipermail/tproxy/attachments/20111114/8d358d34/attachment-0005.obj