[tproxy] tproxy performance

[Balazs Scheidler]

On Tue, 2008-10-14 at 09:15 -0400, Ravi Valmikam wrote:
> This is the setup:
> 
> There are 3 PCs. 
> 
> 1. Client
> 2. A custom TCP proxy based on 2.6.24 + tproxy
> 3. Server
> 
> All pretty high end machines.
> 
> Test sequence:
> 
> Step A. I send a data packet on already established TCP packet. Size
> is an MTU from machine 1.
> Step B. Machine 2 receives it and proxies it.
> Step C. Machine 3 receives it and echoes it back.
> 
> We measure the on-the-wire time in and out of machine (2).
> 
> In pure ipforwarding mode on machine 2, we see 2-3 ms latency.
> 
> In tproxy mode on machine 2, we see 5-6 ms latency (additional 3ms).
> 
> I hope that clarifies the scenario.

Well proxying in userspace definitely adds overheads over simple packet
forwarding:
 * instead of simple packet forwarding, the packets are processed by the
local TCP stack and queued to a socket buffer
 * the userspace application wakes up if there's traffic, since this is
in userspace, it takes some time (remember packet forwarding is all
performed in softirq context, whereas the userspace proxies have to
react on data stored in a socket buffer by the same softirq contex)
 * the userspace application copies data from the receiving socket
buffer and puts that data in the target socket buffer, that copies the
data twice (whereas IP forwarding does not copy packets, except for
maybe in the ethernet driver, but that applies to TCP proxies too)

So there's plenty of overhead, but that's the price of the increased
flexibility you pay for the flexibility that you can do in
userspace/proxies.
 
So the latency is not tproxy specific, it's proxying specific.

If you want to measure tproxy performance, you should measure the
difference between a simple non-transparent proxy and one where tproxy
redirects traffic. I doubt there'd be too much of a difference.

-- 
Bazsi