[tproxy] TPROXY_ASSIGN fails with File exists

[Balazs Scheidler]

On Thu, 2008-01-24 at 11:50 -0800, Pranav Desai wrote:
> Hello Balazs,
> 
> Thanks for the quick reply.
> 
> On Jan 24, 2008 1:29 AM, Balazs Scheidler <bazsi at balabit.hu> wrote:
> >
> > On Wed, 2008-01-23 at 16:40 -0800, Pranav Desai wrote:
> > > Hello All,
> > >
> > > I am testing with the example code foreign-tcp-connect on
> > > kernel-2.6.20.15/cttproxy-2.0.6
> > > When I run the sample in quick successions, I get this message in
> > > /var/log/messages and program returns with errno 17 (File exists).
> > >
> > > IP_TPROXY: socket already assigned, reuse=0, e101a8c0:0f27,
> > > sr->faddr=c000010a:d007, flags=210000,
> > > sr->tv_hashed=1201103313:191693969
> > >
> > > Is it ok to add a check to ignore this ?
> > >
> >
> > This means that the local IP:port pair is already present in the tproxy
> > hash table. This is actually not very good, the assignment should be
> > deleted as soon as the socket is closed.
> >
> > However tracking down these cases proved to be the very problem why we
> > wanted to avoid the current NAT based implementation in the first place.
> > Using this tproxy hash is inherently racy and although we've managed to
> > get it reasonably stable, we encountered similar cases every now and
> > then. And it's very difficult to track down why it happens.
> >
> 
> Is it possible to not use the cache ?. If so what kind of performance
> hit would that be.

This is not a cache. It is inherent in tproxy 2, it contains the mapping
between the local IP:port pairs to the foreign IP:port pairs.

This is the mapping used to enact NATing of the outgoing/incoming
traffic.

> 
> > The information in the log message is follows:
> >  * SO_REUSEADDR was not set for the socket (I can't remember if it's the
> > old or the new socket)
> >  * the local address is 192.168.1.225, port number 9999 or 3879 (can't
> > remember the byte order, it's probably 9999)
> 
> it is 9999.
> 
> >  * the requested foreign address is 10.1.0.192, port number 2000 or
> > 533255
> >  * flags is a bitmask of IPT* values, defined at the beginning of the
> > iptable_tproxy file
> >  * the sr->tv_hashed value is the exact time of the previous
> > registration
> >
> > The problem happens probably because the application requests the same
> > local ip:port, and the previous entry was not yet cleaned up, probably
> > because the kernel side of the socket still exists.
> >
> > Use automatically assigned port numbers and not fixed ones and then it
> > should work.
> 
> We are using tproxy in our proxy server application and we already use
> automatically assigned ports, but its possible that the socket still
> exists, since this seems to happen when there are a few requests
> coming in almost simultaneously. Ours is an event based system using
> epoll.

TProxy cleans up an entry in the tproxy hash whenever an application
calls close() on the socket, and the socket structure is dropped from
the socket hash in the kernel. There might be some time between close()
and the unhashing, but the kernel ensures that it does not reallocate
the same port as long as the old one is still in the hashtable.

If your application does not allocate ports by passing '0' to bind, it
might step on the same local IP:port pair, so generally to make it
reasonably stable you need to ensure that some time passes between port
reallocations.

We did this by using a port "rover" that scans the port range and always
increases sequentially. Using port randomization might cause problems
here.

> 
> The other thing I noticed was that the connect() return EINPROGRESS
> the first time (its non-blocking), which seems normal, and fails with
> EADDRNOTAVAIL the subsequent times for the same src and dest IP pair.

If you use the same tuple (src IP,port -> dst IP:port) then you might
get an EADDRNOTAVAIL from the kernel, this is however not tproxy's
doing.

If you specify SO_REUSEADDR (which you probaly do), you can bind to the
same port but you get an error when you connect.

> 
> Would the new version of tproxy be any better ?

Absolutely. It is a bit volatile now though. There are basically two
versions:
 * tproxy4 released by Panther on BalaBit website is the version we're
going to use in our product
 * tproxy4.1 released by Hidden on his personal homepage is the version
destined for kernel inclusion. This one changes frequently as the kernel
maintainers suggest things.

I'd go for tproxy4 now, and switch to the kernel implementation once it
gets there.

TProxy4 is better as there tproxy does not rely on NAT, but changes the
IP stack to support transparency. No races there and loading
conntrack/nat is not needed.

-- 
Bazsi