[tproxy] TPROXY_ASSIGN fails with File exists

[Balazs Scheidler]

On Fri, 2008-01-25 at 15:19 -0800, Pranav Desai wrote:
> On Jan 25, 2008 1:19 AM, Balazs Scheidler <bazsi at balabit.hu> wrote:
> >
> > On Thu, 2008-01-24 at 11:50 -0800, Pranav Desai wrote:
> > > Hello Balazs,
> > >
> > > Thanks for the quick reply.
> > >
> > > On Jan 24, 2008 1:29 AM, Balazs Scheidler <bazsi at balabit.hu> wrote:
> > > >
> > > > On Wed, 2008-01-23 at 16:40 -0800, Pranav Desai wrote:
> > > > > Hello All,
> > > > >
> > > > > I am testing with the example code foreign-tcp-connect on
> > > > > kernel-2.6.20.15/cttproxy-2.0.6
> > > > > When I run the sample in quick successions, I get this message in
> > > > > /var/log/messages and program returns with errno 17 (File exists).
> > > > >
> > > > > IP_TPROXY: socket already assigned, reuse=0, e101a8c0:0f27,
> > > > > sr->faddr=c000010a:d007, flags=210000,
> > > > > sr->tv_hashed=1201103313:191693969
> > > > >
> > > > > Is it ok to add a check to ignore this ?
> > > > >
> > > >
> > > > This means that the local IP:port pair is already present in the tproxy
> > > > hash table. This is actually not very good, the assignment should be
> > > > deleted as soon as the socket is closed.
> > > >
> > > > However tracking down these cases proved to be the very problem why we
> > > > wanted to avoid the current NAT based implementation in the first place.
> > > > Using this tproxy hash is inherently racy and although we've managed to
> > > > get it reasonably stable, we encountered similar cases every now and
> > > > then. And it's very difficult to track down why it happens.
> > > >
> > >
> > > Is it possible to not use the cache ?. If so what kind of performance
> > > hit would that be.
> >
> > This is not a cache. It is inherent in tproxy 2, it contains the mapping
> > between the local IP:port pairs to the foreign IP:port pairs.
> >
> > This is the mapping used to enact NATing of the outgoing/incoming
> > traffic.
> >
> > >
> > > > The information in the log message is follows:
> > > >  * SO_REUSEADDR was not set for the socket (I can't remember if it's the
> > > > old or the new socket)
> > > >  * the local address is 192.168.1.225, port number 9999 or 3879 (can't
> > > > remember the byte order, it's probably 9999)
> > >
> > > it is 9999.
> > >
> > > >  * the requested foreign address is 10.1.0.192, port number 2000 or
> > > > 533255
> > > >  * flags is a bitmask of IPT* values, defined at the beginning of the
> > > > iptable_tproxy file
> > > >  * the sr->tv_hashed value is the exact time of the previous
> > > > registration
> > > >
> > > > The problem happens probably because the application requests the same
> > > > local ip:port, and the previous entry was not yet cleaned up, probably
> > > > because the kernel side of the socket still exists.
> > > >
> > > > Use automatically assigned port numbers and not fixed ones and then it
> > > > should work.
> > >
> > > We are using tproxy in our proxy server application and we already use
> > > automatically assigned ports, but its possible that the socket still
> > > exists, since this seems to happen when there are a few requests
> > > coming in almost simultaneously. Ours is an event based system using
> > > epoll.
> >
> > TProxy cleans up an entry in the tproxy hash whenever an application
> > calls close() on the socket, and the socket structure is dropped from
> > the socket hash in the kernel. There might be some time between close()
> > and the unhashing, but the kernel ensures that it does not reallocate
> > the same port as long as the old one is still in the hashtable.
> >
> > If your application does not allocate ports by passing '0' to bind, it
> > might step on the same local IP:port pair, so generally to make it
> > reasonably stable you need to ensure that some time passes between port
> > reallocations.
> >
> 
> Thanks for your help, we screwed up in our local ports and were using
> the same ports in some cases.
> 
> One last question, is there any performance advantage of using tproxy4 ?
> 

Sure there is, you don't need to load conntrack/nat, doing a NAT mapping
requires the following additional steps:

for each new connection:
 * look up in the tproxy hash
 * setup a nat mapping

for each packet:
 * look up conntrack entry
 * perform NAT translation


-- 
Bazsi