[tproxy] Need your input regarding some issue with tproxy (may not be related to tproxy)

[Balazs Scheidler]

On Thu, 2008-10-16 at 10:33 -0700, Pranav Desai wrote:
> On Thu, Oct 16, 2008 at 5:37 AM, Balazs Scheidler <bazsi at balabit.hu> wrote:
> > On Wed, 2008-10-15 at 23:22 -0700, Pranav Desai wrote:
> >> On Wed, Oct 15, 2008 at 5:29 PM, Pranav Desai <pranavadesai at gmail.com> wrote:
> >> > Hello,
> >> >
> >> > We have a http proxy server which is in full transparent mode using
> >> > tproxy 2.0.6 + kernel 2.6.20.15.
> >> >
> >> > The iptables rule to redirect port 80 traffic from clients is:
> >> >
> >> >  574K   34M REDIRECT   tcp  --  *      *       0.0.0.0/0
> >> > 0.0.0.0/0           tcp dpts:80:82 redir ports 8001
> >> >
> >> > 8001 is our proxy port. We use tproxy to send connections to web
> >> > servers using the clients IP. Using setsockopt(TPROXY_ASSIGN) ...
> >> >
> >> > The server is at around 240req/s. Most of the time its working fine
> >> > i.e. I see a lot of traffic on port 80 going out using the clients IP,
> >> > but I see quite a few pkts going out using src port 8001 (trace
> >> > below). The rate is not as high port 80 traffic, but still high enough
> >> > to be a concern.
> >> >
> >> > The interesting thing is I don't see any pkts destined to port 8001. I
> >> > have been monitoring this server for an entire day. Hence, I am
> >> > guessing that at times for some reason tproxy is not able to assign
> >> > the foreign address and just uses the local address:port ...
> >> >
> >> > Any ideas or suggestions on how I should go about debugging this. Or
> >> > whether I should be looking somewhere other than the tproxy module.
> >> >
> >> > Its a production server, so I can't get the client side traces, but I
> >> > can get more info from the server. Let me know if you need any other
> >> > information.
> >> >
> >> > I would really appreciate any help I can get.
> >> >
> >> > Thanks
> >> > -- Pranav
> >> >
> >>
> >>
> >> The 10.10.224.6 is the server IP running tproxy.
> >> The 10.1.x.x addresses are the client address.
> >>
> >> So the return traffic or the response seems to be having the problem
> >> where sometimes the src is 10.10.224.6:8001 instead of the origin
> >> server IP:PORT.
> >
> > Are you using bridging as well? Do you have CONFIG_NETFILTER_BRIDGE
> > enabled in your kernel?
> >
> 
> There is no bridging, only bonding.
> 
> > because in that case tcpdump will see the unnated traffic, bridging
> > plays some nasty games with netfilter.
> >
> we thought that tcpdump might be screwing up somewhere, but even on
> external network elements (load balancer) we don't see anything going
> to the tproxy server on port 8001.
> 
> > On the client side of the proxy, the TPROXY_ASSIGN stuff does not really
> > matter, only when going to the server.
> >
> 
> Hmm. so does the netfilter nat rules control the client-side, and
> could that be screwing up somehow. Can the conntrack table be of some
> assistance here ?

Yes, it'd be useful to see the conntrack entry of the invalid packets.

-- 
Bazsi