We have a problem with the following scenario:
1) int fd=socket(AF_INET, SOCK_STREAM, 0)
make fd non-blocking
bind fd to a local address an port 0
setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, 1);
2) setsockopt(fd, SOL_IP, IP_TPROXY_ASSIGN, {client_ip, client_port});
setsockopt(fd, SOL_IP, IP_TPROXY_FLAGS, ITP_CONNECT | ITP_ONCE);
3) connect(fd, {backend_ip, backend_port}); connect() return -1, errno=EAGAIN;
4) send(fd, buffer, sizeof(buffer), 0); send returns -1, errno=ECONNREFUSED;
5) setsockopt(fd, SOL_IP, IP_TPROXY_UNASSIGN); it returns -1, errno=EINVAL;
Our error is the fact that we call send() without checking that connect()
has failed - it fails
because there is no server and we get RST.
When Linux kernel receives RST - it releases socket's source port, and sets
sk->num=0;
After that send, which calls sendmsg in af_inet.c, chooses new source port
via inet_autobind().
inet_autobind() selects new source port, which is used as hash key to
unassign original
socket, and the setsockopt(IP_TPROXY_UNASSIGN) fails to find the socket.
While we will fix out problem with send()-after-connect(), I think that the
same problem can
happen on connected socket - when we call send() twice, and get RST between
the first
and the second send(). Only in this case the second send() will return
ECONNRESET,
but the source port will be changed anyway.
How difficult is to add handling of RST packets to tproxy module? Some way
to remove
entries from the tproxy's hash tables on RST...