- tproxy - lists.balabit.hu

Re: [tproxy] anything known about potential memory leaks in tproxy 1.1.3 ?
by Lennert Buytenhek 06 Apr '04

06 Apr '04

On Tue, Apr 06, 2004 at 11:50:09AM +0200, KOVACS Krisztian wrote: > Hi, Hi, > > Our production firewall/transproxy suddenly became _really_ unresponsive > > a few days ago after ~120 days of uptime, from what seems to have been a > > memory leak. > > > > The machine has 1GB of memory, but the amount of page cache was really small, > > the number of processes waiting for disk I/O was about 30, and at avg 4MB/sec > > reading from disk it was seeking all over the place. Also, bash would > > sometimes say something along the lines of 'resource unavailable' when > > starting subprocesses. > > > > Temporarily disabling the transproxy function by deleting the redirect rule > > caused the disk to go idle (our transproxy-using app intercepts HTTP requests > > and answers the requests locally for static data, fileset is about 5GB but > > ~200MB of it accounts for 90% of the hits.) But every little thing that I > > tried, even an ls or a grep in /etc caused massive disk I/O. > > > > I hastily rebooted the machine and thus have no ways of debugging this > > anymore. But I was wondering, is anything known about memory leaks in the > > 1.1.3 version of the tproxy code? I checked the changelogs, but couldn't > > find anything related to memleaks. > > > > Should I schedule maintenance to move the machine to 1.9.1? Any other > > advice? (It's not a problem for me to schedule the machine to be rebooted > > every 90 days or something like that.) > > We've also found signs that suggest that there may be a possible > memory leak in tproxy (it seemed to leak conntrack entries). However, I > don't have an idea what the problem might be, so it's not fixed yet. > > Could you provide a bit more information? Is this system SMP or UP? > I'd be happy to receive the contents of /proc/slabinfo, if possible, > daily/hourly snapshots. Uniprocessor system, P4 2.4GHz. Two e1000 fiber cards, 2.4.20-20.9 Red Hat kernel plus bridging firewall patch and tproxy patch 1.1.3. Right now about 200 SYNs per second, which is sort-of a quiet time of day. /proc/net/ip_conntrack right now shows about 35k entries, 'netstat -tn' shows about 7k. I've installed a cron job to take regular snapshots of /proc/slabinfo, will get back to you soon. > Moving to 1.9.1 is not recommended in your case, its ABI/API is > incompatible (though trivial to fix in your sources), and probably won't > help. OK. Thanks so far. cheers, Lennert

1 0

anything known about potential memory leaks in tproxy 1.1.3 ?
by Lennert Buytenhek 06 Apr '04

06 Apr '04

Hi, Our production firewall/transproxy suddenly became _really_ unresponsive a few days ago after ~120 days of uptime, from what seems to have been a memory leak. The machine has 1GB of memory, but the amount of page cache was really small, the number of processes waiting for disk I/O was about 30, and at avg 4MB/sec reading from disk it was seeking all over the place. Also, bash would sometimes say something along the lines of 'resource unavailable' when starting subprocesses. Temporarily disabling the transproxy function by deleting the redirect rule caused the disk to go idle (our transproxy-using app intercepts HTTP requests and answers the requests locally for static data, fileset is about 5GB but ~200MB of it accounts for 90% of the hits.) But every little thing that I tried, even an ls or a grep in /etc caused massive disk I/O. I hastily rebooted the machine and thus have no ways of debugging this anymore. But I was wondering, is anything known about memory leaks in the 1.1.3 version of the tproxy code? I checked the changelogs, but couldn't find anything related to memleaks. Should I schedule maintenance to move the machine to 1.9.1? Any other advice? (It's not a problem for me to schedule the machine to be rebooted every 90 days or something like that.) cheers, Lennert

2 2

Re: Development version 1.9.2 released - Oops
by James MacLean 22 Mar '04

22 Mar '04

Hello Everyone, This happened only once. But we have only brought up a kernel with it today :). It appeared to occur when an iptables cmd was issued. Since then (and the PC rebooting) we have not had it happen. We are not using the tproxy filter yet. Also, where is a FAQ on a typical install of this for Squid? I have Squid now patched and I believe I can fuddle through it, but would rather use someone else's notes ;). thanks, JES ksymoops 2.4.5 on i686 2.6.4-rc2. Options used -V (default) -k /proc/ksyms (default) -l /proc/modules (default) -o /lib/modules/2.6.4-rc2/ (default) -m /boot/System.map-2.6.4-rc2 (default) Warning: You did not tell me where to find symbol information. I will assume that the log matches the kernel and modules that are running right now and I'll use the default options above for symbol resolution. If the current kernel and/or modules do not match the log, you can get more accurate output by telling me the kernel version and where to find map, modules, ksyms etc. ksymoops -h explains the options. Error (regular_file): read_ksyms stat /proc/ksyms failed No modules in ksyms, skipping objects No ksyms, skipping lsmod Mar 8 12:07:13 pink kernel: kernel BUG at mm/slab.c:1316! Mar 8 12:07:13 pink kernel: invalid operand: 0000 [#1] Mar 8 12:07:13 pink kernel: CPU: 1 Mar 8 12:07:13 pink kernel: EIP: 0060:[<c013c159>] Not tainted Using defaults from ksymoops -t elf32-i386 -a i386 Mar 8 12:07:13 pink kernel: EFLAGS: 00010202 Mar 8 12:07:13 pink kernel: eax: 00000033 ebx: f6ee3fec ecx: c03b4e30 edx: c0306c7c Mar 8 12:07:13 pink kernel: esi: f8a1dc18 edi: f8a1dc18 ebp: f6ee3e68 esp: e433df3c Mar 8 12:07:13 pink kernel: ds: 007b es: 007b ss: 0068 Mar 8 12:07:13 pink kernel: Stack: c02c8c60 f8a1dc08 00002000 e433df58 f6ee3eb0 c0000000 ffffffc0 00000080 Mar 8 12:07:13 pink kernel: 00000000 f8a20f00 e433c000 c03082f0 f88b40b6 f8a1dc08 00000040 00000080 Mar 8 12:07:13 pink kernel: 00002000 00000000 00000000 00000000 f8a1a3a4 0807d6a0 c030830c f88b400f Mar 8 12:07:13 pink kernel: [<f88b40b6>] ip_nat_init+0x42/0x1bf [iptable_nat] Mar 8 12:07:13 pink kernel: [<f8a1a3a4>] init_or_cleanup+0x2a/0xba [iptable_nat] Mar 8 12:07:13 pink kernel: [<f88b400f>] init+0xf/0x13 [iptable_nat] Mar 8 12:07:13 pink kernel: [<c01346dc>] sys_init_module+0x11d/0x20e Mar 8 12:07:13 pink kernel: [<c0108d7f>] syscall_call+0x7/0xb Mar 8 12:07:13 pink kernel: Code: 0f 0b 24 05 ce 84 2c c0 8b 0b e9 74 ff ff ff 8b 47 40 c7 04 >>EIP; c013c159 <kmem_cache_create+438/506> <===== >>ebx; f6ee3fec <__crc_net_ratelimit+27fb1/13b36f> >>ecx; c03b4e30 <cache_chain_sem+0/14> >>edx; c0306c7c <log_wait+4/c> >>esi; f8a1dc18 <__crc_pm_idle+10dd99/244fc7> >>edi; f8a1dc18 <__crc_pm_idle+10dd99/244fc7> >>ebp; f6ee3e68 <__crc_net_ratelimit+27e2d/13b36f> >>esp; e433df3c <__crc_skb_clone_fraglist+37fe19/9833c6> Code; c013c159 <kmem_cache_create+438/506> 00000000 <_EIP>: Code; c013c159 <kmem_cache_create+438/506> <===== 0: 0f 0b ud2a <===== Code; c013c15b <kmem_cache_create+43a/506> 2: 24 05 and $0x5,%al Code; c013c15d <kmem_cache_create+43c/506> 4: ce into Code; c013c15e <kmem_cache_create+43d/506> 5: 84 2c c0 test %ch,(%eax,%eax,8) Code; c013c161 <kmem_cache_create+440/506> 8: 8b 0b mov (%ebx),%ecx Code; c013c163 <kmem_cache_create+442/506> a: e9 74 ff ff ff jmp ffffff83 <_EIP+0xffffff83> Code; c013c168 <kmem_cache_create+447/506> f: 8b 47 40 mov 0x40(%edi),%eax Code; c013c16b <kmem_cache_create+44a/506> 12: c7 04 00 00 00 00 00 movl $0x0,(%eax,%eax,1) 1 warning and 1 error issued. Results may not be reliable. -- James B. MacLean macleajb(a)ednet.ns.ca Department of Education Nova Scotia, Canada

2 7

Development version 1.9.2 released
by KOVACS Krisztian 27 Feb '04

27 Feb '04

Hi, TProxy version 1.9.2 has been released. This release is the first which supports Linux 2.6 (tested with 2.6.3). Because of the changes needed for Linux 2.6 compatibility, it cannot be used with Linux 2.4. Major changes: * supports Linux 2.6 only * Netfilter NAT reservations are now optional (compile time option) * constants used by the setsockopt calls has been changed, this causes binary incompatibility with all previous versions Hopefully, this will be the last ABI break for the new TProxy version. Unfortunately the README file has not been updated for the new API, but the example programs are fully up-to-date (but still incomplete, they do not cover all features of TProxy). This will be fixed before the stable 2.0 release. The new release is available from http://www.balabit.com/downloads/tproxy/linux-2.4/devel/ MD5: cttproxy-2.6.3-1.9.2.tar.gz 31cae35cf1187a9a027fd830f0e099a8 Any feedback (test results, problems, bugs, etc.) is appreciated. -- Regards, Krisztian KOVACS

1 0

TProxy version 1.2.0 for Linux 2.4.25
by KOVACS Krisztian 23 Feb '04

23 Feb '04

Hi, Because of changes in the Netfilter code in Linux 2.4.25, the TProxy patch for 2.4.24 does not compile with 2.4.25. So I've created a new patch tarball for 2.4.25, which fixes the compile problems with ipt_tproxy.c. Except for this trivial change in ipt_tproxy.c, this is exactly the same as those previous 1.2.0 releases. As always, the release is available at: http://www.balabit.com/downloads/tproxy/linux-2.4/ MD5: cttproxy-2.4.25-1.2.0.tar.gz be031e61415349188f4e6321b2636f3c -- Regards, Krisztian KOVACS

1 0

hi
by bazsi＠balabit.hu 06 Feb '04

06 Feb '04

1 0

new tproxy development version: 1.9.1
by KOVACS Krisztian 29 Jan '04

29 Jan '04

Hi, After almost two months work, a new development version of TProxy has been released: 1.9.1. The 1.9 branch, when declared stable, will become TProxy version 2.0 -- the change in the major version number reflects complete API change, and some semantic changes. The iptables (userspace) patch is still the same, so this version is compatible with iptables compiled for the stable branch of TProxy, however, the sockopt interface has gone through a complete rewrite. New features are: * New API, see the updated examples in the tests directory. This makes TProxy version and compatibility checks possible, and provides easy extensibility. * TPROXY_ALLOC operation: makes it possible to allocate a unique foreign port automatically before the first packet is leaving. * TPROXY_CONNECT operation: informs TProxy about the peer. It helps a lot with NAT reservations, and is _needed_ before setting the ESTABLISHED flag on a sockref. (Instead of the dirty hack of requiring connect()-ing before setting the flag.) This release has a bit more patches inside than the usual: * 01-nat_reservations.diff: NAT reservations patch for the Netfilter NAT core, this is needed for TProxy to avoid a lot of NAT failures * 02-nat_reservations_tproxy_exports.diff: two extra exported functions from the NAT reservations support code * 03-tproxy.diff: transparent proxying support code * 04-nat_delete.diff: extended version of the older nat_delete patch, it is able to delete conntrack entries when a clashing NAT mapping is to be applied This release is available, as always, from: http://www.balabit.com/downloads/tproxy/linux-2.4/devel/ MD5: cttproxy-2.4.23-1.9.1.tar.gz 512fb80b37f8a9f260d3a90084f426ee Since this is the first development release of this completely new branch, its main purpose is to demonstrate the new API and feature set. It is certainly buggy, so handle with care :) Of course, any feedback (test results, problems, bugs, etc.) is appreciated. -- Regards, Krisztian KOVACS

1 0

Error
by bazsi＠balabit.hu 27 Jan '04

27 Jan '04

1 0

Re: [tproxy] tproxy 1.2.0 missing ip_tproxy_sockref_uniq?
by KOVACS Krisztian 09 Jan '04

09 Jan '04

Hi, On Fri, 2004-01-09 at 18:08, Jon Nelson wrote: > > > > > > /* check if the proxy requested a wildcard port, and allocate > > > * a free port if necessary */ > > > if (itp.itp_faddr.s_addr && (itp.itp_fport == 0)) { > > > itp.itp_fport = htons(ip_tproxy_sockref_uniq(itp.itp_faddr, proto)); > > > if (itp.itp_fport == 0) { > > > /* port allocation failed */ > > > res = -EAGAIN; > > > goto read_unlk; > > > } > > > } > > > > > > > Yes, it did. In the 1.2 branch, if a wildcard foreign port is > > specified at ASSIGN, Netfilter's NAT subsystem allocates a foreign port. > > This is very useful for connect()-ing from a wildcard foreign port, > > since it avoids "failed to apply NAT mapping" errors in a lot of cases. > > However, this allocation happens only when the first packet of the > > connection is traversing the Netfilter hook, so you cannot get the exact > > port number _before_ sending the first packet. In case of listening > > I don't think that's true. I hand-applied a patch to add back > ip_tproxy_sockref_uniq (and the code that calls it, shown above) and it > works perfectly. After the ASSIGN I just use QUERY to get the IP > and port which I can then use for my own devices. > It works perfectly, and as expected. Tested with both listening and > connect()'ing sockets. Of course it works, but with these changes you lose the advantages of the ability to let the NAT subsystem choose the protocol-dependent part (TCP/UDP port, in our case). The problem with the uniq() approach was that you cannot be sure that the allocated port won't clash with an already existing connection when the first packet traverses the Netfilter hook, and tproxy tries to apply the pre-allocated, fully specified NAT mapping. The 1.2.0 approach makes transparent "wildcard" connects() perform better and more reliably, but unfortunately this broke "wildcard" listening support (which in some cases can be easily worked around in userspace). This is why IP_TPROXY_ALLOC will be introduced in the next major tproxy version. BTW could you publish your patch, just in case somebody else needs it, too? -- Regards, Krisztian KOVACS

1 0

tproxy 1.2.0 missing ip_tproxy_sockref_uniq?
by Jon Nelson 09 Jan '04

09 Jan '04

I was wondering something. Let's say somebody wants to perform a non-local LISTEN. Since the port isn't known ahead of time, both the local bind(2) operation *and* the ASSIGN use a port of 0 (pick one for me). Of course the 'local' socket (getsockaddr) gets a port, thanks to the kernel. However, getsockopt (QUERY) returns ports of 0 for both local *and* foreign. How does one determine what the 'remote' port is? With 2.4.21-23, there was (seeming) explicit support for this scenario involving the function ip_tproxy_sockref_uniq (from the setsockopt handler): /* check if the proxy requested a wildcard port, and allocate * a free port if necessary */ if (itp.itp_faddr.s_addr && (itp.itp_fport == 0)) { itp.itp_fport = htons(ip_tproxy_sockref_uniq(itp.itp_faddr, proto)); if (itp.itp_fport == 0) { /* port allocation failed */ res = -EAGAIN; goto read_unlk; } } Did support for automatically assigning a foreign port go away between 2.4.21-23 and 1.2.0? -- What do you call a fish with no eyes? A fsh. Jon Nelson <jnelson-tproxy(a)securepipe.com>

2 2