[zorp] Zorp 2.1.5.5 can't handle load

[Sheldon Hearn]

On Fri, 2004-05-21 at 10:58, Balazs Scheidler wrote:

> The solution is to split your single Zorp instances to smaller instances
> working on the same set of connections. This can be achieved by running
> for example 16 instances of HTTP listening on different ports. (for
> example 50080 - 50095) then use 16 packet filter rules to distribute the
> load between processes based on source port for example. 

This works very well, thank you.

When I push a single instance to its maximum threads limit, I soon get
the following:

(zorp_default_http_00@default@balrog/nosession): Too many running
threads, waiting for one to become free; num_threads='1000',
max_threads='1000'
zorp_default_http_00[12741]: (Log thread):
zorp_default_http_00[12741]: (Log thread): GLib-ERROR **: Cannot create
pipe main loop wake-up: Too many open files

Is this one of the serious problems you warned me about with Glib, for
which you have a patched version of GLib as part of your Debian
packages?  Do you have the patches available?

> We have somewhere between 500-600 new connections/sec distributed on 4 
> computers running 16 processes each. And latency is ok. And btw: which tproxy 
> version are you using?

IP_TPROXY: Transparent proxy support initialized 1.2.0

> Do you have more system or userspace CPU time? (vmstat will tell you that)

It's about equally split, with system outweighing userspace 4:3 as total
utilization approaches 50% (about the highest I get with the limited
load I've been able to produce in testing).

Ciao,
Sheldon.