[zorp] Zorp and gzip

[Balazs Scheidler]

On Thu, Jun 05, 2003 at 05:05:51PM -0400, Philippe Lucas wrote:
> I'd like to develop the gzip module, but concerning performances, I'd 
> rather develop it in C. Could you explain me how to retrieve the http 
> data flow in the code to compress it ?

You need to develop a proxy module which is able to en/decompress data on
the fly. Look at one of the simpler proxies like finger or plug
(modules/zorp-module-finger-2.0.0 directory).

Each proxy is stored in a separate shared object and is demand loaded when
necessary. When the object is loaded the function named zorp_module_init()
is called which usually registers a new proxy type:

  z_registry_add("finger", ZR_PROXY, finger_proxy_new);

finger_proxy_new is a C function (right above zorp_module_init) which is
called each time a new finger proxy is to be started:

static ZProxy *
finger_proxy_new(gchar *session_id, ZStream *client, ZPolicyObj *handler, ZProxy *parent_proxy)
{
  FingerProxy  *self;
  
  z_enter();
  self = Z_CAST(z_proxy_new(Z_CLASS(FingerProxy), session_id, client, handler, parent_proxy), FingerProxy);
  
  z_thread_new(session_id, finger_thread, self);
  z_leave();
  return (ZProxy *) self;
}

This creates a new instance of the class FingerProxy, and starts a new
thread using the instance pointer as argument. The thread function is called
finger_thread:

static gpointer
finger_thread(gpointer s)
{
  FingerProxy *self = (FingerProxy *) s;

  z_proxy_enter(self);
  
  finger_config_set_defaults(self);  
  finger_register_vars(self);
  
  ZPROXY_SET_STATE(self, PS_CONFIG);
  if(!z_proxy_config_event(&self->super))
    ZPROXY_SET_FLAG(self, PF_QUIT);
  else
    {
      finger_config_init(self);
      
      ZPROXY_SET_STATE(self, PS_STARTING_UP);
      if(!z_proxy_startup_event(&self->super))
        ZPROXY_SET_FLAG(self,PF_QUIT);
    }

  if (!(ZPROXY_GET_FLAG(self,PF_QUIT)))
    {
      ZPROXY_SET_STATE(self, PS_WORKING);

      finger_main(self);
    }
  ZPROXY_SET_STATE(self, PS_SHUTTING_DOWN);
  z_proxy_shutdown_event(&self->super);
  
  z_proxy_destroy(&self->super);
  z_leave();
  return NULL;
}

This function is the main function of the proxy, it goes through a series of
states: CONFIG, STARTUP, WORKING and SHUTTING_DOWN.

First the proxy initializes internal state in the
finger_config_set_defaults() function, then registers Python accessible
variables in the finger_register_vars() function.

* In the config state a python function named config() is called to set up
  various proxy settings.

* In the startup state a python function named startUp() is called to let
  the Python part initialize things after config() was completed.

* then the working state is entered when the proxy does its job by reading
  and forwarding requests/responses. This again can invoke various Python
  callbacks depending on the current configuration.

* when the protocol logic tells that the protocol is ended, the
  shutting_down state is entered which shuts down the proxy and frees
  associated structures.

As you can see nearly everything is performed in C and Python is only used
when the there's something the proxy cannot do on its own. Therefore Zorp
performs quite well in benchmarks. (our HTTP proxy was measured to be better
than Raptor's HTTPGW and CheckPoint's HTTP Security Server)

Proxies usually use blocking I/O (for example HTTP or FINGER), others use
non-blocking I/O (like PLUG)

Another important object to consider is ZStream. A ZStream is a
bidirectional stream which 

1) can be used as a source/destination of I/O operations (read/write)
2) can be polled whether new data arrived or not

We have the following stream types: ZStreamFD (basic file descriptor),
ZStreamSsl (a stream wrapping data stream in SSL), ZStreamLine (which allows
reading line-by-line) and ZStreamPacket (UDP specific extensions)

An implementation of ZStreamGZip should be quite easy, and adding a GZip
functionality to Plug should not be difficult either.

-- 
Bazsi
PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1