On Thu, Jun 05, 2003 at 05:05:51PM -0400, Philippe Lucas wrote:

I'd like to develop the gzip module, but concerning performances, I'd rather develop it in C. Could you explain me how to retrieve the http data flow in the code to compress it ?

You need to develop a proxy module which is able to en/decompress data on the fly. Look at one of the simpler proxies like finger or plug (modules/zorp-module-finger-2.0.0 directory). Each proxy is stored in a separate shared object and is demand loaded when necessary. When the object is loaded the function named zorp_module_init() is called which usually registers a new proxy type: z_registry_add("finger", ZR_PROXY, finger_proxy_new); finger_proxy_new is a C function (right above zorp_module_init) which is called each time a new finger proxy is to be started: static ZProxy * finger_proxy_new(gchar *session_id, ZStream *client, ZPolicyObj *handler, ZProxy *parent_proxy) { FingerProxy *self; z_enter(); self = Z_CAST(z_proxy_new(Z_CLASS(FingerProxy), session_id, client, handler, parent_proxy), FingerProxy); z_thread_new(session_id, finger_thread, self); z_leave(); return (ZProxy *) self; } This creates a new instance of the class FingerProxy, and starts a new thread using the instance pointer as argument. The thread function is called finger_thread: static gpointer finger_thread(gpointer s) { FingerProxy *self = (FingerProxy *) s; z_proxy_enter(self); finger_config_set_defaults(self); finger_register_vars(self); ZPROXY_SET_STATE(self, PS_CONFIG); if(!z_proxy_config_event(&self->super)) ZPROXY_SET_FLAG(self, PF_QUIT); else { finger_config_init(self); ZPROXY_SET_STATE(self, PS_STARTING_UP); if(!z_proxy_startup_event(&self->super)) ZPROXY_SET_FLAG(self,PF_QUIT); } if (!(ZPROXY_GET_FLAG(self,PF_QUIT))) { ZPROXY_SET_STATE(self, PS_WORKING); finger_main(self); } ZPROXY_SET_STATE(self, PS_SHUTTING_DOWN); z_proxy_shutdown_event(&self->super); z_proxy_destroy(&self->super); z_leave(); return NULL; } This function is the main function of the proxy, it goes through a series of states: CONFIG, STARTUP, WORKING and SHUTTING_DOWN. First the proxy initializes internal state in the finger_config_set_defaults() function, then registers Python accessible variables in the finger_register_vars() function. * In the config state a python function named config() is called to set up various proxy settings. * In the startup state a python function named startUp() is called to let the Python part initialize things after config() was completed. * then the working state is entered when the proxy does its job by reading and forwarding requests/responses. This again can invoke various Python callbacks depending on the current configuration. * when the protocol logic tells that the protocol is ended, the shutting_down state is entered which shuts down the proxy and frees associated structures. As you can see nearly everything is performed in C and Python is only used when the there's something the proxy cannot do on its own. Therefore Zorp performs quite well in benchmarks. (our HTTP proxy was measured to be better than Raptor's HTTPGW and CheckPoint's HTTP Security Server) Proxies usually use blocking I/O (for example HTTP or FINGER), others use non-blocking I/O (like PLUG) Another important object to consider is ZStream. A ZStream is a bidirectional stream which 1) can be used as a source/destination of I/O operations (read/write) 2) can be polled whether new data arrived or not We have the following stream types: ZStreamFD (basic file descriptor), ZStreamSsl (a stream wrapping data stream in SSL), ZStreamLine (which allows reading line-by-line) and ZStreamPacket (UDP specific extensions) An implementation of ZStreamGZip should be quite easy, and adding a GZip functionality to Plug should not be difficult either. -- Bazsi PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1