[syslog-ng] [review request]: MongoDB destination

[Balazs Scheidler]

Hi,

On Mon, 2011-01-17 at 21:36 +0100, Gergely Nagy wrote:
> I prepared three branches, in preparation for a pull request, but
> there's still some minor work left to do, so this is more like a "review
> the structure and plan before the pull request, please" request.
> 
> All three branches are integration branches, and as such, I will rebase
> them from time to time. All three branches are folded into one commit
> over their base, signed-off, and should have proper descriptions.
> 
> * integration/libmongo-client
> 
> This one adds a git submodule, which pulls in the libmongo-client
> library from github to modules/afmongodb/libmongo-client. Also adds
> infrastructure to make autogen.sh, configure and make enter this
> directory and Do The Right Thing.
> 
> At the moment, linking with the supplied libmongo-client is the only
> option, because the library is not yet ready for system-wide install.
> 
> It is easy to extend this framework to use the system library if it
> exists - that is one of the missing features which should go into this
> branch, but isn't there yet.

pulled, thanks.

> 
> * integration/afmongodb/base
> 
> This is the basic, bare-bones driver, without support for dynamic
> values: it depends on the previous one, and doesn't change anything
> outside of modules/afmongodb/, except lib/stats.[ch], where it adds
> SCS_MONGO for stat counter support.
> 
> The current state of the driver is reasonably stable, there are three
> known problems that I am aware of:

also pulled.

> 
>         + Segfaults whenever the writer thread emits an msg_debug() or
>         similar itself. This is actually a bug in ivykis, for which
>         Bazsi had a fix for, but I can't find the fix in his git tree
>         yet (only the log message, but without the patch).

this fix is there too.

>         
>         + Locking issues: it's fairly easy to push the driver into a
>         deadlock, where it will just sit and wait, not doing a thing.
>         I've only seen this happen with flow-control enabled, though.
>         
>         This will have to be solved before I'm confident enough to ask
>         for a pull.

this was solved with a separate commit, so this is also closed.

>         
>         + The libmongo-client library is built as a shared library, and
>         is installed as such. It should be static-only, and not
>         installed, only linked to afmongodb.
>         
>         This will be implemented mostly in libmongo-client, and an
>         update of the integration/libmongo-client branch will pull that
>         in; then this branch will be updated accordingly.

This is currently done in the libmongo-client git tree, but perhaps the
same solution I've used with ivykis could work here too.

e.g. create a configure.gnu script that does:

./configure --disable-shared --enable-static --with-pic $@

which in turn will create a static-only library, that you can link into
libafmongodb.so (because of the PIC compilation).

> 
> * integration/afmongodb/dyn-vars
> 
> Based on integration/afmongodb/base, this adds dynamic_values() support.
> Minor modifications are done to nvtable too, but it's a very small
> patch, and very straightforward too.
> 
> However, as discussed earlier, the goal in the long run is to develop a
> generic framework for this kind of thing, so this here is more like a
> temporary solution so that people can experiment with the key feature
> the mongodb provides over the others.
> 
> Until the generic framework is ready, this, I believe, is an acceptable
> compromise.
> 

I didn't merge this one yet.

-- 
Bazsi