[syslog-ng] [PATCH] tfjson: $(format_json) template function implementation.

[Matthew Hall]

On Mon, Feb 07, 2011 at 10:12:16AM +0100, Gergely Nagy wrote:
> The problem is, we'd pretty much have to backport value-pairs() too, if
> we want to make the 3.2 tfjson port compatible with 3.3. I'm not
> entirely convinced that would be worth it, even though value-pairs() is
> mostly self contained too (at least at the moment, and I don't expect it
> to grow outside of lib/value-pairs.[ch] and a few lines in the config
> grammar files [which don't need to be backported to 3.2]).

It would be really useful for me, trying to find a good solution for 
interaction between SNG and other non-C code.

> > I wonder if there could also be a way to support binary formats like
> > XDR or Google Protocol Buffers or some other kind of RPC format.
> 
> I don't see much that would be against it: template functions are using
> GStrings as results, and technically, we can have binary data in those.
> It all depends how the result is used: if the GString-supplied length is
> used, we're good. If we assume it's a C string, then things will break.
> 
> It probably wouldn't be a big task to verify that we're 'binary clean' -
> and once we are, there's nothing that would prevent one from using
> binary formats.

What could I do to aid in the process of assessing this? It seems a bit 
nebulous and thus scary for an outsider to tackle.

Thinking of the complete use case, it could corrupt the terminal if 
SNG's debug logging was enabled, etc. It would likely need some work to 
make it enterprise ready even if it's capable of passing out binary to 
other apps.

Therefore I think JSON seems to be a good bet until then.

> > I'd like to be able to interface syslog-ng with higher level languages
> > for coding complex logic without adding risk of bugs in my code
> > breaking the core daemon if I start hacking out C modules.
> 
> I had this idea of making it possible to write syslog-ng modules in
> various other languages, but so far, I didn't get past the "oh hey, this
> would be neat" stage.
> 
> But anyway, I see three ways to accomplish something like this:
> 
>         * "Message passing": output in a format that the other language
>         understands easily, and connect the two that way.
>         * Embedded interpreters: Embed python/perl/lua/whatever
>         interpreters, and export selected syslog-ng internals into the
>         language.
>         * Implement language bindings for libsyslog-ng, and write
>         loaders for the various languages (a'la dlopen()).
>         
> But these are merely vague plans, and I'm not even sure either of these
> is a good idea to begin with.

I'll number these 1, 2, and 3 for discussion.

I really like the idea of 1, and think your JSON thing goes a long way 
toward this, hence why I was working on that and hoping for a backport.

I think 2 will work well for simpler higher-level requirements like 
somebody who wants a template function without the headache of making a 
new library or module to get it. I for one would love to be able to code 
up template functions in ${script_language}. Then I could implement more 
efficient binary versions of 1 instead of JSON with lots less work.

I don't know enough about what's in that library to comment in any 
detail on 3 yet, but if the library exports a whole bunch of useful 
symbols, it should not be hard to use SWIG, h2xs, or a few other classic 
tools to manufacture some rudimentary front-ends to allow 
experimentation to begin before settling on a final design for how it 
should work.

Matthew.