[syslog-ng] [PATCH] tfjson: $(format_json) template function implementation.

[Gergely Nagy]

On Mon, 2011-02-07 at 11:59 -0800, Matthew Hall wrote: 
> On Mon, Feb 07, 2011 at 10:12:16AM +0100, Gergely Nagy wrote:
> > The problem is, we'd pretty much have to backport value-pairs() too, if
> > we want to make the 3.2 tfjson port compatible with 3.3. I'm not
> > entirely convinced that would be worth it, even though value-pairs() is
> > mostly self contained too (at least at the moment, and I don't expect it
> > to grow outside of lib/value-pairs.[ch] and a few lines in the config
> > grammar files [which don't need to be backported to 3.2]).
> 
> It would be really useful for me, trying to find a good solution for 
> interaction between SNG and other non-C code.

An unofficial port is easy, as nothing in neither value-pairs() nor
tfjson depends on 3.3 features. I'll happily assist in making both work
for 3.2 - I'm just not convinced that it should be merged back to the
official tree. However, I'm not against it, either, so if Bazsi thinks
its worthwhile, all the better!

> > > I wonder if there could also be a way to support binary formats like
> > > XDR or Google Protocol Buffers or some other kind of RPC format.
> > 
> > I don't see much that would be against it: template functions are using
> > GStrings as results, and technically, we can have binary data in those.
> > It all depends how the result is used: if the GString-supplied length is
> > used, we're good. If we assume it's a C string, then things will break.
> > 
> > It probably wouldn't be a big task to verify that we're 'binary clean' -
> > and once we are, there's nothing that would prevent one from using
> > binary formats.
> 
> What could I do to aid in the process of assessing this? It seems a bit 
> nebulous and thus scary for an outsider to tackle.

Give me a day or two, and I'll come up with a few cases you could test.

> Thinking of the complete use case, it could corrupt the terminal if 
> SNG's debug logging was enabled, etc. It would likely need some work to 
> make it enterprise ready even if it's capable of passing out binary to 
> other apps.

I'm not sure about that... if we emit debug messages where the binary
output is embedded - that would break the terminal, indeed. But we
usually don't do that, I think. But you're correct, this is something
that will need to be investigated: msg_debug() and friends should
correctly handle binary data.

Perhaps an evt_tag_raw_string() is called for - I'll look into that too.

> Therefore I think JSON seems to be a good bet until then.

Aye! It's probably easier to stick with JSON and the parsing overhead
than making sure syslog-ng handles binary output correctly in all cases.
(But, since I haven't looked deeper into this yet, I might be completely
wrong)

> > > I'd like to be able to interface syslog-ng with higher level languages
> > > for coding complex logic without adding risk of bugs in my code
> > > breaking the core daemon if I start hacking out C modules.
> > 
> > I had this idea of making it possible to write syslog-ng modules in
> > various other languages, but so far, I didn't get past the "oh hey, this
> > would be neat" stage.
> > 
> > But anyway, I see three ways to accomplish something like this:
> > 
> >         * "Message passing": output in a format that the other language
> >         understands easily, and connect the two that way.
> >         * Embedded interpreters: Embed python/perl/lua/whatever
> >         interpreters, and export selected syslog-ng internals into the
> >         language.
> >         * Implement language bindings for libsyslog-ng, and write
> >         loaders for the various languages (a'la dlopen()).
> >         
> > But these are merely vague plans, and I'm not even sure either of these
> > is a good idea to begin with.
> 
> I'll number these 1, 2, and 3 for discussion.

[...]

> I think 2 will work well for simpler higher-level requirements like 
> somebody who wants a template function without the headache of making a 
> new library or module to get it. I for one would love to be able to code 
> up template functions in ${script_language}. Then I could implement more 
> efficient binary versions of 1 instead of JSON with lots less work.

Yep, it'd be quite useful for templates. Though, the config file would
look horrid after a while, I imagine. For this, and various other
reasons, option 3 is probably better in the long run, if we want to go
this far.

> I don't know enough about what's in that library to comment in any 
> detail on 3 yet, but if the library exports a whole bunch of useful 
> symbols, it should not be hard to use SWIG, h2xs, or a few other classic 
> tools to manufacture some rudimentary front-ends to allow 
> experimentation to begin before settling on a final design for how it 
> should work.

The basic idea here is that the whole lot of stuff that the current C
modules use would get a binding in $language, and a mechanism would be
added to syslog-ng to load modules written in that language, like if it
was a C module.

Something along the lines of:

@module python:hello_world.py

hello_world.py would implement the module itself, with an API similar to
what we have for C modules right now, and there wouldn't be much
difference in using that instead of a C module. And yes, I realize I
didn't do much of an explaining. My brain dumping functions are sadly
quite primitive :(

This, however, is very intrusive, and challenging to implement. I could
probably hack up a very simple prototype within a week or two, but to
bring it to syslog-ng quality... that'd be a long process.

I suggest we stick to option 1 for now =)

-- 
|8]