On Mon, 2011-02-07 at 11:59 -0800, Matthew Hall wrote:
On Mon, Feb 07, 2011 at 10:12:16AM +0100, Gergely Nagy wrote:
The problem is, we'd pretty much have to backport value-pairs() too, if we want to make the 3.2 tfjson port compatible with 3.3. I'm not entirely convinced that would be worth it, even though value-pairs() is mostly self contained too (at least at the moment, and I don't expect it to grow outside of lib/value-pairs.[ch] and a few lines in the config grammar files [which don't need to be backported to 3.2]).
It would be really useful for me, trying to find a good solution for interaction between SNG and other non-C code.
An unofficial port is easy, as nothing in neither value-pairs() nor tfjson depends on 3.3 features. I'll happily assist in making both work for 3.2 - I'm just not convinced that it should be merged back to the official tree. However, I'm not against it, either, so if Bazsi thinks its worthwhile, all the better!
I wonder if there could also be a way to support binary formats like XDR or Google Protocol Buffers or some other kind of RPC format.
I don't see much that would be against it: template functions are using GStrings as results, and technically, we can have binary data in those. It all depends how the result is used: if the GString-supplied length is used, we're good. If we assume it's a C string, then things will break.
It probably wouldn't be a big task to verify that we're 'binary clean' - and once we are, there's nothing that would prevent one from using binary formats.
What could I do to aid in the process of assessing this? It seems a bit nebulous and thus scary for an outsider to tackle.
Give me a day or two, and I'll come up with a few cases you could test.
Thinking of the complete use case, it could corrupt the terminal if SNG's debug logging was enabled, etc. It would likely need some work to make it enterprise ready even if it's capable of passing out binary to other apps.
I'm not sure about that... if we emit debug messages where the binary output is embedded - that would break the terminal, indeed. But we usually don't do that, I think. But you're correct, this is something that will need to be investigated: msg_debug() and friends should correctly handle binary data. Perhaps an evt_tag_raw_string() is called for - I'll look into that too.
Therefore I think JSON seems to be a good bet until then.
Aye! It's probably easier to stick with JSON and the parsing overhead than making sure syslog-ng handles binary output correctly in all cases. (But, since I haven't looked deeper into this yet, I might be completely wrong)
I'd like to be able to interface syslog-ng with higher level languages for coding complex logic without adding risk of bugs in my code breaking the core daemon if I start hacking out C modules.
I had this idea of making it possible to write syslog-ng modules in various other languages, but so far, I didn't get past the "oh hey, this would be neat" stage.
But anyway, I see three ways to accomplish something like this:
* "Message passing": output in a format that the other language understands easily, and connect the two that way. * Embedded interpreters: Embed python/perl/lua/whatever interpreters, and export selected syslog-ng internals into the language. * Implement language bindings for libsyslog-ng, and write loaders for the various languages (a'la dlopen()).
But these are merely vague plans, and I'm not even sure either of these is a good idea to begin with.
I'll number these 1, 2, and 3 for discussion.
[...]
I think 2 will work well for simpler higher-level requirements like somebody who wants a template function without the headache of making a new library or module to get it. I for one would love to be able to code up template functions in ${script_language}. Then I could implement more efficient binary versions of 1 instead of JSON with lots less work.
Yep, it'd be quite useful for templates. Though, the config file would look horrid after a while, I imagine. For this, and various other reasons, option 3 is probably better in the long run, if we want to go this far.
I don't know enough about what's in that library to comment in any detail on 3 yet, but if the library exports a whole bunch of useful symbols, it should not be hard to use SWIG, h2xs, or a few other classic tools to manufacture some rudimentary front-ends to allow experimentation to begin before settling on a final design for how it should work.
The basic idea here is that the whole lot of stuff that the current C modules use would get a binding in $language, and a mechanism would be added to syslog-ng to load modules written in that language, like if it was a C module. Something along the lines of: @module python:hello_world.py hello_world.py would implement the module itself, with an API similar to what we have for C modules right now, and there wouldn't be much difference in using that instead of a C module. And yes, I realize I didn't do much of an explaining. My brain dumping functions are sadly quite primitive :( This, however, is very intrusive, and challenging to implement. I could probably hack up a very simple prototype within a week or two, but to bring it to syslog-ng quality... that'd be a long process. I suggest we stick to option 1 for now =) -- |8]