[syslog-ng] [RFC]: $(format-json) improvements, and other things

[Gergely Nagy]

Hi!

An old goal we had (and which was a GSoC project for a while)) is to
support nested structures in $(format-json), so that you can have output
that looks like this:

  {"msg":{"text":"Hi, I can has structure!"}}

Sadly, $(format-json) was not able to do that, as all keys within
syslog-ng are stored in a flat structure, that has strong resemblance to
JSON's dot notation, and format-json was not able to translate from that
notation back to nested structures.

With a set of patches I just pushed to my repository at github[1]
(feature/3.4/value-pairs/walk branch), this limitation is no more.

If we have a template like the following:

 $(format-json msg.text='Hi, I can has structure!')

Then the output will be nested, like the first JSON showed in this mail.

The implementation is fairly inefficient at the moment, and has a few
bugs in it still, but the API looks and feels resonable, so I figured
I'll give you all a heads up on what to expect in the not too distant
future.

The branch also merges the tfjson and jsonparser modules, and drops
json-glib support: json-c is required for both format-json and the
json-parser, and they come in the same module now (json-plugin).

The work done on this branch also paves way for major changes within the
MongoDB destination driver: moving away from upserts to inserts, and
then on to batch inserts, from which I expect a significant performance
boost.

Code reviews, comments and whatnot are most appreciated. Do keep in mind
however, that as I work, I will likely rebase the branch a few more
times, without notice.

 [1]: https://github.com/algernon/syslog-ng/commits/feature/3.4/value-pairs/walk

-- 
|8]