[RFC]: $(format-json) improvements, and other things
Hi! An old goal we had (and which was a GSoC project for a while)) is to support nested structures in $(format-json), so that you can have output that looks like this: {"msg":{"text":"Hi, I can has structure!"}} Sadly, $(format-json) was not able to do that, as all keys within syslog-ng are stored in a flat structure, that has strong resemblance to JSON's dot notation, and format-json was not able to translate from that notation back to nested structures. With a set of patches I just pushed to my repository at github[1] (feature/3.4/value-pairs/walk branch), this limitation is no more. If we have a template like the following: $(format-json msg.text='Hi, I can has structure!') Then the output will be nested, like the first JSON showed in this mail. The implementation is fairly inefficient at the moment, and has a few bugs in it still, but the API looks and feels resonable, so I figured I'll give you all a heads up on what to expect in the not too distant future. The branch also merges the tfjson and jsonparser modules, and drops json-glib support: json-c is required for both format-json and the json-parser, and they come in the same module now (json-plugin). The work done on this branch also paves way for major changes within the MongoDB destination driver: moving away from upserts to inserts, and then on to batch inserts, from which I expect a significant performance boost. Code reviews, comments and whatnot are most appreciated. Do keep in mind however, that as I work, I will likely rebase the branch a few more times, without notice. [1]: https://github.com/algernon/syslog-ng/commits/feature/3.4/value-pairs/walk -- |8]
Gergely Nagy <algernon@balabit.hu> writes:
The work done on this branch also paves way for major changes within the MongoDB destination driver: moving away from upserts to inserts, and then on to batch inserts, from which I expect a significant performance boost.
The feature/3.4/value-pairs/walk branch[1] has now been changed so that the mongodb destination will use inserts over updates (and still maintain a possibly nested structure in the db). The performance right now is probably quite poor, there's a lot of things that can - and will - be improved, but not using upserts should already result in lower load on busy sites. I would appreciate any feedback on this, mostly about performance, or any possible bugs. No changes are necessary in the config, just plug 3.4 in, and it should Do The Right Thing(tm). [1]: https://github.com/algernon/syslog-ng/commits/feature/3.4/value-pairs/walk -- |8]
participants (1)
-
Gergely Nagy