is it possible to normalize the jSON keys to lowercase?
Hi, I am using the 'format-json' as my below snip. template t_raw2json { template("$(format-json --scope selected_macros \ --exclude TAGS \ --exclude DATE \ --exclude PRIORITY \ --exclude FACILITY \ --exclude PROGRAM )\n"); }; but, the problem is because sometimes I receive a jSON packet as { "Key": "value" } or {"key": "value}".... this has been a problem to my filter. e.g: destination d_nginx_modsec_log { file("/var/logs/myapp//${j.key:-invalid_key}.log" template(t_raw2json) ); }; Question: is it possible to normalize the jSON keys to lowercase? or someone has other approaches? -- Jorge Pereira
Not right now, but value-pairs does support key transformations (add-prefix and friends), and the same way lowercasing could happen there. However, someone needs to come up with an implementation, as right now, I don't see when I could get to it. The required code is in lib/value-pairs/transforms.c, and then its parsing needs to be added to both the cmdline parser (lib/value-pairs/cmdline.c) and the grammar (vp_rekey_options in cfg-grammar.y) HTH, Bazsi -- Bazsi On Wed, Jan 11, 2017 at 12:03 AM, Jorge Pereira <jpereiran@gmail.com> wrote:
Hi,
I am using the 'format-json' as my below snip.
template t_raw2json { template("$(format-json --scope selected_macros \ --exclude TAGS \ --exclude DATE \ --exclude PRIORITY \ --exclude FACILITY \ --exclude PROGRAM )\n"); };
but, the problem is because sometimes I receive a jSON packet as { "Key": "value" } or {"key": "value}".... this has been a problem to my filter.
e.g:
destination d_nginx_modsec_log { file("/var/logs/myapp//${j.key:-invalid_key}.log" template(t_raw2json) ); };
Question: is it possible to normalize the jSON keys to lowercase? or someone has other approaches? -- Jorge Pereira
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Actually, in the docs we mention a 'lowercase' template function, I think algernon wrote it a few years ago, you can give it a try. https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-o... On Wed, Jan 11, 2017 at 8:43 AM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Not right now, but value-pairs does support key transformations (add-prefix and friends), and the same way lowercasing could happen there.
However, someone needs to come up with an implementation, as right now, I don't see when I could get to it.
The required code is in lib/value-pairs/transforms.c, and then its parsing needs to be added to both the cmdline parser (lib/value-pairs/cmdline.c) and the grammar (vp_rekey_options in cfg-grammar.y)
HTH, Bazsi
-- Bazsi
On Wed, Jan 11, 2017 at 12:03 AM, Jorge Pereira <jpereiran@gmail.com> wrote:
Hi,
I am using the 'format-json' as my below snip.
template t_raw2json { template("$(format-json --scope selected_macros \ --exclude TAGS \ --exclude DATE \ --exclude PRIORITY \ --exclude FACILITY \ --exclude PROGRAM )\n"); };
but, the problem is because sometimes I receive a jSON packet as { "Key": "value" } or {"key": "value}".... this has been a problem to my filter.
e.g:
destination d_nginx_modsec_log { file("/var/logs/myapp//${j.key:-invalid_key}.log" template(t_raw2json) ); };
Question: is it possible to normalize the jSON keys to lowercase? or someone has other approaches? -- Jorge Pereira
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On Wed, Jan 11, 2017 at 09:39:48AM +0100, Fekete, Róbert wrote:
Actually, in the docs we mention a 'lowercase' template function, I think algernon wrote it a few years ago, you can give it a try.
But that will not let you munge the keys, unfortunately
Oh, right. Well, you could make an extra round of formatting+parsing, like incoming-message>format-json>lowercase-the-formatted-json>json-parser>format-json, but that would not be too efficient performance-wise. Or maybe you could use the lowercase function on the incoming message, depending on what it is like. On Wed, Jan 11, 2017 at 9:46 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
On Wed, Jan 11, 2017 at 09:39:48AM +0100, Fekete, Róbert wrote:
Actually, in the docs we mention a 'lowercase' template function, I think algernon wrote it a few years ago, you can give it a try.
But that will not let you munge the keys, unfortunately
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, Thanks for all suggestions. But, I solved the problem doing such hacking creating a new parser() as the below sample. ............... rewrite p_my_key { set("${Key}", value("my_key"), condition("${Key}" != "")); set("${key}", value("my_key"), condition("${key}" != "")); }; ............ destination d_my_log { file("/var/logs/myapp/${my_key:-invalid_key}.log" template(t_raw2json) ); }; -- Jorge Pereira On Wed, Jan 11, 2017 at 7:06 AM, Fekete, Róbert <robert.fekete@balabit.com> wrote:
Oh, right. Well, you could make an extra round of formatting+parsing, like incoming-message>format-json>lowercase-the-formatted-json>json-parser>format-json, but that would not be too efficient performance-wise. Or maybe you could use the lowercase function on the incoming message, depending on what it is like.
On Wed, Jan 11, 2017 at 9:46 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
On Wed, Jan 11, 2017 at 09:39:48AM +0100, Fekete, Róbert wrote:
Actually, in the docs we mention a 'lowercase' template function, I think algernon wrote it a few years ago, you can give it a try.
But that will not let you munge the keys, unfortunately
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (4)
-
Fabien Wernli
-
Fekete, Róbert
-
Jorge Pereira
-
Scheidler, Balázs