Hey there, right now I'm playing around with different json shippers for log files and I'm a bit lost with syslog-ng. I've read the docs but I still don't know how to change the json Fields syslog-ng sends out. In order to get my new Installation approved I have to keep the old field names in mind, for example syslog sends out HOST_FROM but I need source_host. Right now logstash/mutate does the renaming but I don't like to waste performance there. My destination: destination d_logstash_syslog_syslog_new {tcp("consumer.foo.bar" port(6002) template("$(format-json --scope selected_macros --scope nv_pairs)\n") );}; Thanks in advance Daniel
Hi, format-json uses the value-pairs syntax, and I agree that the docs is not very clear on that. Here's the right section: http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-gu... You can rename stuff using the --pair argument: $(format-json --pair name=$VALUE macro resolution is possible after the equal sign, in fact it is a complete template, that may again contain template functions and such. You might even be interested in rekey() where you can apply simple transformation on key names automatically: $(format-json --rekey .cee.* --add-prefix events.) Hope this helps, -- Bazsi On Wed, Jan 28, 2015 at 10:02 AM, Daniel Neubacher < daniel.neubacher@xing.com> wrote:
Hey there,
right now I’m playing around with different json shippers for log files and I’m a bit lost with syslog-ng. I’ve read the docs but I still don’t know how to change the json Fields syslog-ng sends out. In order to get my new Installation approved I have to keep the old field names in mind, for example syslog sends out HOST_FROM but I need source_host. Right now logstash/mutate does the renaming but I don’t like to waste performance there.
My destination:
destination d_logstash_syslog_syslog_new {tcp("consumer.foo.bar" port(6002) template("$(format-json --scope selected_macros --scope nv_pairs)\n") );};
Thanks in advance
Daniel
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Daniel, On Wed, Jan 28, 2015 at 09:02:20AM +0000, Daniel Neubacher wrote:
Hey there, right now I'm playing around with different json shippers for log files and I'm a bit lost with syslog-ng. I've read the docs but I still don't know how to change the json Fields syslog-ng sends out. In order to get my new Installation approved I have to keep the old field names in mind, for example syslog sends out HOST_FROM but I need source_host. Right now logstash/mutate does the renaming but I don't like to waste performance there.
My destination:
destination d_logstash_syslog_syslog_new {tcp("consumer.foo.bar" port(6002) template("$(format-json --scope selected_macros --scope nv_pairs)\n") );};
You can use various helpers to manipulate nv-pairs as described in the relevant section of the documentation [1]. Here are some ideas: * '-p source_host=$HOST_FROM' will add key 'source_host' with the contents of 'HOST_FROM' * '-x HOST_FROM' will remove key 'HOST_FROM' * '-p ISODATE=$ISODATE' will force key 'ISODATE' regardless of the 'scope()' * '-k ISODATE' will do as above but this does not work on earlier versions of syslog-ng * ' --rekey .classifier.* --add-prefix pdb' will add prefix 'pdb' to all macros beginning with '.classifier.' * '--rekey .SDATA.* --replace-prefix .SDATA=.sdata' will replace prefix '.SDATA' with '.sdata' * '--rekey .sdata.foo.* --shift 7' will remove the prefix '.sdata' by removing 7 chars Hoping this helps [1] http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.6-guide...
Thanks guys, the explanation helped me a lot. It seems like I'm only limited a bit because of my old old stable 3.4 :P Hope I can update it soon. This way of transforming the json seems much faster than logstash mutate. The logstash client lost quite a bit CPU usage. -----Ursprüngliche Nachricht----- Von: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] Im Auftrag von Fabien Wernli Gesendet: Mittwoch, 28. Januar 2015 10:36 An: Syslog-ng users' and developers' mailing list Betreff: Re: [syslog-ng] Change JSON Format Hi Daniel, On Wed, Jan 28, 2015 at 09:02:20AM +0000, Daniel Neubacher wrote:
Hey there, right now I'm playing around with different json shippers for log files and I'm a bit lost with syslog-ng. I've read the docs but I still don't know how to change the json Fields syslog-ng sends out. In order to get my new Installation approved I have to keep the old field names in mind, for example syslog sends out HOST_FROM but I need source_host. Right now logstash/mutate does the renaming but I don't like to waste performance there.
My destination:
destination d_logstash_syslog_syslog_new {tcp("consumer.foo.bar" port(6002) template("$(format-json --scope selected_macros --scope nv_pairs)\n") );};
You can use various helpers to manipulate nv-pairs as described in the relevant section of the documentation [1]. Here are some ideas: * '-p source_host=$HOST_FROM' will add key 'source_host' with the contents of 'HOST_FROM' * '-x HOST_FROM' will remove key 'HOST_FROM' * '-p ISODATE=$ISODATE' will force key 'ISODATE' regardless of the 'scope()' * '-k ISODATE' will do as above but this does not work on earlier versions of syslog-ng * ' --rekey .classifier.* --add-prefix pdb' will add prefix 'pdb' to all macros beginning with '.classifier.' * '--rekey .SDATA.* --replace-prefix .SDATA=.sdata' will replace prefix '.SDATA' with '.sdata' * '--rekey .sdata.foo.* --shift 7' will remove the prefix '.sdata' by removing 7 chars Hoping this helps [1] http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.6-guide... ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (3)
-
Daniel Neubacher
-
Fabien Wernli
-
Scheidler, Balázs