[syslog-ng] Change JSON Format

[Daniel Neubacher]

Thanks guys, the explanation helped me a lot. It seems like I'm only limited a bit because of my old old stable 3.4 :P Hope I can update it soon.
This way of transforming the json seems much faster than logstash mutate. The logstash client lost quite a bit CPU usage.

-----Ursprüngliche Nachricht-----
Von: syslog-ng-bounces at lists.balabit.hu [mailto:syslog-ng-bounces at lists.balabit.hu] Im Auftrag von Fabien Wernli
Gesendet: Mittwoch, 28. Januar 2015 10:36
An: Syslog-ng users' and developers' mailing list
Betreff: Re: [syslog-ng] Change JSON Format

Hi Daniel,

On Wed, Jan 28, 2015 at 09:02:20AM +0000, Daniel Neubacher wrote:
> Hey there,
> right now I'm playing around with different json shippers for log files and I'm a bit lost with syslog-ng. I've read the docs but I still don't know how to change the json Fields syslog-ng sends out. In order to get my new Installation approved I have to keep the old field names in mind, for example syslog sends out HOST_FROM but I  need source_host. Right now logstash/mutate does the renaming but I don't like to waste performance there.
> 
> My destination:
> 
> destination d_logstash_syslog_syslog_new {tcp("consumer.foo.bar" 
> port(6002) template("$(format-json --scope selected_macros --scope 
> nv_pairs)\n") );};

You can use various helpers to manipulate nv-pairs as described in the relevant section of the documentation [1]. Here are some ideas:

* '-p source_host=$HOST_FROM' will add key 'source_host' with the contents of 'HOST_FROM'
* '-x HOST_FROM' will remove key 'HOST_FROM'
* '-p ISODATE=$ISODATE' will force key 'ISODATE' regardless of the 'scope()'
* '-k ISODATE' will do as above but this does not work on earlier
	versions of syslog-ng
* ' --rekey .classifier.* --add-prefix pdb' will add prefix 'pdb' to all
  macros beginning with '.classifier.'
* '--rekey .SDATA.* --replace-prefix .SDATA=.sdata' will replace prefix
  '.SDATA' with '.sdata'
* '--rekey .sdata.foo.* --shift 7' will remove the prefix '.sdata' by
  removing 7 chars

Hoping this helps

[1] http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.6-guides/en/syslog-ng-ose-v3.6-guide-admin/html-single/index.html#options-value-pairs
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq