Parsing Messages for Elasticsearch
Hi, I get a lot of name="value" formated messages which I want to receive in syslog-ng and pass them into elasticsearch via the elasticsearch2 module. The message passing works fine, however I'm not able to parse the messages for elasticsearch yet. What I do get in elasticsearch is the whole syslog as a string in the message, what I actually want is all the keys as fields with the value. My elasticsearch destination is configured as followed: destination d_elastic { elasticsearch2( client-lib-dir("/usr/share/elastic-5-lib/lib") client-mode("http") cluster("ng") index("ng-${YEAR}.${MONTH}.${DAY}") type("syslog") cluster-url("http://172.18.1.5:9200/") template("$(format-json --scope nv-pairs --exclude R_DATE --key ISODATE)\n") flush-limit("100") concurrent-requests("10") disk-buffer( disk-buf-size(500000000) dir("/opt/disk-buffer") reliable(yes) ) ); }; I thought the nv-pairs scope would do the trick but it doesn't seem to have any effect on the message. Any idea what I'm doing wrong here or can syslog-ng not accomplish what I want to do at all? Thanks Tim
Hi Tim, On Wed, Dec 21, 2016 at 11:47:46AM +0000, Jentz, Tim wrote:
I thought the nv-pairs scope would do the trick but it doesn't seem to have any effect on the message. Any idea what I'm doing wrong here or can syslog-ng not accomplish what I want to do at all?
No, the `format-json()` function will merely generate JSON for all the syslog-ng macros, e.g. MESSAGE. But your key=value stings are inside the MESSAGE macro, and for them to be extracted you need to parse the content of MESSAGE. Luckily for you there's the `kv-parser()` which will do just that: https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-o...
Hi, This article discusses parsing plus elastic, albeit it does use db-parser() for the parsing piece, where you probably want to use kv-parser() as Fabien has mentioned. https://www.balabit.com/blog/how-to-parse-data-with-syslog-ng-store-in-elast... -- Bazsi On Wed, Dec 21, 2016 at 12:52 PM, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi Tim,
On Wed, Dec 21, 2016 at 11:47:46AM +0000, Jentz, Tim wrote:
I thought the nv-pairs scope would do the trick but it doesn't seem to have any effect on the message. Any idea what I'm doing wrong here or can syslog-ng not accomplish what I want to do at all?
No, the `format-json()` function will merely generate JSON for all the syslog-ng macros, e.g. MESSAGE. But your key=value stings are inside the MESSAGE macro, and for them to be extracted you need to parse the content of MESSAGE.
Luckily for you there's the `kv-parser()` which will do just that:
https://www.balabit.com/documents/syslog-ng-ose- latest-guides/en/syslog-ng-ose-guide-admin/html/key-value-parser.html
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (3)
-
Fabien Wernli
-
Jentz, Tim
-
Scheidler, Balázs