Re: [syslog-ng] Help with ES destination error
Hi Fabien, I have tried the following: - emptying all index/docs in ES - create the test/test index with the CURL in my email - pointing syslog ES destination to the test/test index This resulted in the same error again. I have tried to change the template to just output all nv-pairs and use a complete new index - same error. Grabbing some packet capture now to see if I can spot anything wrong. Marco
On 28 Jan 2018, at 14:19, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi,
The reason I asked you to configure syslogng to index to "test" was to make sure you are in the same conditions as your curl command. You might for instance have a mapping template matching fw-* but not test.
Please either configure syslogng to index to test, or use the same fw- index on the curl cmdline.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, On Wed, Jan 31, 2018 at 10:35:36AM +0000, Marco Mignone wrote:
I have tried to change the template to just output all nv-pairs and use a complete new index - same error.
Grabbing some packet capture now to see if I can spot anything wrong.
Okay, great. Please share your findings. Maybe something ES6.1 doesn't like with the encoding. Haven't tried 6.x yet, sadly.
Hi Fabien, Ok after some deeper investigation I have found out the following: template t_sonicwall { template("$(format-json --scope all-nv-pairs --exclude MESSAGE,SOURCE,src.data,dst.data,cfield,nfield,mfield,fw.ip)"); }; ## DESTINATIONS ## destination d_elasticsearch { elasticsearch2( #client-lib-dir("/jarfiles/") client-lib-dir("/jarfiles/*.jar:/usr/lib/syslog-ng/3.13/java-modules/elastic-jest-client/*.jar:/usr/lib/syslog-ng/3.13/java-modules/") index("fw") # index("fw-${YEAR}.${MONTH}.$(lowercase '${.classifier.class}')") type("syslog") #time-zone("UTC") client_mode("http") cluster("tst-docker-cluster") #cluster_url("http://192.168.32.100:9200") cluster_url("http://tst-es6:9200") skip-cluster-health-check("yes") template(t_sonicwall) flush-limit("1") ); }; The template substitution in the destination does not seem to work - in the packet capture I can see syslog-ng sending just ’t_sonicwall’ as a string instead of replacing that with the template defined in the config file. If in the destination I put the full line: template("$(format-json --scope all-nv-pairs --exclude MESSAGE,SOURCE,src.data,dst.data,cfield,nfield,mfield,fw.ip)"); Things works correctly and a proper JSON object is sent to ES. In my config file I tried both template(t_sonicwall) and template(“t_sonicwall”) as I am never sure what goes in quotes and what doesn’t. I am not sure if this is a bug or just something wrong with the Docker image of syslog-ng - weird I am the only one experiencing this? Thanks, Marco
On 31 Jan 2018, at 10:35, Marco Mignone <info@marcomignone.com> wrote:
Hi Fabien, I have tried the following:
- emptying all index/docs in ES - create the test/test index with the CURL in my email - pointing syslog ES destination to the test/test index
This resulted in the same error again.
I have tried to change the template to just output all nv-pairs and use a complete new index - same error.
Grabbing some packet capture now to see if I can spot anything wrong.
Marco
On 28 Jan 2018, at 14:19, Fabien Wernli <wernli@in2p3.fr <mailto:wernli@in2p3.fr>> wrote:
Hi,
The reason I asked you to configure syslogng to index to "test" was to make sure you are in the same conditions as your curl command. You might for instance have a mapping template matching fw-* but not test.
Please either configure syslogng to index to test, or use the same fw- index on the curl cmdline.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
Furthermore to my previous email looks like the substitution does not work in the ES destination options. Instead in a file destination works perfectly. destination d_json { file("/var/log/json.log" template(t_sonicwall)); }; Marco
On 31 Jan 2018, at 11:26, Marco Mignone <info@marcomignone.com> wrote:
Hi Fabien, Ok after some deeper investigation I have found out the following:
template t_sonicwall { template("$(format-json --scope all-nv-pairs --exclude MESSAGE,SOURCE,src.data,dst.data,cfield,nfield,mfield,fw.ip)"); };
## DESTINATIONS ##
destination d_elasticsearch { elasticsearch2( #client-lib-dir("/jarfiles/") client-lib-dir("/jarfiles/*.jar:/usr/lib/syslog-ng/3.13/java-modules/elastic-jest-client/*.jar:/usr/lib/syslog-ng/3.13/java-modules/") index("fw") # index("fw-${YEAR}.${MONTH}.$(lowercase '${.classifier.class}')") type("syslog") #time-zone("UTC") client_mode("http") cluster("tst-docker-cluster") #cluster_url("http://192.168.32.100:9200 <http://192.168.32.100:9200/>") cluster_url("http://tst-es6:9200 <http://tst-es6:9200/>") skip-cluster-health-check("yes") template(t_sonicwall) flush-limit("1") ); };
The template substitution in the destination does not seem to work - in the packet capture I can see syslog-ng sending just ’t_sonicwall’ as a string instead of replacing that with the template defined in the config file. If in the destination I put the full line: template("$(format-json --scope all-nv-pairs --exclude MESSAGE,SOURCE,src.data,dst.data,cfield,nfield,mfield,fw.ip)");
Things works correctly and a proper JSON object is sent to ES. In my config file I tried both template(t_sonicwall) and template(“t_sonicwall”) as I am never sure what goes in quotes and what doesn’t.
I am not sure if this is a bug or just something wrong with the Docker image of syslog-ng - weird I am the only one experiencing this?
Thanks, Marco
Hi, On Wed, Jan 31, 2018 at 11:26:00AM +0000, Marco Mignone wrote:
The template substitution in the destination does not seem to work - in the packet capture I can see syslog-ng sending just ’t_sonicwall’ as a string instead of replacing that with the template defined in the config file.
That certainly lookls like a bug! Could you please create a minimal config to reproduce the problem, and open an issue on github? Cheers!
Hi Fabien, Hope this is understandable as it is my first one ever. https://github.com/balabit/syslog-ng/issues/1870 <https://github.com/balabit/syslog-ng/issues/1870> Marco
On 1 Feb 2018, at 12:22, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi,
On Wed, Jan 31, 2018 at 11:26:00AM +0000, Marco Mignone wrote:
The template substitution in the destination does not seem to work - in the packet capture I can see syslog-ng sending just ’t_sonicwall’ as a string instead of replacing that with the template defined in the config file.
That certainly lookls like a bug!
Could you please create a minimal config to reproduce the problem, and open an issue on github?
Cheers!
participants (2)
-
Fabien Wernli
-
Marco Mignone