[syslog-ng] Help with ES destination error
Marco Mignone
info at marcomignone.com
Wed Jan 31 11:26:00 UTC 2018
Hi Fabien,
Ok after some deeper investigation I have found out the following:
template t_sonicwall {
template("$(format-json --scope all-nv-pairs --exclude MESSAGE,SOURCE,src.data,dst.data,cfield,nfield,mfield,fw.ip)");
};
## DESTINATIONS ##
destination d_elasticsearch {
elasticsearch2(
#client-lib-dir("/jarfiles/")
client-lib-dir("/jarfiles/*.jar:/usr/lib/syslog-ng/3.13/java-modules/elastic-jest-client/*.jar:/usr/lib/syslog-ng/3.13/java-modules/")
index("fw")
# index("fw-${YEAR}.${MONTH}.$(lowercase '${.classifier.class}')")
type("syslog")
#time-zone("UTC")
client_mode("http")
cluster("tst-docker-cluster")
#cluster_url("http://192.168.32.100:9200")
cluster_url("http://tst-es6:9200")
skip-cluster-health-check("yes")
template(t_sonicwall)
flush-limit("1")
);
};
The template substitution in the destination does not seem to work - in the packet capture I can see syslog-ng sending just ’t_sonicwall’ as a string instead of replacing that with the template defined in the config file.
If in the destination I put the full line:
template("$(format-json --scope all-nv-pairs --exclude MESSAGE,SOURCE,src.data,dst.data,cfield,nfield,mfield,fw.ip)");
Things works correctly and a proper JSON object is sent to ES.
In my config file I tried both template(t_sonicwall) and template(“t_sonicwall”) as I am never sure what goes in quotes and what doesn’t.
I am not sure if this is a bug or just something wrong with the Docker image of syslog-ng - weird I am the only one experiencing this?
Thanks,
Marco
> On 31 Jan 2018, at 10:35, Marco Mignone <info at marcomignone.com> wrote:
>
> Hi Fabien,
> I have tried the following:
>
> - emptying all index/docs in ES
> - create the test/test index with the CURL in my email
> - pointing syslog ES destination to the test/test index
>
> This resulted in the same error again.
>
> I have tried to change the template to just output all nv-pairs and use a complete new index - same error.
>
> Grabbing some packet capture now to see if I can spot anything wrong.
>
> Marco
>
>> On 28 Jan 2018, at 14:19, Fabien Wernli <wernli at in2p3.fr <mailto:wernli at in2p3.fr>> wrote:
>>
>> Hi,
>>
>> The reason I asked you to configure syslogng to index to "test" was to make
>> sure you are in the same conditions as your curl command.
>> You might for instance have a mapping template matching fw-* but not test.
>>
>> Please either configure syslogng to index to test, or use the same fw- index
>> on the curl cmdline.
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng>
>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng>
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
>>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng>
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20180131/c647ee7a/attachment-0001.html>
More information about the syslog-ng
mailing list