[syslog-ng] Elasticsearch 5 and syslog-ng

[Peter Eckel]

Hi, 

I'm currently investigating the Syslog NG -> Elasticsearch 2 destination for a project I'm working on, and started using basically the sample configuration in Peter Czanik's blog article. Thanks, by the way, for the great tutorial. 

There is one thing I'm currently struggling with: On my test system I have a fairly low volume of messages, and there seems to be an issue with flushing the cache of Syslog NG to Elasticsearch. To be precise: It doesn't happen. I can easily force a cache flush by reloading Syslog NG, but if I just keep it sitting there it doesn't log anything at all to Elasticsearch (while logs to files, e.g. /var/log/messages, are happening in real time). 

I tried configuring the flush-limit() in the elasticsearch2 destination, but the configuration prevents syslog-ng from starting with an error message: 

> Nov 23 17:30:28 rpm-test syslog-ng[14527]: Error parsing destination, destination plugin flush-limit not found in /etc/syslog-ng/conf.d/elasticsearch.conf at line 9, column 5:
> Nov 23 17:30:28 rpm-test syslog-ng[14527]: included from /etc/syslog-ng/syslog-ng.conf line 68, column 1
> Nov 23 17:30:28 rpm-test syslog-ng[14527]: flush-limit( 1000 )
> Nov 23 17:30:28 rpm-test syslog-ng[14527]: ^^^^^^^^^^^
> Nov 23 17:30:28 rpm-test syslog-ng[14527]: syslog-ng documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> Nov 23 17:30:28 rpm-test syslog-ng[14527]: mailing list: https://lists.balabit.hu/mailman/listinfo/syslog-ng

If I remove the flush-limit() line, the config loads without a problem. 

After some hours of operation, Syslog NG actually crashed on reload with a memory corruption issue (might be unlelated): 

> Nov 23 17:50:14 rpm-test
> systemd[1]: Started System Logger Daemon.
> Nov 24 11:04:11 rpm-test systemd[1]: Reloaded System Logger Daemon.
> Nov 24 11:04:15 rpm-test syslog-ng[31599]: *** Error in `/usr/sbin/syslog-ng': malloc(): memory corruption (fast): 0x00000000023293df ***
> Nov 24 11:04:15 rpm-test syslog-ng[31599]: ======= Backtrace: =========
> Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libc.so.6(+0x7b184)[0x7f87e29b7184]
> Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libc.so.6(+0x7e877)[0x7f87e29ba877]
> Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libc.so.6(__libc_calloc+0xb4)[0x7f87e29bc2d4]
> Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libglib-2.0.so.0(g_malloc0+0x17)[0x7f87e39e32c7]
> Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(log_multiplexer_new+0x13)[0x7f87e4529833]
> Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(cfg_tree_new_mpx+0x12)[0x7f87e4521872]
> Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x30136)[0x7f87e4522136]
> Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x2fc11)[0x7f87e4521c11]
> Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x2fb2b)[0x7f87e4521b2b]
> Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x2fc11)[0x7f87e4521c11]
> Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x302a9)[0x7f87e45222a9]
> Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x2fc11)[0x7f87e4521c11]
> Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(cfg_tree_compile_rule+0x35)[0x7f87e4522375]
> Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(cfg_tree_compile+0x4b)[0x7f87e45224cb]
> Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(cfg_tree_start+0x16)[0x7f87e4522576]
> Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(cfg_init+0x16e)[0x7f87e451d58e]
> Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x400de)[0x7f87e45320de]
> Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x407d7)[0x7f87e45327d7]
> Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libivykis.so.0(+0x3b4f)[0x7f87e2f1cb4f]
> Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libivykis.so.0(+0x5193)[0x7f87e2f1e193]
> Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libivykis.so.0(+0x5810)[0x7f87e2f1e810]
> Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libivykis.so.0(iv_main+0x44)[0x7f87e2f1f7d4]
> Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(main_loop_run+0x74)[0x7f87e4532734]
> Nov 24 11:04:15 rpm-test syslog-ng[31599]: /usr/sbin/syslog-ng(main+0x1b8)[0x4017b8]
> Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f87e295db15]
> Nov 24 11:04:15 rpm-test syslog-ng[31599]: /usr/sbin/syslog-ng[0x4018dd]

OS Version is CentOS 7.2, Syslog NG 5.8.1 (installed from Peter's repository), Elasticsearch 5.0.1 (installed from Elastic's repo). The machine is a freshly installed system. 

Any ideas are welcome ...

Best regards, 

  Peter.