Elasticsearch 5 and syslog-ng
Hi, For the last six months, Elastic’s communication centered around the upcoming Elastic Stack 5.0. And finally it is here: tons of new features, improved performance and a single version number for all Elastic products. Compatibility with syslog-ng was checked already during the alpha phase of development, as syslog-ng is becoming popular among Elasticsearch users: it can greatly simplify logging to Elasticsearch <https://www.balabit.com/blog/logging-to-elasticsearch-made-simple-with-syslog-ng/> . As Elastic Stack 5.0.0 is now generally available, here is a quick how-to guide to get you started with syslog-ng 3.8.1 and Elasticsearch 5.0.0 on RHEL/CentOS 7: https://www.balabit.com/blog/syslog-ng-and-elasticsearch-5-getting-started-o... Bye, Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik
Hi, I'm currently investigating the Syslog NG -> Elasticsearch 2 destination for a project I'm working on, and started using basically the sample configuration in Peter Czanik's blog article. Thanks, by the way, for the great tutorial. There is one thing I'm currently struggling with: On my test system I have a fairly low volume of messages, and there seems to be an issue with flushing the cache of Syslog NG to Elasticsearch. To be precise: It doesn't happen. I can easily force a cache flush by reloading Syslog NG, but if I just keep it sitting there it doesn't log anything at all to Elasticsearch (while logs to files, e.g. /var/log/messages, are happening in real time). I tried configuring the flush-limit() in the elasticsearch2 destination, but the configuration prevents syslog-ng from starting with an error message:
Nov 23 17:30:28 rpm-test syslog-ng[14527]: Error parsing destination, destination plugin flush-limit not found in /etc/syslog-ng/conf.d/elasticsearch.conf at line 9, column 5: Nov 23 17:30:28 rpm-test syslog-ng[14527]: included from /etc/syslog-ng/syslog-ng.conf line 68, column 1 Nov 23 17:30:28 rpm-test syslog-ng[14527]: flush-limit( 1000 ) Nov 23 17:30:28 rpm-test syslog-ng[14527]: ^^^^^^^^^^^ Nov 23 17:30:28 rpm-test syslog-ng[14527]: syslog-ng documentation: http://www.balabit.com/support/documentation/?product=syslog-ng Nov 23 17:30:28 rpm-test syslog-ng[14527]: mailing list: https://lists.balabit.hu/mailman/listinfo/syslog-ng
If I remove the flush-limit() line, the config loads without a problem. After some hours of operation, Syslog NG actually crashed on reload with a memory corruption issue (might be unlelated):
Nov 23 17:50:14 rpm-test systemd[1]: Started System Logger Daemon. Nov 24 11:04:11 rpm-test systemd[1]: Reloaded System Logger Daemon. Nov 24 11:04:15 rpm-test syslog-ng[31599]: *** Error in `/usr/sbin/syslog-ng': malloc(): memory corruption (fast): 0x00000000023293df *** Nov 24 11:04:15 rpm-test syslog-ng[31599]: ======= Backtrace: ========= Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libc.so.6(+0x7b184)[0x7f87e29b7184] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libc.so.6(+0x7e877)[0x7f87e29ba877] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libc.so.6(__libc_calloc+0xb4)[0x7f87e29bc2d4] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libglib-2.0.so.0(g_malloc0+0x17)[0x7f87e39e32c7] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(log_multiplexer_new+0x13)[0x7f87e4529833] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(cfg_tree_new_mpx+0x12)[0x7f87e4521872] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x30136)[0x7f87e4522136] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x2fc11)[0x7f87e4521c11] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x2fb2b)[0x7f87e4521b2b] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x2fc11)[0x7f87e4521c11] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x302a9)[0x7f87e45222a9] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x2fc11)[0x7f87e4521c11] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(cfg_tree_compile_rule+0x35)[0x7f87e4522375] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(cfg_tree_compile+0x4b)[0x7f87e45224cb] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(cfg_tree_start+0x16)[0x7f87e4522576] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(cfg_init+0x16e)[0x7f87e451d58e] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x400de)[0x7f87e45320de] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x407d7)[0x7f87e45327d7] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libivykis.so.0(+0x3b4f)[0x7f87e2f1cb4f] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libivykis.so.0(+0x5193)[0x7f87e2f1e193] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libivykis.so.0(+0x5810)[0x7f87e2f1e810] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libivykis.so.0(iv_main+0x44)[0x7f87e2f1f7d4] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(main_loop_run+0x74)[0x7f87e4532734] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /usr/sbin/syslog-ng(main+0x1b8)[0x4017b8] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f87e295db15] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /usr/sbin/syslog-ng[0x4018dd]
OS Version is CentOS 7.2, Syslog NG 5.8.1 (installed from Peter's repository), Elasticsearch 5.0.1 (installed from Elastic's repo). The machine is a freshly installed system. Any ideas are welcome ... Best regards, Peter.
OK, I got the first one figured out myself ... flush-limit( 1000 ) does not work. flush-limit( "1000" ) does, which is inconsistent with the usual syslog-ng.conf behaviour. And setting it to "1" actually fixed the slow propagation to ES for me. Now I need to see whether the crash still occurs. Another question that came up: Is it possible to do flushing based on a time interval and not on a message count in the queue? It's rather unsatisfactory to wait for critical messages to appear in ES. Best regards, Peter.
On 24 Nov 2016, at 12:28, Peter Eckel <lists@eckel-edv.de> wrote:
I'm currently investigating the Syslog NG -> Elasticsearch 2 destination for a project I'm working on, and started using basically the sample configuration in Peter Czanik's blog article. Thanks, by the way, for the great tutorial.
There is one thing I'm currently struggling with: On my test system I have a fairly low volume of messages, and there seems to be an issue with flushing the cache of Syslog NG to Elasticsearch. To be precise: It doesn't happen. I can easily force a cache flush by reloading Syslog NG, but if I just keep it sitting there it doesn't log anything at all to Elasticsearch (while logs to files, e.g. /var/log/messages, are happening in real time).
I tried configuring the flush-limit() in the elasticsearch2 destination, but the configuration prevents syslog-ng from starting with an error message:
Nov 23 17:30:28 rpm-test syslog-ng[14527]: Error parsing destination, destination plugin flush-limit not found in /etc/syslog-ng/conf.d/elasticsearch.conf at line 9, column 5: Nov 23 17:30:28 rpm-test syslog-ng[14527]: included from /etc/syslog-ng/syslog-ng.conf line 68, column 1 Nov 23 17:30:28 rpm-test syslog-ng[14527]: flush-limit( 1000 ) Nov 23 17:30:28 rpm-test syslog-ng[14527]: ^^^^^^^^^^^ Nov 23 17:30:28 rpm-test syslog-ng[14527]: syslog-ng documentation: http://www.balabit.com/support/documentation/?product=syslog-ng Nov 23 17:30:28 rpm-test syslog-ng[14527]: mailing list: https://lists.balabit.hu/mailman/listinfo/syslog-ng
If I remove the flush-limit() line, the config loads without a problem.
[fixed]
After some hours of operation, Syslog NG actually crashed on reload with a memory corruption issue (might be unlelated):
Nov 23 17:50:14 rpm-test systemd[1]: Started System Logger Daemon. Nov 24 11:04:11 rpm-test systemd[1]: Reloaded System Logger Daemon. Nov 24 11:04:15 rpm-test syslog-ng[31599]: *** Error in `/usr/sbin/syslog-ng': malloc(): memory corruption (fast): 0x00000000023293df *** Nov 24 11:04:15 rpm-test syslog-ng[31599]: ======= Backtrace: ========= Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libc.so.6(+0x7b184)[0x7f87e29b7184] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libc.so.6(+0x7e877)[0x7f87e29ba877] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libc.so.6(__libc_calloc+0xb4)[0x7f87e29bc2d4] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libglib-2.0.so.0(g_malloc0+0x17)[0x7f87e39e32c7] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(log_multiplexer_new+0x13)[0x7f87e4529833] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(cfg_tree_new_mpx+0x12)[0x7f87e4521872] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x30136)[0x7f87e4522136] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x2fc11)[0x7f87e4521c11] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x2fb2b)[0x7f87e4521b2b] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x2fc11)[0x7f87e4521c11] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x302a9)[0x7f87e45222a9] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x2fc11)[0x7f87e4521c11] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(cfg_tree_compile_rule+0x35)[0x7f87e4522375] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(cfg_tree_compile+0x4b)[0x7f87e45224cb] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(cfg_tree_start+0x16)[0x7f87e4522576] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(cfg_init+0x16e)[0x7f87e451d58e] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x400de)[0x7f87e45320de] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x407d7)[0x7f87e45327d7] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libivykis.so.0(+0x3b4f)[0x7f87e2f1cb4f] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libivykis.so.0(+0x5193)[0x7f87e2f1e193] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libivykis.so.0(+0x5810)[0x7f87e2f1e810] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libivykis.so.0(iv_main+0x44)[0x7f87e2f1f7d4] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(main_loop_run+0x74)[0x7f87e4532734] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /usr/sbin/syslog-ng(main+0x1b8)[0x4017b8] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f87e295db15] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /usr/sbin/syslog-ng[0x4018dd]
[status unknown]
OS Version is CentOS 7.2, Syslog NG 5.8.1 (installed from Peter's repository), Elasticsearch 5.0.1 (installed from Elastic's repo). The machine is a freshly installed system.
Hi, This patch changes the grammar, so it accepts number tokens as well: https://github.com/balabit/syslog-ng/tree/f/java-should-accept-numbers-as-op... testing would be appreciated as this is a "blind" patch, I haven't tried it as I don't personally use elasticsearch/java destinations. The memory corruption problem was already raised in a different thread, but I don't know about any exact outcome from those emails, maybe @lbudai does. -- Bazsi On Thu, Nov 24, 2016 at 3:06 PM, Peter Eckel <lists@eckel-edv.de> wrote:
OK, I got the first one figured out myself ...
flush-limit( 1000 )
does not work.
flush-limit( "1000" )
does, which is inconsistent with the usual syslog-ng.conf behaviour. And setting it to "1" actually fixed the slow propagation to ES for me.
Now I need to see whether the crash still occurs.
Another question that came up: Is it possible to do flushing based on a time interval and not on a message count in the queue? It's rather unsatisfactory to wait for critical messages to appear in ES.
Best regards,
Peter.
On 24 Nov 2016, at 12:28, Peter Eckel <lists@eckel-edv.de> wrote:
I'm currently investigating the Syslog NG -> Elasticsearch 2 destination for a project I'm working on, and started using basically the sample configuration in Peter Czanik's blog article. Thanks, by the way, for the great tutorial.
There is one thing I'm currently struggling with: On my test system I have a fairly low volume of messages, and there seems to be an issue with flushing the cache of Syslog NG to Elasticsearch. To be precise: It doesn't happen. I can easily force a cache flush by reloading Syslog NG, but if I just keep it sitting there it doesn't log anything at all to Elasticsearch (while logs to files, e.g. /var/log/messages, are happening in real time).
I tried configuring the flush-limit() in the elasticsearch2 destination, but the configuration prevents syslog-ng from starting with an error message:
Nov 23 17:30:28 rpm-test syslog-ng[14527]: Error parsing destination, destination plugin flush-limit not found in /etc/syslog-ng/conf.d/elasticsearch.conf at line 9, column 5: Nov 23 17:30:28 rpm-test syslog-ng[14527]: included from /etc/syslog-ng/syslog-ng.conf line 68, column 1 Nov 23 17:30:28 rpm-test syslog-ng[14527]: flush-limit( 1000 ) Nov 23 17:30:28 rpm-test syslog-ng[14527]: ^^^^^^^^^^^ Nov 23 17:30:28 rpm-test syslog-ng[14527]: syslog-ng documentation: http://www.balabit.com/support/documentation/?product=syslog-ng Nov 23 17:30:28 rpm-test syslog-ng[14527]: mailing list: https://lists.balabit.hu/mailman/listinfo/syslog-ng
If I remove the flush-limit() line, the config loads without a problem.
[fixed]
After some hours of operation, Syslog NG actually crashed on reload with
a memory corruption issue (might be unlelated):
Nov 23 17:50:14 rpm-test systemd[1]: Started System Logger Daemon. Nov 24 11:04:11 rpm-test systemd[1]: Reloaded System Logger Daemon. Nov 24 11:04:15 rpm-test syslog-ng[31599]: *** Error in
`/usr/sbin/syslog-ng': malloc(): memory corruption (fast): 0x00000000023293df ***
Nov 24 11:04:15 rpm-test syslog-ng[31599]: ======= Backtrace: ========= Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libc.so.6(+0x7b184)[ 0x7f87e29b7184] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libc.so.6(+0x7e877)[ 0x7f87e29ba877] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libc.so.6(__libc_ calloc+0xb4)[0x7f87e29bc2d4] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libglib-2.0.so.0(g_ malloc0+0x17)[0x7f87e39e32c7] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0( log_multiplexer_new+0x13)[0x7f87e4529833] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0( cfg_tree_new_mpx+0x12)[0x7f87e4521872] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x30136)[0x7f87e4522136] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x2fc11)[0x7f87e4521c11] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x2fb2b)[0x7f87e4521b2b] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x2fc11)[0x7f87e4521c11] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x302a9)[0x7f87e45222a9] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x2fc11)[0x7f87e4521c11] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0( cfg_tree_compile_rule+0x35)[0x7f87e4522375] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0( cfg_tree_compile+0x4b)[0x7f87e45224cb] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0( cfg_tree_start+0x16)[0x7f87e4522576] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0( cfg_init+0x16e)[0x7f87e451d58e] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x400de)[0x7f87e45320de] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x407d7)[0x7f87e45327d7] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libivykis.so.0(+0x3b4f)[0x7f87e2f1cb4f] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libivykis.so.0(+0x5193)[0x7f87e2f1e193] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libivykis.so.0(+0x5810)[0x7f87e2f1e810] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libivykis.so.0(iv_main+0x44)[0x7f87e2f1f7d4] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0( main_loop_run+0x74)[0x7f87e4532734] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /usr/sbin/syslog-ng(main+ 0x1b8)[0x4017b8] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f87e295db15] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /usr/sbin/syslog-ng[0x4018dd]
[status unknown]
OS Version is CentOS 7.2, Syslog NG 5.8.1 (installed from Peter's
repository), Elasticsearch 5.0.1 (installed from Elastic's repo). The machine is a freshly installed system.
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Baszi, thanks for the extremely quick response and the patch - I just implemented it, repackaged and tested the config, and now the usual syntax works just fine. Thanks! As for the crash I will dive deeper into this after I have investigated my alternative solution of using the http() destination. My aim is to get the basic functionality of the elasticsearch() destination with the http() destination instead, as I want to get rid of the Syslog NG daemon having a dependency from Java. Best regards, Peter.
Hi, @Bazsi: could you send a PR with the grammar changes? L. On Thu, Nov 24, 2016 at 4:13 PM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Hi,
This patch changes the grammar, so it accepts number tokens as well:
https://github.com/balabit/syslog-ng/tree/f/java-should- accept-numbers-as-options
testing would be appreciated as this is a "blind" patch, I haven't tried it as I don't personally use elasticsearch/java destinations.
The memory corruption problem was already raised in a different thread, but I don't know about any exact outcome from those emails, maybe @lbudai does.
-- Bazsi
On Thu, Nov 24, 2016 at 3:06 PM, Peter Eckel <lists@eckel-edv.de> wrote:
OK, I got the first one figured out myself ...
flush-limit( 1000 )
does not work.
flush-limit( "1000" )
does, which is inconsistent with the usual syslog-ng.conf behaviour. And setting it to "1" actually fixed the slow propagation to ES for me.
Now I need to see whether the crash still occurs.
Another question that came up: Is it possible to do flushing based on a time interval and not on a message count in the queue? It's rather unsatisfactory to wait for critical messages to appear in ES.
Best regards,
Peter.
On 24 Nov 2016, at 12:28, Peter Eckel <lists@eckel-edv.de> wrote:
I'm currently investigating the Syslog NG -> Elasticsearch 2 destination for a project I'm working on, and started using basically the sample configuration in Peter Czanik's blog article. Thanks, by the way, for the great tutorial.
There is one thing I'm currently struggling with: On my test system I have a fairly low volume of messages, and there seems to be an issue with flushing the cache of Syslog NG to Elasticsearch. To be precise: It doesn't happen. I can easily force a cache flush by reloading Syslog NG, but if I just keep it sitting there it doesn't log anything at all to Elasticsearch (while logs to files, e.g. /var/log/messages, are happening in real time).
I tried configuring the flush-limit() in the elasticsearch2 destination, but the configuration prevents syslog-ng from starting with an error message:
Nov 23 17:30:28 rpm-test syslog-ng[14527]: Error parsing destination, destination plugin flush-limit not found in /etc/syslog-ng/conf.d/elasticsearch.conf at line 9, column 5: Nov 23 17:30:28 rpm-test syslog-ng[14527]: included from /etc/syslog-ng/syslog-ng.conf line 68, column 1 Nov 23 17:30:28 rpm-test syslog-ng[14527]: flush-limit( 1000 ) Nov 23 17:30:28 rpm-test syslog-ng[14527]: ^^^^^^^^^^^ Nov 23 17:30:28 rpm-test syslog-ng[14527]: syslog-ng documentation: http://www.balabit.com/support/documentation/?product=syslog-ng Nov 23 17:30:28 rpm-test syslog-ng[14527]: mailing list: https://lists.balabit.hu/mailman/listinfo/syslog-ng
If I remove the flush-limit() line, the config loads without a problem.
[fixed]
After some hours of operation, Syslog NG actually crashed on reload
with a memory corruption issue (might be unlelated):
Nov 23 17:50:14 rpm-test systemd[1]: Started System Logger Daemon. Nov 24 11:04:11 rpm-test systemd[1]: Reloaded System Logger Daemon. Nov 24 11:04:15 rpm-test syslog-ng[31599]: *** Error in
`/usr/sbin/syslog-ng': malloc(): memory corruption (fast): 0x00000000023293df ***
Nov 24 11:04:15 rpm-test syslog-ng[31599]: ======= Backtrace: ========= Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libc.so.6(+0x7b184)[0x7f87e29b7184] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libc.so.6(+0x7e877)[0x7f87e29ba877] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libc.so.6(__libc_calloc+0xb4)[0x7f87e29bc2d4] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libglib-2.0.so.0(g_malloc0+0x17)[0x7f87e39e32c7] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(log_multiplexer_new+0x13)[0x7f87e4529833] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(cfg_tree_new_mpx+0x12)[0x7f87e4521872] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x30136)[0x7f87e4522136] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x2fc11)[0x7f87e4521c11] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x2fb2b)[0x7f87e4521b2b] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x2fc11)[0x7f87e4521c11] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x302a9)[0x7f87e45222a9] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x2fc11)[0x7f87e4521c11] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(cfg_tree_compile_rule+0x35)[0x7f87e4522375] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(cfg_tree_compile+0x4b)[0x7f87e45224cb] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(cfg_tree_start+0x16)[0x7f87e4522576] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(cfg_init+0x16e)[0x7f87e451d58e] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x400de)[0x7f87e45320de] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x407d7)[0x7f87e45327d7] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libivykis.so.0(+0x3b4f)[0x7f87e2f1cb4f] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libivykis.so.0(+0x5193)[0x7f87e2f1e193] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libivykis.so.0(+0x5810)[0x7f87e2f1e810] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libivykis.so.0(iv_main+0x44)[0x7f87e2f1f7d4] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(main_loop_run+0x74)[0x7f87e4532734] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /usr/sbin/syslog-ng(main+0x1b8)[0x4017b8] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f87e295db15] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /usr/sbin/syslog-ng[0x4018dd]
[status unknown]
OS Version is CentOS 7.2, Syslog NG 5.8.1 (installed from Peter's
repository), Elasticsearch 5.0.1 (installed from Elastic's repo). The machine is a freshly installed system.
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Did exactly that. I just didnt want to submit the pr until some testing was done. On Nov 25, 2016 3:05 PM, "Budai, László" <laszlo.budai@balabit.com> wrote:
Hi,
@Bazsi: could you send a PR with the grammar changes?
L.
On Thu, Nov 24, 2016 at 4:13 PM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Hi,
This patch changes the grammar, so it accepts number tokens as well:
https://github.com/balabit/syslog-ng/tree/f/java-should-acce pt-numbers-as-options
testing would be appreciated as this is a "blind" patch, I haven't tried it as I don't personally use elasticsearch/java destinations.
The memory corruption problem was already raised in a different thread, but I don't know about any exact outcome from those emails, maybe @lbudai does.
-- Bazsi
On Thu, Nov 24, 2016 at 3:06 PM, Peter Eckel <lists@eckel-edv.de> wrote:
OK, I got the first one figured out myself ...
flush-limit( 1000 )
does not work.
flush-limit( "1000" )
does, which is inconsistent with the usual syslog-ng.conf behaviour. And setting it to "1" actually fixed the slow propagation to ES for me.
Now I need to see whether the crash still occurs.
Another question that came up: Is it possible to do flushing based on a time interval and not on a message count in the queue? It's rather unsatisfactory to wait for critical messages to appear in ES.
Best regards,
Peter.
On 24 Nov 2016, at 12:28, Peter Eckel <lists@eckel-edv.de> wrote:
I'm currently investigating the Syslog NG -> Elasticsearch 2 destination for a project I'm working on, and started using basically the sample configuration in Peter Czanik's blog article. Thanks, by the way, for the great tutorial.
There is one thing I'm currently struggling with: On my test system I have a fairly low volume of messages, and there seems to be an issue with flushing the cache of Syslog NG to Elasticsearch. To be precise: It doesn't happen. I can easily force a cache flush by reloading Syslog NG, but if I just keep it sitting there it doesn't log anything at all to Elasticsearch (while logs to files, e.g. /var/log/messages, are happening in real time).
I tried configuring the flush-limit() in the elasticsearch2 destination, but the configuration prevents syslog-ng from starting with an error message:
Nov 23 17:30:28 rpm-test syslog-ng[14527]: Error parsing destination, destination plugin flush-limit not found in /etc/syslog-ng/conf.d/elasticsearch.conf at line 9, column 5: Nov 23 17:30:28 rpm-test syslog-ng[14527]: included from /etc/syslog-ng/syslog-ng.conf line 68, column 1 Nov 23 17:30:28 rpm-test syslog-ng[14527]: flush-limit( 1000 ) Nov 23 17:30:28 rpm-test syslog-ng[14527]: ^^^^^^^^^^^ Nov 23 17:30:28 rpm-test syslog-ng[14527]: syslog-ng documentation: http://www.balabit.com/support/documentation/?product=syslog-ng Nov 23 17:30:28 rpm-test syslog-ng[14527]: mailing list: https://lists.balabit.hu/mailman/listinfo/syslog-ng
If I remove the flush-limit() line, the config loads without a problem.
[fixed]
After some hours of operation, Syslog NG actually crashed on reload
with a memory corruption issue (might be unlelated):
Nov 23 17:50:14 rpm-test systemd[1]: Started System Logger Daemon. Nov 24 11:04:11 rpm-test systemd[1]: Reloaded System Logger Daemon. Nov 24 11:04:15 rpm-test syslog-ng[31599]: *** Error in
`/usr/sbin/syslog-ng': malloc(): memory corruption (fast): 0x00000000023293df ***
Nov 24 11:04:15 rpm-test syslog-ng[31599]: ======= Backtrace: ========= Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libc.so.6(+0x7b184)[0x7f87e29b7184] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libc.so.6(+0x7e877)[0x7f87e29ba877] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libc.so.6(__libc_calloc+0xb4)[0x7f87e29bc2d4] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libglib-2.0.so.0(g_malloc0+0x17)[0x7f87e39e32c7] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(log_multiplexer_new+0x13)[0x7f87e4529833] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(cfg_tree_new_mpx+0x12)[0x7f87e4521872] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x30136)[0x7f87e4522136] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x2fc11)[0x7f87e4521c11] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x2fb2b)[0x7f87e4521b2b] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x2fc11)[0x7f87e4521c11] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x302a9)[0x7f87e45222a9] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x2fc11)[0x7f87e4521c11] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(cfg_tree_compile_rule+0x35)[0x7f87e4522375] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(cfg_tree_compile+0x4b)[0x7f87e45224cb] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(cfg_tree_start+0x16)[0x7f87e4522576] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(cfg_init+0x16e)[0x7f87e451d58e] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x400de)[0x7f87e45320de] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(+0x407d7)[0x7f87e45327d7] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libivykis.so.0(+0x3b4f)[0x7f87e2f1cb4f] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libivykis.so.0(+0x5193)[0x7f87e2f1e193] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libivykis.so.0(+0x5810)[0x7f87e2f1e810] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libivykis.so.0(iv_main+0x44)[0x7f87e2f1f7d4] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libsyslog-ng-3.8.so.0(main_loop_run+0x74)[0x7f87e4532734] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /usr/sbin/syslog-ng(main+0x1b8)[0x4017b8] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f87e295db15] Nov 24 11:04:15 rpm-test syslog-ng[31599]: /usr/sbin/syslog-ng[0x4018dd]
[status unknown]
OS Version is CentOS 7.2, Syslog NG 5.8.1 (installed from Peter's
repository), Elasticsearch 5.0.1 (installed from Elastic's repo). The machine is a freshly installed system.
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Balazs,
Did exactly that. I just didnt want to submit the pr until some testing was done.
just in case the information got lost: I've done basic testing and your patch works for me. Best regards, Peter.
Thanks, I submitted the pr based on that. On Nov 27, 2016 1:22 PM, "Peter Eckel" <lists@eckel-edv.de> wrote:
Hi Balazs,
Did exactly that. I just didnt want to submit the pr until some testing was done.
just in case the information got lost: I've done basic testing and your patch works for me.
Best regards,
Peter. ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (5)
-
Balazs Scheidler
-
Budai, László
-
Czanik, Péter
-
Peter Eckel
-
Scheidler, Balázs