[syslog-ng] Elasticsearch 5 and syslog-ng

Sat Nov 26 13:38:28 UTC 2016

Did exactly that. I just didnt want to submit the pr until some testing was
done.

On Nov 25, 2016 3:05 PM, "Budai, László" <laszlo.budai at balabit.com> wrote:

> Hi,
>
> @Bazsi: could you send a PR with the grammar changes?
>
>
> L.
>
> On Thu, Nov 24, 2016 at 4:13 PM, Scheidler, Balázs <
> balazs.scheidler at balabit.com> wrote:
>
>> Hi,
>>
>> This patch changes the grammar, so it accepts number tokens as well:
>>
>> https://github.com/balabit/syslog-ng/tree/f/java-should-acce
>> pt-numbers-as-options
>>
>> testing would be appreciated as this is a "blind" patch, I haven't tried
>> it as I don't personally use elasticsearch/java destinations.
>>
>> The memory corruption problem was already raised in a different thread,
>> but I don't know about any exact outcome from those emails, maybe @lbudai
>> does.
>>
>>
>> --
>> Bazsi
>>
>> On Thu, Nov 24, 2016 at 3:06 PM, Peter Eckel <lists at eckel-edv.de> wrote:
>>
>>> OK, I got the first one figured out myself ...
>>>
>>>   flush-limit( 1000 )
>>>
>>> does not work.
>>>
>>>   flush-limit( "1000" )
>>>
>>> does, which is inconsistent with the usual syslog-ng.conf behaviour. And
>>> setting it to "1" actually fixed the slow propagation to ES for me.
>>>
>>> Now I need to see whether the crash still occurs.
>>>
>>> Another question that came up: Is it possible to do flushing based on a
>>> time interval and not on a message count in the queue? It's rather
>>> unsatisfactory to wait for critical messages to appear in ES.
>>>
>>> Best regards,
>>>
>>>   Peter.
>>>
>>> > On 24 Nov 2016, at 12:28, Peter Eckel <lists at eckel-edv.de> wrote:
>>> >
>>> > I'm currently investigating the Syslog NG -> Elasticsearch 2
>>> destination for a project I'm working on, and started using basically the
>>> sample configuration in Peter Czanik's blog article. Thanks, by the way,
>>> for the great tutorial.
>>> >
>>> > There is one thing I'm currently struggling with: On my test system I
>>> have a fairly low volume of messages, and there seems to be an issue with
>>> flushing the cache of Syslog NG to Elasticsearch. To be precise: It doesn't
>>> happen. I can easily force a cache flush by reloading Syslog NG, but if I
>>> just keep it sitting there it doesn't log anything at all to Elasticsearch
>>> (while logs to files, e.g. /var/log/messages, are happening in real time).
>>> >
>>> > I tried configuring the flush-limit() in the elasticsearch2
>>> destination, but the configuration prevents syslog-ng from starting with an
>>> error message:
>>> >
>>> >> Nov 23 17:30:28 rpm-test syslog-ng[14527]: Error parsing destination,
>>> destination plugin flush-limit not found in /etc/syslog-ng/conf.d/elasticsearch.conf
>>> at line 9, column 5:
>>> >> Nov 23 17:30:28 rpm-test syslog-ng[14527]: included from
>>> /etc/syslog-ng/syslog-ng.conf line 68, column 1
>>> >> Nov 23 17:30:28 rpm-test syslog-ng[14527]: flush-limit( 1000 )
>>> >> Nov 23 17:30:28 rpm-test syslog-ng[14527]: ^^^^^^^^^^^
>>> >> Nov 23 17:30:28 rpm-test syslog-ng[14527]: syslog-ng documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> >> Nov 23 17:30:28 rpm-test syslog-ng[14527]: mailing list:
>>> https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> >
>>> > If I remove the flush-limit() line, the config loads without a problem.
>>>
>>> [fixed]
>>>
>>> >
>>> > After some hours of operation, Syslog NG actually crashed on reload
>>> with a memory corruption issue (might be unlelated):
>>> >
>>> >> Nov 23 17:50:14 rpm-test
>>> >> systemd[1]: Started System Logger Daemon.
>>> >> Nov 24 11:04:11 rpm-test systemd[1]: Reloaded System Logger Daemon.
>>> >> Nov 24 11:04:15 rpm-test syslog-ng[31599]: *** Error in
>>> `/usr/sbin/syslog-ng': malloc(): memory corruption (fast):
>>> 0x00000000023293df ***
>>> >> Nov 24 11:04:15 rpm-test syslog-ng[31599]: ======= Backtrace:
>>> =========
>>> >> Nov 24 11:04:15 rpm-test syslog-ng[31599]:
>>> /lib64/libc.so.6(+0x7b184)[0x7f87e29b7184]
>>> >> Nov 24 11:04:15 rpm-test syslog-ng[31599]:
>>> /lib64/libc.so.6(+0x7e877)[0x7f87e29ba877]
>>> >> Nov 24 11:04:15 rpm-test syslog-ng[31599]:
>>> /lib64/libc.so.6(__libc_calloc+0xb4)[0x7f87e29bc2d4]
>>> >> Nov 24 11:04:15 rpm-test syslog-ng[31599]:
>>> /lib64/libglib-2.0.so.0(g_malloc0+0x17)[0x7f87e39e32c7]
>>> >> Nov 24 11:04:15 rpm-test syslog-ng[31599]:
>>> /lib64/libsyslog-ng-3.8.so.0(log_multiplexer_new+0x13)[0x7f87e4529833]
>>> >> Nov 24 11:04:15 rpm-test syslog-ng[31599]:
>>> /lib64/libsyslog-ng-3.8.so.0(cfg_tree_new_mpx+0x12)[0x7f87e4521872]
>>> >> Nov 24 11:04:15 rpm-test syslog-ng[31599]:
>>> /lib64/libsyslog-ng-3.8.so.0(+0x30136)[0x7f87e4522136]
>>> >> Nov 24 11:04:15 rpm-test syslog-ng[31599]:
>>> /lib64/libsyslog-ng-3.8.so.0(+0x2fc11)[0x7f87e4521c11]
>>> >> Nov 24 11:04:15 rpm-test syslog-ng[31599]:
>>> /lib64/libsyslog-ng-3.8.so.0(+0x2fb2b)[0x7f87e4521b2b]
>>> >> Nov 24 11:04:15 rpm-test syslog-ng[31599]:
>>> /lib64/libsyslog-ng-3.8.so.0(+0x2fc11)[0x7f87e4521c11]
>>> >> Nov 24 11:04:15 rpm-test syslog-ng[31599]:
>>> /lib64/libsyslog-ng-3.8.so.0(+0x302a9)[0x7f87e45222a9]
>>> >> Nov 24 11:04:15 rpm-test syslog-ng[31599]:
>>> /lib64/libsyslog-ng-3.8.so.0(+0x2fc11)[0x7f87e4521c11]
>>> >> Nov 24 11:04:15 rpm-test syslog-ng[31599]:
>>> /lib64/libsyslog-ng-3.8.so.0(cfg_tree_compile_rule+0x35)[0x7f87e4522375]
>>> >> Nov 24 11:04:15 rpm-test syslog-ng[31599]:
>>> /lib64/libsyslog-ng-3.8.so.0(cfg_tree_compile+0x4b)[0x7f87e45224cb]
>>> >> Nov 24 11:04:15 rpm-test syslog-ng[31599]:
>>> /lib64/libsyslog-ng-3.8.so.0(cfg_tree_start+0x16)[0x7f87e4522576]
>>> >> Nov 24 11:04:15 rpm-test syslog-ng[31599]:
>>> /lib64/libsyslog-ng-3.8.so.0(cfg_init+0x16e)[0x7f87e451d58e]
>>> >> Nov 24 11:04:15 rpm-test syslog-ng[31599]:
>>> /lib64/libsyslog-ng-3.8.so.0(+0x400de)[0x7f87e45320de]
>>> >> Nov 24 11:04:15 rpm-test syslog-ng[31599]:
>>> /lib64/libsyslog-ng-3.8.so.0(+0x407d7)[0x7f87e45327d7]
>>> >> Nov 24 11:04:15 rpm-test syslog-ng[31599]:
>>> /lib64/libivykis.so.0(+0x3b4f)[0x7f87e2f1cb4f]
>>> >> Nov 24 11:04:15 rpm-test syslog-ng[31599]:
>>> /lib64/libivykis.so.0(+0x5193)[0x7f87e2f1e193]
>>> >> Nov 24 11:04:15 rpm-test syslog-ng[31599]:
>>> /lib64/libivykis.so.0(+0x5810)[0x7f87e2f1e810]
>>> >> Nov 24 11:04:15 rpm-test syslog-ng[31599]:
>>> /lib64/libivykis.so.0(iv_main+0x44)[0x7f87e2f1f7d4]
>>> >> Nov 24 11:04:15 rpm-test syslog-ng[31599]:
>>> /lib64/libsyslog-ng-3.8.so.0(main_loop_run+0x74)[0x7f87e4532734]
>>> >> Nov 24 11:04:15 rpm-test syslog-ng[31599]:
>>> /usr/sbin/syslog-ng(main+0x1b8)[0x4017b8]
>>> >> Nov 24 11:04:15 rpm-test syslog-ng[31599]:
>>> /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f87e295db15]
>>> >> Nov 24 11:04:15 rpm-test syslog-ng[31599]:
>>> /usr/sbin/syslog-ng[0x4018dd]
>>>
>>> [status unknown]
>>>
>>> >
>>> > OS Version is CentOS 7.2, Syslog NG 5.8.1 (installed from Peter's
>>> repository), Elasticsearch 5.0.1 (installed from Elastic's repo). The
>>> machine is a freshly installed system.
>>>
>>> ____________________________________________________________
>>> __________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation: http://www.balabit.com/support
>>> /documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>
>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=
>> syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20161126/a781e9f1/attachment.html>