Hi Russell, First of all - I'm glad to see more of us working on this. Now: - There are a couple of options in the syslog-ng-incubator that provide some elasticsearch destinations using Perl, Python and Lua scripts. I have done some basic testing and it looks like the Lua one has more features, but I am having library issues with it so I may try to use the Perl module and try to add some of these features (e.g. template() is missing in the current Elasticsearch.pm so using that to format-json seems out of the question at the moment) - However with syslog-ng OSE built with redis and json support, it is easily possible to do this: syslog-ng (using patterndb & format-json) => redis => logstash (with no pattern matching) => elasticsearch You still have logstash (and all it's java wonderfulness) in the middle, but it is a pretty minimal configuration just for the convenience of linking redis & elasticsearch and it seems to run pretty well. So far on a single 32G RAM, 8 CPU box running all the pieces I top out around 5000 events per second (EPS) before elasticsearch has performance issues. I am pretty confident if I split this out into shards and ran multiple machines it would be my best "production" bet right now. (I set a 4GB limit for elasticsearch and have it lock the memory) - Clearly there is also the option of using a program destination and letting something external feed it to elasticsearch. Please let me know how you proceed and let's see if we can figure out a decent architecture for this "stack". Thanks! Jim On 10/22/2014 07:17 PM, Russell Fulton wrote:
Hi
We are already using the open source version of syslog-ng and I am about to set up some elastic search instances and would much prefer to feed data direct from syslog-ng rather than go through logstash (I already have a heap of patterndb parsers and performance should be way better!)
I have spent an hour or so with Google and have found various references to elastic search destination being available but I can find no mention of it in the release notes for 3.6.1. I have also downloaded the the tarball and unpacked it but could not find any evidence of the module , nore is there any mention of it in the manual.
As of now what is the recommended way of getting parsed data from OS syslog-ng into ES?
Thanks, Russell
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq