[syslog-ng] Elasticsearch destination

[Jim Hendrick]

Hi Russell,

  First of all - I'm glad to see more of us working on this.

Now:
  - There are a couple of options in the syslog-ng-incubator that
provide some elasticsearch destinations using Perl, Python and Lua
scripts. I have done some basic testing and it looks like the Lua one
has more features, but I am having library issues with it so I may try
to use the Perl module and try to add some of these features (e.g.
template() is missing in the current Elasticsearch.pm so using that to
format-json seems out of the question at the moment)

 - However with syslog-ng OSE built with redis and json support, it is
easily possible to do this:
syslog-ng (using patterndb & format-json) => redis => logstash (with no
pattern matching) => elasticsearch

   You still have logstash (and all it's java wonderfulness) in the
middle, but it is a pretty minimal configuration just for the
convenience of linking redis & elasticsearch and it seems to run pretty
well.

   So far on a single 32G RAM, 8 CPU box running all the pieces I top
out around 5000 events per second (EPS) before elasticsearch has
performance issues. I am pretty confident if I split this out into
shards and ran multiple machines it would be my best "production" bet
right now. (I set a 4GB limit for elasticsearch and have it lock the memory)

  - Clearly there is also the option of using a program destination and
letting something external feed it to elasticsearch.

Please let me know how you proceed and let's see if we can figure out a
decent architecture for this "stack".

Thanks!
Jim



On 10/22/2014 07:17 PM, Russell Fulton wrote:
> Hi
>
> We are already using the open source version of syslog-ng and I am about to set up some elastic search instances and would much prefer to feed data direct from syslog-ng rather than go through logstash (I already have a heap of patterndb parsers and performance should be way better!)
>
> I have spent an hour or so with Google and have found various references to elastic search destination being available but I can find no mention of it in the release notes for 3.6.1.  I have also downloaded the the tarball and unpacked it but could not find any evidence of the module , nore is there any mention of it in the manual.
>
> As of now what is the recommended way of getting parsed data from OS syslog-ng into ES?
>
> Thanks, Russell
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>