[syslog-ng] Elasticsearch destination

[Evan Rempel]

We are putting up this stack as well and will report
on success. My concern is that you imply that Elasticsearch
will have a performance issue on a single node around
5000 EPS. Is that really what you are saying.

Our previous kick at this can we ran into a performance
limit around 4000 EPS but I thought it was logstash.
Now that we are removing that, I was hoping for a lot more
performance.

We were hoping to get to a total solution of around
100,000 EPS using 4 or 5 elasticsearch nodes.

I'll let you know how we make out.

Evan.

On 10/22/2014 06:28 PM, Jim Hendrick wrote:
> Hi Russell,
>
>    First of all - I'm glad to see more of us working on this.
>
> Now:
>    - There are a couple of options in the syslog-ng-incubator that
> provide some elasticsearch destinations using Perl, Python and Lua
> scripts. I have done some basic testing and it looks like the Lua one
> has more features, but I am having library issues with it so I may try
> to use the Perl module and try to add some of these features (e.g.
> template() is missing in the current Elasticsearch.pm so using that to
> format-json seems out of the question at the moment)
>
>   - However with syslog-ng OSE built with redis and json support, it is
> easily possible to do this:
> syslog-ng (using patterndb & format-json) => redis => logstash (with no
> pattern matching) => elasticsearch
>
>     You still have logstash (and all it's java wonderfulness) in the
> middle, but it is a pretty minimal configuration just for the
> convenience of linking redis & elasticsearch and it seems to run pretty
> well.
>
>     So far on a single 32G RAM, 8 CPU box running all the pieces I top
> out around 5000 events per second (EPS) before elasticsearch has
> performance issues. I am pretty confident if I split this out into
> shards and ran multiple machines it would be my best "production" bet
> right now. (I set a 4GB limit for elasticsearch and have it lock the memory)
>
>    - Clearly there is also the option of using a program destination and
> letting something external feed it to elasticsearch.
>
> Please let me know how you proceed and let's see if we can figure out a
> decent architecture for this "stack".
>
> Thanks!
> Jim
>
>
>
> On 10/22/2014 07:17 PM, Russell Fulton wrote:
>> Hi
>>
>> We are already using the open source version of syslog-ng and I am about to set up some elastic search instances and would much prefer to feed data direct from syslog-ng rather than go through logstash (I already have a heap of patterndb parsers and performance should be way better!)
>>
>> I have spent an hour or so with Google and have found various references to elastic search destination being available but I can find no mention of it in the release notes for 3.6.1.  I have also downloaded the the tarball and unpacked it but could not find any evidence of the module , nore is there any mention of it in the manual.
>>
>> As of now what is the recommended way of getting parsed data from OS syslog-ng into ES?
>>
>> Thanks, Russell
>>