Hi, Just realized I never replied to you. I spent some time this past week trying to get the incubator module working with very limited success (mostly due to this being a part of one of my job responsibilities) I did get syslog-ng & patterndb doing what I was doing in logstash & grok and it seems much faster. I will work on this more hopefully later this week, but I did want to reply Thanks! Jim On 10/03/2014 04:12 AM, Fabien Wernli wrote:
Hi Jim,
On Fri, Oct 03, 2014 at 12:33:41AM +0000, Jim Hendrick wrote:
syslog-ng ==> redis ==> logstash ==> elasticsearch ==> apache ==> kibana We've been using the following stack for over a year: syslog-ng ==> logstash ==> elasticsearch
For various reasons, one being performance, we recently switched to: syslog-ng ==> elasticsearch
This was done thanks to the syslog-ng-incubator perl module. I've set up a small github repository where you can see our configuration [1].
(I topped out today sending ~7000 events per second, and saw an insane amount of swapping going on) I've had tremendous issues with LS when the workload was darting up. Since we switched to perl, we still have issues, but they're certainly not performance related: with a single perl destination we could easily keep up 10k events per second on a mediumish virtual machine.
Is anyone aware of any plans to implement an elasticsearch destination? The upcoming 3.6 version will ship with a "native" elasticsearch destination, which currently however is only a wrapper script.
I'd highly appreciate if you could test a similar config to ours, in order to share some experience.
Cheers
[1] https://github.com/faxm0dem/syslog_ng-elasticsearch
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq