<html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"></head><body ><div>Thanks! I will. </div><div>Jim</div><div><br></div><div><br></div><div><div style="font-size:9px;color:#575757">Sent from my Verizon Wireless 4G LTE smartphone</div></div><div></div><br><br>-------- Original message --------<br>From: Fabien Wernli <wernli@in2p3.fr> <br>Date:10/03/2014 4:12 AM (GMT-05:00) <br>To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> <br>Subject: Re: [syslog-ng] syslog-ng as "shipper" into ELK stack <br><br>Hi Jim,<br><br>On Fri, Oct 03, 2014 at 12:33:41AM +0000, Jim Hendrick wrote:<br>> syslog-ng ==> redis ==> logstash ==> elasticsearch ==> apache ==> kibana<br><br>We've been using the following stack for over a year:<br>syslog-ng ==> logstash ==> elasticsearch<br><br>For various reasons, one being performance, we recently switched to:<br>syslog-ng ==> elasticsearch<br><br>This was done thanks to the syslog-ng-incubator perl module. I've set up a<br>small github repository where you can see our configuration [1].<br><br>> (I topped out today sending ~7000 events per second, and saw an insane <br>> amount of swapping going on)<br><br>I've had tremendous issues with LS when the workload was darting up.<br>Since we switched to perl, we still have issues, but they're certainly not<br>performance related: with a single perl destination we could easily keep up<br>10k events per second on a mediumish virtual machine.<br><br>> Is anyone aware of any plans to implement an elasticsearch destination?<br><br>The upcoming 3.6 version will ship with a "native" elasticsearch<br>destination, which currently however is only a wrapper script.<br><br>I'd highly appreciate if you could test a similar config to ours, in order<br>to share some experience.<br><br>Cheers<br><br>[1] https://github.com/faxm0dem/syslog_ng-elasticsearch<br><br>______________________________________________________________________________<br>Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<br>Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<br>FAQ: http://www.balabit.com/wiki/syslog-ng-faq<br><br></body>