[syslog-ng] syslog-ng as "shipper" into ELK stack

[Jim Hendrick]

Thanks!  I will.  
Jim


Sent from my Verizon Wireless 4G LTE smartphone


-------- Original message --------
From: Fabien Wernli <wernli at in2p3.fr> 
Date:10/03/2014  4:12 AM  (GMT-05:00) 
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu> 
Subject: Re: [syslog-ng] syslog-ng as "shipper" into ELK stack 

Hi Jim,

On Fri, Oct 03, 2014 at 12:33:41AM +0000, Jim Hendrick wrote:
> syslog-ng ==> redis ==> logstash ==> elasticsearch ==> apache ==> kibana

We've been using the following stack for over a year:
syslog-ng ==> logstash ==> elasticsearch

For various reasons, one being performance, we recently switched to:
syslog-ng ==> elasticsearch

This was done thanks to the syslog-ng-incubator perl module. I've set up a
small github repository where you can see our configuration [1].

> (I topped out today sending ~7000 events per second, and saw an insane 
> amount of swapping going on)

I've had tremendous issues with LS when the workload was darting up.
Since we switched to perl, we still have issues, but they're certainly not
performance related: with a single perl destination we could easily keep up
10k events per second on a mediumish virtual machine.

> Is anyone aware of any plans to implement an elasticsearch destination?

The upcoming 3.6 version will ship with a "native" elasticsearch
destination, which currently however is only a wrapper script.

I'd highly appreciate if you could test a similar config to ours, in order
to share some experience.

Cheers

[1] https://github.com/faxm0dem/syslog_ng-elasticsearch

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20141003/526cdaae/attachment.htm