[syslog-ng] syslog-ng as "shipper" into ELK stack

[Fabien Wernli]

Hi Jim,

On Fri, Oct 03, 2014 at 12:33:41AM +0000, Jim Hendrick wrote:
> syslog-ng ==> redis ==> logstash ==> elasticsearch ==> apache ==> kibana

We've been using the following stack for over a year:
syslog-ng ==> logstash ==> elasticsearch

For various reasons, one being performance, we recently switched to:
syslog-ng ==> elasticsearch

This was done thanks to the syslog-ng-incubator perl module. I've set up a
small github repository where you can see our configuration [1].

> (I topped out today sending ~7000 events per second, and saw an insane 
> amount of swapping going on)

I've had tremendous issues with LS when the workload was darting up.
Since we switched to perl, we still have issues, but they're certainly not
performance related: with a single perl destination we could easily keep up
10k events per second on a mediumish virtual machine.

> Is anyone aware of any plans to implement an elasticsearch destination?

The upcoming 3.6 version will ship with a "native" elasticsearch
destination, which currently however is only a wrapper script.

I'd highly appreciate if you could test a similar config to ours, in order
to share some experience.

Cheers

[1] https://github.com/faxm0dem/syslog_ng-elasticsearch