[syslog-ng] syslog-ng as "shipper" into ELK stack

[Jim Hendrick]

Hi,

   I am working on configuring Elasticsearch, Logstash & Kibana (ELK) to 
test it as a backend search tool for large volumes of logs.

I decided to put Redis in front of Logstash as a "broker" for the 
incoming logs, and syslog-ng as the "shipper" so it looks like this:

syslog-ng ==> redis ==> logstash ==> elasticsearch ==> apache ==> kibana

It works very well using the redis destination in syslog-ng, although I 
am having performance problems with logstash & elasticsearch default 
configurations keeping up.

(I topped out today sending ~7000 events per second, and saw an insane 
amount of swapping going on)

Not so much a specific question (I'll be working on heap & thread 
settings and am pretty confident I can get it to handle at least this 
moderate load) but I was wondering if anyone else is working in this area.

Also, in this configuration logstash is simply "parsing" the data it 
pulls from redis and sending it into elasticsearch.

Seems like something syslog-ng might be able to do directly.

Is anyone aware of any plans to implement an elasticsearch destination?

Feel free to contact me on or off list if you want to discuss this.

Thanks!!
Jim