[syslog-ng] syslog-ng as "shipper" into ELK stack
jrhendri at roadrunner.com
Fri Oct 3 02:33:41 CEST 2014
I am working on configuring Elasticsearch, Logstash & Kibana (ELK) to
test it as a backend search tool for large volumes of logs.
I decided to put Redis in front of Logstash as a "broker" for the
incoming logs, and syslog-ng as the "shipper" so it looks like this:
syslog-ng ==> redis ==> logstash ==> elasticsearch ==> apache ==> kibana
It works very well using the redis destination in syslog-ng, although I
am having performance problems with logstash & elasticsearch default
configurations keeping up.
(I topped out today sending ~7000 events per second, and saw an insane
amount of swapping going on)
Not so much a specific question (I'll be working on heap & thread
settings and am pretty confident I can get it to handle at least this
moderate load) but I was wondering if anyone else is working in this area.
Also, in this configuration logstash is simply "parsing" the data it
pulls from redis and sending it into elasticsearch.
Seems like something syslog-ng might be able to do directly.
Is anyone aware of any plans to implement an elasticsearch destination?
Feel free to contact me on or off list if you want to discuss this.
More information about the syslog-ng