syslog-ng as "shipper" into ELK stack
Hi, I am working on configuring Elasticsearch, Logstash & Kibana (ELK) to test it as a backend search tool for large volumes of logs. I decided to put Redis in front of Logstash as a "broker" for the incoming logs, and syslog-ng as the "shipper" so it looks like this: syslog-ng ==> redis ==> logstash ==> elasticsearch ==> apache ==> kibana It works very well using the redis destination in syslog-ng, although I am having performance problems with logstash & elasticsearch default configurations keeping up. (I topped out today sending ~7000 events per second, and saw an insane amount of swapping going on) Not so much a specific question (I'll be working on heap & thread settings and am pretty confident I can get it to handle at least this moderate load) but I was wondering if anyone else is working in this area. Also, in this configuration logstash is simply "parsing" the data it pulls from redis and sending it into elasticsearch. Seems like something syslog-ng might be able to do directly. Is anyone aware of any plans to implement an elasticsearch destination? Feel free to contact me on or off list if you want to discuss this. Thanks!! Jim
Hi Jim, On Fri, Oct 03, 2014 at 12:33:41AM +0000, Jim Hendrick wrote:
syslog-ng ==> redis ==> logstash ==> elasticsearch ==> apache ==> kibana
We've been using the following stack for over a year: syslog-ng ==> logstash ==> elasticsearch For various reasons, one being performance, we recently switched to: syslog-ng ==> elasticsearch This was done thanks to the syslog-ng-incubator perl module. I've set up a small github repository where you can see our configuration [1].
(I topped out today sending ~7000 events per second, and saw an insane amount of swapping going on)
I've had tremendous issues with LS when the workload was darting up. Since we switched to perl, we still have issues, but they're certainly not performance related: with a single perl destination we could easily keep up 10k events per second on a mediumish virtual machine.
Is anyone aware of any plans to implement an elasticsearch destination?
The upcoming 3.6 version will ship with a "native" elasticsearch destination, which currently however is only a wrapper script. I'd highly appreciate if you could test a similar config to ours, in order to share some experience. Cheers [1] https://github.com/faxm0dem/syslog_ng-elasticsearch
Hi, Just realized I never replied to you. I spent some time this past week trying to get the incubator module working with very limited success (mostly due to this being a part of one of my job responsibilities) I did get syslog-ng & patterndb doing what I was doing in logstash & grok and it seems much faster. I will work on this more hopefully later this week, but I did want to reply Thanks! Jim On 10/03/2014 04:12 AM, Fabien Wernli wrote:
Hi Jim,
On Fri, Oct 03, 2014 at 12:33:41AM +0000, Jim Hendrick wrote:
syslog-ng ==> redis ==> logstash ==> elasticsearch ==> apache ==> kibana We've been using the following stack for over a year: syslog-ng ==> logstash ==> elasticsearch
For various reasons, one being performance, we recently switched to: syslog-ng ==> elasticsearch
This was done thanks to the syslog-ng-incubator perl module. I've set up a small github repository where you can see our configuration [1].
(I topped out today sending ~7000 events per second, and saw an insane amount of swapping going on) I've had tremendous issues with LS when the workload was darting up. Since we switched to perl, we still have issues, but they're certainly not performance related: with a single perl destination we could easily keep up 10k events per second on a mediumish virtual machine.
Is anyone aware of any plans to implement an elasticsearch destination? The upcoming 3.6 version will ship with a "native" elasticsearch destination, which currently however is only a wrapper script.
I'd highly appreciate if you could test a similar config to ours, in order to share some experience.
Cheers
[1] https://github.com/faxm0dem/syslog_ng-elasticsearch
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On Thu, Oct 2, 2014 at 9:33 PM, Jim Hendrick <jrhendri@roadrunner.com> wrote:
Hi,
I am working on configuring Elasticsearch, Logstash & Kibana (ELK) to test it as a backend search tool for large volumes of logs.
I decided to put Redis in front of Logstash as a "broker" for the incoming logs, and syslog-ng as the "shipper" so it looks like this:
syslog-ng ==> redis ==> logstash ==> elasticsearch ==> apache ==> kibana
I've been using the following: syslog-ng => rabbitmq => elasticsearch syslog-ng + patterndb to parse logs and write then in json format on rabbitmq, after that is just use elasticsearch amqp river to consume the queue.
participants (3)
-
Alexandre Biancalana
-
Fabien Wernli
-
Jim Hendrick