[syslog-ng] syslog-ng as "shipper" into ELK stack

[Radu Gheorghe]

Hi Jim,

With rabbitmq you have the advantage that you can install the RabbitMQ
river and have Elasticsearch pull logs from Rabbit instead of having
another [moving] piece pull logs from Rabbit and push them to ES. So you'd
have a simpler setup that also makes sure ES isn't overwhelmed (because ES
is pulling).

There are some problems with this approach:
- the river only runs on one node at a time, which may become a bottleneck
- rivers are deprecated (or will be) so the ES side isn't actively
maintained. I've seen failover issues (node running the river goes down,
another node should start the river but doesn't) which needed river delete
+ recreate to kick the process in again

Logstash started by recommending RabbitMQ as the queue between two Logstash
instances, but now moved to Redis. Apparently the reason is that Redis
plays nicely with Logstash, and Rabbit didn't, here's a quote from the guide
<http://logstash.net/docs/1.1.1/tutorials/getting-started-centralized>:

"Previous versions of this guide used AMQP via RabbitMQ. Due to the
complexity of AMQP as well as performance issues related to the Bunny
driver we use, we're now recommending Redis instead."

Best regards,
Radu
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Sat, Oct 4, 2014 at 5:09 AM, Jim Hendrick <jrhendri at roadrunner.com>
wrote:

> Thanks. Why rabbitmq  instead of redis?  Is it faster, or does it offer
> some additional  functions?
>
> Jim
>
>
> Sent from my Verizon Wireless 4G LTE smartphone
>
>
> -------- Original message --------
> From: Alexandre Biancalana <biancalana at gmail.com>
> Date:10/03/2014 7:01 PM (GMT-05:00)
> To: Syslog-ng users' and developers' mailing list <
> syslog-ng at lists.balabit.hu>
> Subject: Re: [syslog-ng] syslog-ng as "shipper" into ELK stack
>
>
> On Thu, Oct 2, 2014 at 9:33 PM, Jim Hendrick <jrhendri at roadrunner.com>
> wrote:
>
>> Hi,
>>
>>    I am working on configuring Elasticsearch, Logstash & Kibana (ELK) to
>> test it as a backend search tool for large volumes of logs.
>>
>> I decided to put Redis in front of Logstash as a "broker" for the
>> incoming logs, and syslog-ng as the "shipper" so it looks like this:
>>
>> syslog-ng ==> redis ==> logstash ==> elasticsearch ==> apache ==> kibana
>>
>
> I've been using the following:
>
> syslog-ng => rabbitmq => elasticsearch
>
> syslog-ng + patterndb to parse logs and write then in json format on
> rabbitmq, after that is just use elasticsearch amqp river to consume the
> queue.
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20141006/4e17eeab/attachment.htm