[syslog-ng] syslog-ng as "shipper" into ELK stack

[Alexandre Biancalana]

On Mon, Oct 6, 2014 at 9:36 AM, Radu Gheorghe <radu.gheorghe at sematext.com>
wrote:

> Hi Jim,
>
> With rabbitmq you have the advantage that you can install the RabbitMQ
> river and have Elasticsearch pull logs from Rabbit instead of having
> another [moving] piece pull logs from Rabbit and push them to ES. So you'd
> have a simpler setup that also makes sure ES isn't overwhelmed (because ES
> is pulling).
>

Another point that can be an advantage is that with Redis you are limited
to the RAM memory available on the machine (all your data need to fit in
memory) , with RabbitMQ you have disk persistence that can help in cases
where you need to stop ES consumption by any reason.


>
> There are some problems with this approach:
> - the river only runs on one node at a time, which may become a bottleneck
> - rivers are deprecated (or will be) so the ES side isn't actively
> maintained. I've seen failover issues (node running the river goes down,
> another node should start the river but doesn't) which needed river delete
> + recreate to kick the process in again
>

I didn't know that, I will check it out.


>
> Logstash started by recommending RabbitMQ as the queue between two
> Logstash instances, but now moved to Redis. Apparently the reason is that
> Redis plays nicely with Logstash, and Rabbit didn't, here's a quote from
> the guide
> <http://logstash.net/docs/1.1.1/tutorials/getting-started-centralized>:
>
> "Previous versions of this guide used AMQP via RabbitMQ. Due to the
> complexity of AMQP as well as performance issues related to the Bunny
> driver we use, we're now recommending Redis instead."
>
> Best regards,
> Radu
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
>
> On Sat, Oct 4, 2014 at 5:09 AM, Jim Hendrick <jrhendri at roadrunner.com>
> wrote:
>
>> Thanks. Why rabbitmq  instead of redis?  Is it faster, or does it offer
>> some additional  functions?
>>
>> Jim
>>
>>
>> Sent from my Verizon Wireless 4G LTE smartphone
>>
>>
>> -------- Original message --------
>> From: Alexandre Biancalana <biancalana at gmail.com>
>> Date:10/03/2014 7:01 PM (GMT-05:00)
>> To: Syslog-ng users' and developers' mailing list <
>> syslog-ng at lists.balabit.hu>
>> Subject: Re: [syslog-ng] syslog-ng as "shipper" into ELK stack
>>
>>
>> On Thu, Oct 2, 2014 at 9:33 PM, Jim Hendrick <jrhendri at roadrunner.com>
>> wrote:
>>
>>> Hi,
>>>
>>>    I am working on configuring Elasticsearch, Logstash & Kibana (ELK) to
>>> test it as a backend search tool for large volumes of logs.
>>>
>>> I decided to put Redis in front of Logstash as a "broker" for the
>>> incoming logs, and syslog-ng as the "shipper" so it looks like this:
>>>
>>> syslog-ng ==> redis ==> logstash ==> elasticsearch ==> apache ==> kibana
>>>
>>
>> I've been using the following:
>>
>> syslog-ng => rabbitmq => elasticsearch
>>
>> syslog-ng + patterndb to parse logs and write then in json format on
>> rabbitmq, after that is just use elasticsearch amqp river to consume the
>> queue.
>>
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20141006/a2b24638/attachment.htm