Hi Jim,
With rabbitmq you have the advantage that you can install the RabbitMQ river and have Elasticsearch pull logs from Rabbit instead of having another [moving] piece pull logs from Rabbit and push them to ES. So you'd have a simpler setup that also makes sure ES isn't overwhelmed (because ES is pulling).
There are some problems with this approach:
- the river only runs on one node at a time, which may become a bottleneck
- rivers are deprecated (or will be) so the ES side isn't actively maintained. I've seen failover issues (node running the river goes down, another node should start the river but doesn't) which needed river delete + recreate to kick the process in again
Logstash started by recommending RabbitMQ as the queue between two Logstash instances, but now moved to Redis. Apparently the reason is that Redis plays nicely with Logstash, and Rabbit didn't, here's a quote from the
guide:
"Previous versions of this guide used AMQP via RabbitMQ. Due to the complexity of AMQP as well as performance issues related to the Bunny driver we use, we're now recommending Redis instead."