<html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"></head><body ><div>Thanks!I am on vacation this week but I look forward to trying out these suggestions. </div><div>Jim</div><div><br></div><div><br></div><div><div style="font-size:9px;color:#575757">Sent from my Verizon Wireless 4G LTE smartphone</div></div><div></div><br><br>-------- Original message --------<br>From: Alexandre Biancalana <biancalana@gmail.com> <br>Date:10/06/2014 11:42 AM (GMT-05:00) <br>To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> <br>Subject: Re: [syslog-ng] syslog-ng as "shipper" into ELK stack <br><br><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Oct 6, 2014 at 9:36 AM, Radu Gheorghe <span dir="ltr"><<a href="mailto:radu.gheorghe@sematext.com" target="_blank">radu.gheorghe@sematext.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div>Hi Jim,<br><br></div>With rabbitmq you have the advantage that you can install the RabbitMQ river and have Elasticsearch pull logs from Rabbit instead of having another [moving] piece pull logs from Rabbit and push them to ES. So you'd have a simpler setup that also makes sure ES isn't overwhelmed (because ES is pulling).<br></div></div></div></div></div></div></blockquote><div><br></div><div>Another point that can be an advantage is that with Redis you are limited to the RAM memory available on the machine (all your data need to fit in memory) , with RabbitMQ you have disk persistence that can help in cases where you need to stop ES consumption by any reason.<br> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div><br></div>There are some problems with this approach:<br></div>- the river only runs on one node at a time, which may become a bottleneck<br></div>- rivers are deprecated (or will be) so the ES side isn't actively maintained. I've seen failover issues (node running the river goes down, another node should start the river but doesn't) which needed river delete + recreate to kick the process in again<br></div></div></div></blockquote><div><br>I didn't know that, I will check it out.<br> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><br></div>Logstash started by recommending RabbitMQ as the queue between two Logstash instances, but now moved to Redis. Apparently the reason is that Redis plays nicely with Logstash, and Rabbit didn't, here's a quote from the <a href="http://logstash.net/docs/1.1.1/tutorials/getting-started-centralized" target="_blank">guide</a>:<br><br>"Previous versions of this guide used AMQP via RabbitMQ. Due to the complexity of AMQP as well as performance issues related to the Bunny driver we use, we're now recommending Redis instead."<br></div><div><div><div><div><div><div class="gmail_extra"><br></div><div class="gmail_extra">Best regards,<br>Radu<br clear="all"></div><div class="gmail_extra"><div><div dir="ltr"><div>--</div><div>Performance Monitoring * Log Analytics * Search Analytics</div><div><span style="font-family:arial,sans-serif;font-size:13px">Solr & Elasticsearch Support * </span><a href="http://sematext.com/" style="font-size:13px;font-family:arial,sans-serif" target="_blank">http://sematext.com/</a></div></div></div>
<br><div class="gmail_quote"><div><div class="h5">On Sat, Oct 4, 2014 at 5:09 AM, Jim Hendrick <span dir="ltr"><<a href="mailto:jrhendri@roadrunner.com" target="_blank">jrhendri@roadrunner.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="h5"><div><div>Thanks. Why rabbitmq instead of redis? Is it faster, or does it offer some additional functions? </div><span><div><br></div><div>Jim</div><div><br></div><div><br></div><div><div style="font-size:9px;color:rgb(87,87,87)">Sent from my Verizon Wireless 4G LTE smartphone</div></div><div></div><br><br>-------- Original message --------<br></span><span>From: Alexandre Biancalana <<a href="mailto:biancalana@gmail.com" target="_blank">biancalana@gmail.com</a>> <br>Date:10/03/2014 7:01 PM (GMT-05:00) <br>To: Syslog-ng users' and developers' mailing list <<a href="mailto:syslog-ng@lists.balabit.hu" target="_blank">syslog-ng@lists.balabit.hu</a>> <br>Subject: Re: [syslog-ng] syslog-ng as "shipper" into ELK stack <br><br></span><div><div><div dir="ltr"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Oct 2, 2014 at 9:33 PM, Jim Hendrick <span dir="ltr"><<a href="mailto:jrhendri@roadrunner.com" target="_blank">jrhendri@roadrunner.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
I am working on configuring Elasticsearch, Logstash & Kibana (ELK) to<br>
test it as a backend search tool for large volumes of logs.<br>
<br>
I decided to put Redis in front of Logstash as a "broker" for the<br>
incoming logs, and syslog-ng as the "shipper" so it looks like this:<br>
<br>
syslog-ng ==> redis ==> logstash ==> elasticsearch ==> apache ==> kibana<br></blockquote><div><br></div><div>I've been using the following:<br><br>syslog-ng => rabbitmq => elasticsearch<br><br></div><div>syslog-ng + patterndb to parse logs and write then in json format on rabbitmq, after that is just use elasticsearch amqp river to consume the queue.<br></div></div><br></div></div>
</div></div></div><br></div></div><span class="">______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
<br></span></blockquote></div><br></div></div></div></div></div></div></div>
<br>______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div></div>
</body>