<div dir="ltr"><div><div><div><div>Hi Nick!<br><br></div>Unfortunately, redis destination cannot do batch inserts, that's why it's very slow. But why do you want to use redis?<br></div>Syslog-ng can send messages directly to logstash. In incubator, you can find and elastricsearch destination, although it is <br>
only a slightly better in performance than the redis one.<br><br></div>BR,<br></div>Viktor<br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Jul 21, 2014 at 3:53 PM, Nicholas Radonicich <span dir="ltr"><<a href="mailto:Nicholas.Radonicich@cogeco.com" target="_blank">Nicholas.Radonicich@cogeco.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div link="blue" vlink="purple" lang="EN-CA">
<p></p>
<div>
<p class="MsoNormal">Hello, hoping someone might be able to help... I am attempting to build a syslog server with outputs to Logstash/Elasticsearch via redis but am having issues with dropped messages in such that i cannot get over ~4000 messages per second
out to a destination (TCP or redis) without the rest being dropped. I have tried various tuning on the tcp output with flush_lines() and threaded() to no avail, as well as some things on the inputs. Both redis and logstash (used for tcp inputs) were on the
localhost.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">redis-benchmark has no issues doing >70000 LPUSH/second and the file based destinations of syslog-ng are not dropping messages (we are sending almost 80k/s) so I don’t know where I am going wrong.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">loggen was accepting > 110000mps its just on the outputs it seems.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Thanks, Nick<u></u><u></u></p>
</div>
<br>
<font style="FONT-FAMILY:Verdana;COLOR:silver;FONT-SIZE:7pt" color="silver" face="Verdana" size="1"><span lang="EN">
<p><font color="#a5a5a5">The information in this message, including in all attachments, is confidential or privileged. In the event you have received this message in error and are not the intended recipient, you are hereby advised that any use, copying or reproduction
of this document is strictly forbidden. Please notify immediately the sender of this error and destroy this message, including its attachments, as the case may be.<font face="Times New Roman"><font size="3">
</font></font></font></p>
</span></font><font size="1">
<p><font color="#a5a5a5">L'information apparaissant dans ce message électronique et dans les documents qui y sont joints est de nature confidentielle ou privilégiée. Si ce message vous est parvenu par erreur et que vous n'en êtes pas le destinataire visé, vous
êtes par les présentes avisé que toute utilisation, copie ou distribution de ce message est strictement interdite. Vous êtes donc prié d’en informer immédiatement l’expéditeur et de détruire ce message, ainsi que les documents qui y sont joints, le cas échéant.</font></p>
<p></p></font><font color="black"></font>
<p></p>
<p></p>
<p></p>
<p></p>
<p></p>
<p> </p>
<p><font color="#a5a5a5" face="Verdana" size="1">Ce courriel provient de
</font><span style="font-family:Verdana;font-size:xx-small;color:#a5a5a5"><a href="mailto:Nicholas.Radonicich@cogeco.com" target="_blank">Nicholas.Radonicich@cogeco.com</a></span><font color="#a5a5a5" face="Verdana" size="1"> . Pour assurer la livraison de futurs envois, veuillez inclure la présente adresse courriel à
votre carnet d’adresses ou votre liste d’expéditeurs autorisés.</font></p>
<p><font color="#a5a5a5" face="Verdana" size="1">Si vous ne souhaitez plus recevoir de messages promotionnels de la part de Cogeco, veuillez envoyer un courriel à <a href="mailto:desabonnement@cogeco.com?Subject=Unsubscribe+to+mail+from+Nicholas.Radonicich@cogeco.com" title="" target="_blank">desabonnement@cogeco.com</a>.</font></p>
<p><a href="http://www.cogeco.ca/cable/entreprise/cca/vie_privee.html" target="_blank"><font color="#a5a5a5" face="Verdana" size="1">Politique en matière de protection des renseignements personnels de Cogeco et Engagement en matière
d’anti-spam</font></a><font color="#a5a5a5" face="Verdana" size="1"> – </font><a href="http://www.cogeco.ca/web/qc/fr/residentiel/soutien/contact_telephone.php" target="_blank"><font color="#a5a5a5" face="Verdana" size="1">Contactez-nous</font></a></p>
<p><font color="#a5a5a5" face="Verdana" size="1">Cogeco Câble Canada, 5 Place Ville-Marie, Bureau 1700, Montréal, Québec, H3B 0B3</font></p>
<p><font color="#a5a5a5" face="Verdana" size="1">--</font></p>
<p><font color="#a5a5a5" face="Verdana" size="1">This email is from
</font><span style="font-family:Verdana;font-size:xx-small;color:#a5a5a5"><a href="mailto:Nicholas.Radonicich@cogeco.com" target="_blank">Nicholas.Radonicich@cogeco.com</a></span><font color="#a5a5a5" face="Verdana" size="1"> . To ensure the delivery of future emails, please add the current email address to your address
book or safe senders list.</font></p>
<p><font color="#a5a5a5" face="Verdana" size="1">If you no longer wish to receive promotional emails from Cogeco, please send an email
<font color="#a5a5a5">to</font><font color="#a5a5a5"> <a href="mailto:unsubscribe@cogeco.com?Subject=Unsubscribe+to+mail+from+Nicholas.Radonicich@cogeco.com" title="" target="_blank">unsubscribe@cogeco.com</a>.</font></font></p>
<p><a href="http://www.cogeco.ca/cable/corporate/cgo/privacy.html" target="_blank"><font color="#a5a5a5" face="Verdana" size="1">Privacy Policy and Anti-spam Commitment</font></a><font color="#a5a5a5" face="Verdana" size="1"> -
</font><a href="http://www.cogeco.ca/web/on/en/residential/support/contact_phone.php" target="_blank"><font color="#a5a5a5" face="Verdana" size="1">Contact us</font></a><font color="#a5a5a5" face="Verdana" size="1">
</font></p>
<p><font color="#a5a5a5" face="Verdana" size="1">Cogeco Cable Canada, 5 Place Ville-Marie, Suite 1700, Montreal, Quebec, H3B 0B3</font></p>
<p><font face="Verdana" size="1"></font> </p>
<p></p>
</div>
<br>______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div>