<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<!-- Template generated by Exclaimer Mail Disclaimers on 02:40:28 Monday, 21 July 2014 -->
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css">P.d54469d5-a5ab-4e52-9fa1-4d811d2d76c4 {
MARGIN: 0cm 0cm 0pt
}
LI.d54469d5-a5ab-4e52-9fa1-4d811d2d76c4 {
MARGIN: 0cm 0cm 0pt
}
DIV.d54469d5-a5ab-4e52-9fa1-4d811d2d76c4 {
MARGIN: 0cm 0cm 0pt
}
TABLE.d54469d5-a5ab-4e52-9fa1-4d811d2d76c4Table {
MARGIN: 0cm 0cm 0pt
}
DIV.Section1 {
page: Section1
}
</style><!-- Template generated by Exclaimer Mail Disclaimers on 02:40:28 Monday, 21 July 2014 --><style type="text/css">P.80ad36b4-0d57-40dc-a6f0-bdcaa3455b69 {
MARGIN: 0cm 0cm 0pt
}
LI.80ad36b4-0d57-40dc-a6f0-bdcaa3455b69 {
MARGIN: 0cm 0cm 0pt
}
DIV.80ad36b4-0d57-40dc-a6f0-bdcaa3455b69 {
MARGIN: 0cm 0cm 0pt
}
TABLE.80ad36b4-0d57-40dc-a6f0-bdcaa3455b69Table {
MARGIN: 0cm 0cm 0pt
}
DIV.Section1 {
page: Section1
}
</style>
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-CA" link="blue" vlink="purple">
<p class="d54469d5-a5ab-4e52-9fa1-4d811d2d76c4"></p>
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I tried direct to logstash via TCP output and SYSLOG output and ran into the same problem. Today I tried the unix-stream output with the same results of about
3500-4000 messages before I get dropped on those outputs (again the file output was fine). My next step is to try syslog-ng -> syslog-ng with the TCP output just to test.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
</div>
<br>
<font style="FONT-FAMILY: Verdana; COLOR: silver; FONT-SIZE: 7pt" color="silver" size="1" face="Verdana"><span lang="EN">
<p><font color="#a5a5a5">The information in this message, including in all attachments, is confidential or privileged. In the event you have received this message in error and are not the intended recipient, you are hereby advised that any use, copying or reproduction
of this document is strictly forbidden. Please notify immediately the sender of this error and destroy this message, including its attachments, as the case may be.<font face="Times New Roman"><font size="3">
</font></p>
</font></font><font size="1">
<p><font color="#a5a5a5">L'information apparaissant dans ce message électronique et dans les documents qui y sont joints est de nature confidentielle ou privilégiée. Si ce message vous est parvenu par erreur et que vous n'en êtes pas le destinataire visé, vous
êtes par les présentes avisé que toute utilisation, copie ou distribution de ce message est strictement interdite. Vous êtes donc prié d’en informer immédiatement l’expéditeur et de détruire ce message, ainsi que les documents qui y sont joints, le cas échéant.</font></font><font face="Times New Roman"><br>
</p>
<p class="80ad36b4-0d57-40dc-a6f0-bdcaa3455b69"></font></span></font></p>
<div class="WordSection1"></div>
<p></p>
<p></p>
<p></p>
<p></p>
<p class="d54469d5-a5ab-4e52-9fa1-4d811d2d76c4"> </p>
<p class="d54469d5-a5ab-4e52-9fa1-4d811d2d76c4"><font color="#a5a5a5" size="1" face="Verdana">Ce courriel provient de
</font><span style="font-family:Verdana; font-size:xx-small; color: #A5A5A5; ">Nicholas.Radonicich@cogeco.com</span><font color="#a5a5a5" size="1" face="Verdana"> . Pour assurer la livraison de futurs envois, veuillez inclure la présente adresse courriel à
votre carnet d’adresses ou votre liste d’expéditeurs autorisés.</font></p>
<p class="d54469d5-a5ab-4e52-9fa1-4d811d2d76c4"><font face="Verdana"><font size="1"><font color="#a5a5a5">Si vous ne souhaitez plus recevoir de messages promotionnels de la part de Cogeco, veuillez envoyer un courriel à
<a href="mailto:desabonnement@cogeco.com?Subject=Unsubscribe to mail from Nicholas.Radonicich@cogeco.com" title="" target="">
desabonnement@cogeco.com</a></font><font color="#a5a5a5">.</font></font></font></p>
<p class="d54469d5-a5ab-4e52-9fa1-4d811d2d76c4"><a href="http://www.cogeco.ca/cable/entreprise/cca/vie_privee.html"><font color="#a5a5a5" size="1" face="Verdana">Politique en matière de protection des renseignements personnels de Cogeco et Engagement en matière
d’anti-spam</font></a><font color="#a5a5a5" size="1" face="Verdana"> – </font><a href="http://www.cogeco.ca/web/qc/fr/residentiel/soutien/contact_telephone.php"><font color="#a5a5a5" size="1" face="Verdana">Contactez-nous</font></a></p>
<p class="d54469d5-a5ab-4e52-9fa1-4d811d2d76c4"><font color="#a5a5a5" size="1" face="Verdana">Cogeco Câble Canada, 5 Place Ville-Marie, Bureau 1700, Montréal, Québec, H3B 0B3</font></p>
<p class="d54469d5-a5ab-4e52-9fa1-4d811d2d76c4"><font color="#a5a5a5" size="1" face="Verdana">--</font></p>
<p class="d54469d5-a5ab-4e52-9fa1-4d811d2d76c4"><font color="#a5a5a5" size="1" face="Verdana">This email is from
</font><span style="font-family:Verdana; font-size:xx-small; color: #A5A5A5; ">Nicholas.Radonicich@cogeco.com</span><font color="#a5a5a5" size="1" face="Verdana"> . To ensure the delivery of future emails, please add the current email address to your address
book or safe senders list.</font></p>
<p class="d54469d5-a5ab-4e52-9fa1-4d811d2d76c4"><font color="#a5a5a5" size="1" face="Verdana">If you no longer wish to receive promotional emails from Cogeco, please send an email to
<a href="mailto:unsubscribe@cogeco.com?Subject=Unsubscribe to mail from Nicholas.Radonicich@cogeco.com" title="" target="">
unsubscribe@cogeco.com</a>.</font></p>
<p class="d54469d5-a5ab-4e52-9fa1-4d811d2d76c4"><a href="http://www.cogeco.ca/cable/corporate/cgo/privacy.html"><font color="#a5a5a5" size="1" face="Verdana">Privacy Policy and Anti-spam Commitment</font></a><font color="#a5a5a5" size="1" face="Verdana"> -
</font><a href="http://www.cogeco.ca/web/on/en/residential/support/contact_phone.php"><font color="#a5a5a5" size="1" face="Verdana">Contact us</font></a><font color="#a5a5a5" size="1" face="Verdana">
</font></p>
<p class="d54469d5-a5ab-4e52-9fa1-4d811d2d76c4"><font color="#a5a5a5" size="1" face="Verdana">Cogeco Cable Canada, 5 Place Ville-Marie, Suite 1700, Montreal, Quebec, H3B 0B3</font></p>
<p class="d54469d5-a5ab-4e52-9fa1-4d811d2d76c4"><font size="1" face="Verdana"></font> </p>
<p class="d54469d5-a5ab-4e52-9fa1-4d811d2d76c4"><font style="FONT-FAMILY: Verdana; COLOR: silver; FONT-SIZE: 7pt" color="silver" size="1" face="Verdana"><span lang="EN"></p>
<p><font size="1"></p>
<p><font face="Times New Roman"></p>
<p class="80ad36b4-0d57-40dc-a6f0-bdcaa3455b69"></p>
<div class="WordSection1">
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu]
<b>On Behalf Of </b>Tusa Viktor<br>
<b>Sent:</b> July-21-14 2:22 PM<br>
<b>To:</b> Syslog-ng users' and developers' mailing list<br>
<b>Subject:</b> Re: [syslog-ng] redis/tcp destination throughput and dropped messages<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Hi Nick!<o:p></o:p></p>
</div>
<p class="MsoNormal">Unfortunately, redis destination cannot do batch inserts, that's why it's very slow. But why do you want to use redis?<o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Syslog-ng can send messages directly to logstash. In incubator, you can find and elastricsearch destination, although it is
<br>
only a slightly better in performance than the redis one.<o:p></o:p></p>
</div>
<p class="MsoNormal">BR,<o:p></o:p></p>
</div>
<p class="MsoNormal">Viktor<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Mon, Jul 21, 2014 at 3:53 PM, Nicholas Radonicich <<a href="mailto:Nicholas.Radonicich@cogeco.com" target="_blank">Nicholas.Radonicich@cogeco.com</a>> wrote:<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hello, hoping someone might be able to help... I am attempting to build a syslog server with outputs to Logstash/Elasticsearch via redis but am having issues with dropped messages
in such that i cannot get over ~4000 messages per second out to a destination (TCP or redis) without the rest being dropped. I have tried various tuning on the tcp output with flush_lines() and threaded() to no avail, as well as some things on the inputs.
Both redis and logstash (used for tcp inputs) were on the localhost.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">redis-benchmark has no issues doing >70000 LPUSH/second and the file based destinations of syslog-ng are not dropping messages (we are sending almost 80k/s) so I don’t know where
I am going wrong.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">loggen was accepting > 110000mps its just on the outputs it seems.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thanks, Nick<o:p></o:p></p>
</div>
<p class="MsoNormal"><span lang="EN" style="font-size:7.0pt;font-family:"Verdana","sans-serif";color:silver"><o:p> </o:p></span></p>
<p><span lang="EN" style="font-size:7.0pt;font-family:"Verdana","sans-serif";color:#A5A5A5">The information in this message, including in all attachments, is confidential or privileged. In the event you have received this message in error and are not the intended
recipient, you are hereby advised that any use, copying or reproduction of this document is strictly forbidden. Please notify immediately the sender of this error and destroy this message, including its attachments, as the case may be.</span><span lang="EN" style="color:#A5A5A5">
</span><span lang="EN" style="font-size:7.0pt;font-family:"Verdana","sans-serif";color:silver"><o:p></o:p></span></p>
<p><span style="font-size:7.5pt;color:#A5A5A5">L'information apparaissant dans ce message électronique et dans les documents qui y sont joints est de nature confidentielle ou privilégiée. Si ce message vous est parvenu par erreur et que vous n'en êtes pas le
destinataire visé, vous êtes par les présentes avisé que toute utilisation, copie ou distribution de ce message est strictement interdite. Vous êtes donc prié d’en informer immédiatement l’expéditeur et de détruire ce message, ainsi que les documents qui y
sont joints, le cas échéant.</span><span style="font-size:7.5pt"><o:p></o:p></span></p>
<p> <o:p></o:p></p>
<p><span style="font-size:7.5pt;font-family:"Verdana","sans-serif";color:#A5A5A5">Ce courriel provient de
<a href="mailto:Nicholas.Radonicich@cogeco.com" target="_blank">Nicholas.Radonicich@cogeco.com</a> . Pour assurer la livraison de futurs envois, veuillez inclure la présente adresse courriel à votre carnet d’adresses ou votre liste d’expéditeurs autorisés.</span><o:p></o:p></p>
<p><span style="font-size:7.5pt;font-family:"Verdana","sans-serif";color:#A5A5A5">Si vous ne souhaitez plus recevoir de messages promotionnels de la part de Cogeco, veuillez envoyer un courriel à <a href="mailto:desabonnement@cogeco.com?Subject=Unsubscribe+to+mail+from+Nicholas.Radonicich@cogeco.com" target="_blank" title="">desabonnement@cogeco.com</a>.</span><o:p></o:p></p>
<p><a href="http://www.cogeco.ca/cable/entreprise/cca/vie_privee.html" target="_blank"><span style="font-size:7.5pt;font-family:"Verdana","sans-serif";color:#A5A5A5">Politique en matière de protection des renseignements personnels de Cogeco et Engagement en
matière d’anti-spam</span></a><span style="font-size:7.5pt;font-family:"Verdana","sans-serif";color:#A5A5A5"> –
</span><a href="http://www.cogeco.ca/web/qc/fr/residentiel/soutien/contact_telephone.php" target="_blank"><span style="font-size:7.5pt;font-family:"Verdana","sans-serif";color:#A5A5A5">Contactez-nous</span></a><o:p></o:p></p>
<p><span style="font-size:7.5pt;font-family:"Verdana","sans-serif";color:#A5A5A5">Cogeco Câble Canada, 5 Place Ville-Marie, Bureau 1700, Montréal, Québec, H3B 0B3</span><o:p></o:p></p>
<p><span style="font-size:7.5pt;font-family:"Verdana","sans-serif";color:#A5A5A5">--</span><o:p></o:p></p>
<p><span style="font-size:7.5pt;font-family:"Verdana","sans-serif";color:#A5A5A5">This email is from
<a href="mailto:Nicholas.Radonicich@cogeco.com" target="_blank">Nicholas.Radonicich@cogeco.com</a> . To ensure the delivery of future emails, please add the current email address to your address book or safe senders list.</span><o:p></o:p></p>
<p><span style="font-size:7.5pt;font-family:"Verdana","sans-serif";color:#A5A5A5">If you no longer wish to receive promotional emails from Cogeco, please send an email to <a href="mailto:unsubscribe@cogeco.com?Subject=Unsubscribe+to+mail+from+Nicholas.Radonicich@cogeco.com" target="_blank" title="">unsubscribe@cogeco.com</a>.</span><o:p></o:p></p>
<p><a href="http://www.cogeco.ca/cable/corporate/cgo/privacy.html" target="_blank"><span style="font-size:7.5pt;font-family:"Verdana","sans-serif";color:#A5A5A5">Privacy Policy and Anti-spam Commitment</span></a><span style="font-size:7.5pt;font-family:"Verdana","sans-serif";color:#A5A5A5">
- </span><a href="http://www.cogeco.ca/web/on/en/residential/support/contact_phone.php" target="_blank"><span style="font-size:7.5pt;font-family:"Verdana","sans-serif";color:#A5A5A5">Contact us</span></a><span style="font-size:7.5pt;font-family:"Verdana","sans-serif";color:#A5A5A5">
</span><o:p></o:p></p>
<p><span style="font-size:7.5pt;font-family:"Verdana","sans-serif";color:#A5A5A5">Cogeco Cable Canada, 5 Place Ville-Marie, Suite 1700, Montreal, Quebec, H3B 0B3</span><o:p></o:p></p>
<p> <o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">
https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">
http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<p></p>
</font>
<p></p>
</font>
<p></p>
</span></font>
<p></p>
<p></p>
<p></p>
</body>
</html>