[syslog-ng] redis/tcp destination throughput and dropped messages

[Nicholas Radonicich]

I tried direct to logstash via TCP output and SYSLOG output and ran into the same problem. Today I tried the unix-stream output with the same results of about 3500-4000 messages before I get dropped on those outputs (again the file output was fine). My next step is to try syslog-ng -> syslog-ng with the TCP output just to test.




The information in this message, including in all attachments, is confidential or privileged. In the event you have received this message in error and are not the intended recipient, you are hereby advised that any use, copying or reproduction of this document is strictly forbidden. Please notify immediately the sender of this error and destroy this message, including its attachments, as the case may be.

L'information apparaissant dans ce message électronique et dans les documents qui y sont joints est de nature confidentielle ou privilégiée. Si ce message vous est parvenu par erreur et que vous n'en êtes pas le destinataire visé, vous êtes par les présentes avisé que toute utilisation, copie ou distribution de ce message est strictement interdite. Vous êtes donc prié d’en informer immédiatement l’expéditeur et de détruire ce message, ainsi que les documents qui y sont joints, le cas échéant.



Ce courriel provient de Nicholas.Radonicich at cogeco.com . Pour assurer la livraison de futurs envois, veuillez inclure la présente adresse courriel à votre carnet d’adresses ou votre liste d’expéditeurs autorisés.

Si vous ne souhaitez plus recevoir de messages promotionnels de la part de Cogeco, veuillez envoyer un courriel à desabonnement at cogeco.com<mailto:desabonnement at cogeco.com?Subject=Unsubscribe%20to%20mail%20from%20Nicholas.Radonicich at cogeco.com>.

Politique en matière de protection des renseignements personnels de Cogeco et Engagement en matière d’anti-spam<http://www.cogeco.ca/cable/entreprise/cca/vie_privee.html> – Contactez-nous<http://www.cogeco.ca/web/qc/fr/residentiel/soutien/contact_telephone.php>

Cogeco Câble Canada, 5 Place Ville-Marie, Bureau 1700, Montréal, Québec, H3B 0B3

--

This email is from Nicholas.Radonicich at cogeco.com . To ensure the delivery of future emails, please add the current email address to your address book or safe senders list.

If you no longer wish to receive promotional emails from Cogeco, please send an email to unsubscribe at cogeco.com<mailto:unsubscribe at cogeco.com?Subject=Unsubscribe%20to%20mail%20from%20Nicholas.Radonicich at cogeco.com>.

Privacy Policy and Anti-spam Commitment<http://www.cogeco.ca/cable/corporate/cgo/privacy.html> - Contact us<http://www.cogeco.ca/web/on/en/residential/support/contact_phone.php>

Cogeco Cable Canada, 5 Place Ville-Marie, Suite 1700, Montreal, Quebec, H3B 0B3


From: syslog-ng-bounces at lists.balabit.hu [mailto:syslog-ng-bounces at lists.balabit.hu] On Behalf Of Tusa Viktor
Sent: July-21-14 2:22 PM
To: Syslog-ng users' and developers' mailing list
Subject: Re: [syslog-ng] redis/tcp destination throughput and dropped messages

Hi Nick!
Unfortunately, redis destination cannot do batch inserts, that's why it's very slow. But why do you want to use redis?
Syslog-ng can send messages directly to logstash. In incubator, you can find and elastricsearch destination, although it is
only a slightly better in performance than the redis one.
BR,
Viktor

On Mon, Jul 21, 2014 at 3:53 PM, Nicholas Radonicich <Nicholas.Radonicich at cogeco.com<mailto:Nicholas.Radonicich at cogeco.com>> wrote:
Hello, hoping someone might be able to help... I am attempting to build a syslog server with outputs to Logstash/Elasticsearch via redis but am having issues with dropped messages in such that i cannot get over ~4000 messages per second out to a destination (TCP or redis) without the rest being dropped. I have tried various tuning on the tcp output with flush_lines() and threaded() to no avail, as well as some things on the inputs. Both redis and logstash (used for tcp inputs) were on the localhost.

redis-benchmark has no issues doing >70000 LPUSH/second and the file based destinations of syslog-ng are not dropping messages (we are sending almost 80k/s) so I don’t know where I am going wrong.

loggen was accepting > 110000mps its just on the outputs it seems.



Thanks, Nick


The information in this message, including in all attachments, is confidential or privileged. In the event you have received this message in error and are not the intended recipient, you are hereby advised that any use, copying or reproduction of this document is strictly forbidden. Please notify immediately the sender of this error and destroy this message, including its attachments, as the case may be.

L'information apparaissant dans ce message électronique et dans les documents qui y sont joints est de nature confidentielle ou privilégiée. Si ce message vous est parvenu par erreur et que vous n'en êtes pas le destinataire visé, vous êtes par les présentes avisé que toute utilisation, copie ou distribution de ce message est strictement interdite. Vous êtes donc prié d’en informer immédiatement l’expéditeur et de détruire ce message, ainsi que les documents qui y sont joints, le cas échéant.



Ce courriel provient de Nicholas.Radonicich at cogeco.com<mailto:Nicholas.Radonicich at cogeco.com> . Pour assurer la livraison de futurs envois, veuillez inclure la présente adresse courriel à votre carnet d’adresses ou votre liste d’expéditeurs autorisés.

Si vous ne souhaitez plus recevoir de messages promotionnels de la part de Cogeco, veuillez envoyer un courriel à desabonnement at cogeco.com<mailto:desabonnement at cogeco.com?Subject=Unsubscribe+to+mail+from+Nicholas.Radonicich at cogeco.com>.

Politique en matière de protection des renseignements personnels de Cogeco et Engagement en matière d’anti-spam<http://www.cogeco.ca/cable/entreprise/cca/vie_privee.html> – Contactez-nous<http://www.cogeco.ca/web/qc/fr/residentiel/soutien/contact_telephone.php>

Cogeco Câble Canada, 5 Place Ville-Marie, Bureau 1700, Montréal, Québec, H3B 0B3

--

This email is from Nicholas.Radonicich at cogeco.com<mailto:Nicholas.Radonicich at cogeco.com> . To ensure the delivery of future emails, please add the current email address to your address book or safe senders list.

If you no longer wish to receive promotional emails from Cogeco, please send an email to unsubscribe at cogeco.com<mailto:unsubscribe at cogeco.com?Subject=Unsubscribe+to+mail+from+Nicholas.Radonicich at cogeco.com>.

Privacy Policy and Anti-spam Commitment<http://www.cogeco.ca/cable/corporate/cgo/privacy.html> - Contact us<http://www.cogeco.ca/web/on/en/residential/support/contact_phone.php>

Cogeco Cable Canada, 5 Place Ville-Marie, Suite 1700, Montreal, Quebec, H3B 0B3



______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20140721/8e5a792d/attachment-0001.htm