[syslog-ng] redis/tcp destination throughput and dropped messages

[Nicholas Radonicich]

Hello, hoping someone might be able to help... I am attempting to build a syslog server with outputs to Logstash/Elasticsearch via redis but am having issues with dropped messages in such that i cannot get over ~4000 messages per second out to a destination (TCP or redis) without the rest being dropped. I have tried various tuning on the tcp output with flush_lines() and threaded() to no avail, as well as some things on the inputs. Both redis and logstash (used for tcp inputs) were on the localhost.

redis-benchmark has no issues doing >70000 LPUSH/second and the file based destinations of syslog-ng are not dropping messages (we are sending almost 80k/s) so I don’t know where I am going wrong.

loggen was accepting > 110000mps its just on the outputs it seems.



Thanks, Nick


The information in this message, including in all attachments, is confidential or privileged. In the event you have received this message in error and are not the intended recipient, you are hereby advised that any use, copying or reproduction of this document is strictly forbidden. Please notify immediately the sender of this error and destroy this message, including its attachments, as the case may be.

L'information apparaissant dans ce message électronique et dans les documents qui y sont joints est de nature confidentielle ou privilégiée. Si ce message vous est parvenu par erreur et que vous n'en êtes pas le destinataire visé, vous êtes par les présentes avisé que toute utilisation, copie ou distribution de ce message est strictement interdite. Vous êtes donc prié d’en informer immédiatement l’expéditeur et de détruire ce message, ainsi que les documents qui y sont joints, le cas échéant.



Ce courriel provient de Nicholas.Radonicich at cogeco.com . Pour assurer la livraison de futurs envois, veuillez inclure la présente adresse courriel à votre carnet d’adresses ou votre liste d’expéditeurs autorisés.

Si vous ne souhaitez plus recevoir de messages promotionnels de la part de Cogeco, veuillez envoyer un courriel à desabonnement at cogeco.com<mailto:desabonnement at cogeco.com?Subject=Unsubscribe%20to%20mail%20from%20Nicholas.Radonicich at cogeco.com>.

Politique en matière de protection des renseignements personnels de Cogeco et Engagement en matière d’anti-spam<http://www.cogeco.ca/cable/entreprise/cca/vie_privee.html> – Contactez-nous<http://www.cogeco.ca/web/qc/fr/residentiel/soutien/contact_telephone.php>

Cogeco Câble Canada, 5 Place Ville-Marie, Bureau 1700, Montréal, Québec, H3B 0B3

--

This email is from Nicholas.Radonicich at cogeco.com . To ensure the delivery of future emails, please add the current email address to your address book or safe senders list.

If you no longer wish to receive promotional emails from Cogeco, please send an email to unsubscribe at cogeco.com<mailto:unsubscribe at cogeco.com?Subject=Unsubscribe%20to%20mail%20from%20Nicholas.Radonicich at cogeco.com>.

Privacy Policy and Anti-spam Commitment<http://www.cogeco.ca/cable/corporate/cgo/privacy.html> - Contact us<http://www.cogeco.ca/web/on/en/residential/support/contact_phone.php>

Cogeco Cable Canada, 5 Place Ville-Marie, Suite 1700, Montreal, Quebec, H3B 0B3


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20140721/1a29106f/attachment.htm