[syslog-ng] redis/tcp destination throughput and dropped messages

[Tusa Viktor]

Hi Nick!

Unfortunately, redis destination cannot do batch inserts, that's why it's
very slow. But why do you want to use redis?
Syslog-ng can send messages directly to logstash. In incubator, you can
find and elastricsearch destination, although it is
only a slightly better in performance than the redis one.

BR,
Viktor


On Mon, Jul 21, 2014 at 3:53 PM, Nicholas Radonicich <
Nicholas.Radonicich at cogeco.com> wrote:

>   Hello, hoping someone might be able to help... I am attempting to build
> a syslog server with outputs to Logstash/Elasticsearch via redis but am
> having issues with dropped messages in such that i cannot get over ~4000
> messages per second out to a destination (TCP or redis) without the rest
> being dropped. I have tried various tuning on the tcp output with
> flush_lines() and threaded() to no avail, as well as some things on the
> inputs. Both redis and logstash (used for tcp inputs) were on the localhost.
>
>
>
> redis-benchmark has no issues doing >70000 LPUSH/second and the file based
> destinations of syslog-ng are not dropping messages (we are sending almost
> 80k/s) so I don’t know where I am going wrong.
>
>
>
> loggen was accepting > 110000mps its just on the outputs it seems.
>
>
>
>
>
>
>
> Thanks, Nick
>
>  The information in this message, including in all attachments, is
> confidential or privileged. In the event you have received this message in
> error and are not the intended recipient, you are hereby advised that any
> use, copying or reproduction of this document is strictly forbidden. Please
> notify immediately the sender of this error and destroy this message,
> including its attachments, as the case may be.
>
> L'information apparaissant dans ce message électronique et dans les
> documents qui y sont joints est de nature confidentielle ou privilégiée. Si
> ce message vous est parvenu par erreur et que vous n'en êtes pas le
> destinataire visé, vous êtes par les présentes avisé que toute utilisation,
> copie ou distribution de ce message est strictement interdite. Vous êtes
> donc prié d’en informer immédiatement l’expéditeur et de détruire ce
> message, ainsi que les documents qui y sont joints, le cas échéant.
>
>
>
> Ce courriel provient de Nicholas.Radonicich at cogeco.com . Pour assurer la
> livraison de futurs envois, veuillez inclure la présente adresse courriel à
> votre carnet d’adresses ou votre liste d’expéditeurs autorisés.
>
> Si vous ne souhaitez plus recevoir de messages promotionnels de la part de
> Cogeco, veuillez envoyer un courriel à desabonnement at cogeco.com
> <desabonnement at cogeco.com?Subject=Unsubscribe+to+mail+from+Nicholas.Radonicich at cogeco.com>
> .
>
> Politique en matière de protection des renseignements personnels de Cogeco
> et Engagement en matière d’anti-spam
> <http://www.cogeco.ca/cable/entreprise/cca/vie_privee.html> –
> Contactez-nous
> <http://www.cogeco.ca/web/qc/fr/residentiel/soutien/contact_telephone.php>
>
> Cogeco Câble Canada, 5 Place Ville-Marie, Bureau 1700, Montréal, Québec,
> H3B 0B3
>
> --
>
> This email is from Nicholas.Radonicich at cogeco.com . To ensure the
> delivery of future emails, please add the current email address to your
> address book or safe senders list.
>
> If you no longer wish to receive promotional emails from Cogeco, please
> send an email to unsubscribe at cogeco.com
> <unsubscribe at cogeco.com?Subject=Unsubscribe+to+mail+from+Nicholas.Radonicich at cogeco.com>
> .
>
> Privacy Policy and Anti-spam Commitment
> <http://www.cogeco.ca/cable/corporate/cgo/privacy.html> - Contact us
> <http://www.cogeco.ca/web/on/en/residential/support/contact_phone.php>
>
> Cogeco Cable Canada, 5 Place Ville-Marie, Suite 1700, Montreal, Quebec,
> H3B 0B3
>
>
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20140721/482ff20a/attachment-0001.htm