[syslog-ng] Kafka performance

[Laszlo Budai (lbudai)]

Could you open an issue at github?
Do you have a test system where you can install a not released syslog-ng? (We can provide you rpm or deb packages.)
If you have a test system, you could replace kafka with file destination to see if the bottleneck is at kafka (or maybe at the source, or other logpipe element). Or just run the perf tool...

L.
________________________________
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of Ronald Fenner <rfenner at gamecircus.com>
Sent: Thursday, August 6, 2020 1:47 AM
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu>
Subject: [syslog-ng] Kafka performance

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

I'm trying to figure out why the performance of syslog-ng sending to kafka has tanked considerably. Everything was running fine up until 7/30 and then all of sudden syslog-ng is having problems pushing messages to kafka at a quick pace.

I've updated to 3.28.1 and it's still having problems. I was using disk buffering but I removed it an d doesn't seem to be the bottleneck.

I feel it's something with syslog ng as I can use the kafka perf test to push messages at a very quick rate to the cluster. I was able to push 1 million 1k test messages in several seconds. Syslog-ng seems to be able to only push a couple of messages a second.

Any thoughts on what could be tanking it's performance? It's running on Amazon Linux 2.

I've rebuilt our base image with the latest version so we can switch between kafka-c and the java version however since the kafka-c topic can't be templated at the moment until a pull request is released I can't switch to it to see if it fairs better.


Ronald Fenner
Network Architect
Game Circus LLC.

rfenner at gamecircus.com<mailto:rfenner at gamecircus.com>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20200806/06e2d4e6/attachment.html>