[syslog-ng] Kafka performance

[Ronald Fenner]

I'm trying to figure out why the performance of syslog-ng sending to kafka has tanked considerably. Everything was running fine up until 7/30 and then all of sudden syslog-ng is having problems pushing messages to kafka at a quick pace.

I've updated to 3.28.1 and it's still having problems. I was using disk buffering but I removed it an d doesn't seem to be the bottleneck.

I feel it's something with syslog ng as I can use the kafka perf test to push messages at a very quick rate to the cluster. I was able to push 1 million 1k test messages in several seconds. Syslog-ng seems to be able to only push a couple of messages a second.

Any thoughts on what could be tanking it's performance? It's running on Amazon Linux 2.

I've rebuilt our base image with the latest version so we can switch between kafka-c and the java version however since the kafka-c topic can't be templated at the moment until a pull request is released I can't switch to it to see if it fairs better.


Ronald Fenner
Network Architect
Game Circus LLC.

rfenner at gamecircus.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20200805/b2a17a0c/attachment.html>