After more investigation including looking at the packets going between the instance and the kafka cluster I found the problem. It appears that one of the tpoics didn't exist that messages were trying to push to. This seems like was blocking all message going to any of the other topics that existed. Once I added the topic on the cluster's side messages started being written. I'm guessing syslog-ng was getting stuck trying to write to the non-existent topic and then not writing any of hte messages to topics that actually existed. Ronald Fenner Network Architect Game Circus LLC. rfenner@gamecircus.com
On Aug 6, 2020, at 12:01 AM, Ronald Fenner <rfenner@gamecircus.com> wrote:
I build from source so if provided with a tar ball I can drop it into the build process and generate test image to see what happens. I know it's not an issue with it receiving the messages they are sent to a unix pipe and if I enable trace I see the messages go screaming by when I tail the log.
The problems seems to be on the sending side and I'm not sure why.
You mentioned a perf tool what is it so I can look for it and then see how to run it.
I'll also go open an issue on GitHub.
Ronald Fenner Network Architect Game Circus LLC.
rfenner@gamecircus.com <mailto:rfenner@gamecircus.com>
On Aug 5, 2020, at 11:51 PM, Laszlo Budai (lbudai) <Laszlo.Budai@oneidentity.com <mailto:Laszlo.Budai@oneidentity.com>> wrote:
Could you open an issue at github? Do you have a test system where you can install a not released syslog-ng? (We can provide you rpm or deb packages.) If you have a test system, you could replace kafka with file destination to see if the bottleneck is at kafka (or maybe at the source, or other logpipe element). Or just run the perf tool...
L. From: syslog-ng <syslog-ng-bounces@lists.balabit.hu <mailto:syslog-ng-bounces@lists.balabit.hu>> on behalf of Ronald Fenner <rfenner@gamecircus.com <mailto:rfenner@gamecircus.com>> Sent: Thursday, August 6, 2020 1:47 AM To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu <mailto:syslog-ng@lists.balabit.hu>> Subject: [syslog-ng] Kafka performance
CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.
I'm trying to figure out why the performance of syslog-ng sending to kafka has tanked considerably. Everything was running fine up until 7/30 and then all of sudden syslog-ng is having problems pushing messages to kafka at a quick pace.
I've updated to 3.28.1 and it's still having problems. I was using disk buffering but I removed it an d doesn't seem to be the bottleneck.
I feel it's something with syslog ng as I can use the kafka perf test to push messages at a very quick rate to the cluster. I was able to push 1 million 1k test messages in several seconds. Syslog-ng seems to be able to only push a couple of messages a second.
Any thoughts on what could be tanking it's performance? It's running on Amazon Linux 2.
I've rebuilt our base image with the latest version so we can switch between kafka-c and the java version however since the kafka-c topic can't be templated at the moment until a pull request is released I can't switch to it to see if it fairs better.
Ronald Fenner Network Architect Game Circus LLC.
rfenner@gamecircus.com <mailto:rfenner@gamecircus.com> ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>