[syslog-ng] [External] Re: librdkafka config property error on startup

Thu Sep 9 14:05:30 UTC 2021

Hi SKP,

I assume you have obtained syslog-ng-mod-rdkafka from https://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/xUbuntu_18.04/amd64/
Can you please confirm that? If it is true, then unfortunately it will not work.

tl;dr: You have to obtain librdkafka version at least 1.0.0 (if using syslog-ng 3.31.2) from other sources. Building it from git source might be an option: https://github.com/edenhill/librdkafka
Alternatively, you can use a more recent platform, like Ubuntu 20.04, which provides librdkafka 1.2.1.

In more details:

The above mentioned OBS repository was a personal project, and it has been discontinued.
The syslog-ng-mod-rdkafka is available for Ubuntu 18.04 from the OBS repository, but the necessary librdkafka package is not available there.
The OBS repository have provided librdkafka in the past, but this does not seem to be the case anymore.

We are actively working on providing a syslog-ng APT repository with stable releases and nightly packages that will be maintained by the syslog-ng development team. The first steps for this have been merged (#3763), we will post to this mailing list announcing this new repository when we are ready.

Cheers,
Attila

________________________________
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of Sanjay Patel <skp at smarthive.io>
Sent: Thursday, September 2, 2021 5:32 PM
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu>
Subject: Re: [syslog-ng] [External] Re: librdkafka config property error on startup

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

What does hard coded mean? What parameters need to be passed during startup?

* the version of syslog-ng
3.31

* the version of librdkafka library
librdkafka1/bionic,now 0.11.3-1build1 amd64
librdkafka-dev/bionic 0.11.3-1build1 amd64
librdkafka++1/bionic 0.11.3-1build1 amd64

* the version of your kafka server

kafka_2.12-2.8.0:

Thank you for all your help,

SKP

-----Original Message-----
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> On Behalf Of Peter Kokai (pkokai)
Sent: Thursday, September 2, 2021 3:09 AM
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu>
Subject: [External] Re: [syslog-ng] librdkafka config property error on startup

Hello,

This property is hard coded into the syslog-ng's kafka module.
As far as I can see the latest kafka version is 2.8: https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FFuture%2Brelease%2Bplan&data=04%7C01%7Cattila.szakacs%40balabit.com%7C0a0bc6f318584b40455608d96e26ee48%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637661935776168028%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=qJWnZK2PQDzF1rEvxJHShGrsxEN7mqSYLHDa8PKdHDE%3D&reserved=0>

Could you please share the following with us:
* the version of syslog-ng
* the version of librdkafka library
* the version of your kafka server

--
Kokan

________________________________________
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of Sanjay Patel <skp at smarthive.io>
Sent: 02 September 2021 08:57
To: syslog-ng at lists.balabit.hu
Subject: [syslog-ng] librdkafka config property error on startup

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

I am trying to use Kafka-c with Syslog-ng. I have version Kafka 3.31 of Kafka installed, and I have syslog-ng-mod-rdkafka installed. When I start Syslog-ng, I get the following error. I can't figure out where it is getting this setting from. Syslog starts, but it does not send any logs to Kafka.

kafka: error setting librdkafka config property; name='topic.partitioner', value='murmur2_random', error='No such configuration property:

My Kafka config looks as follows:

@include "scl.conf"
destination d_kafka {
  kafka-c(
    bootstrap-servers("10.254.164.15:9092")
    topic("incoming")
  );
};
log {
  source(s_network);
  destination(d_kafka);
};

Any help will be deeply appreciated.

SKP

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=04%7C01%7Cattila.szakacs%40balabit.com%7C0a0bc6f318584b40455608d96e26ee48%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637661935776178023%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=AH3%2FBiatgNTXolCPzRAfFKaOyTFuX6d%2F92h%2BLpQzECs%3D&reserved=0>
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=04%7C01%7Cattila.szakacs%40balabit.com%7C0a0bc6f318584b40455608d96e26ee48%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637661935776178023%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ye8s%2B3RkO3z14wcvw2mGr03qsFwyxMbidZJkzfMDkZM%3D&reserved=0>
FAQ: http://www.balabit.com/wiki/syslog-ng-faq<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=04%7C01%7Cattila.szakacs%40balabit.com%7C0a0bc6f318584b40455608d96e26ee48%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637661935776188018%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=96scYzgR7J9ZcW8niXWL0ELlqKXQv5tCgxENkM2drwU%3D&reserved=0>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20210909/7b99ee85/attachment.html>