[syslog-ng] Error trying to use kafka-c

[Peter Kokai (pkokai)]

Hello,

The log suggests that syslog-ng cannot find the module containing kafka-c destination plugin; despite of your configuration argument list seems reasonable.

syslog-ng has an option --module-registry, that does module discovery and displays information about the modules.
First it would be a good idea to check if that list has kafka in the module list, or if that command write out any error regarding the libkafka.so module.
If the list does not have kafka and there is no error you should check if the /opt/syslog-ng/lib/syslog-ng directory has libkafka.so at all.
If there is an error message that is most likely a missing dependency for librdkafka, that you can fix either installing or correcting LD_LIBRARY_PATH.

--
Kokan

On Tue, Aug 04, 2020 at 06:05:58PM -0500, Ronald Fenner wrote:
> CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.
> 
> I'm trying to switch from the java version of kafka to the c version and have built syslog with the following config:
> 
> ./configure --prefix=/opt/syslog-ng --enable-json --enable-python --with-python=2.7 --enable-kafka
> 
> However when I try syntax check the config it's returning this error:
> Error parsing destination statement, destination plugin kafka-c not found in block destination kafka() at /opt/syslog-ng/share/syslog-ng/include/scl/kafka/kafka.conf:24:4:5-4:12:
> 1
> 2       #Start Block block destination kafka() at /opt/syslog-ng/share/syslog-ng/include/scl/kafka/kafka.conf:24
> 3
> 4----->     kafka-c(bootstrap_servers("kafka.service.consul:9092") topic("syslog-ng-production") );
> 4----->     ^^^^^^^
> 5
> 6       #End Block block destination kafka() at /opt/syslog-ng/share/syslog-ng/include/scl/kafka/kafka.conf:24
> 7
> 
> Included from /opt/syslog-ng/etc/syslog-ng.conf:33:5-36:6:
> 28      source s_rtl_raw_backup {
> 29          file("/var/gamecircus/log/rtl-raw-backup.log" flags(no-parse) log_msg_size(1000000));
> 30      };
> 31
> 32      destination d_kafka_unstructured {
> 33---->     kafka (
> 33---->     ^^^^^^^^
> 34---->         bootstrap-servers("kafka.service.consul:9092")
> 35---->         topic("syslog-ng-`DeployedEnvironment`")
> 36---->     );
> 37      };
> 38
> 
> 
> The top of my config looks like this
> 
> @version: 3.28
> @include "scl.conf"
> @include "conf.d/gc-parsers.conf"
> @define kafka-implementation kafka-c
> 
> Everything looks correct just not sure why it's having a hard time finding it.
> 
> 
> Ronald Fenner
> Network Architect
> Game Circus LLC.
> 
> rfenner at gamecircus.com<mailto:rfenner at gamecircus.com>
> 

> ______________________________________________________________________________
> Member info: https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7C98c1e77f51fa481f262f08d838caf6b6%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637321791667263481&sdata=uIkIQMnfTXxy9c0R0WM3MpaKXbBcjoXCb7bmJpR1lws%3D&reserved=0
> Documentation: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7C98c1e77f51fa481f262f08d838caf6b6%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637321791667263481&sdata=1Xj9sTfU8tJIeoUzN5pt4U71f8i1XYuBg2uRndi2f4I%3D&reserved=0
> FAQ: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7C98c1e77f51fa481f262f08d838caf6b6%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637321791667263481&sdata=iAIgISDCX8dSFUpEW26UPy84foQlOHk8QNNlIQK94OQ%3D&reserved=0
>