I'm using syslog-ng 3 (3.23.1) Config version: 3.23 Installer-Version: 3.23.1 Revision: Compile-Date: Oct 21 2019 21:48:06 Module-Directory: /opt/syslog-ng/lib/syslog-ng Module-Path: /opt/syslog-ng/lib/syslog-ng Include-Path: /opt/syslog-ng/share/syslog-ng/include Error opening plugin module; module='mod-java', error='libjvm.so: cannot open shared object file: No such file or directory' Error opening plugin module; module='kafka', error='librdkafka.so.1: cannot open shared object file: No such file or directory' Available-Modules: add-contextual-data,affile,afprog,afsocket,afstomp,afuser,appmodel,basicfuncs,cef,confgen,cryptofuncs,csvparser,date,dbparser,disk-buffer,examples,tfgetent,graphite,hook-commands,http,json-plugin,kvformat,linux-kmsg-format,map-value-pairs,pseudofile,mod-python,snmptrapd-parser,stardate,syslogformat,system-source,tags-parser,xml Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: off Enable-Systemd: off I'm using the java version of Kafka as the C version was recently released and I haven't upped my version of syslog-ng yet. My kafka destination looks like: destination d_kafka_environment { kafka ( client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/kafka.jar:/opt/kafka/libs/") kafka-bootstrap-servers("kafka.service.consul:9092") topic("${topic}.${environment}") template("$(format-json --scope nv_pairs --exclude MESSAGE)\n") ); }; I had tried placing the disk-buffer inside the kafka block and inside the destination block without success. I was using this block for the disk-buffer # disk-buffer( # mem-buff-size(10000000) # disk-buf-size(10000000000) # reliable(yes) # dir("/opt/syslog-ng/disk-buffers/kafka_environment") # ) Ronald Fenner Network Architect Game Circus LLC. rfenner@gamecircus.com
On Jan 10, 2020, at 1:08 AM, Peter Kokai (pkokai) <Peter.Kokai@oneidentity.com> wrote:
Hello,
Every destination has a queue by default memory, and it is possible to replace with disk based buffer. If that does not work that is probably a bug. Could you please write the version of syslog-ng you are using ? Do you use the java or the C based kafka destination ?
Please include the config and error.
-- Kokan
On Thu, Jan 09, 2020 at 03:07:44PM -0600, Ronald Fenner wrote:
CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.
I can't see to me get syslog-ng to start if I add the disk-buffer to the destination. From the docs page it looks like it should be added to the kafka section but it doesn't work and doesn't work if I put it with in the destination block.
Is this tied to a specific destination type?
We are trying to make out messages going to kafka more reliable when the cluster is unavailable.
Ronald Fenner Network Architect Game Circus LLC.
rfenner@gamecircus.com<mailto:rfenner@gamecircus.com>
______________________________________________________________________________ Member info: https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7CPeter.Kokai%40oneidentity.com%7Cb72a9458366446f63e9908d79547fcd2%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142008740844164&sdata=rsTQvFjkAJPdawx18O7jSuHbbcKTZFiR8qTbUBbgEH4%3D&reserved=0 Documentation: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7CPeter.Kokai%40oneidentity.com%7Cb72a9458366446f63e9908d79547fcd2%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142008740854159&sdata=13KCnq80UC9fGKLIDzLkFt%2FiGeo6KBNthfbqaT9UQiQ%3D&reserved=0 FAQ: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7CPeter.Kokai%40oneidentity.com%7Cb72a9458366446f63e9908d79547fcd2%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142008740854159&sdata=yeie8XJtlm1czeAefxbz%2FFXsmrf%2BxFZe4xHkpbrt1t8%3D&reserved=0
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq