[syslog-ng] Disk-buffer in kafka-destinations

[Ronald Fenner]

I'm using 
syslog-ng 3 (3.23.1)
Config version: 3.23
Installer-Version: 3.23.1
Revision:
Compile-Date: Oct 21 2019 21:48:06
Module-Directory: /opt/syslog-ng/lib/syslog-ng
Module-Path: /opt/syslog-ng/lib/syslog-ng
Include-Path: /opt/syslog-ng/share/syslog-ng/include
Error opening plugin module; module='mod-java', error='libjvm.so: cannot open shared object file: No such file or directory'
Error opening plugin module; module='kafka', error='librdkafka.so.1: cannot open shared object file: No such file or directory'
Available-Modules: add-contextual-data,affile,afprog,afsocket,afstomp,afuser,appmodel,basicfuncs,cef,confgen,cryptofuncs,csvparser,date,dbparser,disk-buffer,examples,tfgetent,graphite,hook-commands,http,json-plugin,kvformat,linux-kmsg-format,map-value-pairs,pseudofile,mod-python,snmptrapd-parser,stardate,syslogformat,system-source,tags-parser,xml
Enable-Debug: off
Enable-GProf: off
Enable-Memtrace: off
Enable-IPv6: on
Enable-Spoof-Source: off
Enable-TCP-Wrapper: off
Enable-Linux-Caps: off
Enable-Systemd: off

I'm using the java version of Kafka as the C version was recently released and I haven't upped my version of syslog-ng yet.

My kafka destination looks like:

destination d_kafka_environment {
    kafka (
        client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/kafka.jar:/opt/kafka/libs/")
        kafka-bootstrap-servers("kafka.service.consul:9092")
        topic("${topic}.${environment}")
        template("$(format-json --scope nv_pairs --exclude MESSAGE)\n")
    );
};

I had tried placing the disk-buffer inside the kafka block and inside the destination block without success.

I was using this block for the disk-buffer
#    disk-buffer(
#        mem-buff-size(10000000)
#        disk-buf-size(10000000000)
#        reliable(yes)
#        dir("/opt/syslog-ng/disk-buffers/kafka_environment")
#    )

Ronald Fenner
Network Architect
Game Circus LLC.

rfenner at gamecircus.com

> On Jan 10, 2020, at 1:08 AM, Peter Kokai (pkokai) <Peter.Kokai at oneidentity.com> wrote:
> 
> Hello,
> 
> Every destination has a queue by default memory, and it is possible to replace with disk based buffer. If that does not work that is probably a bug.
> Could you please write the version of syslog-ng you are using ?
> Do you use the java or the C based kafka destination ?
> 
> Please include the config and error.
> 
> --
> Kokan
> 
> On Thu, Jan 09, 2020 at 03:07:44PM -0600, Ronald Fenner wrote:
>> CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.
>> 
>> I can't see to me get syslog-ng to start if I add the disk-buffer to the destination.
>> From the docs page it looks like it should be added to the kafka section but it doesn't work and doesn't work if I put it with in the destination block.
>> 
>> Is this tied to a specific destination type?
>> 
>> We are trying to make out messages going to kafka more reliable when the cluster is unavailable.
>> 
>> Ronald Fenner
>> Network Architect
>> Game Circus LLC.
>> 
>> rfenner at gamecircus.com<mailto:rfenner at gamecircus.com>
>> 
> 
>> ______________________________________________________________________________
>> Member info: https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7CPeter.Kokai%40oneidentity.com%7Cb72a9458366446f63e9908d79547fcd2%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142008740844164&sdata=rsTQvFjkAJPdawx18O7jSuHbbcKTZFiR8qTbUBbgEH4%3D&reserved=0
>> Documentation: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7CPeter.Kokai%40oneidentity.com%7Cb72a9458366446f63e9908d79547fcd2%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142008740854159&sdata=13KCnq80UC9fGKLIDzLkFt%2FiGeo6KBNthfbqaT9UQiQ%3D&reserved=0
>> FAQ: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7CPeter.Kokai%40oneidentity.com%7Cb72a9458366446f63e9908d79547fcd2%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142008740854159&sdata=yeie8XJtlm1czeAefxbz%2FFXsmrf%2BxFZe4xHkpbrt1t8%3D&reserved=0
>> 
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>