[syslog-ng] Disk-buffer in kafka-destinations

Fri Jan 10 18:09:22 UTC 2020

Hello,

You have a typo `mem-buff-size` -> `mem-buf-size`.
Let us know if you have further trouble.

--
Kokan
On Fri, Jan 10, 2020 at 10:38:50AM -0600, Ronald Fenner wrote:
> CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.
> 
> 
> I'm using
> syslog-ng 3 (3.23.1)
> Config version: 3.23
> Installer-Version: 3.23.1
> Revision:
> Compile-Date: Oct 21 2019 21:48:06
> Module-Directory: /opt/syslog-ng/lib/syslog-ng
> Module-Path: /opt/syslog-ng/lib/syslog-ng
> Include-Path: /opt/syslog-ng/share/syslog-ng/include
> Error opening plugin module; module='mod-java', error='libjvm.so: cannot open shared object file: No such file or directory'
> Error opening plugin module; module='kafka', error='librdkafka.so.1: cannot open shared object file: No such file or directory'
> Available-Modules: add-contextual-data,affile,afprog,afsocket,afstomp,afuser,appmodel,basicfuncs,cef,confgen,cryptofuncs,csvparser,date,dbparser,disk-buffer,examples,tfgetent,graphite,hook-commands,http,json-plugin,kvformat,linux-kmsg-format,map-value-pairs,pseudofile,mod-python,snmptrapd-parser,stardate,syslogformat,system-source,tags-parser,xml
> Enable-Debug: off
> Enable-GProf: off
> Enable-Memtrace: off
> Enable-IPv6: on
> Enable-Spoof-Source: off
> Enable-TCP-Wrapper: off
> Enable-Linux-Caps: off
> Enable-Systemd: off
> 
> I'm using the java version of Kafka as the C version was recently released and I haven't upped my version of syslog-ng yet.
> 
> My kafka destination looks like:
> 
> destination d_kafka_environment {
>     kafka (
>         client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/kafka.jar:/opt/kafka/libs/")
>         kafka-bootstrap-servers("kafka.service.consul:9092")
>         topic("${topic}.${environment}")
>         template("$(format-json --scope nv_pairs --exclude MESSAGE)\n")
>     );
> };
> 
> I had tried placing the disk-buffer inside the kafka block and inside the destination block without success.
> 
> I was using this block for the disk-buffer
> #    disk-buffer(
> #        mem-buff-size(10000000)
> #        disk-buf-size(10000000000)
> #        reliable(yes)
> #        dir("/opt/syslog-ng/disk-buffers/kafka_environment")
> #    )
> 
> Ronald Fenner
> Network Architect
> Game Circus LLC.
> 
> rfenner at gamecircus.com
> 
> > On Jan 10, 2020, at 1:08 AM, Peter Kokai (pkokai) <Peter.Kokai at oneidentity.com> wrote:
> >
> > Hello,
> >
> > Every destination has a queue by default memory, and it is possible to replace with disk based buffer. If that does not work that is probably a bug.
> > Could you please write the version of syslog-ng you are using ?
> > Do you use the java or the C based kafka destination ?
> >
> > Please include the config and error.
> >
> > --
> > Kokan
> >
> > On Thu, Jan 09, 2020 at 03:07:44PM -0600, Ronald Fenner wrote:
> >> CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.
> >>
> >> I can't see to me get syslog-ng to start if I add the disk-buffer to the destination.
> >> From the docs page it looks like it should be added to the kafka section but it doesn't work and doesn't work if I put it with in the destination block.
> >>
> >> Is this tied to a specific destination type?
> >>
> >> We are trying to make out messages going to kafka more reliable when the cluster is unavailable.
> >>
> >> Ronald Fenner
> >> Network Architect
> >> Game Circus LLC.
> >>
> >> rfenner at gamecircus.com<mailto:rfenner at gamecircus.com>
> >>
> >
> >> ______________________________________________________________________________
> >> Member info: https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408966399&sdata=y%2B%2FEXYVMpa1siBC%2Bf29Aw66EtxqTZygBNPzS9ZwiDTE%3D&reserved=0
> >> Documentation: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408966399&sdata=lR%2FSGG3gLGDIaFFaC5xiTDn8i5OioHXXWdzoQXKveOY%3D&reserved=0
> >> FAQ: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408976387&sdata=wP7WS%2Fb4bHF6VjCW0CnEGHQnUhslSwg89qC06qn52pI%3D&reserved=0
> >>
> > ______________________________________________________________________________
> > Member info: https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408976387&sdata=HqV%2FD9%2Bdn5%2B%2BB5fyENstMbwRpfydRH7nWKaNIudxNUg%3D&reserved=0
> > Documentation: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408976387&sdata=%2BHrQkHUbr6UpQducSiqu6xck%2FAbjeLFTwfGkKxeispg%3D&reserved=0
> > FAQ: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408976387&sdata=wP7WS%2Fb4bHF6VjCW0CnEGHQnUhslSwg89qC06qn52pI%3D&reserved=0
> >
> 
> ______________________________________________________________________________
> Member info: https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408976387&sdata=HqV%2FD9%2Bdn5%2B%2BB5fyENstMbwRpfydRH7nWKaNIudxNUg%3D&reserved=0
> Documentation: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408976387&sdata=%2BHrQkHUbr6UpQducSiqu6xck%2FAbjeLFTwfGkKxeispg%3D&reserved=0
> FAQ: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408976387&sdata=wP7WS%2Fb4bHF6VjCW0CnEGHQnUhslSwg89qC06qn52pI%3D&reserved=0
> 