[syslog-ng] Disk-buffer in kafka-destinations
Ronald Fenner
rfenner at gamecircus.com
Fri Jan 10 18:40:55 UTC 2020
Yep that was it. Thanks.
Ronald Fenner
Network Architect
Game Circus LLC.
rfenner at gamecircus.com
> On Jan 10, 2020, at 12:09 PM, Peter Kokai (pkokai) <Peter.Kokai at oneidentity.com> wrote:
>
> Hello,
>
> You have a typo `mem-buff-size` -> `mem-buf-size`.
> Let us know if you have further trouble.
>
> --
> Kokan
> On Fri, Jan 10, 2020 at 10:38:50AM -0600, Ronald Fenner wrote:
>> CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.
>>
>>
>> I'm using
>> syslog-ng 3 (3.23.1)
>> Config version: 3.23
>> Installer-Version: 3.23.1
>> Revision:
>> Compile-Date: Oct 21 2019 21:48:06
>> Module-Directory: /opt/syslog-ng/lib/syslog-ng
>> Module-Path: /opt/syslog-ng/lib/syslog-ng
>> Include-Path: /opt/syslog-ng/share/syslog-ng/include
>> Error opening plugin module; module='mod-java', error='libjvm.so: cannot open shared object file: No such file or directory'
>> Error opening plugin module; module='kafka', error='librdkafka.so.1: cannot open shared object file: No such file or directory'
>> Available-Modules: add-contextual-data,affile,afprog,afsocket,afstomp,afuser,appmodel,basicfuncs,cef,confgen,cryptofuncs,csvparser,date,dbparser,disk-buffer,examples,tfgetent,graphite,hook-commands,http,json-plugin,kvformat,linux-kmsg-format,map-value-pairs,pseudofile,mod-python,snmptrapd-parser,stardate,syslogformat,system-source,tags-parser,xml
>> Enable-Debug: off
>> Enable-GProf: off
>> Enable-Memtrace: off
>> Enable-IPv6: on
>> Enable-Spoof-Source: off
>> Enable-TCP-Wrapper: off
>> Enable-Linux-Caps: off
>> Enable-Systemd: off
>>
>> I'm using the java version of Kafka as the C version was recently released and I haven't upped my version of syslog-ng yet.
>>
>> My kafka destination looks like:
>>
>> destination d_kafka_environment {
>> kafka (
>> client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/kafka.jar:/opt/kafka/libs/")
>> kafka-bootstrap-servers("kafka.service.consul:9092")
>> topic("${topic}.${environment}")
>> template("$(format-json --scope nv_pairs --exclude MESSAGE)\n")
>> );
>> };
>>
>> I had tried placing the disk-buffer inside the kafka block and inside the destination block without success.
>>
>> I was using this block for the disk-buffer
>> # disk-buffer(
>> # mem-buff-size(10000000)
>> # disk-buf-size(10000000000)
>> # reliable(yes)
>> # dir("/opt/syslog-ng/disk-buffers/kafka_environment")
>> # )
>>
>> Ronald Fenner
>> Network Architect
>> Game Circus LLC.
>>
>> rfenner at gamecircus.com
>>
>>> On Jan 10, 2020, at 1:08 AM, Peter Kokai (pkokai) <Peter.Kokai at oneidentity.com> wrote:
>>>
>>> Hello,
>>>
>>> Every destination has a queue by default memory, and it is possible to replace with disk based buffer. If that does not work that is probably a bug.
>>> Could you please write the version of syslog-ng you are using ?
>>> Do you use the java or the C based kafka destination ?
>>>
>>> Please include the config and error.
>>>
>>> --
>>> Kokan
>>>
>>> On Thu, Jan 09, 2020 at 03:07:44PM -0600, Ronald Fenner wrote:
>>>> CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.
>>>>
>>>> I can't see to me get syslog-ng to start if I add the disk-buffer to the destination.
>>>> From the docs page it looks like it should be added to the kafka section but it doesn't work and doesn't work if I put it with in the destination block.
>>>>
>>>> Is this tied to a specific destination type?
>>>>
>>>> We are trying to make out messages going to kafka more reliable when the cluster is unavailable.
>>>>
>>>> Ronald Fenner
>>>> Network Architect
>>>> Game Circus LLC.
>>>>
>>>> rfenner at gamecircus.com<mailto:rfenner at gamecircus.com>
>>>>
>>>
>>>> ______________________________________________________________________________
>>>> Member info: https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408966399&sdata=y%2B%2FEXYVMpa1siBC%2Bf29Aw66EtxqTZygBNPzS9ZwiDTE%3D&reserved=0 <https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408966399&sdata=y%2B%2FEXYVMpa1siBC%2Bf29Aw66EtxqTZygBNPzS9ZwiDTE%3D&reserved=0>
>>>> Documentation: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408966399&sdata=lR%2FSGG3gLGDIaFFaC5xiTDn8i5OioHXXWdzoQXKveOY%3D&reserved=0 <https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408966399&sdata=lR%2FSGG3gLGDIaFFaC5xiTDn8i5OioHXXWdzoQXKveOY%3D&reserved=0>
>>>> FAQ: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408976387&sdata=wP7WS%2Fb4bHF6VjCW0CnEGHQnUhslSwg89qC06qn52pI%3D&reserved=0 <https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408976387&sdata=wP7WS%2Fb4bHF6VjCW0CnEGHQnUhslSwg89qC06qn52pI%3D&reserved=0>
>>>>
>>> ______________________________________________________________________________
>>> Member info: https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408976387&sdata=HqV%2FD9%2Bdn5%2B%2BB5fyENstMbwRpfydRH7nWKaNIudxNUg%3D&reserved=0 <https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408976387&sdata=HqV%2FD9%2Bdn5%2B%2BB5fyENstMbwRpfydRH7nWKaNIudxNUg%3D&reserved=0>
>>> Documentation: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408976387&sdata=%2BHrQkHUbr6UpQducSiqu6xck%2FAbjeLFTwfGkKxeispg%3D&reserved=0 <https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408976387&sdata=%2BHrQkHUbr6UpQducSiqu6xck%2FAbjeLFTwfGkKxeispg%3D&reserved=0>
>>> FAQ: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408976387&sdata=wP7WS%2Fb4bHF6VjCW0CnEGHQnUhslSwg89qC06qn52pI%3D&reserved=0 <https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408976387&sdata=wP7WS%2Fb4bHF6VjCW0CnEGHQnUhslSwg89qC06qn52pI%3D&reserved=0>
>>>
>>
>> ______________________________________________________________________________
>> Member info: https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408976387&sdata=HqV%2FD9%2Bdn5%2B%2BB5fyENstMbwRpfydRH7nWKaNIudxNUg%3D&reserved=0 <https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408976387&sdata=HqV%2FD9%2Bdn5%2B%2BB5fyENstMbwRpfydRH7nWKaNIudxNUg%3D&reserved=0>
>> Documentation: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408976387&sdata=%2BHrQkHUbr6UpQducSiqu6xck%2FAbjeLFTwfGkKxeispg%3D&reserved=0 <https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408976387&sdata=%2BHrQkHUbr6UpQducSiqu6xck%2FAbjeLFTwfGkKxeispg%3D&reserved=0>
>> FAQ: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408976387&sdata=wP7WS%2Fb4bHF6VjCW0CnEGHQnUhslSwg89qC06qn52pI%3D&reserved=0 <https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408976387&sdata=wP7WS%2Fb4bHF6VjCW0CnEGHQnUhslSwg89qC06qn52pI%3D&reserved=0>
>>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng>
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20200110/553335c1/attachment-0001.html>
More information about the syslog-ng
mailing list