On Jan 10, 2020, at 12:09 PM, Peter Kokai (pkokai) <Peter.Kokai@oneidentity.com> wrote:Hello,
You have a typo `mem-buff-size` -> `mem-buf-size`.
Let us know if you have further trouble.
--
Kokan
On Fri, Jan 10, 2020 at 10:38:50AM -0600, Ronald Fenner wrote:CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.______________________________________________________________________________
I'm using
syslog-ng 3 (3.23.1)
Config version: 3.23
Installer-Version: 3.23.1
Revision:
Compile-Date: Oct 21 2019 21:48:06
Module-Directory: /opt/syslog-ng/lib/syslog-ng
Module-Path: /opt/syslog-ng/lib/syslog-ng
Include-Path: /opt/syslog-ng/share/syslog-ng/include
Error opening plugin module; module='mod-java', error='libjvm.so: cannot open shared object file: No such file or directory'
Error opening plugin module; module='kafka', error='librdkafka.so.1: cannot open shared object file: No such file or directory'
Available-Modules: add-contextual-data,affile,afprog,afsocket,afstomp,afuser,appmodel,basicfuncs,cef,confgen,cryptofuncs,csvparser,date,dbparser,disk-buffer,examples,tfgetent,graphite,hook-commands,http,json-plugin,kvformat,linux-kmsg-format,map-value-pairs,pseudofile,mod-python,snmptrapd-parser,stardate,syslogformat,system-source,tags-parser,xml
Enable-Debug: off
Enable-GProf: off
Enable-Memtrace: off
Enable-IPv6: on
Enable-Spoof-Source: off
Enable-TCP-Wrapper: off
Enable-Linux-Caps: off
Enable-Systemd: off
I'm using the java version of Kafka as the C version was recently released and I haven't upped my version of syslog-ng yet.
My kafka destination looks like:
destination d_kafka_environment {
kafka (
client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/kafka.jar:/opt/kafka/libs/")
kafka-bootstrap-servers("kafka.service.consul:9092")
topic("${topic}.${environment}")
template("$(format-json --scope nv_pairs --exclude MESSAGE)\n")
);
};
I had tried placing the disk-buffer inside the kafka block and inside the destination block without success.
I was using this block for the disk-buffer
# disk-buffer(
# mem-buff-size(10000000)
# disk-buf-size(10000000000)
# reliable(yes)
# dir("/opt/syslog-ng/disk-buffers/kafka_environment")
# )
Ronald Fenner
Network Architect
Game Circus LLC.
rfenner@gamecircus.comOn Jan 10, 2020, at 1:08 AM, Peter Kokai (pkokai) <Peter.Kokai@oneidentity.com> wrote:
Hello,
Every destination has a queue by default memory, and it is possible to replace with disk based buffer. If that does not work that is probably a bug.
Could you please write the version of syslog-ng you are using ?
Do you use the java or the C based kafka destination ?
Please include the config and error.
--
Kokan
On Thu, Jan 09, 2020 at 03:07:44PM -0600, Ronald Fenner wrote:CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.
I can't see to me get syslog-ng to start if I add the disk-buffer to the destination.
From the docs page it looks like it should be added to the kafka section but it doesn't work and doesn't work if I put it with in the destination block.
Is this tied to a specific destination type?
We are trying to make out messages going to kafka more reliable when the cluster is unavailable.
Ronald Fenner
Network Architect
Game Circus LLC.
rfenner@gamecircus.com<mailto:rfenner@gamecircus.com>____________________________________________________________________________________________________________________________________________________________
Member info: https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408966399&sdata=y%2B%2FEXYVMpa1siBC%2Bf29Aw66EtxqTZygBNPzS9ZwiDTE%3D&reserved=0
Documentation: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408966399&sdata=lR%2FSGG3gLGDIaFFaC5xiTDn8i5OioHXXWdzoQXKveOY%3D&reserved=0
FAQ: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408976387&sdata=wP7WS%2Fb4bHF6VjCW0CnEGHQnUhslSwg89qC06qn52pI%3D&reserved=0
Member info: https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408976387&sdata=HqV%2FD9%2Bdn5%2B%2BB5fyENstMbwRpfydRH7nWKaNIudxNUg%3D&reserved=0
Documentation: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408976387&sdata=%2BHrQkHUbr6UpQducSiqu6xck%2FAbjeLFTwfGkKxeispg%3D&reserved=0
FAQ: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408976387&sdata=wP7WS%2Fb4bHF6VjCW0CnEGHQnUhslSwg89qC06qn52pI%3D&reserved=0
______________________________________________________________________________
Member info: https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408976387&sdata=HqV%2FD9%2Bdn5%2B%2BB5fyENstMbwRpfydRH7nWKaNIudxNUg%3D&reserved=0
Documentation: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408976387&sdata=%2BHrQkHUbr6UpQducSiqu6xck%2FAbjeLFTwfGkKxeispg%3D&reserved=0
FAQ: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7Cc2cc9de46a404287783208d795eb96fe%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637142711408976387&sdata=wP7WS%2Fb4bHF6VjCW0CnEGHQnUhslSwg89qC06qn52pI%3D&reserved=0
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq