Re: [syslog-ng] Syslog-ng 3.7.3 + Elasticsearch V2.3 - Error init pipeline
Hi, it should work with the latest 1.x. I'm working on my 3.8 OBS repo, but having some problems. When I have the alpha packages, I'll send a mail. Regards, Laszlo Budai Sent from my HTC ----- Reply message ----- From: "Marco Mignone" <info@marcomignone.com> To: "Syslog-ng users' and developers' mailing list" <syslog-ng@lists.balabit.hu> Subject: [syslog-ng] Syslog-ng 3.7.3 + Elasticsearch V2.3 - Error init pipeline Date: Thu, Jun 16, 2016 17:34 Hi Peter, Thanks for the quick reply. I thought there was some hack or way around it. I will then try with ElasticSearch1.X. Will the latest version of 1.x work or is there a specific version? Thanks again for your time. Marco
On 16 Jun 2016, at 16:26, Czanik, Péter <peter.czanik@balabit.com> wrote:
Hi,
syslog-ng 3.7 only works with Elasticsearch 1.X. For Elasticsearch 2.X you need syslog-ng 3.8, which is still under development (alpha).
Bye, Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik
On Thu, Jun 16, 2016 at 5:11 PM, Marco Mignone <info@marcomignone.com> wrote:
Hi All, I am trying to setup syslog-ng to use elasticsearch as its destination on Ubuntu 14.04. This the version of syslog (the unofficial versions installed from laszlo_budai rep:
syslog-ng 3.7.3 Installer-Version: 3.7.3 Revision: 3.7.3-8 Compile-Date: Jun 1 2016 16:33:00 Available-Modules: basicfuncs,linux-kmsg-format,riemann,afuser,afstomp,afprog,json-plugin,afsmtp,affile,csvparser,mod-java,pseudofile,confgen,afsocket,afamqp,redis,sdjournal,kvformat,syslogformat,afsql,system-source,mod-python,graphite,dbparser,geoip-plugin,afmongodb,cryptofuncs
Elastic search is:
"name" : "Theresa Cassidy", "cluster_name" : "elasticsearch", "version" : { "number" : "2.3.3", "build_hash" : "218bdf10790eef486ff2c41a3df5cfa32dadcfde", "build_timestamp" : "2016-05-17T15:40:04Z", "build_snapshot" : false, "lucene_version" : “5.5.0"
And my custom configuration in /etc/syslog-ng/conf.d/test.conf which is:
@module mod-java
source s_net { udp(); tcp(); };
destination d_elastic { elasticsearch( index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("test") client_lib_dir("/usr/share/elasticsearch/lib") ); };
log { source(s_net); destination(d_elastic); flags(flow-control); };
When I try to launch syslog in debug mode this is what I get:
[2016-06-16T15:54:29.378356] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/syslog-ng-core.jar; [2016-06-16T15:54:29.382446] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/syslog-ng-core.jar; [2016-06-16T15:54:29.382660] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/kafka.jar; [2016-06-16T15:54:29.382862] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/http.jar; [2016-06-16T15:54:29.383052] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/hdfs.jar; [2016-06-16T15:54:29.383258] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/elastic.jar; [2016-06-16T15:54:29.383479] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/syslog-ng-common.jar; [2016-06-16T15:54:29.383670] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/log4j-1.2.16.jar; [2016-06-16T15:54:29.383917] Add path to classpath: /usr/share/elasticsearch/lib/guava-18.0.jar; [2016-06-16T15:54:29.384098] Add path to classpath: /usr/share/elasticsearch/lib/jna-4.1.0.jar; [2016-06-16T15:54:29.384293] Add path to classpath: /usr/share/elasticsearch/lib/spatial4j-0.5.jar; [2016-06-16T15:54:29.384494] Add path to classpath: /usr/share/elasticsearch/lib/compress-lzf-1.0.2.jar; [2016-06-16T15:54:29.386104] Add path to classpath: /usr/share/elasticsearch/lib/lucene-spatial-5.5.0.jar; [2016-06-16T15:54:29.386342] Add path to classpath: /usr/share/elasticsearch/lib/lucene-backward-codecs-5.5.0.jar; [2016-06-16T15:54:29.386507] Add path to classpath: /usr/share/elasticsearch/lib/t-digest-3.0.jar; [2016-06-16T15:54:29.386677] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.6.6.jar; [2016-06-16T15:54:29.386865] Add path to classpath: /usr/share/elasticsearch/lib/lucene-memory-5.5.0.jar; [2016-06-16T15:54:29.387044] Add path to classpath: /usr/share/elasticsearch/lib/jackson-core-2.6.6.jar; [2016-06-16T15:54:29.387216] Add path to classpath: /usr/share/elasticsearch/lib/joda-time-2.8.2.jar; [2016-06-16T15:54:29.387394] Add path to classpath: /usr/share/elasticsearch/lib/lucene-join-5.5.0.jar; [2016-06-16T15:54:29.387673] Add path to classpath: /usr/share/elasticsearch/lib/lucene-grouping-5.5.0.jar; [2016-06-16T15:54:29.388476] Add path to classpath: /usr/share/elasticsearch/lib/HdrHistogram-2.1.6.jar; [2016-06-16T15:54:29.388647] Add path to classpath: /usr/share/elasticsearch/lib/compiler-0.8.13.jar; [2016-06-16T15:54:29.388818] Add path to classpath: /usr/share/elasticsearch/lib/netty-3.10.5.Final.jar; [2016-06-16T15:54:29.388972] Add path to classpath: /usr/share/elasticsearch/lib/lucene-misc-5.5.0.jar; [2016-06-16T15:54:29.389518] Add path to classpath: /usr/share/elasticsearch/lib/lucene-analyzers-common-5.5.0.jar; [2016-06-16T15:54:29.389711] Add path to classpath: /usr/share/elasticsearch/lib/lucene-sandbox-5.5.0.jar; [2016-06-16T15:54:29.390094] Add path to classpath: /usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar; [2016-06-16T15:54:29.390283] Add path to classpath: /usr/share/elasticsearch/lib/commons-cli-1.3.1.jar; [2016-06-16T15:54:29.390488] Add path to classpath: /usr/share/elasticsearch/lib/snakeyaml-1.15.jar; [2016-06-16T15:54:29.390659] Add path to classpath: /usr/share/elasticsearch/lib/jsr166e-1.1.0.jar; [2016-06-16T15:54:29.390935] Add path to classpath: /usr/share/elasticsearch/lib/lucene-core-5.5.0.jar; [2016-06-16T15:54:29.391176] Add path to classpath: /usr/share/elasticsearch/lib/lucene-suggest-5.5.0.jar; [2016-06-16T15:54:29.394616] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.6.6.jar; [2016-06-16T15:54:29.395279] Add path to classpath: /usr/share/elasticsearch/lib/log4j-1.2.17.jar; [2016-06-16T15:54:29.395458] Add path to classpath: /usr/share/elasticsearch/lib/joda-convert-1.2.jar; [2016-06-16T15:54:29.395970] Add path to classpath: /usr/share/elasticsearch/lib/hppc-0.7.1.jar; [2016-06-16T15:54:29.396734] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-smile-2.6.6.jar; [2016-06-16T15:54:29.397919] Add path to classpath: /usr/share/elasticsearch/lib/lucene-queryparser-5.5.0.jar; [2016-06-16T15:54:29.398106] Add path to classpath: /usr/share/elasticsearch/lib/lucene-spatial3d-5.5.0.jar; [2016-06-16T15:54:29.398281] Add path to classpath: /usr/share/elasticsearch/lib/lucene-queries-5.5.0.jar; [2016-06-16T15:54:29.398440] Add path to classpath: /usr/share/elasticsearch/lib/apache-log4j-extras-1.2.17.jar; [2016-06-16T15:54:29.398610] Add path to classpath: /usr/share/elasticsearch/lib/lucene-highlighter-5.5.0.jar; [2016-06-16T15:54:29.398784] Add path to classpath: /usr/share/elasticsearch/lib/jts-1.13.jar; [2016-06-16T15:54:29.398925] Add path to classpath: /usr/share/elasticsearch/lib/securesm-1.0.jar; [2016-06-16T15:54:29.501879] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/syslog-ng-core.jar; [2016-06-16T15:54:29.519443] Error initializing message pipeline;
Was anyone able to make this work or suggest a way to fix this? I hope I am on the right place and if not apologies in advance.
Thanks, Marco
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Thanks guys, You’re great. I can confirm it works with ES1.7.5 and Kibana 4.1 Thanks, Marco
On 16 Jun 2016, at 16:48, Laszlo Budai <laszlo.budai@outlook.com> wrote:
Hi,
it should work with the latest 1.x.
I'm working on my 3.8 OBS repo, but having some problems. When I have the alpha packages, I'll send a mail.
Regards, Laszlo Budai
Sent from my HTC
----- Reply message ----- From: "Marco Mignone" <info@marcomignone.com> To: "Syslog-ng users' and developers' mailing list" <syslog-ng@lists.balabit.hu> Subject: [syslog-ng] Syslog-ng 3.7.3 + Elasticsearch V2.3 - Error init pipeline Date: Thu, Jun 16, 2016 17:34
Hi Peter, Thanks for the quick reply. I thought there was some hack or way around it. I will then try with ElasticSearch1.X. Will the latest version of 1.x work or is there a specific version?
Thanks again for your time.
Marco
On 16 Jun 2016, at 16:26, Czanik, Péter <peter.czanik@balabit.com> wrote:
Hi,
syslog-ng 3.7 only works with Elasticsearch 1.X. For Elasticsearch 2.X you need syslog-ng 3.8, which is still under development (alpha).
Bye, Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik
On Thu, Jun 16, 2016 at 5:11 PM, Marco Mignone <info@marcomignone.com> wrote:
Hi All, I am trying to setup syslog-ng to use elasticsearch as its destination on Ubuntu 14.04. This the version of syslog (the unofficial versions installed from laszlo_budai rep:
syslog-ng 3.7.3 Installer-Version: 3.7.3 Revision: 3.7.3-8 Compile-Date: Jun 1 2016 16:33:00 Available-Modules: basicfuncs,linux-kmsg-format,riemann,afuser,afstomp,afprog,json-plugin,afsmtp,affile,csvparser,mod-java,pseudofile,confgen,afsocket,afamqp,redis,sdjournal,kvformat,syslogformat,afsql,system-source,mod-python,graphite,dbparser,geoip-plugin,afmongodb,cryptofuncs
Elastic search is:
"name" : "Theresa Cassidy", "cluster_name" : "elasticsearch", "version" : { "number" : "2.3.3", "build_hash" : "218bdf10790eef486ff2c41a3df5cfa32dadcfde", "build_timestamp" : "2016-05-17T15:40:04Z", "build_snapshot" : false, "lucene_version" : “5.5.0"
And my custom configuration in /etc/syslog-ng/conf.d/test.conf which is:
@module mod-java
source s_net { udp(); tcp(); };
destination d_elastic { elasticsearch( index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("test") client_lib_dir("/usr/share/elasticsearch/lib") ); };
log { source(s_net); destination(d_elastic); flags(flow-control); };
When I try to launch syslog in debug mode this is what I get:
[2016-06-16T15:54:29.378356] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/syslog-ng-core.jar; [2016-06-16T15:54:29.382446] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/syslog-ng-core.jar; [2016-06-16T15:54:29.382660] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/kafka.jar; [2016-06-16T15:54:29.382862] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/http.jar; [2016-06-16T15:54:29.383052] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/hdfs.jar; [2016-06-16T15:54:29.383258] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/elastic.jar; [2016-06-16T15:54:29.383479] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/syslog-ng-common.jar; [2016-06-16T15:54:29.383670] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/log4j-1.2.16.jar; [2016-06-16T15:54:29.383917] Add path to classpath: /usr/share/elasticsearch/lib/guava-18.0.jar; [2016-06-16T15:54:29.384098] Add path to classpath: /usr/share/elasticsearch/lib/jna-4.1.0.jar; [2016-06-16T15:54:29.384293] Add path to classpath: /usr/share/elasticsearch/lib/spatial4j-0.5.jar; [2016-06-16T15:54:29.384494] Add path to classpath: /usr/share/elasticsearch/lib/compress-lzf-1.0.2.jar; [2016-06-16T15:54:29.386104] Add path to classpath: /usr/share/elasticsearch/lib/lucene-spatial-5.5.0.jar; [2016-06-16T15:54:29.386342] Add path to classpath: /usr/share/elasticsearch/lib/lucene-backward-codecs-5.5.0.jar; [2016-06-16T15:54:29.386507] Add path to classpath: /usr/share/elasticsearch/lib/t-digest-3.0.jar; [2016-06-16T15:54:29.386677] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.6.6.jar; [2016-06-16T15:54:29.386865] Add path to classpath: /usr/share/elasticsearch/lib/lucene-memory-5.5.0.jar; [2016-06-16T15:54:29.387044] Add path to classpath: /usr/share/elasticsearch/lib/jackson-core-2.6.6.jar; [2016-06-16T15:54:29.387216] Add path to classpath: /usr/share/elasticsearch/lib/joda-time-2.8.2.jar; [2016-06-16T15:54:29.387394] Add path to classpath: /usr/share/elasticsearch/lib/lucene-join-5.5.0.jar; [2016-06-16T15:54:29.387673] Add path to classpath: /usr/share/elasticsearch/lib/lucene-grouping-5.5.0.jar; [2016-06-16T15:54:29.388476] Add path to classpath: /usr/share/elasticsearch/lib/HdrHistogram-2.1.6.jar; [2016-06-16T15:54:29.388647] Add path to classpath: /usr/share/elasticsearch/lib/compiler-0.8.13.jar; [2016-06-16T15:54:29.388818] Add path to classpath: /usr/share/elasticsearch/lib/netty-3.10.5.Final.jar; [2016-06-16T15:54:29.388972] Add path to classpath: /usr/share/elasticsearch/lib/lucene-misc-5.5.0.jar; [2016-06-16T15:54:29.389518] Add path to classpath: /usr/share/elasticsearch/lib/lucene-analyzers-common-5.5.0.jar; [2016-06-16T15:54:29.389711] Add path to classpath: /usr/share/elasticsearch/lib/lucene-sandbox-5.5.0.jar; [2016-06-16T15:54:29.390094] Add path to classpath: /usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar; [2016-06-16T15:54:29.390283] Add path to classpath: /usr/share/elasticsearch/lib/commons-cli-1.3.1.jar; [2016-06-16T15:54:29.390488] Add path to classpath: /usr/share/elasticsearch/lib/snakeyaml-1.15.jar; [2016-06-16T15:54:29.390659] Add path to classpath: /usr/share/elasticsearch/lib/jsr166e-1.1.0.jar; [2016-06-16T15:54:29.390935] Add path to classpath: /usr/share/elasticsearch/lib/lucene-core-5.5.0.jar; [2016-06-16T15:54:29.391176] Add path to classpath: /usr/share/elasticsearch/lib/lucene-suggest-5.5.0.jar; [2016-06-16T15:54:29.394616] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.6.6.jar; [2016-06-16T15:54:29.395279] Add path to classpath: /usr/share/elasticsearch/lib/log4j-1.2.17.jar; [2016-06-16T15:54:29.395458] Add path to classpath: /usr/share/elasticsearch/lib/joda-convert-1.2.jar; [2016-06-16T15:54:29.395970] Add path to classpath: /usr/share/elasticsearch/lib/hppc-0.7.1.jar; [2016-06-16T15:54:29.396734] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-smile-2.6.6.jar; [2016-06-16T15:54:29.397919] Add path to classpath: /usr/share/elasticsearch/lib/lucene-queryparser-5.5.0.jar; [2016-06-16T15:54:29.398106] Add path to classpath: /usr/share/elasticsearch/lib/lucene-spatial3d-5.5.0.jar; [2016-06-16T15:54:29.398281] Add path to classpath: /usr/share/elasticsearch/lib/lucene-queries-5.5.0.jar; [2016-06-16T15:54:29.398440] Add path to classpath: /usr/share/elasticsearch/lib/apache-log4j-extras-1.2.17.jar; [2016-06-16T15:54:29.398610] Add path to classpath: /usr/share/elasticsearch/lib/lucene-highlighter-5.5.0.jar; [2016-06-16T15:54:29.398784] Add path to classpath: /usr/share/elasticsearch/lib/jts-1.13.jar; [2016-06-16T15:54:29.398925] Add path to classpath: /usr/share/elasticsearch/lib/securesm-1.0.jar; [2016-06-16T15:54:29.501879] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/syslog-ng-core.jar; [2016-06-16T15:54:29.519443] Error initializing message pipeline;
Was anyone able to make this work or suggest a way to fix this? I hope I am on the right place and if not apologies in advance.
Thanks, Marco
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (2)
-
Laszlo Budai
-
Marco Mignone