Re: [syslog-ng] Web Visualize logs (Ivan Adji - Krstev)
Hi, For using syslog-ng together with elasticsearch and Kibana this is a pretty good description, also giving you a good understanding of what it enables you to do: https://czanik.blogs.balabit.com/2015/10/how-to-parse-data-with-syslog-ng-st... https://czanik.blogs.balabit.com/2015/12/elasticsearch-and-syslog-ng-fast-an... /Istvan
Thanks for the doc. I read them and i have the following error: *Error parsing destination, destination plugin java not found in /etc/syslog-ng/syslog-ng.conf at line 57, column 3:** ** java( *I have configure as follow: @version:3.7 @include "scl.conf" @module mod-java **source s_sys { system(); internal(); network(ip(0.0.0.0) port(6514) flags(syslog-protocol) transport("tls") tls(key_file("/etc/syslog-ng/cert.d/serverkey.pem") cert_file("/etc/syslog-ng/cert.d/servercert.pem") ca_dir("/etc/syslog-ng/ca.d") ) ); }; parser pattern_db { db-parser( file("/etc/syslog-ng/patterndb.d/patterndb.xml") ); }; destination d_es { java( class-path("/usr/lib64/syslog-ng/java-modules/*.jar:/usr/share/elasticsearch/lib/*.jar") class-name("org.syslog_ng.elasticsearch.ElasticSearchDestination") option("index", "syslog-ng_${YEAR}.${MONTH}.${DAY}") option("type", "test") option("cluster", "czpcluster") option("flush_limit", "100") option( "message_template", "$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n") ); }; I have installed just Syslog-NG and Elasticsearch ? Do i need to install somethign else ? On 05/26/2016 04:53 PM, Szabó, István wrote:
Hi,
For using syslog-ng together with elasticsearch and Kibana this is a pretty good description, also giving you a good understanding of what it enables you to do:
https://czanik.blogs.balabit.com/2015/10/how-to-parse-data-with-syslog-ng-st...
https://czanik.blogs.balabit.com/2015/12/elasticsearch-and-syslog-ng-fast-an...
/Istvan
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Your build of syslog-ng seems to lack java support. It should be present in 3.7.x releases, but it also depends on who compiled them. Where did you get your binaries from? BTW, someone builds syslog-ng docker images that has java support. You might try that to get through the initial hassles. Then later you may decide whether to use the docker in production or rather get a recent and properly built binary. Bazsi On May 27, 2016 10:16 AM, "Ivan Adji - Krstev" <akivanradix@gmail.com> wrote:
Thanks for the doc. I read them and i have the following error:
*Error parsing destination, destination plugin java not found in /etc/syslog-ng/syslog-ng.conf at line 57, column 3:*
* java( *I have configure as follow: @version:3.7 @include "scl.conf" @module mod-java
source s_sys { system(); internal(); network(ip(0.0.0.0) port(6514) flags(syslog-protocol) transport("tls") tls(key_file("/etc/syslog-ng/cert.d/serverkey.pem") cert_file("/etc/syslog-ng/cert.d/servercert.pem") ca_dir("/etc/syslog-ng/ca.d") ) );
};
parser pattern_db { db-parser( file("/etc/syslog-ng/patterndb.d/patterndb.xml") ); };
destination d_es { java(
class-path("/usr/lib64/syslog-ng/java-modules/*.jar:/usr/share/elasticsearch/lib/*.jar") class-name("org.syslog_ng.elasticsearch.ElasticSearchDestination") option("index", "syslog-ng_${YEAR}.${MONTH}.${DAY}") option("type", "test") option("cluster", "czpcluster") option("flush_limit", "100") option( "message_template", "$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n") ); };
I have installed just Syslog-NG and Elasticsearch ? Do i need to install somethign else ?
On 05/26/2016 04:53 PM, Szabó, István wrote:
Hi,
For using syslog-ng together with elasticsearch and Kibana this is a pretty good description, also giving you a good understanding of what it enables you to do:
https://czanik.blogs.balabit.com/2015/10/how-to-parse-data-with-syslog-ng-st...
https://czanik.blogs.balabit.com/2015/12/elasticsearch-and-syslog-ng-fast-an...
/Istvan
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Bazsi, I get syslog from: "https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng37/repo/epel-7/czani..." add the repo and then "yum install syslog-ng" after that i have download the Elasticsearch and install it and that is it. Im using CentOS 7. Ivan On 05/27/2016 11:57 AM, Balazs Scheidler wrote:
Your build of syslog-ng seems to lack java support. It should be present in 3.7.x releases, but it also depends on who compiled them. Where did you get your binaries from?
BTW, someone builds syslog-ng docker images that has java support. You might try that to get through the initial hassles. Then later you may decide whether to use the docker in production or rather get a recent and properly built binary.
Bazsi
On May 27, 2016 10:16 AM, "Ivan Adji - Krstev" <akivanradix@gmail.com <mailto:akivanradix@gmail.com>> wrote:
Thanks for the doc. I read them and i have the following error:
*Error parsing destination, destination plugin java not found in /etc/syslog-ng/syslog-ng.conf at line 57, column 3:** ** java(
*I have configure as follow: @version:3.7 @include "scl.conf" @module mod-java
source s_sys { system(); internal(); network(ip(0.0.0.0) port(6514) flags(syslog-protocol) transport("tls") tls(key_file("/etc/syslog-ng/cert.d/serverkey.pem") cert_file("/etc/syslog-ng/cert.d/servercert.pem") ca_dir("/etc/syslog-ng/ca.d") ) );
};
parser pattern_db { db-parser( file("/etc/syslog-ng/patterndb.d/patterndb.xml") ); };
destination d_es { java(
class-path("/usr/lib64/syslog-ng/java-modules/*.jar:/usr/share/elasticsearch/lib/*.jar") class-name("org.syslog_ng.elasticsearch.ElasticSearchDestination") option("index", "syslog-ng_${YEAR}.${MONTH}.${DAY}") option("type", "test") option("cluster", "czpcluster") option("flush_limit", "100") option( "message_template", "$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n") ); };
I have installed just Syslog-NG and Elasticsearch ? Do i need to install somethign else ?
On 05/26/2016 04:53 PM, Szabó, István wrote:
Hi,
For using syslog-ng together with elasticsearch and Kibana this is a pretty good description, also giving you a good understanding of what it enables you to do:
https://czanik.blogs.balabit.com/2015/10/how-to-parse-data-with-syslog-ng-st...
https://czanik.blogs.balabit.com/2015/12/elasticsearch-and-syslog-ng-fast-an...
/Istvan
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On Fri, May 27, 2016 at 12:08:21PM +0200, Ivan Adji - Krstev wrote:
Hi Bazsi, I get syslog from: "https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng37/repo/epel-7/czani..." add the repo and then "yum install syslog-ng" after that i have download the Elasticsearch and install it and that is it. Im using CentOS 7.
you also want the package syslog-ng-java
Yes i install that too.. still nothing. Ivan On 05/27/2016 12:09 PM, Fabien Wernli wrote:
On Fri, May 27, 2016 at 12:08:21PM +0200, Ivan Adji - Krstev wrote:
Hi Bazsi, I get syslog from: "https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng37/repo/epel-7/czani..." add the repo and then "yum install syslog-ng" after that i have download the Elasticsearch and install it and that is it. Im using CentOS 7. you also want the package syslog-ng-java
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hopefully Peter Czanik can help you then, as he prepared those packages. On May 27, 2016 11:10 AM, "Ivan Adji - Krstev" <akivanradix@gmail.com> wrote:
Yes i install that too.. still nothing.
Ivan
On 05/27/2016 12:09 PM, Fabien Wernli wrote:
On Fri, May 27, 2016 at 12:08:21PM +0200, Ivan Adji - Krstev wrote:
Hi Bazsi, I get syslog from:"https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng37/repo/epel-7/czani..." <https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng37/repo/epel-7/czanik-syslog-ng37-epel-7.repo> add the repo and then "yum install syslog-ng" after that i have download the Elasticsearch and install it and that is it. Im using CentOS 7.
you also want the package syslog-ng-java
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
So should i contact him directly or should i wait here to reply on this list ? Ivan On 05/27/2016 12:13 PM, Scheidler, Balázs wrote:
Hopefully Peter Czanik can help you then, as he prepared those packages.
On May 27, 2016 11:10 AM, "Ivan Adji - Krstev" <akivanradix@gmail.com <mailto:akivanradix@gmail.com>> wrote:
Yes i install that too.. still nothing.
Ivan
On 05/27/2016 12:09 PM, Fabien Wernli wrote:
On Fri, May 27, 2016 at 12:08:21PM +0200, Ivan Adji - Krstev wrote:
Hi Bazsi, I get syslog from: "https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng37/repo/epel-7/czani..." <https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng37/repo/epel-7/czanik-syslog-ng37-epel-7.repo> add the repo and then "yum install syslog-ng" after that i have download the Elasticsearch and install it and that is it. Im using CentOS 7.
you also want the package syslog-ng-java
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On Fri, May 27, 2016 at 12:14:45PM +0200, Ivan Adji - Krstev wrote:
So should i contact him directly or should i wait here to reply on this list ?
he reads the list, so just wait :) Can you send the output of the following command please: syslog-ng -V
OK, Hmmm interesting error [root@syslogserver ~]# syslog-ng -V syslog-ng 3.7.3 Installer-Version: 3.7.3 Revision: Compile-Date: Mar 24 2016 10:55:14 Error opening plugin module; module='mod-java', error='libjvm.so: cannot open shared object file: No such file or directory' Available-Modules: afamqp,affile,afprog,afsocket,afstomp,afuser,basicfuncs,confgen,cryptofuncs,csvparser,dbparser,graphite,json-plugin,kvformat,linux-kmsg-format,pseudofile,sdjournal,syslogformat,system-source Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: off Ivan On 05/27/2016 12:18 PM, Fabien Wernli wrote:
On Fri, May 27, 2016 at 12:14:45PM +0200, Ivan Adji - Krstev wrote:
So should i contact him directly or should i wait here to reply on this list ? he reads the list, so just wait :)
Can you send the output of the following command please:
syslog-ng -V
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On Fri, May 27, 2016 at 12:19:17PM +0200, Ivan Adji - Krstev wrote:
Error opening plugin module; module='mod-java', error='libjvm.so: cannot open shared object file: No such file or directory'
You can get the explanation in another blog post from Peter: https://czanik.blogs.balabit.com/2016/03/troubleshooting-java-support-in-sys...
Hi, To enable Java support you need at least the "syslog-ng" and "syslog-ng-java" packages from that repository. Optionally you can also install the "syslog-ng-java-hack" package, which includes all the necessary JAR files, or you can also point your config to the JAR files of your Elasticsearch installation. Note, that syslog-ng 3.7 only supports Elasticsearch 1.X. You will also need to point syslog-ng to libjvm.so. There are multiple ways: https://czanik.blogs.balabit.com/2016/03/troubleshooting-java-support-in-sys... My personal preference is the ld.so.conf trick, but note that it has side effects if you have multiple Java versions on your system. Bye, Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik On Fri, May 27, 2016 at 12:14 PM, Ivan Adji - Krstev <akivanradix@gmail.com> wrote:
So should i contact him directly or should i wait here to reply on this list ?
Ivan
On 05/27/2016 12:13 PM, Scheidler, Balázs wrote:
Hopefully Peter Czanik can help you then, as he prepared those packages. On May 27, 2016 11:10 AM, "Ivan Adji - Krstev" <akivanradix@gmail.com> wrote:
Yes i install that too.. still nothing.
Ivan
On 05/27/2016 12:09 PM, Fabien Wernli wrote:
On Fri, May 27, 2016 at 12:08:21PM +0200, Ivan Adji - Krstev wrote:
Hi Bazsi, I get syslog from:"https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng37/repo/epel-7/czani..." <https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng37/repo/epel-7/czanik-syslog-ng37-epel-7.repo> add the repo and then "yum install syslog-ng" after that i have download the Elasticsearch and install it and that is it. Im using CentOS 7.
you also want the package syslog-ng-java
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Well that one i fix it ... export the new path of the libjvm.so file and it works. But now i have another error :) *Error stating pattern database file, no automatic reload will be performed; error='No such file or directory'* . . *Add path to classpath: /usr/share/elasticsearch/lib/spatial4j-0.5.jar;** **[2016-05-27T06:38:30.933808] Add path to classpath: /usr/share/elasticsearch/lib/t-digest-3.0.jar;** **[2016-05-27T06:38:31.287344] Add path to classpath: //usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar;** **[2016-05-27T06:38:31.333759] Error initializing message pipeline;** * And i have no idea what is this problem as im using ES for the first time. This is what i have: source s_sys { system(); internal(); network(ip(0.0.0.0) port(6514) flags(syslog-protocol) transport("tls") tls(key_file("/etc/syslog-ng/cert.d/serverkey.pem") cert_file("/etc/syslog-ng/cert.d/servercert.pem") ca_dir("/etc/syslog-ng/ca.d") ) ); }; parser pattern_db { db-parser( file("/etc/syslog-ng/patterndb.d/patterndb.xml") ); }; destination d_es { java( class-path("/usr/lib64/syslog-ng/java-modules/*.jar:/usr/share/elasticsearch/lib/*.jar") class-name("org.syslog_ng.elasticsearch.ElasticSearchDestination") option("index", "syslog-ng_${YEAR}.${MONTH}.${DAY}") option("type", "test") option("cluster", "czpcluster") option("flush_limit", "100") option( "message_template", "$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n") ); }; Kind regards Ivan On 05/27/2016 12:22 PM, Czanik, Péter wrote:
Hi,
To enable Java support you need at least the "syslog-ng" and "syslog-ng-java" packages from that repository. Optionally you can also install the "syslog-ng-java-hack" package, which includes all the necessary JAR files, or you can also point your config to the JAR files of your Elasticsearch installation. Note, that syslog-ng 3.7 only supports Elasticsearch 1.X.
You will also need to point syslog-ng to libjvm.so. There are multiple ways: https://czanik.blogs.balabit.com/2016/03/troubleshooting-java-support-in-sys... My personal preference is the ld.so.conf trick, but note that it has side effects if you have multiple Java versions on your system.
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com <mailto:peter.czanik@balabit.com>> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik
On Fri, May 27, 2016 at 12:14 PM, Ivan Adji - Krstev <akivanradix@gmail.com <mailto:akivanradix@gmail.com>> wrote:
So should i contact him directly or should i wait here to reply on this list ?
Ivan
On 05/27/2016 12:13 PM, Scheidler, Balázs wrote:
Hopefully Peter Czanik can help you then, as he prepared those packages.
On May 27, 2016 11:10 AM, "Ivan Adji - Krstev" <akivanradix@gmail.com <mailto:akivanradix@gmail.com>> wrote:
Yes i install that too.. still nothing.
Ivan
On 05/27/2016 12:09 PM, Fabien Wernli wrote:
On Fri, May 27, 2016 at 12:08:21PM +0200, Ivan Adji - Krstev wrote:
Hi Bazsi, I get syslog from: "https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng37/repo/epel-7/czani..." <https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng37/repo/epel-7/czanik-syslog-ng37-epel-7.repo> add the repo and then "yum install syslog-ng" after that i have download the Elasticsearch and install it and that is it. Im using CentOS 7.
you also want the package syslog-ng-java
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
option("cluster", "czpcluster") Unless your cluster is called "czpcluster", the above option won't work. This value is just an example (derived from my nick name :-) ). Bye, Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik On Fri, May 27, 2016 at 12:42 PM, Ivan Adji - Krstev <akivanradix@gmail.com> wrote:
Well that one i fix it ... export the new path of the libjvm.so file and it works. But now i have another error :)
*Error stating pattern database file, no automatic reload will be performed; error='No such file or directory'* . . *Add path to classpath: /usr/share/elasticsearch/lib/spatial4j-0.5.jar;* *[2016-05-27T06:38:30.933808] Add path to classpath: /usr/share/elasticsearch/lib/t-digest-3.0.jar;* *[2016-05-27T06:38:31.287344] Add path to classpath: //usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar;* *[2016-05-27T06:38:31.333759] Error initializing message pipeline;*
And i have no idea what is this problem as im using ES for the first time.
This is what i have:
source s_sys { system(); internal(); network(ip(0.0.0.0) port(6514) flags(syslog-protocol) transport("tls") tls(key_file("/etc/syslog-ng/cert.d/serverkey.pem") cert_file("/etc/syslog-ng/cert.d/servercert.pem") ca_dir("/etc/syslog-ng/ca.d") ) );
}; parser pattern_db { db-parser( file("/etc/syslog-ng/patterndb.d/patterndb.xml") ); }; destination d_es { java(
class-path("/usr/lib64/syslog-ng/java-modules/*.jar:/usr/share/elasticsearch/lib/*.jar") class-name("org.syslog_ng.elasticsearch.ElasticSearchDestination") option("index", "syslog-ng_${YEAR}.${MONTH}.${DAY}") option("type", "test") option("cluster", "czpcluster") option("flush_limit", "100") option( "message_template", "$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n") ); };
Kind regards Ivan
On 05/27/2016 12:22 PM, Czanik, Péter wrote:
Hi,
To enable Java support you need at least the "syslog-ng" and "syslog-ng-java" packages from that repository. Optionally you can also install the "syslog-ng-java-hack" package, which includes all the necessary JAR files, or you can also point your config to the JAR files of your Elasticsearch installation. Note, that syslog-ng 3.7 only supports Elasticsearch 1.X.
You will also need to point syslog-ng to libjvm.so. There are multiple ways: https://czanik.blogs.balabit.com/2016/03/troubleshooting-java-support-in-sys... My personal preference is the ld.so.conf trick, but note that it has side effects if you have multiple Java versions on your system.
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik
On Fri, May 27, 2016 at 12:14 PM, Ivan Adji - Krstev < akivanradix@gmail.com> wrote:
So should i contact him directly or should i wait here to reply on this list ?
Ivan
On 05/27/2016 12:13 PM, Scheidler, Balázs wrote:
Hopefully Peter Czanik can help you then, as he prepared those packages. On May 27, 2016 11:10 AM, "Ivan Adji - Krstev" <akivanradix@gmail.com> wrote:
Yes i install that too.. still nothing.
Ivan
On 05/27/2016 12:09 PM, Fabien Wernli wrote:
On Fri, May 27, 2016 at 12:08:21PM +0200, Ivan Adji - Krstev wrote:
Hi Bazsi, I get syslog from:"https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng37/repo/epel-7/czani..." <https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng37/repo/epel-7/czanik-syslog-ng37-epel-7.repo> add the repo and then "yum install syslog-ng" after that i have download the Elasticsearch and install it and that is it. Im using CentOS 7.
you also want the package syslog-ng-java
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
So the cluster should be the cluster name of the ES ? How can i see this name ? There is nothing in the elasticsearch.yml file, do i have to specify there ? Kind regards Ivan On 05/27/2016 12:45 PM, Czanik, Péter wrote:
option("cluster", "czpcluster")
Unless your cluster is called "czpcluster", the above option won't work. This value is just an example (derived from my nick name :-) ).
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com <mailto:peter.czanik@balabit.com>> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik
On Fri, May 27, 2016 at 12:42 PM, Ivan Adji - Krstev <akivanradix@gmail.com <mailto:akivanradix@gmail.com>> wrote:
Well that one i fix it ... export the new path of the libjvm.so file and it works. But now i have another error :)
*Error stating pattern database file, no automatic reload will be performed; error='No such file or directory'* . . *Add path to classpath: /usr/share/elasticsearch/lib/spatial4j-0.5.jar;** **[2016-05-27T06:38:30.933808] Add path to classpath: /usr/share/elasticsearch/lib/t-digest-3.0.jar;** **[2016-05-27T06:38:31.287344] Add path to classpath: //usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar;** **[2016-05-27T06:38:31.333759] Error initializing message pipeline;** * And i have no idea what is this problem as im using ES for the first time.
This is what i have:
source s_sys { system(); internal(); network(ip(0.0.0.0) port(6514) flags(syslog-protocol) transport("tls") tls(key_file("/etc/syslog-ng/cert.d/serverkey.pem") cert_file("/etc/syslog-ng/cert.d/servercert.pem") ca_dir("/etc/syslog-ng/ca.d") ) );
}; parser pattern_db { db-parser( file("/etc/syslog-ng/patterndb.d/patterndb.xml") ); }; destination d_es { java(
class-path("/usr/lib64/syslog-ng/java-modules/*.jar:/usr/share/elasticsearch/lib/*.jar") class-name("org.syslog_ng.elasticsearch.ElasticSearchDestination") option("index", "syslog-ng_${YEAR}.${MONTH}.${DAY}") option("type", "test") option("cluster", "czpcluster") option("flush_limit", "100") option( "message_template", "$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n") ); };
Kind regards Ivan
On 05/27/2016 12:22 PM, Czanik, Péter wrote:
Hi,
To enable Java support you need at least the "syslog-ng" and "syslog-ng-java" packages from that repository. Optionally you can also install the "syslog-ng-java-hack" package, which includes all the necessary JAR files, or you can also point your config to the JAR files of your Elasticsearch installation. Note, that syslog-ng 3.7 only supports Elasticsearch 1.X.
You will also need to point syslog-ng to libjvm.so. There are multiple ways: https://czanik.blogs.balabit.com/2016/03/troubleshooting-java-support-in-sys... My personal preference is the ld.so.conf trick, but note that it has side effects if you have multiple Java versions on your system.
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com <mailto:peter.czanik@balabit.com>> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik
On Fri, May 27, 2016 at 12:14 PM, Ivan Adji - Krstev <akivanradix@gmail.com <mailto:akivanradix@gmail.com>> wrote:
So should i contact him directly or should i wait here to reply on this list ?
Ivan
On 05/27/2016 12:13 PM, Scheidler, Balázs wrote:
Hopefully Peter Czanik can help you then, as he prepared those packages.
On May 27, 2016 11:10 AM, "Ivan Adji - Krstev" <akivanradix@gmail.com <mailto:akivanradix@gmail.com>> wrote:
Yes i install that too.. still nothing.
Ivan
On 05/27/2016 12:09 PM, Fabien Wernli wrote:
On Fri, May 27, 2016 at 12:08:21PM +0200, Ivan Adji - Krstev wrote:
Hi Bazsi, I get syslog from: "https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng37/repo/epel-7/czani..." <https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng37/repo/epel-7/czanik-syslog-ng37-epel-7.repo> add the repo and then "yum install syslog-ng" after that i have download the Elasticsearch and install it and that is it. Im using CentOS 7.
you also want the package syslog-ng-java
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On Fri, May 27, 2016 at 01:43:29PM +0200, Ivan Adji - Krstev wrote:
So the cluster should be the cluster name of the ES ? How can i see this name ? There is nothing in the elasticsearch.yml file, do i have to specify there ?
by default, it's elasticsearch
Well i try default one i try add cluster.neame: in the .yml file, but still nothing still the same error. [2016-05-27T08:14:29.003329] Error stating pattern database file, no automatic reload will be performed; error='No such file or directory' [2016-05-27T08:14:29.758816] Add path to classpath: //usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar; [2016-05-27T08:14:29.759989] Add path to classpath: /usr/lib64/syslog-ng/java-modules/dummy.jar; [2016-05-27T08:14:29.760284] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic.jar; [2016-05-27T08:14:29.760573] Add path to classpath: /usr/lib64/syslog-ng/java-modules/hdfs.jar; [2016-05-27T08:14:29.760923] Add path to classpath: /usr/lib64/syslog-ng/java-modules/http.jar; [2016-05-27T08:14:29.761256] Add path to classpath: /usr/lib64/syslog-ng/java-modules/java-modules.jar; [2016-05-27T08:14:29.761555] Add path to classpath: /usr/lib64/syslog-ng/java-modules/kafka.jar; [2016-05-27T08:14:29.761888] Add path to classpath: /usr/lib64/syslog-ng/java-modules/log4j-1.2.16.jar; [2016-05-27T08:14:29.762193] Add path to classpath: /usr/lib64/syslog-ng/java-modules/syslog-ng-common.jar; [2016-05-27T08:14:29.762511] Add path to classpath: /usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar; [2016-05-27T08:14:29.762854] Add path to classpath: /usr/share/elasticsearch/lib/HdrHistogram-2.1.6.jar; [2016-05-27T08:14:29.763170] Add path to classpath: /usr/share/elasticsearch/lib/apache-log4j-extras-1.2.17.jar; [2016-05-27T08:14:29.763510] Add path to classpath: /usr/share/elasticsearch/lib/commons-cli-1.3.1.jar; [2016-05-27T08:14:29.763859] Add path to classpath: /usr/share/elasticsearch/lib/compiler-0.8.13.jar; [2016-05-27T08:14:29.764128] Add path to classpath: /usr/share/elasticsearch/lib/compress-lzf-1.0.2.jar; [2016-05-27T08:14:29.764449] Add path to classpath: /usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar; [2016-05-27T08:14:29.764773] Add path to classpath: /usr/share/elasticsearch/lib/guava-18.0.jar; [2016-05-27T08:14:29.765098] Add path to classpath: /usr/share/elasticsearch/lib/hppc-0.7.1.jar; [2016-05-27T08:14:29.765392] Add path to classpath: /usr/share/elasticsearch/lib/jackson-core-2.6.6.jar; [2016-05-27T08:14:29.765737] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.6.6.jar; [2016-05-27T08:14:29.766062] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-smile-2.6.6.jar; [2016-05-27T08:14:29.766395] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.6.6.jar; [2016-05-27T08:14:29.766945] Add path to classpath: /usr/share/elasticsearch/lib/jna-4.1.0.jar; [2016-05-27T08:14:29.767245] Add path to classpath: /usr/share/elasticsearch/lib/joda-convert-1.2.jar; [2016-05-27T08:14:29.767557] Add path to classpath: /usr/share/elasticsearch/lib/joda-time-2.8.2.jar; [2016-05-27T08:14:29.767950] Add path to classpath: /usr/share/elasticsearch/lib/jsr166e-1.1.0.jar; [2016-05-27T08:14:29.768251] Add path to classpath: /usr/share/elasticsearch/lib/jts-1.13.jar; [2016-05-27T08:14:29.768544] Add path to classpath: /usr/share/elasticsearch/lib/log4j-1.2.17.jar; [2016-05-27T08:14:29.768876] Add path to classpath: /usr/share/elasticsearch/lib/lucene-analyzers-common-5.5.0.jar; [2016-05-27T08:14:29.769225] Add path to classpath: /usr/share/elasticsearch/lib/lucene-backward-codecs-5.5.0.jar; [2016-05-27T08:14:29.769570] Add path to classpath: /usr/share/elasticsearch/lib/lucene-core-5.5.0.jar; [2016-05-27T08:14:29.769900] Add path to classpath: /usr/share/elasticsearch/lib/lucene-grouping-5.5.0.jar; [2016-05-27T08:14:29.770194] Add path to classpath: /usr/share/elasticsearch/lib/lucene-highlighter-5.5.0.jar; [2016-05-27T08:14:29.770514] Add path to classpath: /usr/share/elasticsearch/lib/lucene-join-5.5.0.jar; [2016-05-27T08:14:29.770888] Add path to classpath: /usr/share/elasticsearch/lib/lucene-memory-5.5.0.jar; [2016-05-27T08:14:29.771187] Add path to classpath: /usr/share/elasticsearch/lib/lucene-misc-5.5.0.jar; [2016-05-27T08:14:29.771497] Add path to classpath: /usr/share/elasticsearch/lib/lucene-queries-5.5.0.jar; [2016-05-27T08:14:29.771844] Add path to classpath: /usr/share/elasticsearch/lib/lucene-queryparser-5.5.0.jar; [2016-05-27T08:14:29.772179] Add path to classpath: /usr/share/elasticsearch/lib/lucene-sandbox-5.5.0.jar; [2016-05-27T08:14:29.772466] Add path to classpath: /usr/share/elasticsearch/lib/lucene-spatial-5.5.0.jar; [2016-05-27T08:14:29.772811] Add path to classpath: /usr/share/elasticsearch/lib/lucene-spatial3d-5.5.0.jar; [2016-05-27T08:14:29.773098] Add path to classpath: /usr/share/elasticsearch/lib/lucene-suggest-5.5.0.jar; [2016-05-27T08:14:29.773447] Add path to classpath: /usr/share/elasticsearch/lib/netty-3.10.5.Final.jar; [2016-05-27T08:14:29.773783] Add path to classpath: /usr/share/elasticsearch/lib/securesm-1.0.jar; [2016-05-27T08:14:29.774055] Add path to classpath: /usr/share/elasticsearch/lib/snakeyaml-1.15.jar; [2016-05-27T08:14:29.774326] Add path to classpath: /usr/share/elasticsearch/lib/spatial4j-0.5.jar; [2016-05-27T08:14:29.774572] Add path to classpath: /usr/share/elasticsearch/lib/t-digest-3.0.jar; [2016-05-27T08:14:32.237017] Add path to classpath: //usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar; [2016-05-27T08:14:32.372708] Error initializing message pipeline; Ivan On 05/27/2016 02:13 PM, Fabien Wernli wrote:
On Fri, May 27, 2016 at 01:43:29PM +0200, Ivan Adji - Krstev wrote:
So the cluster should be the cluster name of the ES ? How can i see this name ? There is nothing in the elasticsearch.yml file, do i have to specify there ? by default, it's elasticsearch
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, As I mentioned in a previous post: syslog-ng 3.7 only supports Elasticsearch 1.X. Looking at your logs reveals, that you use Elasticsearch 2.3.3 Bye, Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik On Fri, May 27, 2016 at 2:15 PM, Ivan Adji - Krstev <akivanradix@gmail.com> wrote:
Well i try default one i try add cluster.neame: in the .yml file, but still nothing still the same error.
[2016-05-27T08:14:29.003329] Error stating pattern database file, no automatic reload will be performed; error='No such file or directory' [2016-05-27T08:14:29.758816] Add path to classpath: //usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar; [2016-05-27T08:14:29.759989] Add path to classpath: /usr/lib64/syslog-ng/java-modules/dummy.jar; [2016-05-27T08:14:29.760284] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic.jar; [2016-05-27T08:14:29.760573] Add path to classpath: /usr/lib64/syslog-ng/java-modules/hdfs.jar; [2016-05-27T08:14:29.760923] Add path to classpath: /usr/lib64/syslog-ng/java-modules/http.jar; [2016-05-27T08:14:29.761256] Add path to classpath: /usr/lib64/syslog-ng/java-modules/java-modules.jar; [2016-05-27T08:14:29.761555] Add path to classpath: /usr/lib64/syslog-ng/java-modules/kafka.jar; [2016-05-27T08:14:29.761888] Add path to classpath: /usr/lib64/syslog-ng/java-modules/log4j-1.2.16.jar; [2016-05-27T08:14:29.762193] Add path to classpath: /usr/lib64/syslog-ng/java-modules/syslog-ng-common.jar; [2016-05-27T08:14:29.762511] Add path to classpath: /usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar; [2016-05-27T08:14:29.762854] Add path to classpath: /usr/share/elasticsearch/lib/HdrHistogram-2.1.6.jar; [2016-05-27T08:14:29.763170] Add path to classpath: /usr/share/elasticsearch/lib/apache-log4j-extras-1.2.17.jar; [2016-05-27T08:14:29.763510] Add path to classpath: /usr/share/elasticsearch/lib/commons-cli-1.3.1.jar; [2016-05-27T08:14:29.763859] Add path to classpath: /usr/share/elasticsearch/lib/compiler-0.8.13.jar; [2016-05-27T08:14:29.764128] Add path to classpath: /usr/share/elasticsearch/lib/compress-lzf-1.0.2.jar; [2016-05-27T08:14:29.764449] Add path to classpath: /usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar; [2016-05-27T08:14:29.764773] Add path to classpath: /usr/share/elasticsearch/lib/guava-18.0.jar; [2016-05-27T08:14:29.765098] Add path to classpath: /usr/share/elasticsearch/lib/hppc-0.7.1.jar; [2016-05-27T08:14:29.765392] Add path to classpath: /usr/share/elasticsearch/lib/jackson-core-2.6.6.jar; [2016-05-27T08:14:29.765737] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.6.6.jar; [2016-05-27T08:14:29.766062] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-smile-2.6.6.jar; [2016-05-27T08:14:29.766395] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.6.6.jar; [2016-05-27T08:14:29.766945] Add path to classpath: /usr/share/elasticsearch/lib/jna-4.1.0.jar; [2016-05-27T08:14:29.767245] Add path to classpath: /usr/share/elasticsearch/lib/joda-convert-1.2.jar; [2016-05-27T08:14:29.767557] Add path to classpath: /usr/share/elasticsearch/lib/joda-time-2.8.2.jar; [2016-05-27T08:14:29.767950] Add path to classpath: /usr/share/elasticsearch/lib/jsr166e-1.1.0.jar; [2016-05-27T08:14:29.768251] Add path to classpath: /usr/share/elasticsearch/lib/jts-1.13.jar; [2016-05-27T08:14:29.768544] Add path to classpath: /usr/share/elasticsearch/lib/log4j-1.2.17.jar; [2016-05-27T08:14:29.768876] Add path to classpath: /usr/share/elasticsearch/lib/lucene-analyzers-common-5.5.0.jar; [2016-05-27T08:14:29.769225] Add path to classpath: /usr/share/elasticsearch/lib/lucene-backward-codecs-5.5.0.jar; [2016-05-27T08:14:29.769570] Add path to classpath: /usr/share/elasticsearch/lib/lucene-core-5.5.0.jar; [2016-05-27T08:14:29.769900] Add path to classpath: /usr/share/elasticsearch/lib/lucene-grouping-5.5.0.jar; [2016-05-27T08:14:29.770194] Add path to classpath: /usr/share/elasticsearch/lib/lucene-highlighter-5.5.0.jar; [2016-05-27T08:14:29.770514] Add path to classpath: /usr/share/elasticsearch/lib/lucene-join-5.5.0.jar; [2016-05-27T08:14:29.770888] Add path to classpath: /usr/share/elasticsearch/lib/lucene-memory-5.5.0.jar; [2016-05-27T08:14:29.771187] Add path to classpath: /usr/share/elasticsearch/lib/lucene-misc-5.5.0.jar; [2016-05-27T08:14:29.771497] Add path to classpath: /usr/share/elasticsearch/lib/lucene-queries-5.5.0.jar; [2016-05-27T08:14:29.771844] Add path to classpath: /usr/share/elasticsearch/lib/lucene-queryparser-5.5.0.jar; [2016-05-27T08:14:29.772179] Add path to classpath: /usr/share/elasticsearch/lib/lucene-sandbox-5.5.0.jar; [2016-05-27T08:14:29.772466] Add path to classpath: /usr/share/elasticsearch/lib/lucene-spatial-5.5.0.jar; [2016-05-27T08:14:29.772811] Add path to classpath: /usr/share/elasticsearch/lib/lucene-spatial3d-5.5.0.jar; [2016-05-27T08:14:29.773098] Add path to classpath: /usr/share/elasticsearch/lib/lucene-suggest-5.5.0.jar; [2016-05-27T08:14:29.773447] Add path to classpath: /usr/share/elasticsearch/lib/netty-3.10.5.Final.jar; [2016-05-27T08:14:29.773783] Add path to classpath: /usr/share/elasticsearch/lib/securesm-1.0.jar; [2016-05-27T08:14:29.774055] Add path to classpath: /usr/share/elasticsearch/lib/snakeyaml-1.15.jar; [2016-05-27T08:14:29.774326] Add path to classpath: /usr/share/elasticsearch/lib/spatial4j-0.5.jar; [2016-05-27T08:14:29.774572] Add path to classpath: /usr/share/elasticsearch/lib/t-digest-3.0.jar; [2016-05-27T08:14:32.237017] Add path to classpath: //usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar; [2016-05-27T08:14:32.372708] Error initializing message pipeline;
Ivan
On 05/27/2016 02:13 PM, Fabien Wernli wrote:
On Fri, May 27, 2016 at 01:43:29PM +0200, Ivan Adji - Krstev wrote:
So the cluster should be the cluster name of the ES ? How can i see this name ? There is nothing in the elasticsearch.yml file, do i have to specify there ?
by default, it's elasticsearch
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hmm but some additional log messages would be great to have any initialization failed. Or am I the only one missing some hints there? Viktor? On May 27, 2016 2:17 PM, "Czanik, Péter" <peter.czanik@balabit.com> wrote:
Hi,
As I mentioned in a previous post: syslog-ng 3.7 only supports Elasticsearch 1.X. Looking at your logs reveals, that you use Elasticsearch 2.3.3
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik
On Fri, May 27, 2016 at 2:15 PM, Ivan Adji - Krstev <akivanradix@gmail.com
wrote:
Well i try default one i try add cluster.neame: in the .yml file, but still nothing still the same error.
[2016-05-27T08:14:29.003329] Error stating pattern database file, no automatic reload will be performed; error='No such file or directory' [2016-05-27T08:14:29.758816] Add path to classpath: //usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar; [2016-05-27T08:14:29.759989] Add path to classpath: /usr/lib64/syslog-ng/java-modules/dummy.jar; [2016-05-27T08:14:29.760284] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic.jar; [2016-05-27T08:14:29.760573] Add path to classpath: /usr/lib64/syslog-ng/java-modules/hdfs.jar; [2016-05-27T08:14:29.760923] Add path to classpath: /usr/lib64/syslog-ng/java-modules/http.jar; [2016-05-27T08:14:29.761256] Add path to classpath: /usr/lib64/syslog-ng/java-modules/java-modules.jar; [2016-05-27T08:14:29.761555] Add path to classpath: /usr/lib64/syslog-ng/java-modules/kafka.jar; [2016-05-27T08:14:29.761888] Add path to classpath: /usr/lib64/syslog-ng/java-modules/log4j-1.2.16.jar; [2016-05-27T08:14:29.762193] Add path to classpath: /usr/lib64/syslog-ng/java-modules/syslog-ng-common.jar; [2016-05-27T08:14:29.762511] Add path to classpath: /usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar; [2016-05-27T08:14:29.762854] Add path to classpath: /usr/share/elasticsearch/lib/HdrHistogram-2.1.6.jar; [2016-05-27T08:14:29.763170] Add path to classpath: /usr/share/elasticsearch/lib/apache-log4j-extras-1.2.17.jar; [2016-05-27T08:14:29.763510] Add path to classpath: /usr/share/elasticsearch/lib/commons-cli-1.3.1.jar; [2016-05-27T08:14:29.763859] Add path to classpath: /usr/share/elasticsearch/lib/compiler-0.8.13.jar; [2016-05-27T08:14:29.764128] Add path to classpath: /usr/share/elasticsearch/lib/compress-lzf-1.0.2.jar; [2016-05-27T08:14:29.764449] Add path to classpath: /usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar; [2016-05-27T08:14:29.764773] Add path to classpath: /usr/share/elasticsearch/lib/guava-18.0.jar; [2016-05-27T08:14:29.765098] Add path to classpath: /usr/share/elasticsearch/lib/hppc-0.7.1.jar; [2016-05-27T08:14:29.765392] Add path to classpath: /usr/share/elasticsearch/lib/jackson-core-2.6.6.jar; [2016-05-27T08:14:29.765737] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.6.6.jar; [2016-05-27T08:14:29.766062] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-smile-2.6.6.jar; [2016-05-27T08:14:29.766395] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.6.6.jar; [2016-05-27T08:14:29.766945] Add path to classpath: /usr/share/elasticsearch/lib/jna-4.1.0.jar; [2016-05-27T08:14:29.767245] Add path to classpath: /usr/share/elasticsearch/lib/joda-convert-1.2.jar; [2016-05-27T08:14:29.767557] Add path to classpath: /usr/share/elasticsearch/lib/joda-time-2.8.2.jar; [2016-05-27T08:14:29.767950] Add path to classpath: /usr/share/elasticsearch/lib/jsr166e-1.1.0.jar; [2016-05-27T08:14:29.768251] Add path to classpath: /usr/share/elasticsearch/lib/jts-1.13.jar; [2016-05-27T08:14:29.768544] Add path to classpath: /usr/share/elasticsearch/lib/log4j-1.2.17.jar; [2016-05-27T08:14:29.768876] Add path to classpath: /usr/share/elasticsearch/lib/lucene-analyzers-common-5.5.0.jar; [2016-05-27T08:14:29.769225] Add path to classpath: /usr/share/elasticsearch/lib/lucene-backward-codecs-5.5.0.jar; [2016-05-27T08:14:29.769570] Add path to classpath: /usr/share/elasticsearch/lib/lucene-core-5.5.0.jar; [2016-05-27T08:14:29.769900] Add path to classpath: /usr/share/elasticsearch/lib/lucene-grouping-5.5.0.jar; [2016-05-27T08:14:29.770194] Add path to classpath: /usr/share/elasticsearch/lib/lucene-highlighter-5.5.0.jar; [2016-05-27T08:14:29.770514] Add path to classpath: /usr/share/elasticsearch/lib/lucene-join-5.5.0.jar; [2016-05-27T08:14:29.770888] Add path to classpath: /usr/share/elasticsearch/lib/lucene-memory-5.5.0.jar; [2016-05-27T08:14:29.771187] Add path to classpath: /usr/share/elasticsearch/lib/lucene-misc-5.5.0.jar; [2016-05-27T08:14:29.771497] Add path to classpath: /usr/share/elasticsearch/lib/lucene-queries-5.5.0.jar; [2016-05-27T08:14:29.771844] Add path to classpath: /usr/share/elasticsearch/lib/lucene-queryparser-5.5.0.jar; [2016-05-27T08:14:29.772179] Add path to classpath: /usr/share/elasticsearch/lib/lucene-sandbox-5.5.0.jar; [2016-05-27T08:14:29.772466] Add path to classpath: /usr/share/elasticsearch/lib/lucene-spatial-5.5.0.jar; [2016-05-27T08:14:29.772811] Add path to classpath: /usr/share/elasticsearch/lib/lucene-spatial3d-5.5.0.jar; [2016-05-27T08:14:29.773098] Add path to classpath: /usr/share/elasticsearch/lib/lucene-suggest-5.5.0.jar; [2016-05-27T08:14:29.773447] Add path to classpath: /usr/share/elasticsearch/lib/netty-3.10.5.Final.jar; [2016-05-27T08:14:29.773783] Add path to classpath: /usr/share/elasticsearch/lib/securesm-1.0.jar; [2016-05-27T08:14:29.774055] Add path to classpath: /usr/share/elasticsearch/lib/snakeyaml-1.15.jar; [2016-05-27T08:14:29.774326] Add path to classpath: /usr/share/elasticsearch/lib/spatial4j-0.5.jar; [2016-05-27T08:14:29.774572] Add path to classpath: /usr/share/elasticsearch/lib/t-digest-3.0.jar; [2016-05-27T08:14:32.237017] Add path to classpath: //usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar; [2016-05-27T08:14:32.372708] Error initializing message pipeline;
Ivan
On 05/27/2016 02:13 PM, Fabien Wernli wrote:
On Fri, May 27, 2016 at 01:43:29PM +0200, Ivan Adji - Krstev wrote:
So the cluster should be the cluster name of the ES ? How can i see this name ? There is nothing in the elasticsearch.yml file, do i have to specify there ?
by default, it's elasticsearch
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Again i have the following error Error stating pattern database file, no automatic reload will be performed; error='No such file or directory' It works but this error remind Ivan On 05/27/2016 02:47 PM, Scheidler, Balázs wrote:
Hmm but some additional log messages would be great to have any initialization failed. Or am I the only one missing some hints there?
Viktor?
On May 27, 2016 2:17 PM, "Czanik, Péter" <peter.czanik@balabit.com <mailto:peter.czanik@balabit.com>> wrote:
Hi,
As I mentioned in a previous post: syslog-ng 3.7 only supports Elasticsearch 1.X. Looking at your logs reveals, that you use Elasticsearch 2.3.3
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com <mailto:peter.czanik@balabit.com>> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik
On Fri, May 27, 2016 at 2:15 PM, Ivan Adji - Krstev <akivanradix@gmail.com <mailto:akivanradix@gmail.com>> wrote:
Well i try default one i try add cluster.neame: in the .yml file, but still nothing still the same error.
[2016-05-27T08:14:29.003329 <tel:29.003329>] Error stating pattern database file, no automatic reload will be performed; error='No such file or directory' [2016-05-27T08:14:29.758816 <tel:29.758816>] Add path to classpath: //usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar; [2016-05-27T08:14:29.759989 <tel:29.759989>] Add path to classpath: /usr/lib64/syslog-ng/java-modules/dummy.jar; [2016-05-27T08:14:29.760284 <tel:29.760284>] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic.jar; [2016-05-27T08:14:29.760573 <tel:29.760573>] Add path to classpath: /usr/lib64/syslog-ng/java-modules/hdfs.jar; [2016-05-27T08:14:29.760923 <tel:29.760923>] Add path to classpath: /usr/lib64/syslog-ng/java-modules/http.jar; [2016-05-27T08:14:29.761256 <tel:29.761256>] Add path to classpath: /usr/lib64/syslog-ng/java-modules/java-modules.jar; [2016-05-27T08:14:29.761555 <tel:29.761555>] Add path to classpath: /usr/lib64/syslog-ng/java-modules/kafka.jar; [2016-05-27T08:14:29.761888 <tel:29.761888>] Add path to classpath: /usr/lib64/syslog-ng/java-modules/log4j-1.2.16.jar; [2016-05-27T08:14:29.762193 <tel:29.762193>] Add path to classpath: /usr/lib64/syslog-ng/java-modules/syslog-ng-common.jar; [2016-05-27T08:14:29.762511 <tel:29.762511>] Add path to classpath: /usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar; [2016-05-27T08:14:29.762854 <tel:29.762854>] Add path to classpath: /usr/share/elasticsearch/lib/HdrHistogram-2.1.6.jar; [2016-05-27T08:14:29.763170 <tel:29.763170>] Add path to classpath: /usr/share/elasticsearch/lib/apache-log4j-extras-1.2.17.jar; [2016-05-27T08:14:29.763510 <tel:29.763510>] Add path to classpath: /usr/share/elasticsearch/lib/commons-cli-1.3.1.jar; [2016-05-27T08:14:29.763859 <tel:29.763859>] Add path to classpath: /usr/share/elasticsearch/lib/compiler-0.8.13.jar; [2016-05-27T08:14:29.764128 <tel:29.764128>] Add path to classpath: /usr/share/elasticsearch/lib/compress-lzf-1.0.2.jar; [2016-05-27T08:14:29.764449 <tel:29.764449>] Add path to classpath: /usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar; [2016-05-27T08:14:29.764773 <tel:29.764773>] Add path to classpath: /usr/share/elasticsearch/lib/guava-18.0.jar; [2016-05-27T08:14:29.765098 <tel:29.765098>] Add path to classpath: /usr/share/elasticsearch/lib/hppc-0.7.1.jar; [2016-05-27T08:14:29.765392 <tel:29.765392>] Add path to classpath: /usr/share/elasticsearch/lib/jackson-core-2.6.6.jar; [2016-05-27T08:14:29.765737 <tel:29.765737>] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.6.6.jar; [2016-05-27T08:14:29.766062 <tel:29.766062>] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-smile-2.6.6.jar; [2016-05-27T08:14:29.766395 <tel:29.766395>] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.6.6.jar; [2016-05-27T08:14:29.766945 <tel:29.766945>] Add path to classpath: /usr/share/elasticsearch/lib/jna-4.1.0.jar; [2016-05-27T08:14:29.767245 <tel:29.767245>] Add path to classpath: /usr/share/elasticsearch/lib/joda-convert-1.2.jar; [2016-05-27T08:14:29.767557 <tel:29.767557>] Add path to classpath: /usr/share/elasticsearch/lib/joda-time-2.8.2.jar; [2016-05-27T08:14:29.767950 <tel:29.767950>] Add path to classpath: /usr/share/elasticsearch/lib/jsr166e-1.1.0.jar; [2016-05-27T08:14:29.768251 <tel:29.768251>] Add path to classpath: /usr/share/elasticsearch/lib/jts-1.13.jar; [2016-05-27T08:14:29.768544 <tel:29.768544>] Add path to classpath: /usr/share/elasticsearch/lib/log4j-1.2.17.jar; [2016-05-27T08:14:29.768876 <tel:29.768876>] Add path to classpath: /usr/share/elasticsearch/lib/lucene-analyzers-common-5.5.0.jar; [2016-05-27T08:14:29.769225 <tel:29.769225>] Add path to classpath: /usr/share/elasticsearch/lib/lucene-backward-codecs-5.5.0.jar; [2016-05-27T08:14:29.769570 <tel:29.769570>] Add path to classpath: /usr/share/elasticsearch/lib/lucene-core-5.5.0.jar; [2016-05-27T08:14:29.769900 <tel:29.769900>] Add path to classpath: /usr/share/elasticsearch/lib/lucene-grouping-5.5.0.jar; [2016-05-27T08:14:29.770194 <tel:29.770194>] Add path to classpath: /usr/share/elasticsearch/lib/lucene-highlighter-5.5.0.jar; [2016-05-27T08:14:29.770514 <tel:29.770514>] Add path to classpath: /usr/share/elasticsearch/lib/lucene-join-5.5.0.jar; [2016-05-27T08:14:29.770888 <tel:29.770888>] Add path to classpath: /usr/share/elasticsearch/lib/lucene-memory-5.5.0.jar; [2016-05-27T08:14:29.771187 <tel:29.771187>] Add path to classpath: /usr/share/elasticsearch/lib/lucene-misc-5.5.0.jar; [2016-05-27T08:14:29.771497 <tel:29.771497>] Add path to classpath: /usr/share/elasticsearch/lib/lucene-queries-5.5.0.jar; [2016-05-27T08:14:29.771844 <tel:29.771844>] Add path to classpath: /usr/share/elasticsearch/lib/lucene-queryparser-5.5.0.jar; [2016-05-27T08:14:29.772179 <tel:29.772179>] Add path to classpath: /usr/share/elasticsearch/lib/lucene-sandbox-5.5.0.jar; [2016-05-27T08:14:29.772466 <tel:29.772466>] Add path to classpath: /usr/share/elasticsearch/lib/lucene-spatial-5.5.0.jar; [2016-05-27T08:14:29.772811 <tel:29.772811>] Add path to classpath: /usr/share/elasticsearch/lib/lucene-spatial3d-5.5.0.jar; [2016-05-27T08:14:29.773098 <tel:29.773098>] Add path to classpath: /usr/share/elasticsearch/lib/lucene-suggest-5.5.0.jar; [2016-05-27T08:14:29.773447 <tel:29.773447>] Add path to classpath: /usr/share/elasticsearch/lib/netty-3.10.5.Final.jar; [2016-05-27T08:14:29.773783 <tel:29.773783>] Add path to classpath: /usr/share/elasticsearch/lib/securesm-1.0.jar; [2016-05-27T08:14:29.774055 <tel:29.774055>] Add path to classpath: /usr/share/elasticsearch/lib/snakeyaml-1.15.jar; [2016-05-27T08:14:29.774326 <tel:29.774326>] Add path to classpath: /usr/share/elasticsearch/lib/spatial4j-0.5.jar; [2016-05-27T08:14:29.774572 <tel:29.774572>] Add path to classpath: /usr/share/elasticsearch/lib/t-digest-3.0.jar; [2016-05-27T08:14:32.237017 <tel:32.237017>] Add path to classpath: //usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar; [2016-05-27T08:14:32.372708 <tel:32.372708>] Error initializing message pipeline;
Ivan
On 05/27/2016 02:13 PM, Fabien Wernli wrote:
On Fri, May 27, 2016 at 01:43:29PM +0200, Ivan Adji - Krstev wrote:
So the cluster should be the cluster name of the ES ? How can i see this name ? There is nothing in the elasticsearch.yml file, do i have to specify there ?
by default, it's elasticsearch
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, Do you use patterndb at all? If not, you should remove patterndb related settings from your configuration. If you use it, correct the path name to the XML file. Bye, Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik On Fri, May 27, 2016 at 2:59 PM, Ivan Adji - Krstev <akivanradix@gmail.com> wrote:
Again i have the following error Error stating pattern database file, no automatic reload will be performed; error='No such file or directory' It works but this error remind
Ivan
On 05/27/2016 02:47 PM, Scheidler, Balázs wrote:
Hmm but some additional log messages would be great to have any initialization failed. Or am I the only one missing some hints there?
Viktor? On May 27, 2016 2:17 PM, "Czanik, Péter" <peter.czanik@balabit.com> wrote:
Hi,
As I mentioned in a previous post: syslog-ng 3.7 only supports Elasticsearch 1.X. Looking at your logs reveals, that you use Elasticsearch 2.3.3
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik
On Fri, May 27, 2016 at 2:15 PM, Ivan Adji - Krstev < <akivanradix@gmail.com>akivanradix@gmail.com> wrote:
Well i try default one i try add cluster.neame: in the .yml file, but still nothing still the same error.
[2016-05-27T08:14:29.003329] Error stating pattern database file, no automatic reload will be performed; error='No such file or directory' [2016-05-27T08:14:29.758816] Add path to classpath: //usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar; [2016-05-27T08:14:29.759989] Add path to classpath: /usr/lib64/syslog-ng/java-modules/dummy.jar; [2016-05-27T08:14:29.760284] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic.jar; [2016-05-27T08:14:29.760573] Add path to classpath: /usr/lib64/syslog-ng/java-modules/hdfs.jar; [2016-05-27T08:14:29.760923] Add path to classpath: /usr/lib64/syslog-ng/java-modules/http.jar; [2016-05-27T08:14:29.761256] Add path to classpath: /usr/lib64/syslog-ng/java-modules/java-modules.jar; [2016-05-27T08:14:29.761555] Add path to classpath: /usr/lib64/syslog-ng/java-modules/kafka.jar; [2016-05-27T08:14:29.761888] Add path to classpath: /usr/lib64/syslog-ng/java-modules/log4j-1.2.16.jar; [2016-05-27T08:14:29.762193] Add path to classpath: /usr/lib64/syslog-ng/java-modules/syslog-ng-common.jar; [2016-05-27T08:14:29.762511] Add path to classpath: /usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar; [2016-05-27T08:14:29.762854] Add path to classpath: /usr/share/elasticsearch/lib/HdrHistogram-2.1.6.jar; [2016-05-27T08:14:29.763170] Add path to classpath: /usr/share/elasticsearch/lib/apache-log4j-extras-1.2.17.jar; [2016-05-27T08:14:29.763510] Add path to classpath: /usr/share/elasticsearch/lib/commons-cli-1.3.1.jar; [2016-05-27T08:14:29.763859] Add path to classpath: /usr/share/elasticsearch/lib/compiler-0.8.13.jar; [2016-05-27T08:14:29.764128] Add path to classpath: /usr/share/elasticsearch/lib/compress-lzf-1.0.2.jar; [2016-05-27T08:14:29.764449] Add path to classpath: /usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar; [2016-05-27T08:14:29.764773] Add path to classpath: /usr/share/elasticsearch/lib/guava-18.0.jar; [2016-05-27T08:14:29.765098] Add path to classpath: /usr/share/elasticsearch/lib/hppc-0.7.1.jar; [2016-05-27T08:14:29.765392] Add path to classpath: /usr/share/elasticsearch/lib/jackson-core-2.6.6.jar; [2016-05-27T08:14:29.765737] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.6.6.jar; [2016-05-27T08:14:29.766062] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-smile-2.6.6.jar; [2016-05-27T08:14:29.766395] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.6.6.jar; [2016-05-27T08:14:29.766945] Add path to classpath: /usr/share/elasticsearch/lib/jna-4.1.0.jar; [2016-05-27T08:14:29.767245] Add path to classpath: /usr/share/elasticsearch/lib/joda-convert-1.2.jar; [2016-05-27T08:14:29.767557] Add path to classpath: /usr/share/elasticsearch/lib/joda-time-2.8.2.jar; [2016-05-27T08:14:29.767950] Add path to classpath: /usr/share/elasticsearch/lib/jsr166e-1.1.0.jar; [2016-05-27T08:14:29.768251] Add path to classpath: /usr/share/elasticsearch/lib/jts-1.13.jar; [2016-05-27T08:14:29.768544] Add path to classpath: /usr/share/elasticsearch/lib/log4j-1.2.17.jar; [2016-05-27T08:14:29.768876] Add path to classpath: /usr/share/elasticsearch/lib/lucene-analyzers-common-5.5.0.jar; [2016-05-27T08:14:29.769225] Add path to classpath: /usr/share/elasticsearch/lib/lucene-backward-codecs-5.5.0.jar; [2016-05-27T08:14:29.769570] Add path to classpath: /usr/share/elasticsearch/lib/lucene-core-5.5.0.jar; [2016-05-27T08:14:29.769900] Add path to classpath: /usr/share/elasticsearch/lib/lucene-grouping-5.5.0.jar; [2016-05-27T08:14:29.770194] Add path to classpath: /usr/share/elasticsearch/lib/lucene-highlighter-5.5.0.jar; [2016-05-27T08:14:29.770514] Add path to classpath: /usr/share/elasticsearch/lib/lucene-join-5.5.0.jar; [2016-05-27T08:14:29.770888] Add path to classpath: /usr/share/elasticsearch/lib/lucene-memory-5.5.0.jar; [2016-05-27T08:14:29.771187] Add path to classpath: /usr/share/elasticsearch/lib/lucene-misc-5.5.0.jar; [2016-05-27T08:14:29.771497] Add path to classpath: /usr/share/elasticsearch/lib/lucene-queries-5.5.0.jar; [2016-05-27T08:14:29.771844] Add path to classpath: /usr/share/elasticsearch/lib/lucene-queryparser-5.5.0.jar; [2016-05-27T08:14:29.772179] Add path to classpath: /usr/share/elasticsearch/lib/lucene-sandbox-5.5.0.jar; [2016-05-27T08:14:29.772466] Add path to classpath: /usr/share/elasticsearch/lib/lucene-spatial-5.5.0.jar; [2016-05-27T08:14:29.772811] Add path to classpath: /usr/share/elasticsearch/lib/lucene-spatial3d-5.5.0.jar; [2016-05-27T08:14:29.773098] Add path to classpath: /usr/share/elasticsearch/lib/lucene-suggest-5.5.0.jar; [2016-05-27T08:14:29.773447] Add path to classpath: /usr/share/elasticsearch/lib/netty-3.10.5.Final.jar; [2016-05-27T08:14:29.773783] Add path to classpath: /usr/share/elasticsearch/lib/securesm-1.0.jar; [2016-05-27T08:14:29.774055] Add path to classpath: /usr/share/elasticsearch/lib/snakeyaml-1.15.jar; [2016-05-27T08:14:29.774326] Add path to classpath: /usr/share/elasticsearch/lib/spatial4j-0.5.jar; [2016-05-27T08:14:29.774572] Add path to classpath: /usr/share/elasticsearch/lib/t-digest-3.0.jar; [2016-05-27T08:14:32.237017] Add path to classpath: //usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar; [2016-05-27T08:14:32.372708] Error initializing message pipeline;
Ivan
On 05/27/2016 02:13 PM, Fabien Wernli wrote:
On Fri, May 27, 2016 at 01:43:29PM +0200, Ivan Adji - Krstev wrote:
So the cluster should be the cluster name of the ES ? How can i see this name ? There is nothing in the elasticsearch.yml file, do i have to specify there ?
by default, it's elasticsearch
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
From the server where ES is running curl http://localhost:9200 <http://localhost:9200/>
On May 27, 2016, at 7:43 AM, Ivan Adji - Krstev <akivanradix@gmail.com> wrote:
So the cluster should be the cluster name of the ES ? How can i see this name ? There is nothing in the elasticsearch.yml file, do i have to specify there ?
Kind regards Ivan
On 05/27/2016 12:45 PM, Czanik, Péter wrote:
option("cluster", "czpcluster")
Unless your cluster is called "czpcluster", the above option won't work. This value is just an example (derived from my nick name :-) ).
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com <mailto:peter.czanik@balabit.com>> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ <http://czanik.blogs.balabit.com/> https://twitter.com/PCzanik <https://twitter.com/PCzanik> On Fri, May 27, 2016 at 12:42 PM, Ivan Adji - Krstev <akivanradix@gmail.com <mailto:akivanradix@gmail.com>> wrote: Well that one i fix it ... export the new path of the libjvm.so file and it works. But now i have another error :)
Error stating pattern database file, no automatic reload will be performed; error='No such file or directory' . . Add path to classpath: /usr/share/elasticsearch/lib/spatial4j-0.5.jar; [2016-05-27T06:38:30.933808] Add path to classpath: /usr/share/elasticsearch/lib/t-digest-3.0.jar; [2016-05-27T06:38:31.287344] Add path to classpath: //usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar; [2016-05-27T06:38:31.333759] Error initializing message pipeline;
And i have no idea what is this problem as im using ES for the first time.
This is what i have:
source s_sys { system(); internal(); network(ip(0.0.0.0) port(6514) flags(syslog-protocol) transport("tls") tls(key_file("/etc/syslog-ng/cert.d/serverkey.pem") cert_file("/etc/syslog-ng/cert.d/servercert.pem") ca_dir("/etc/syslog-ng/ca.d") ) );
}; parser pattern_db { db-parser( file("/etc/syslog-ng/patterndb.d/patterndb.xml") ); }; destination d_es { java( class-path("/usr/lib64/syslog-ng/java-modules/*.jar:/usr/share/elasticsearch/lib/*.jar") class-name("org.syslog_ng.elasticsearch.ElasticSearchDestination") option("index", "syslog-ng_${YEAR}.${MONTH}.${DAY}") option("type", "test") option("cluster", "czpcluster") option("flush_limit", "100") option( "message_template", "$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n") ); };
Kind regards Ivan
On 05/27/2016 12:22 PM, Czanik, Péter wrote:
Hi,
To enable Java support you need at least the "syslog-ng" and "syslog-ng-java" packages from that repository. Optionally you can also install the "syslog-ng-java-hack" package, which includes all the necessary JAR files, or you can also point your config to the JAR files of your Elasticsearch installation. Note, that syslog-ng 3.7 only supports Elasticsearch 1.X.
You will also need to point syslog-ng to libjvm.so. There are multiple ways: <https://czanik.blogs.balabit.com/2016/03/troubleshooting-java-support-in-syslog-ng/>https://czanik.blogs.balabit.com/2016/03/troubleshooting-java-support-in-syslog-ng/ <https://czanik.blogs.balabit.com/2016/03/troubleshooting-java-support-in-syslog-ng/> My personal preference is the ld.so.conf trick, but note that it has side effects if you have multiple Java versions on your system.
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com <mailto:peter.czanik@balabit.com>> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ <http://czanik.blogs.balabit.com/> https://twitter.com/PCzanik <https://twitter.com/PCzanik> On Fri, May 27, 2016 at 12:14 PM, Ivan Adji - Krstev < <mailto:akivanradix@gmail.com>akivanradix@gmail.com <mailto:akivanradix@gmail.com>> wrote: So should i contact him directly or should i wait here to reply on this list ?
Ivan
On 05/27/2016 12:13 PM, Scheidler, Balázs wrote:
Hopefully Peter Czanik can help you then, as he prepared those packages.
On May 27, 2016 11:10 AM, "Ivan Adji - Krstev" <akivanradix@gmail.com <mailto:akivanradix@gmail.com>> wrote: Yes i install that too.. still nothing.
Ivan
On 05/27/2016 12:09 PM, Fabien Wernli wrote:
On Fri, May 27, 2016 at 12:08:21PM +0200, Ivan Adji - Krstev wrote:
Hi Bazsi, I get syslog from: "https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng37/repo/epel-7/czani..." <https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng37/repo/epel-7/czanik-syslog-ng37-epel-7.repo> add the repo and then "yum install syslog-ng" after that i have download the Elasticsearch and install it and that is it. Im using CentOS 7. you also want the package syslog-ng-java
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: <https://lists.balabit.hu/mailman/listinfo/syslog-ng>https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: <http://www.balabit.com/support/documentation/?product=syslog-ng>http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
I have the elasticsearch destination working correctly, but if I try to use multiple rsources, I get an error destination d_elasticsearch_0 { elasticsearch( index("flare-${YEAR}.${MONTH}.${DAY}.${HOUR}") type("flare") cluster("es-dcs-dev") client_mode("node") flush_limit("10000") template("$MESSAGE ") resource("/etc/elasticsearch/elasticsearch.yml") ); }; works fine, but if I add a second resource destination d_elasticsearch_0 { elasticsearch( index("flare-${YEAR}.${MONTH}.${DAY}.${HOUR}") type("flare") cluster("es-dcs-dev") client_mode("node") flush_limit("10000") template("$MESSAGE ") resource("/etc/elasticsearch/elasticsearch.yml;/etc/elasticsearch/logging.yml") ); }; I get an error Starting system logger (syslog-ng.esgateway): [2016-06-07T09:01:58.694633] org.syslog_ng.elasticsearch_v2.client.ESNodeClient.loadConfigFile:69 - Can't load settings from file, file = '/etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/logging.yml', reason = 'Failed to open stream for url [/etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/logging.yml]'; Can anyone shed some light on how to configure multiple resources? -- Evan
Hi, is that possible to set multiple resources for a node? We can use only one resource file. What is your use case that needs this feature? Maybe you have splitted resource files and what we should have to do is just merge the files into a single resource file? Laszlo Budai On Tuesday, June 7, 2016, Evan Rempel <erempel@uvic.ca> wrote:
I have the elasticsearch destination working correctly, but if I try to use multiple rsources, I get an error
destination d_elasticsearch_0 { elasticsearch( index("flare-${YEAR}.${MONTH}.${DAY}.${HOUR}") type("flare") cluster("es-dcs-dev") client_mode("node") flush_limit("10000") template("$MESSAGE ") resource("/etc/elasticsearch/elasticsearch.yml") ); };
works fine, but if I add a second resource
destination d_elasticsearch_0 { elasticsearch( index("flare-${YEAR}.${MONTH}.${DAY}.${HOUR}") type("flare") cluster("es-dcs-dev") client_mode("node") flush_limit("10000") template("$MESSAGE ")
resource("/etc/elasticsearch/elasticsearch.yml;/etc/elasticsearch/logging.yml") ); };
I get an error
Starting system logger (syslog-ng.esgateway): [2016-06-07T09:01:58.694633] org.syslog_ng.elasticsearch_v2.client.ESNodeClient.loadConfigFile:69 - Can't load settings from file, file = '/etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/logging.yml', reason = 'Failed to open stream for url [/etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/logging.yml]';
Can anyone shed some light on how to configure multiple resources?
-- Evan
There are two configuration files that elastic search uses. One is for general elasticsearch functionality (elasticsearch.yml) and the other is specific to logging. I want to be able to control the logging of the elasticsearch instance that is being launched by syslog-ng. Since the syslog-ng documentation states that you can have a list of resources seperated by ";" I thought I could use the logging.yml file as well. On 06/07/2016 08:56 PM, Budai, László wrote:
Hi,
is that possible to set multiple resources for a node? We can use only one resource file.
What is your use case that needs this feature? Maybe you have splitted resource files and what we should have to do is just merge the files into a single resource file?
Laszlo Budai
On Tuesday, June 7, 2016, Evan Rempel <erempel@uvic.ca <mailto:erempel@uvic.ca>> wrote:
I have the elasticsearch destination working correctly, but if I try to use multiple rsources, I get an error
destination d_elasticsearch_0 { elasticsearch( index("flare-${YEAR}.${MONTH}.${DAY}.${HOUR}") type("flare") cluster("es-dcs-dev") client_mode("node") flush_limit("10000") template("$MESSAGE ") resource("/etc/elasticsearch/elasticsearch.yml") ); };
works fine, but if I add a second resource
destination d_elasticsearch_0 { elasticsearch( index("flare-${YEAR}.${MONTH}.${DAY}.${HOUR}") type("flare") cluster("es-dcs-dev") client_mode("node") flush_limit("10000") template("$MESSAGE ") resource("/etc/elasticsearch/elasticsearch.yml;/etc/elasticsearch/logging.yml") ); };
I get an error
Starting system logger (syslog-ng.esgateway): [2016-06-07T09:01:58.694633] org.syslog_ng.elasticsearch_v2.client.ESNodeClient.loadConfigFile:69 - Can't load settings from file, file = '/etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/logging.yml', reason = 'Failed to open stream for url [/etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/logging.yml]';
Can anyone shed some light on how to configure multiple resources?
-- Evan
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (9)
-
Balazs Scheidler
-
Budai, László
-
Czanik, Péter
-
Evan Rempel
-
Fabien Wernli
-
Ivan Adji - Krstev
-
Scheidler, Balázs
-
Scot Needy
-
Szabó, István