[syslog-ng] Elasticsearch with Syslog-ng
Ivan Adji - Krstev
akivanradix at gmail.com
Tue May 31 23:47:15 CEST 2016
Hi Andras,
I have done the following:
destination d_es {
elasticsearch2(
index("syslog-ng_${YEAR}.${MONTH}.${DAY}")
type("test")
cluster("elasticsearch")
);
};
And have the following errors:
[2016-05-31T17:45:53.016537] Add path to classpath:
//usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar;
[2016-05-31T17:45:53.017488] Add path to classpath:
/usr/lib64/syslog-ng/java-modules/commons-codec-1.9.jar;
[2016-05-31T17:45:53.017853] Add path to classpath:
/usr/lib64/syslog-ng/java-modules/commons-lang3-3.4.jar;
[2016-05-31T17:45:53.018120] Add path to classpath:
/usr/lib64/syslog-ng/java-modules/commons-logging-1.2.jar;
[2016-05-31T17:45:53.018409] Add path to classpath:
/usr/lib64/syslog-ng/java-modules/dummy.jar;
[2016-05-31T17:45:53.018694] Add path to classpath:
/usr/lib64/syslog-ng/java-modules/elastic-v2.jar;
[2016-05-31T17:45:53.018989] Add path to classpath:
/usr/lib64/syslog-ng/java-modules/elastic.jar;
[2016-05-31T17:45:53.019244] Add path to classpath:
/usr/lib64/syslog-ng/java-modules/gson-2.6.2.jar;
[2016-05-31T17:45:53.019506] Add path to classpath:
/usr/lib64/syslog-ng/java-modules/guava-19.0.jar;
[2016-05-31T17:45:53.019796] Add path to classpath:
/usr/lib64/syslog-ng/java-modules/hdfs.jar;
[2016-05-31T17:45:53.020099] Add path to classpath:
/usr/lib64/syslog-ng/java-modules/http.jar;
[2016-05-31T17:45:53.020365] Add path to classpath:
/usr/lib64/syslog-ng/java-modules/httpasyncclient-4.1.1.jar;
[2016-05-31T17:45:53.020651] Add path to classpath:
/usr/lib64/syslog-ng/java-modules/httpclient-4.5.2.jar;
[2016-05-31T17:45:53.020966] Add path to classpath:
/usr/lib64/syslog-ng/java-modules/httpcore-4.4.4.jar;
[2016-05-31T17:45:53.021267] Add path to classpath:
/usr/lib64/syslog-ng/java-modules/httpcore-nio-4.4.4.jar;
[2016-05-31T17:45:53.021567] Add path to classpath:
/usr/lib64/syslog-ng/java-modules/jest-2.0.2.jar;
[2016-05-31T17:45:53.022362] Add path to classpath:
/usr/lib64/syslog-ng/java-modules/jest-common-2.0.2.jar;
[2016-05-31T17:45:53.022668] Add path to classpath:
/usr/lib64/syslog-ng/java-modules/kafka.jar;
[2016-05-31T17:45:53.022970] Add path to classpath:
/usr/lib64/syslog-ng/java-modules/log4j-1.2.16.jar;
[2016-05-31T17:45:53.023262] Add path to classpath:
/usr/lib64/syslog-ng/java-modules/slf4j-api-1.7.13.jar;
[2016-05-31T17:45:53.023523] Add path to classpath:
/usr/lib64/syslog-ng/java-modules/syslog-ng-common.jar;
[2016-05-31T17:45:53.023833] Add path to classpath:
/usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar;
[2016-05-31T17:45:53.335411] Add path to classpath:
//usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar;
[2016-05-31T17:45:53.343531] Error initializing message pipeline;
So if i go with the previous configuration is all ok. So any idea what
is wrong with this configuration ?
Ivan
On 05/31/2016 08:18 PM, Mitzki, András wrote:
> Sorry about that (it was a quick copy-paste). The proper configuration
> snippet would be:
>
> [syslog-ng.conf:]
> @version: 3.7
> @module "mod-java"
> @include "scl.conf"
> ...
> [your global options]
> ...
> [your source drivers]
> ....
> destination d_es {
> elasticsearch2(
> [your elasticsearch options in the following way for
> example: cluster("syslog-ng") ]
> );
> };
> ...
> [your logpath]
>
> You can find other configuration examples at :
> https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html-single/index.html#configuring-destinations-elasticsearch
>
> Best Regards,
> Andras
>
> On Tue, May 31, 2016 at 8:06 PM, Ivan Adji - Krstev
> <akivanradix at gmail.com <mailto:akivanradix at gmail.com>> wrote:
>
> Hi Andres,
>
> The classn_name was the problem, you are right. This change help
> me to fix the problem. I try the example you have send
>
> elasticsearch2(
> index("syslog-ng_${YEAR}.${MONTH}.${DAY}")
> type("test")
> cluster("syslog-ng")
> );
>
> And on this one i have the following error:
> 2016-05-31T14:05:36.700320] Using /dev/log Unix socket with
> systemd is not possible. Changing to systemd-syslog source, which
> supports socket activation.;
> Error parsing config, root plugin elasticsearch2 not found in
> /etc/syslog-ng/syslog-ng.conf at line 55, column 1:
>
> elasticsearch2(
>
> Is there any difference between the two of them.
>
> Ivan
>
> On 05/31/2016 07:38 PM, Mitzki, András wrote:
>> Hi Ivan,
>>
>> I think the problem is in
>> the: class-name("org.syslog_ng.elasticsearch.ElasticSearchDestination").
>> You have installed Elasticsearch version 2.3.3 (seen in: Add path
>> to classpath:
>> /usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar;), and with
>> the above line you say that let syslog-ng use the Elasticsearch
>> version 1 java module.
>> To use syslog-ng with Elasticsearch version 2 java module the
>> proper class-name would
>> be: class_name("org.syslog_ng.elasticsearch_v2.ElasticSearchDestination")
>> Or you can configure syslog-ng with the help of scl-s, where you
>> can skip the class-name() option. In this case an example
>> configuration can be the following :
>>
>> @module mod-java
>> @include "scl.conf"
>>
>> elasticsearch2(
>> index("syslog-ng_${YEAR}.${MONTH}.${DAY}")
>> type("test")
>> cluster("syslog-ng")
>> );
>>
>> Best regards,
>> Andras
>>
>> On Tue, May 31, 2016 at 5:18 PM, Ivan Adji - Krstev
>> <akivanradix at gmail.com <mailto:akivanradix at gmail.com>> wrote:
>>
>> Hi all,
>> I have the following error when i trying to run syslog-ng.
>>
>> [2016-05-31T11:17:31.298897] Seeking the journal to the last
>> cursor position;
>> cursor='s=6b605b6b981a471ea3d1abe52861ae82;i=21a8;b=42ab72c238874f7d8afc3c2cf0fea52d;m=38dbdc6d9;t=533be9608d083;x=140acfae276a2e55'
>> [2016-05-31T11:17:31.299921] Module loaded and initialized
>> successfully; module='syslogformat'
>> [2016-05-31T11:17:31.299958] Failed to acquire
>> /run/systemd/journal/syslog socket, disabling systemd-syslog
>> source;
>> [2016-05-31T11:17:31.796854] Add path to classpath:
>> //usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar;
>> [2016-05-31T11:17:31.797915] Add path to classpath:
>> /usr/lib64/syslog-ng/java-modules/commons-codec-1.9.jar;
>> [2016-05-31T11:17:31.798191] Add path to classpath:
>> /usr/lib64/syslog-ng/java-modules/commons-lang3-3.4.jar;
>> [2016-05-31T11:17:31.798466] Add path to classpath:
>> /usr/lib64/syslog-ng/java-modules/commons-logging-1.2.jar;
>> [2016-05-31T11:17:31.798821] Add path to classpath:
>> /usr/lib64/syslog-ng/java-modules/dummy.jar;
>> [2016-05-31T11:17:31.799058] Add path to classpath:
>> /usr/lib64/syslog-ng/java-modules/elastic-v2.jar;
>> [2016-05-31T11:17:31.799296] Add path to classpath:
>> /usr/lib64/syslog-ng/java-modules/elastic.jar;
>> [2016-05-31T11:17:31.799503] Add path to classpath:
>> /usr/lib64/syslog-ng/java-modules/gson-2.6.2.jar;
>> [2016-05-31T11:17:31.799778] Add path to classpath:
>> /usr/lib64/syslog-ng/java-modules/guava-19.0.jar;
>> [2016-05-31T11:17:31.799988] Add path to classpath:
>> /usr/lib64/syslog-ng/java-modules/hdfs.jar;
>> [2016-05-31T11:17:31.800249] Add path to classpath:
>> /usr/lib64/syslog-ng/java-modules/http.jar;
>> [2016-05-31T11:17:31.800477] Add path to classpath:
>> /usr/lib64/syslog-ng/java-modules/httpasyncclient-4.1.1.jar;
>> [2016-05-31T11:17:31.800684] Add path to classpath:
>> /usr/lib64/syslog-ng/java-modules/httpclient-4.5.2.jar;
>> [2016-05-31T11:17:31.800940] Add path to classpath:
>> /usr/lib64/syslog-ng/java-modules/httpcore-4.4.4.jar;
>> [2016-05-31T11:17:31.801215] Add path to classpath:
>> /usr/lib64/syslog-ng/java-modules/httpcore-nio-4.4.4.jar;
>> [2016-05-31T11:17:31.801449] Add path to classpath:
>> /usr/lib64/syslog-ng/java-modules/jest-2.0.2.jar;
>> [2016-05-31T11:17:31.801667] Add path to classpath:
>> /usr/lib64/syslog-ng/java-modules/jest-common-2.0.2.jar;
>> [2016-05-31T11:17:31.801932] Add path to classpath:
>> /usr/lib64/syslog-ng/java-modules/kafka.jar;
>> [2016-05-31T11:17:31.802119] Add path to classpath:
>> /usr/lib64/syslog-ng/java-modules/log4j-1.2.16.jar;
>> [2016-05-31T11:17:31.802353] Add path to classpath:
>> /usr/lib64/syslog-ng/java-modules/slf4j-api-1.7.13.jar;
>> [2016-05-31T11:17:31.802569] Add path to classpath:
>> /usr/lib64/syslog-ng/java-modules/syslog-ng-common.jar;
>> [2016-05-31T11:17:31.802849] Add path to classpath:
>> /usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar;
>> [2016-05-31T11:17:31.803043] Add path to classpath:
>> /usr/share/elasticsearch/lib/HdrHistogram-2.1.6.jar;
>> [2016-05-31T11:17:31.803264] Add path to classpath:
>> /usr/share/elasticsearch/lib/apache-log4j-extras-1.2.17.jar;
>> [2016-05-31T11:17:31.803497] Add path to classpath:
>> /usr/share/elasticsearch/lib/commons-cli-1.3.1.jar;
>> [2016-05-31T11:17:31.803746] Add path to classpath:
>> /usr/share/elasticsearch/lib/compiler-0.8.13.jar;
>> [2016-05-31T11:17:31.803992] Add path to classpath:
>> /usr/share/elasticsearch/lib/compress-lzf-1.0.2.jar;
>> [2016-05-31T11:17:31.804249] Add path to classpath:
>> /usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar;
>> [2016-05-31T11:17:31.804436] Add path to classpath:
>> /usr/share/elasticsearch/lib/guava-18.0.jar;
>> [2016-05-31T11:17:31.804651] Add path to classpath:
>> /usr/share/elasticsearch/lib/hppc-0.7.1.jar;
>> [2016-05-31T11:17:31.804877] Add path to classpath:
>> /usr/share/elasticsearch/lib/jackson-core-2.6.6.jar;
>> [2016-05-31T11:17:31.805109] Add path to classpath:
>> /usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.6.6.jar;
>> [2016-05-31T11:17:31.805383] Add path to classpath:
>> /usr/share/elasticsearch/lib/jackson-dataformat-smile-2.6.6.jar;
>> [2016-05-31T11:17:31.805663] Add path to classpath:
>> /usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.6.6.jar;
>> [2016-05-31T11:17:31.805908] Add path to classpath:
>> /usr/share/elasticsearch/lib/jna-4.1.0.jar;
>> [2016-05-31T11:17:31.806160] Add path to classpath:
>> /usr/share/elasticsearch/lib/joda-convert-1.2.jar;
>> [2016-05-31T11:17:31.806402] Add path to classpath:
>> /usr/share/elasticsearch/lib/joda-time-2.8.2.jar;
>> [2016-05-31T11:17:31.806654] Add path to classpath:
>> /usr/share/elasticsearch/lib/jsr166e-1.1.0.jar;
>> [2016-05-31T11:17:31.806889] Add path to classpath:
>> /usr/share/elasticsearch/lib/jts-1.13.jar;
>> [2016-05-31T11:17:31.807087] Add path to classpath:
>> /usr/share/elasticsearch/lib/log4j-1.2.17.jar;
>> [2016-05-31T11:17:31.807260] Add path to classpath:
>> /usr/share/elasticsearch/lib/lucene-analyzers-common-5.5.0.jar;
>> [2016-05-31T11:17:31.807476] Add path to classpath:
>> /usr/share/elasticsearch/lib/lucene-backward-codecs-5.5.0.jar;
>> [2016-05-31T11:17:31.807759] Add path to classpath:
>> /usr/share/elasticsearch/lib/lucene-core-5.5.0.jar;
>> [2016-05-31T11:17:31.808003] Add path to classpath:
>> /usr/share/elasticsearch/lib/lucene-grouping-5.5.0.jar;
>> [2016-05-31T11:17:31.808261] Add path to classpath:
>> /usr/share/elasticsearch/lib/lucene-highlighter-5.5.0.jar;
>> [2016-05-31T11:17:31.808476] Add path to classpath:
>> /usr/share/elasticsearch/lib/lucene-join-5.5.0.jar;
>> [2016-05-31T11:17:31.808653] Add path to classpath:
>> /usr/share/elasticsearch/lib/lucene-memory-5.5.0.jar;
>> [2016-05-31T11:17:31.808929] Add path to classpath:
>> /usr/share/elasticsearch/lib/lucene-misc-5.5.0.jar;
>> [2016-05-31T11:17:31.809140] Add path to classpath:
>> /usr/share/elasticsearch/lib/lucene-queries-5.5.0.jar;
>> [2016-05-31T11:17:31.809362] Add path to classpath:
>> /usr/share/elasticsearch/lib/lucene-queryparser-5.5.0.jar;
>> [2016-05-31T11:17:31.809595] Add path to classpath:
>> /usr/share/elasticsearch/lib/lucene-sandbox-5.5.0.jar;
>> [2016-05-31T11:17:31.809823] Add path to classpath:
>> /usr/share/elasticsearch/lib/lucene-spatial-5.5.0.jar;
>> [2016-05-31T11:17:31.810023] Add path to classpath:
>> /usr/share/elasticsearch/lib/lucene-spatial3d-5.5.0.jar;
>> [2016-05-31T11:17:31.810229] Add path to classpath:
>> /usr/share/elasticsearch/lib/lucene-suggest-5.5.0.jar;
>> [2016-05-31T11:17:31.810427] Add path to classpath:
>> /usr/share/elasticsearch/lib/netty-3.10.5.Final.jar;
>> [2016-05-31T11:17:31.810628] Add path to classpath:
>> /usr/share/elasticsearch/lib/securesm-1.0.jar;
>> [2016-05-31T11:17:31.810885] Add path to classpath:
>> /usr/share/elasticsearch/lib/snakeyaml-1.15.jar;
>> [2016-05-31T11:17:31.811065] Add path to classpath:
>> /usr/share/elasticsearch/lib/spatial4j-0.5.jar;
>> [2016-05-31T11:17:31.811279] Add path to classpath:
>> /usr/share/elasticsearch/lib/t-digest-3.0.jar;
>> [2016-05-31T11:17:33.037026] Add path to classpath:
>> //usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar;
>> [2016-05-31T11:17:33.073362] Error initializing message pipeline;
>>
>> Im running with Elasticsearch with the following configuration:
>> destination d_es {
>> java(
>>
>> class-path("/usr/lib64/syslog-ng/java-modules/*.jar:/usr/share/elasticsearch/lib/*.jar")
>>
>> class-name("org.syslog_ng.elasticsearch.ElasticSearchDestination")
>> option("index", "syslog-ng_${YEAR}.${MONTH}.${DAY}")
>> option("type", "test")
>> option("cluster", "elasticsearch")
>> # option("flush_limit", "100")
>> option( "message_template", "$(format-json --scope
>> rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n")
>> );
>> };
>>
>> Any hints on this ?
>>
>> Ivan
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20160531/91d39a30/attachment-0001.htm
More information about the syslog-ng
mailing list