<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">
<div>root@ES6:/etc/syslog-ng# more syslog-ng.conf</div>
<div>@version:3.13</div>
<div>@module mod-java</div>
<div>@include "scl.conf"</div>
<div>options {</div>
<div> flush_lines(0);</div>
<div> keep_hostname(yes);</div>
<div> normalize_hostnames(yes);</div>
<div> threaded(yes);</div>
<div>};</div>
<div>source s_local { system(); internal(); };</div>
<div>source s_network { syslog(transport(tcp)); };</div>
<div>destination d_all { file ("/var/log/all.log"); };</div>
<div>destination d_elastic {</div>
<div> elasticsearch2(</div>
<div> client-lib-dir("/usr/lib/syslog-ng/3.13/java-modules/elastic-jest-client/*.jar:/usr/share/elasticsearch/lib/:/usr/lib/syslog-ng/3.13/java-modules/") --- adding path </div>
<div> client_mode("http")</div>
<div> cluster_url("http://192.168.1.75:9200")</div>
<div> index("syslog-ng_${YEAR}.${MONTH}.${DAY}")</div>
<div> type("syslog")</div>
<div> cluster("test")</div>
<div> flush-limit("1000")</div>
<div> template("$(format-json --scope rfc5424 --scope nv-pairs --exclude DATE --key ISODATE)")</div>
<div> time-zone("UTC")</div>
<div> );</div>
<div>};</div>
<div>log { source(s_network); destination(d_elastic); };</div>
<div>log { source(s_local); destination(d_all); };</div>
<div><br>
</div>
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">
fix the error, but i will test and come back.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">
R!</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of hari ram <hariram@hotmail.com><br>
<b>Sent:</b> 14 December 2017 23:04<br>
<b>To:</b> syslog-ng@lists.balabit.hu<br>
<b>Subject:</b> [syslog-ng] SYSLOG-NG issue with ES 6.X</font>
<div> </div>
</div>
<div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">
Hi</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">
I have installed SYSLOG-NG 3.13.2 on ubunutu, try to send logs to ES 6.0 i failed to do so, here is my inputs.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">
<div>root@ES6:/etc/syslog-ng# syslog-ng -V</div>
<div>syslog-ng 3 (3.13.2)</div>
<div>Config version: 3.13</div>
<div>Installer-Version: 3.13.2</div>
<div>Revision: 3.13.2-1</div>
<div>Compile-Date: Dec 5 2017 13:24:07</div>
<div>Module-Directory: /usr/lib/syslog-ng/3.13</div>
<div>Module-Path: /usr/lib/syslog-ng/3.13</div>
<div>Available-Modules: afuser,mod-python,afstomp,http,afsql,disk-buffer,mod-java,cef,pseudofile,sdjournal,kvformat,xml,csvparser,snmptrapd-parser,appmodel,confgen,pacctformat,linux-kmsg-format,dbparser,system-source,map-value-pairs,add-contextual-data,date,syslogformat,afamqp,geoip2-plugin,tfgetent,graphite,afmongodb,cryptofuncs,geoip-plugin,afsmtp,afsocket,redis,affile,stardate,basicfuncs,riemann,json-plugin,tags-parser,afprog</div>
<div>Enable-Debug: off</div>
<div>Enable-GProf: off</div>
<div>Enable-Memtrace: off</div>
<div>Enable-IPv6: on</div>
<div>Enable-Spoof-Source: on</div>
<div>Enable-TCP-Wrapper: on</div>
<div>Enable-Linux-Caps: on</div>
<div>Enable-Systemd: on</div>
<div><br>
</div>
===</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">
<div>root@ES6:/etc/syslog-ng# more syslog-ng.conf</div>
<div>@version:3.13</div>
<div>@module mod-java</div>
<div>@include "scl.conf"</div>
<div>options {</div>
<div> flush_lines(0);</div>
<div> keep_hostname(yes);</div>
<div> normalize_hostnames(yes);</div>
<div> threaded(yes);</div>
<div>};</div>
<div>source s_local { system(); internal(); };</div>
<div>source s_network { syslog(transport(tcp)); };</div>
<div>destination d_all { file ("/var/log/all.log"); };</div>
<div>destination d_elastic {</div>
<div> elasticsearch2(</div>
<div> client-lib-dir("/usr/share/elasticsearch/lib/:/usr/lib/syslog-ng/3.13/java-modules/")</div>
<div> client_mode("http")</div>
<div> cluster_url("http://192.168.1.75:9200")</div>
<div> index("syslog-ng_${YEAR}.${MONTH}.${DAY}")</div>
<div> type("syslog")</div>
<div> cluster("test")</div>
<div> flush-limit("1000")</div>
<div> template("$(format-json --scope rfc5424 --scope nv-pairs --exclude DATE --key ISODATE)")</div>
<div> time-zone("UTC")</div>
<div> );</div>
<div>};</div>
<div>log { source(s_network); destination(d_elastic); };</div>
<div>log { source(s_local); destination(d_all); };</div>
<div><br>
</div>
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">
===</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">
<div>root@ES6:/etc/syslog-ng# ls /usr/share/elasticsearch/lib/</div>
<div>elasticsearch-6.0.1.jar jackson-dataformat-smile-2.8.6.jar jopt-simple-5.0.2.jar lucene-analyzers-common-7.0.1.jar lucene-join-7.0.1.jar lucene-sandbox-7.0.1.jar plugin-cli-6.0.1.jar</div>
<div>HdrHistogram-2.1.9.jar jackson-dataformat-yaml-2.8.6.jar jts-1.13.jar lucene-backward-codecs-7.0.1.jar lucene-memory-7.0.1.jar lucene-spatial3d-7.0.1.jar securesm-1.2.jar</div>
<div>hppc-0.7.1.jar java-version-checker-6.0.1.jar log4j-1.2-api-2.9.1.jar lucene-core-7.0.1.jar lucene-misc-7.0.1.jar lucene-spatial-7.0.1.jar snakeyaml-1.15.jar</div>
<div>jackson-core-2.8.6.jar jna-4.4.0-1.jar log4j-api-2.9.1.jar lucene-grouping-7.0.1.jar lucene-queries-7.0.1.jar lucene-spatial-extras-7.0.1.jar spatial4j-0.6.jar</div>
<div>jackson-dataformat-cbor-2.8.6.jar joda-time-2.9.5.jar log4j-core-2.9.1.jar lucene-highlighter-7.0.1.jar lucene-queryparser-7.0.1.jar lucene-suggest-7.0.1.jar t-digest-3.0.jar</div>
<div><br>
</div>
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">
====</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">
<div>root@ES6:/etc/syslog-ng# ls /usr/lib/syslog-ng/3.13/java-modules/</div>
<div>elastic.jar elastic-jest-client elastic-v2.jar hdfs.jar http.jar kafka.jar log4j-1.2.16.jar syslog-ng-common.jar syslog-ng-core.jar</div>
<div><br>
</div>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">
==</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">
root@ES6:/etc/syslog-ng# syslog-ng -Fevd<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">
<div>[2017-12-14T23:04:21.552408] Compiling #unnamed sequence [log] at [source generator system:14:12]</div>
<div>[2017-12-14T23:04:21.552510] Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf:10:35]</div>
<div>[2017-12-14T23:04:21.552632] Compiling d_all reference [destination] at [/etc/syslog-ng/syslog-ng.conf:27:24]</div>
<div>[2017-12-14T23:04:21.552715] Compiling d_all sequence [destination] at [/etc/syslog-ng/syslog-ng.conf:12:1]</div>
<div>[2017-12-14T23:04:21.552781] Compiling #unnamed junction [log] at [/etc/syslog-ng/syslog-ng.conf:12:20]</div>
<div>[2017-12-14T23:04:21.552884] Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf:12:21]</div>
<div>[2017-12-14T23:04:21.553211] Module loaded and initialized successfully; module='syslogformat'</div>
<div>[2017-12-14T23:04:21.553425] Processing the time zone file (32bit part); filename='/usr/share/zoneinfo/UTC'</div>
<div>[2017-12-14T23:04:21.671696] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/syslog-ng-core.jar;</div>
<div>[2017-12-14T23:04:21.672418] Add path to classpath: /usr/share/elasticsearch/lib/;</div>
<div>[2017-12-14T23:04:21.673641] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/syslog-ng-core.jar;</div>
<div>[2017-12-14T23:04:21.673912] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/syslog-ng-common.jar;</div>
<div>[2017-12-14T23:04:21.674218] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/hdfs.jar;</div>
<div>[2017-12-14T23:04:21.674704] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/http.jar;</div>
<div>[2017-12-14T23:04:21.675858] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/kafka.jar;</div>
<div>[2017-12-14T23:04:21.676116] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/log4j-1.2.16.jar;</div>
<div>[2017-12-14T23:04:21.676322] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/elastic-v2.jar;</div>
<div>[2017-12-14T23:04:21.676484] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/elastic.jar;</div>
<div>[2017-12-14T23:04:21.741649] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/syslog-ng-core.jar;</div>
<div>[2017-12-14T23:04:21.746168] Error initializing message pipeline; plugin name='java', location='#buffer:2:3'</div>
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">
Any suggestions ?</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">
R!</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">
<br>
</div>
</div>
</body>
</html>