<div dir="ltr"><div><div>Hi,<br><br></div>this seems like a bug (I guess resolving the `java-module-dir` in the scl file failes somehow and this is why you have to set the classpath manually).</div><div>Could you share the content of your etc/scl.conf?</div><div></div><div></div><div><br></div><div>regards,</div><div>Laszlo Budai<br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jan 2, 2018 at 9:47 PM, Marco Mignone <span dir="ltr"><<a href="mailto:info@marcomignone.com" target="_blank">info@marcomignone.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">I have forgot to say that I was using the syslog-ng Docker image -> balabit/syslog-ng:latest<div><br></div><div>Thanks,</div><div>Marco<br><div><br><div><blockquote type="cite"><div>On 2 Jan 2018, at 15:41, Marco Mignone <<a href="mailto:info@marcomignone.com" target="_blank">info@marcomignone.com</a>> wrote:</div><br class="m_4915803031001585147Apple-interchange-newline"><div><div style="word-wrap:break-word">Hi All,<div>That worked for me too but I have few questions:</div><div><br></div><div>- Is this the expected behaviour?</div><div>- Do we still need to add the *.jar library files from the ES distribution?</div><div>- The client-lib-dir function seems to need *.jar when multiple paths are specified, apart from the last path in the line - is this correct?</div><div><br></div><div><br></div><div>My path in the ES destination:</div><div><br></div><div>client-lib-dir(“/esjarfiles/*.<wbr>jar:/usr/lib/syslog-ng/3.13/<wbr>java-modules/elastic-jest-<wbr>client/*.jar:/usr/lib/syslog-<wbr>ng/3.13/java-modules/“)</div><div><br></div><div>Thanks,</div><div>Marco</div><div><br><div><blockquote type="cite"><div>On 14 Dec 2017, at 23:08, hari ram <<a href="mailto:hariram@hotmail.com" target="_blank">hariram@hotmail.com</a>> wrote:</div><br class="m_4915803031001585147Apple-interchange-newline"><div><div style="font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-family:Calibri,Helvetica,sans-serif;font-size:12pt;background-color:rgba(0,0,0,0)"><div>root@ES6:/etc/syslog-ng# more syslog-ng.conf</div><div>@version:3.13</div><div>@module mod-java</div><div>@include "scl.conf"</div><div>options {</div><div> flush_lines(0);</div><div> keep_hostname(yes);</div><div> normalize_hostnames(yes);</div><div> threaded(yes);</div><div>};</div><div>source s_local { system(); internal(); };</div><div>source s_network { syslog(transport(tcp)); };</div><div>destination d_all { file ("/var/log/all.log"); };</div><div>destination d_elastic {</div><div> elasticsearch2(</div><div> client-lib-dir("/usr/lib/<wbr>syslog-ng/3.13/java-modules/<wbr>elastic-jest-client/*.jar:/<wbr>usr/share/elasticsearch/lib/:/<wbr>usr/lib/syslog-ng/3.13/java-<wbr>modules/") --- adding path </div><div> client_mode("http")</div><div> cluster_url("<a href="http://192.168.1.75:9200/" target="_blank">http://192.168.1.<wbr>75:9200</a>")</div><div> index("syslog-ng_${YEAR}.${<wbr>MONTH}.${DAY}")</div><div> type("syslog")</div><div> cluster("test")</div><div> flush-limit("1000")</div><div> template("$(format-json --scope rfc5424 --scope nv-pairs --exclude DATE --key ISODATE)")</div><div> time-zone("UTC")</div><div> );</div><div>};</div><div>log { source(s_network); destination(d_elastic); };</div><div>log { source(s_local); destination(d_all); };</div><div><br></div><br></div><div style="font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-family:Calibri,Helvetica,sans-serif;font-size:12pt;background-color:rgba(0,0,0,0)"><br></div><div style="font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-family:Calibri,Helvetica,sans-serif;font-size:12pt;background-color:rgba(0,0,0,0)">fix the error, but i will test and come back.</div><div style="font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-family:Calibri,Helvetica,sans-serif;font-size:12pt;background-color:rgba(0,0,0,0)"><br></div><div style="font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-family:Calibri,Helvetica,sans-serif;font-size:12pt;background-color:rgba(0,0,0,0)">R!</div><hr style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;display:inline-block;width:756.546875px"><span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;display:inline!important"></span><div id="m_4915803031001585147divRplyFwdMsg" dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><font style="font-size:11pt" face="Calibri, sans-serif"><b>From:</b><span class="m_4915803031001585147Apple-converted-space"> </span>syslog-ng <<a href="mailto:syslog-ng-bounces@lists.balabit.hu" target="_blank">syslog-ng-bounces@lists.<wbr>balabit.hu</a>> on behalf of hari ram <<a href="mailto:hariram@hotmail.com" target="_blank">hariram@hotmail.com</a>><br><b>Sent:</b><span class="m_4915803031001585147Apple-converted-space"> </span>14 December 2017 23:04<br><b>To:</b><span class="m_4915803031001585147Apple-converted-space"> </span><a href="mailto:syslog-ng@lists.balabit.hu" target="_blank">syslog-ng@lists.balabit.hu</a><br><b>Subject:</b><span class="m_4915803031001585147Apple-converted-space"> </span>[syslog-ng] SYSLOG-NG issue with ES 6.X</font><div> </div></div><div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;background-color:rgba(0,0,0,0)">Hi</div><div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;background-color:rgba(0,0,0,0)"><br></div><div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;background-color:rgba(0,0,0,0)">I have installed SYSLOG-NG 3.13.2 on ubunutu, try to send logs to ES 6.0 i failed to do so, here is my inputs.</div><div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;background-color:rgba(0,0,0,0)"><br></div><div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;background-color:rgba(0,0,0,0)"><div>root@ES6:/etc/syslog-ng# syslog-ng -V</div><div>syslog-ng 3 (3.13.2)</div><div>Config version: 3.13</div><div>Installer-Version: 3.13.2</div><div>Revision: 3.13.2-1</div><div>Compile-Date: Dec 5 2017 13:24:07</div><div>Module-Directory: /usr/lib/syslog-ng/3.13</div><div>Module-Path: /usr/lib/syslog-ng/3.13</div><div>Available-Modules: afuser,mod-python,afstomp,<wbr>http,afsql,disk-buffer,mod-<wbr>java,cef,pseudofile,sdjournal,<wbr>kvformat,xml,csvparser,<wbr>snmptrapd-parser,appmodel,<wbr>confgen,pacctformat,linux-<wbr>kmsg-format,dbparser,system-<wbr>source,map-value-pairs,add-<wbr>contextual-data,date,<wbr>syslogformat,afamqp,geoip2-<wbr>plugin,tfgetent,graphite,<wbr>afmongodb,cryptofuncs,geoip-<wbr>plugin,afsmtp,afsocket,redis,<wbr>affile,stardate,basicfuncs,<wbr>riemann,json-plugin,tags-<wbr>parser,afprog</div><div>Enable-Debug: off</div><div>Enable-GProf: off</div><div>Enable-Memtrace: off</div><div>Enable-IPv6: on</div><div>Enable-Spoof-Source: on</div><div>Enable-TCP-Wrapper: on</div><div>Enable-Linux-Caps: on</div><div>Enable-Systemd: on</div><div><br></div>===</div><div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;background-color:rgba(0,0,0,0)"><br></div><div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;background-color:rgba(0,0,0,0)"><div>root@ES6:/etc/syslog-ng# more syslog-ng.conf</div><div>@version:3.13</div><div>@module mod-java</div><div>@include "scl.conf"</div><div>options {</div><div> flush_lines(0);</div><div> keep_hostname(yes);</div><div> normalize_hostnames(yes);</div><div> threaded(yes);</div><div>};</div><div>source s_local { system(); internal(); };</div><div>source s_network { syslog(transport(tcp)); };</div><div>destination d_all { file ("/var/log/all.log"); };</div><div>destination d_elastic {</div><div> elasticsearch2(</div><div> client-lib-dir("/usr/share/<wbr>elasticsearch/lib/:/usr/lib/<wbr>syslog-ng/3.13/java-modules/")</div><div> client_mode("http")</div><div> cluster_url("<a href="http://192.168.1.75:9200/" target="_blank">http://192.168.1.<wbr>75:9200</a>")</div><div> index("syslog-ng_${YEAR}.${<wbr>MONTH}.${DAY}")</div><div> type("syslog")</div><div> cluster("test")</div><div> flush-limit("1000")</div><div> template("$(format-json --scope rfc5424 --scope nv-pairs --exclude DATE --key ISODATE)")</div><div> time-zone("UTC")</div><div> );</div><div>};</div><div>log { source(s_network); destination(d_elastic); };</div><div>log { source(s_local); destination(d_all); };</div><div><br></div><br></div><div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;background-color:rgba(0,0,0,0)">===</div><div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;background-color:rgba(0,0,0,0)"><br></div><div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;background-color:rgba(0,0,0,0)"><div>root@ES6:/etc/syslog-ng# ls /usr/share/elasticsearch/lib/</div><div>elasticsearch-6.0.1.jar jackson-dataformat-smile-2.8.<wbr>6.jar jopt-simple-5.0.2.jar lucene-analyzers-common-7.0.1.<wbr>jar lucene-join-7.0.1.jar lucene-sandbox-7.0.1.jar plugin-cli-6.0.1.jar</div><div>HdrHistogram-2.1.9.jar jackson-dataformat-yaml-2.8.<wbr>6.jar jts-1.13.jar lucene-backward-codecs-7.0.1.<wbr>jar lucene-memory-7.0.1.jar lucene-spatial3d-7.0.1.jar securesm-1.2.jar</div><div>hppc-0.7.1.jar java-version-checker-6.0.1.<wbr>jar log4j-1.2-api-2.9.1.jar lucene-core-7.0.1.jar lucene-misc-7.0.1.jar lucene-spatial-7.0.1.jar snakeyaml-1.15.jar</div><div>jackson-core-2.8.6.jar jna-4.4.0-1.jar log4j-api-2.9.1.jar lucene-grouping-7.0.1.jar lucene-queries-7.0.1.jar lucene-spatial-extras-7.0.1.<wbr>jar spatial4j-0.6.jar</div><div>jackson-dataformat-cbor-2.8.6.<wbr>jar joda-time-2.9.5.jar log4j-core-2.9.1.jar lucene-highlighter-7.0.1.jar lucene-queryparser-7.0.1.jar lucene-suggest-7.0.1.jar t-digest-3.0.jar</div><div><br></div><br></div><div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;background-color:rgba(0,0,0,0)">====</div><div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;background-color:rgba(0,0,0,0)"><br></div><div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;background-color:rgba(0,0,0,0)"><div>root@ES6:/etc/syslog-ng# ls /usr/lib/syslog-ng/3.13/java-<wbr>modules/</div><div>elastic.jar elastic-jest-client elastic-v2.jar hdfs.jar http.jar kafka.jar log4j-1.2.16.jar syslog-ng-common.jar syslog-ng-core.jar</div><div><br></div></div><div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;background-color:rgba(0,0,0,0)">==</div><div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;background-color:rgba(0,0,0,0)"><br></div><div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;background-color:rgba(0,0,0,0)">root@ES6:/etc/syslog-ng# syslog-ng -Fevd<br></div><div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;background-color:rgba(0,0,0,0)"><br></div><div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;background-color:rgba(0,0,0,0)"><div>[2017-12-14T23:04:21.552408] Compiling #unnamed sequence [log] at [source generator system:14:12]</div><div>[2017-12-14T23:04:21.552510] Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.<wbr>conf:10:35]</div><div>[2017-12-14T23:04:21.552632] Compiling d_all reference [destination] at [/etc/syslog-ng/syslog-ng.<wbr>conf:27:24]</div><div>[2017-12-14T23:04:21.552715] Compiling d_all sequence [destination] at [/etc/syslog-ng/syslog-ng.<wbr>conf:12:1]</div><div>[2017-12-14T23:04:21.552781] Compiling #unnamed junction [log] at [/etc/syslog-ng/syslog-ng.<wbr>conf:12:20]</div><div>[2017-12-14T23:04:21.552884] Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.<wbr>conf:12:21]</div><div>[2017-12-14T23:04:21.553211] Module loaded and initialized successfully; module='syslogformat'</div><div>[2017-12-14T23:04:21.553425] Processing the time zone file (32bit part); filename='/usr/share/zoneinfo/<wbr>UTC'</div><div>[2017-12-14T23:04:21.671696] Add path to classpath: /usr/lib/syslog-ng/3.13/java-<wbr>modules/syslog-ng-core.jar;</div><div>[2017-12-14T23:04:21.672418] Add path to classpath: /usr/share/elasticsearch/lib/;</div><div>[2017-12-14T23:04:21.673641] Add path to classpath: /usr/lib/syslog-ng/3.13/java-<wbr>modules/syslog-ng-core.jar;</div><div>[2017-12-14T23:04:21.673912] Add path to classpath: /usr/lib/syslog-ng/3.13/java-<wbr>modules/syslog-ng-common.jar;</div><div>[2017-12-14T23:04:21.674218] Add path to classpath: /usr/lib/syslog-ng/3.13/java-<wbr>modules/hdfs.jar;</div><div>[2017-12-14T23:04:21.674704] Add path to classpath: /usr/lib/syslog-ng/3.13/java-<wbr>modules/http.jar;</div><div>[2017-12-14T23:04:21.675858] Add path to classpath: /usr/lib/syslog-ng/3.13/java-<wbr>modules/kafka.jar;</div><div>[2017-12-14T23:04:21.676116] Add path to classpath: /usr/lib/syslog-ng/3.13/java-<wbr>modules/log4j-1.2.16.jar;</div><div>[2017-12-14T23:04:21.676322] Add path to classpath: /usr/lib/syslog-ng/3.13/java-<wbr>modules/elastic-v2.jar;</div><div>[2017-12-14T23:04:21.676484] Add path to classpath: /usr/lib/syslog-ng/3.13/java-<wbr>modules/elastic.jar;</div><div>[2017-12-14T23:04:21.741649] Add path to classpath: /usr/lib/syslog-ng/3.13/java-<wbr>modules/syslog-ng-core.jar;</div><div>[2017-12-14T23:04:21.746168] Error initializing message pipeline; plugin name='java', location='#buffer:2:3'</div><br></div><div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;background-color:rgba(0,0,0,0)"><br></div><div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;background-color:rgba(0,0,0,0)">Any suggestions ?</div><div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;background-color:rgba(0,0,0,0)"><br></div><div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;background-color:rgba(0,0,0,0)">R!</div><div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;background-color:rgba(0,0,0,0)"><br></div></div><span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;display:inline!important">______________________________<wbr>______________________________<wbr>__________________</span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;display:inline!important">Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/<wbr>mailman/listinfo/syslog-ng</a></span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;display:inline!important">Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/<wbr>support/documentation/?<wbr>product=syslog-ng</a></span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;display:inline!important">FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/<wbr>syslog-ng-faq</a></span></div></blockquote></div><br></div></div>______________________________<wbr>______________________________<wbr>__________________<br>Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/<wbr>mailman/listinfo/syslog-ng</a><br>Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/<wbr>support/documentation/?<wbr>product=syslog-ng</a><br>FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/<wbr>syslog-ng-faq</a><br><br></div></blockquote></div><br></div></div></div><br>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/<wbr>mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/<wbr>support/documentation/?<wbr>product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/<wbr>syslog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div>