Error initializing message pipeline;
Hello, First time poster, go easy! :) I’m using syslog-ng for network logging and have been fine running it thus far with no problems. Having decided to go that step further and use Elasticsearch 5.1.1 and Kibana for visualisation, I’ve managed to get both those (along with nginx) working, but currently have an issue with starting syslog-ng. I think it’s java and associated libraries, but according to documentation I’ve read so far I can’t seem to work out what isn’t working correctly/where I should be pathing correct .jar files from? Yum installed from czanik-syslog-ng39-epel-7.repo [root@SOMESERVER syslog-ng]# syslog-ng -V syslog-ng 3.9.1 Installer-Version: 3.9.1 Revision: Module-Directory: //usr/lib64/syslog-ng Module-Path: //usr/lib64/syslog-ng Available-Modules: add-contextual-data,afamqp,affile,afprog,afsocket,afstomp,afuser,basicfuncs,cef,confgen,cryptofuncs,csvparser,date,dbparser,disk-buffer,graphite,json-plugin,kvformat,linux-kmsg-format,pseudofile,sdjournal,syslogformat,system-source,mod-java Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on [root@SOMESERVER syslog-ng]# java -version java version "1.8.0_112" Java(TM) SE Runtime Environment (build 1.8.0_112-b15) Java HotSpot(TM) 64-Bit Server VM (build 25.112-b15, mixed mode) [root@SOMESERVER syslog-ng]# set | grep LD LD_LIBRARY_PATH=/usr/java/jre1.8.0_112/lib/amd64/server Config:- @version:3.9 @include "scl.conf" @module mod-java options { threaded(yes); }; source s_syslog { udp(ip(0.0.0.0) port(514)); }; destination d_elastic { elasticsearch2( index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("test") cluster("someserver") client-mode("transport") template("$(format-json --scope rfc5424 --scope nv-pairs --exclude DATE --key ISODATE)") ); }; log { source(s_syslog); destination(d_elastic); flags(flow-control); }; Output:- [root@SOMESERVER syslog-ng]# syslog-ng -Fevd -f /etc/syslog-ng/syslog-ng.conf [2017-01-17T14:43:33.704051] Systemd is detected as the running init system; [2017-01-17T14:43:33.704988] Starting to read include file; filename='/etc/syslog-ng/scl.conf', depth='1' [2017-01-17T14:43:33.705231] Adding include file; filename='/usr/share/syslog-ng/include/scl/apache/apache.conf', depth='2' [2017-01-17T14:43:33.705240] Adding include file; filename='/usr/share/syslog-ng/include/scl/cim/template.conf', depth='2' [2017-01-17T14:43:33.705244] Adding include file; filename='/usr/share/syslog-ng/include/scl/elasticsearch/plugin.conf', depth='2' [2017-01-17T14:43:33.705249] Adding include file; filename='/usr/share/syslog-ng/include/scl/graphite/plugin.conf', depth='2' [2017-01-17T14:43:33.705253] Adding include file; filename='/usr/share/syslog-ng/include/scl/hdfs/plugin.conf', depth='2' [2017-01-17T14:43:33.705257] Adding include file; filename='/usr/share/syslog-ng/include/scl/kafka/plugin.conf', depth='2' [2017-01-17T14:43:33.705261] Adding include file; filename='/usr/share/syslog-ng/include/scl/loggly/loggly.conf', depth='2' [2017-01-17T14:43:33.705266] Adding include file; filename='/usr/share/syslog-ng/include/scl/logmatic/logmatic.conf', depth='2' [2017-01-17T14:43:33.705270] Adding include file; filename='/usr/share/syslog-ng/include/scl/mbox/mbox.conf', depth='2' [2017-01-17T14:43:33.705274] Adding include file; filename='/usr/share/syslog-ng/include/scl/nodejs/plugin.conf', depth='2' [2017-01-17T14:43:33.705278] Adding include file; filename='/usr/share/syslog-ng/include/scl/pacct/plugin.conf', depth='2' [2017-01-17T14:43:33.705282] Adding include file; filename='/usr/share/syslog-ng/include/scl/rewrite/cc-mask.conf', depth='2' [2017-01-17T14:43:33.705286] Adding include file; filename='/usr/share/syslog-ng/include/scl/solaris/plugin.conf', depth='2' [2017-01-17T14:43:33.705290] Adding include file; filename='/usr/share/syslog-ng/include/scl/syslogconf/plugin.conf', depth='2' [2017-01-17T14:43:33.705294] Adding include file; filename='/usr/share/syslog-ng/include/scl/system/plugin.conf', depth='2' [2017-01-17T14:43:33.705302] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/apache/apache.conf', depth='2' [2017-01-17T14:43:33.705357] Reading path for candidate modules; path='//usr/lib64/syslog-ng' [2017-01-17T14:43:33.705387] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='add-contextual-data.so', module='add-contextual-data' [2017-01-17T14:43:33.705607] Registering candidate plugin; module='add-contextual-data', context='parser', name='add_contextual_data', preference='0' [2017-01-17T14:43:33.705643] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='afamqp.so', module='afamqp' [2017-01-17T14:43:33.705806] Registering candidate plugin; module='afamqp', context='destination', name='amqp', preference='0' [2017-01-17T14:43:33.705836] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='affile.so', module='affile' [2017-01-17T14:43:33.706009] Registering candidate plugin; module='affile', context='source', name='file', preference='0' [2017-01-17T14:43:33.706019] Registering candidate plugin; module='affile', context='source', name='pipe', preference='0' [2017-01-17T14:43:33.706024] Registering candidate plugin; module='affile', context='destination', name='file', preference='0' [2017-01-17T14:43:33.706029] Registering candidate plugin; module='affile', context='destination', name='pipe', preference='0' [2017-01-17T14:43:33.706056] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='afprog.so', module='afprog' [2017-01-17T14:43:33.706207] Registering candidate plugin; module='afprog', context='source', name='program', preference='0' [2017-01-17T14:43:33.706216] Registering candidate plugin; module='afprog', context='destination', name='program', preference='0' [2017-01-17T14:43:33.706240] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='afsocket.so', module='afsocket' [2017-01-17T14:43:33.706560] Registering candidate plugin; module='afsocket', context='source', name='unix-stream', preference='100' [2017-01-17T14:43:33.706570] Registering candidate plugin; module='afsocket', context='destination', name='unix-stream', preference='100' [2017-01-17T14:43:33.706575] Registering candidate plugin; module='afsocket', context='source', name='unix-dgram', preference='100' [2017-01-17T14:43:33.706580] Registering candidate plugin; module='afsocket', context='destination', name='unix-dgram', preference='100' [2017-01-17T14:43:33.706585] Registering candidate plugin; module='afsocket', context='source', name='tcp', preference='100' [2017-01-17T14:43:33.706590] Registering candidate plugin; module='afsocket', context='destination', name='tcp', preference='100' [2017-01-17T14:43:33.706596] Registering candidate plugin; module='afsocket', context='source', name='tcp6', preference='100' [2017-01-17T14:43:33.706601] Registering candidate plugin; module='afsocket', context='destination', name='tcp6', preference='100' [2017-01-17T14:43:33.706605] Registering candidate plugin; module='afsocket', context='source', name='udp', preference='100' [2017-01-17T14:43:33.706610] Registering candidate plugin; module='afsocket', context='destination', name='udp', preference='100' [2017-01-17T14:43:33.706614] Registering candidate plugin; module='afsocket', context='source', name='udp6', preference='100' [2017-01-17T14:43:33.706619] Registering candidate plugin; module='afsocket', context='destination', name='udp6', preference='100' [2017-01-17T14:43:33.706624] Registering candidate plugin; module='afsocket', context='source', name='syslog', preference='100' [2017-01-17T14:43:33.706628] Registering candidate plugin; module='afsocket', context='destination', name='syslog', preference='100' [2017-01-17T14:43:33.706807] Registering candidate plugin; module='afsocket', context='source', name='network', preference='100' [2017-01-17T14:43:33.706815] Registering candidate plugin; module='afsocket', context='destination', name='network', preference='100' [2017-01-17T14:43:33.706819] Registering candidate plugin; module='afsocket', context='source', name='systemd-syslog', preference='100' [2017-01-17T14:43:33.706927] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='afstomp.so', module='afstomp' [2017-01-17T14:43:33.707133] Registering candidate plugin; module='afstomp', context='destination', name='stomp', preference='0' [2017-01-17T14:43:33.707195] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='afuser.so', module='afuser' [2017-01-17T14:43:33.707350] Registering candidate plugin; module='afuser', context='destination', name='usertty', preference='0' [2017-01-17T14:43:33.707379] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='basicfuncs.so', module='basicfuncs' [2017-01-17T14:43:33.707552] Registering candidate plugin; module='basicfuncs', context='template-func', name='grep', preference='0' [2017-01-17T14:43:33.707561] Registering candidate plugin; module='basicfuncs', context='template-func', name='if', preference='0' [2017-01-17T14:43:33.707566] Registering candidate plugin; module='basicfuncs', context='template-func', name='or', preference='0' [2017-01-17T14:43:33.707570] Registering candidate plugin; module='basicfuncs', context='template-func', name='echo', preference='0' [2017-01-17T14:43:33.707575] Registering candidate plugin; module='basicfuncs', context='template-func', name='length', preference='0' [2017-01-17T14:43:33.707584] Registering candidate plugin; module='basicfuncs', context='template-func', name='substr', preference='0' [2017-01-17T14:43:33.707589] Registering candidate plugin; module='basicfuncs', context='template-func', name='strip', preference='0' [2017-01-17T14:43:33.707594] Registering candidate plugin; module='basicfuncs', context='template-func', name='sanitize', preference='0' [2017-01-17T14:43:33.707598] Registering candidate plugin; module='basicfuncs', context='template-func', name='lowercase', preference='0' [2017-01-17T14:43:33.707603] Registering candidate plugin; module='basicfuncs', context='template-func', name='uppercase', preference='0' [2017-01-17T14:43:33.707608] Registering candidate plugin; module='basicfuncs', context='template-func', name='replace-delimiter', preference='0' [2017-01-17T14:43:33.707612] Registering candidate plugin; module='basicfuncs', context='template-func', name='padding', preference='0' [2017-01-17T14:43:33.707617] Registering candidate plugin; module='basicfuncs', context='template-func', name='+', preference='0' [2017-01-17T14:43:33.707622] Registering candidate plugin; module='basicfuncs', context='template-func', name='-', preference='0' [2017-01-17T14:43:33.707626] Registering candidate plugin; module='basicfuncs', context='template-func', name='*', preference='0' [2017-01-17T14:43:33.707631] Registering candidate plugin; module='basicfuncs', context='template-func', name='/', preference='0' [2017-01-17T14:43:33.707635] Registering candidate plugin; module='basicfuncs', context='template-func', name='%', preference='0' [2017-01-17T14:43:33.707639] Registering candidate plugin; module='basicfuncs', context='template-func', name='sum', preference='0' [2017-01-17T14:43:33.707644] Registering candidate plugin; module='basicfuncs', context='template-func', name='min', preference='0' [2017-01-17T14:43:33.707648] Registering candidate plugin; module='basicfuncs', context='template-func', name='max', preference='0' [2017-01-17T14:43:33.707653] Registering candidate plugin; module='basicfuncs', context='template-func', name='average', preference='0' [2017-01-17T14:43:33.707657] Registering candidate plugin; module='basicfuncs', context='template-func', name='ipv4-to-int', preference='0' [2017-01-17T14:43:33.707662] Registering candidate plugin; module='basicfuncs', context='template-func', name='indent-multi-line', preference='0' [2017-01-17T14:43:33.707794] Registering candidate plugin; module='basicfuncs', context='template-func', name='context-length', preference='0' [2017-01-17T14:43:33.707803] Registering candidate plugin; module='basicfuncs', context='template-func', name='env', preference='0' [2017-01-17T14:43:33.707808] Registering candidate plugin; module='basicfuncs', context='template-func', name='template', preference='0' [2017-01-17T14:43:33.707836] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='cef.so', module='cef' [2017-01-17T14:43:33.707990] Registering candidate plugin; module='cef', context='template-func', name='format-cef-extension', preference='0' [2017-01-17T14:43:33.708035] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='confgen.so', module='confgen' [2017-01-17T14:43:33.708183] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='cryptofuncs.so', module='cryptofuncs' [2017-01-17T14:43:33.708327] Registering candidate plugin; module='cryptofuncs', context='template-func', name='uuid', preference='0' [2017-01-17T14:43:33.708336] Registering candidate plugin; module='cryptofuncs', context='template-func', name='hash', preference='0' [2017-01-17T14:43:33.708341] Registering candidate plugin; module='cryptofuncs', context='template-func', name='sha1', preference='0' [2017-01-17T14:43:33.708346] Registering candidate plugin; module='cryptofuncs', context='template-func', name='sha256', preference='0' [2017-01-17T14:43:33.708350] Registering candidate plugin; module='cryptofuncs', context='template-func', name='sha512', preference='0' [2017-01-17T14:43:33.708355] Registering candidate plugin; module='cryptofuncs', context='template-func', name='md4', preference='0' [2017-01-17T14:43:33.708359] Registering candidate plugin; module='cryptofuncs', context='template-func', name='md5', preference='0' [2017-01-17T14:43:33.708381] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='csvparser.so', module='csvparser' [2017-01-17T14:43:33.708530] Registering candidate plugin; module='csvparser', context='parser', name='csv-parser', preference='0' [2017-01-17T14:43:33.708557] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='date.so', module='date' [2017-01-17T14:43:33.708697] Registering candidate plugin; module='date', context='parser', name='date-parser', preference='0' [2017-01-17T14:43:33.708724] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='dbparser.so', module='dbparser' [2017-01-17T14:43:33.708920] Registering candidate plugin; module='dbparser', context='parser', name='db-parser', preference='0' [2017-01-17T14:43:33.708929] Registering candidate plugin; module='dbparser', context='parser', name='grouping-by', preference='0' [2017-01-17T14:43:33.708965] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='disk-buffer.so', module='disk-buffer' [2017-01-17T14:43:33.709124] Registering candidate plugin; module='disk-buffer', context='inner-dest', name='disk_buffer', preference='0' [2017-01-17T14:43:33.709157] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='graphite.so', module='graphite' [2017-01-17T14:43:33.709293] Registering candidate plugin; module='graphite', context='template-func', name='graphite_output', preference='0' [2017-01-17T14:43:33.709318] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='json-plugin.so', module='json-plugin' [2017-01-17T14:43:33.709526] Registering candidate plugin; module='json-plugin', context='parser', name='json-parser', preference='0' [2017-01-17T14:43:33.709537] Registering candidate plugin; module='json-plugin', context='template-func', name='format_json', preference='0' [2017-01-17T14:43:33.709568] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='kvformat.so', module='kvformat' [2017-01-17T14:43:33.709711] Registering candidate plugin; module='kvformat', context='parser', name='kv-parser', preference='0' [2017-01-17T14:43:33.709720] Registering candidate plugin; module='kvformat', context='parser', name='linux-audit-parser', preference='0' [2017-01-17T14:43:33.709822] Registering candidate plugin; module='kvformat', context='template-func', name='format-welf', preference='0' [2017-01-17T14:43:33.709886] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='linux-kmsg-format.so', module='linux-kmsg-format' [2017-01-17T14:43:33.710060] Registering candidate plugin; module='linux-kmsg-format', context='format', name='linux-kmsg', preference='0' [2017-01-17T14:43:33.710088] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='pseudofile.so', module='pseudofile' [2017-01-17T14:43:33.710231] Registering candidate plugin; module='pseudofile', context='destination', name='pseudofile', preference='0' [2017-01-17T14:43:33.710256] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='sdjournal.so', module='sdjournal' [2017-01-17T14:43:33.710396] Registering candidate plugin; module='sdjournal', context='source', name='systemd-journal', preference='0' [2017-01-17T14:43:33.710422] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='syslogformat.so', module='syslogformat' [2017-01-17T14:43:33.710578] Registering candidate plugin; module='syslogformat', context='format', name='syslog', preference='0' [2017-01-17T14:43:33.710591] Registering candidate plugin; module='syslogformat', context='parser', name='syslog-parser', preference='0' [2017-01-17T14:43:33.710625] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='system-source.so', module='system-source' [2017-01-17T14:43:33.710783] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='mod-java.so', module='mod-java' [2017-01-17T14:43:33.712715] Registering candidate plugin; module='mod-java', context='destination', name='java', preference='0' [2017-01-17T14:43:33.712978] Finishing include; filename='/usr/share/syslog-ng/include/scl/apache/apache.conf', depth='2' [2017-01-17T14:43:33.713009] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/cim/template.conf', depth='2' [2017-01-17T14:43:33.713309] Module loaded and initialized successfully; module='json-plugin' [2017-01-17T14:43:33.713473] Finishing include; filename='/usr/share/syslog-ng/include/scl/cim/template.conf', depth='2' [2017-01-17T14:43:33.713496] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/elasticsearch/plugin.conf', depth='2' [2017-01-17T14:43:33.713674] Finishing include; filename='/usr/share/syslog-ng/include/scl/elasticsearch/plugin.conf', depth='2' [2017-01-17T14:43:33.713693] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/graphite/plugin.conf', depth='2' [2017-01-17T14:43:33.713773] Finishing include; filename='/usr/share/syslog-ng/include/scl/graphite/plugin.conf', depth='2' [2017-01-17T14:43:33.713793] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/hdfs/plugin.conf', depth='2' [2017-01-17T14:43:33.713880] Finishing include; filename='/usr/share/syslog-ng/include/scl/hdfs/plugin.conf', depth='2' [2017-01-17T14:43:33.713898] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/kafka/plugin.conf', depth='2' [2017-01-17T14:43:33.713997] Finishing include; filename='/usr/share/syslog-ng/include/scl/kafka/plugin.conf', depth='2' [2017-01-17T14:43:33.714017] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/loggly/loggly.conf', depth='2' [2017-01-17T14:43:33.714133] Finishing include; filename='/usr/share/syslog-ng/include/scl/loggly/loggly.conf', depth='2' [2017-01-17T14:43:33.714151] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/logmatic/logmatic.conf', depth='2' [2017-01-17T14:43:33.714267] Finishing include; filename='/usr/share/syslog-ng/include/scl/logmatic/logmatic.conf', depth='2' [2017-01-17T14:43:33.714286] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/mbox/mbox.conf', depth='2' [2017-01-17T14:43:33.714350] Finishing include; filename='/usr/share/syslog-ng/include/scl/mbox/mbox.conf', depth='2' [2017-01-17T14:43:33.714369] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/nodejs/plugin.conf', depth='2' [2017-01-17T14:43:33.714440] Finishing include; filename='/usr/share/syslog-ng/include/scl/nodejs/plugin.conf', depth='2' [2017-01-17T14:43:33.714458] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/pacct/plugin.conf', depth='2' [2017-01-17T14:43:33.714521] Finishing include; filename='/usr/share/syslog-ng/include/scl/pacct/plugin.conf', depth='2' [2017-01-17T14:43:33.714538] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/rewrite/cc-mask.conf', depth='2' [2017-01-17T14:43:33.714611] Global value changed; define='balabit.credit-card-regexp', value='(?P<1>:4[0-9]{12}(?:[0-9]{3})?|5[1-5][0-9]{14}|6(?:011|5[0-9][0-9])[0-9]{12}|3[47][0-9]{13}|3(?:0[0-5]|[68][0-9])[0-9]{11}|(?:2131|1800|35d{3})d{11})' [2017-01-17T14:43:33.714655] Finishing include; filename='/usr/share/syslog-ng/include/scl/rewrite/cc-mask.conf', depth='2' [2017-01-17T14:43:33.714673] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/solaris/plugin.conf', depth='2' [2017-01-17T14:43:33.714994] Finishing include; filename='/usr/share/syslog-ng/include/scl/solaris/plugin.conf', depth='2' [2017-01-17T14:43:33.715018] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/syslogconf/plugin.conf', depth='2' [2017-01-17T14:43:33.715254] Module loaded and initialized successfully; module='confgen' [2017-01-17T14:43:33.715268] Finishing include; filename='/usr/share/syslog-ng/include/scl/syslogconf/plugin.conf', depth='2' [2017-01-17T14:43:33.715287] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/system/plugin.conf', depth='2' [2017-01-17T14:43:33.715490] Module loaded and initialized successfully; module='system-source' [2017-01-17T14:43:33.715503] Finishing include; filename='/usr/share/syslog-ng/include/scl/system/plugin.conf', depth='2' [2017-01-17T14:43:33.715521] Finishing include; filename='/etc/syslog-ng/scl.conf', depth='1' [2017-01-17T14:43:33.716821] Module loaded and initialized successfully; module='mod-java' [2017-01-17T14:43:33.717202] Module loaded and initialized successfully; module='afsocket' [2017-01-17T14:43:33.717529] Finishing include; content='destination block elasticsearch2', depth='1' [2017-01-17T14:43:33.717869] Compiling #unnamed sequence [log] at [/etc/syslog-ng/syslog-ng.conf:23:3] [2017-01-17T14:43:33.717877] Compiling s_syslog reference [source] at [/etc/syslog-ng/syslog-ng.conf:23:3] [2017-01-17T14:43:33.717881] Compiling s_syslog sequence [source] at [/etc/syslog-ng/syslog-ng.conf:10:1] [2017-01-17T14:43:33.717884] Compiling #unnamed junction [log] at [/etc/syslog-ng/syslog-ng.conf:10:18] [2017-01-17T14:43:33.717887] Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf:10:19] [2017-01-17T14:43:33.717895] Compiling d_elastic reference [destination] at [/etc/syslog-ng/syslog-ng.conf:24:3] [2017-01-17T14:43:33.717899] Compiling d_elastic sequence [destination] at [/etc/syslog-ng/syslog-ng.conf:12:1] [2017-01-17T14:43:33.717902] Compiling #unnamed junction [log] at [/etc/syslog-ng/syslog-ng.conf:12:24] [2017-01-17T14:43:33.717905] Compiling #unnamed single [log] at [#buffer:2:3] [2017-01-17T14:43:33.718096] Module loaded and initialized successfully; module='syslogformat' [2017-01-17T14:43:33.824194] Add path to classpath: //usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar; [2017-01-17T14:43:33.824696] Add path to classpath: /usr/lib64/syslog-ng/java-modules/dummy.jar; [2017-01-17T14:43:33.824868] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-v2.jar; [2017-01-17T14:43:33.825032] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic.jar; [2017-01-17T14:43:33.825169] Add path to classpath: /usr/lib64/syslog-ng/java-modules/hdfs.jar; [2017-01-17T14:43:33.825322] Add path to classpath: /usr/lib64/syslog-ng/java-modules/http.jar; [2017-01-17T14:43:33.825470] Add path to classpath: /usr/lib64/syslog-ng/java-modules/kafka.jar; [2017-01-17T14:43:33.825629] Add path to classpath: /usr/lib64/syslog-ng/java-modules/log4j-1.2.16.jar; [2017-01-17T14:43:33.825781] Add path to classpath: /usr/lib64/syslog-ng/java-modules/syslog-ng-common.jar; [2017-01-17T14:43:33.825936] Add path to classpath: /usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar; [2017-01-17T14:43:33.826108] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-jest-client/commons-codec-1.9.jar; [2017-01-17T14:43:33.826286] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-jest-client/commons-lang3-3.4.jar; [2017-01-17T14:43:33.826459] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-jest-client/commons-logging-1.2.jar; [2017-01-17T14:43:33.826633] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-jest-client/gson-2.6.2.jar; [2017-01-17T14:43:33.826801] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-jest-client/guava-19.0.jar; [2017-01-17T14:43:33.826981] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-jest-client/httpasyncclient-4.1.1.jar; [2017-01-17T14:43:33.827156] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-jest-client/httpclient-4.5.2.jar; [2017-01-17T14:43:33.827343] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-jest-client/httpcore-4.4.4.jar; [2017-01-17T14:43:33.827520] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-jest-client/httpcore-nio-4.4.4.jar; [2017-01-17T14:43:33.827694] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-jest-client/jest-2.0.2.jar; [2017-01-17T14:43:33.827851] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-jest-client/jest-common-2.0.2.jar; [2017-01-17T14:43:33.828061] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-jest-client/slf4j-api-1.7.13.jar; [2017-01-17T14:43:33.828225] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-jest-client/slf4j-simple-1.7.13.jar; [2017-01-17T14:43:33.901817] Add path to classpath: //usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar; [2017-01-17T14:43:33.906737] Error initializing message pipeline; I’m sure I’m missing something basic – perhaps correct pathing of the ES .jar files? Many thanks for your help in advance. Damian Bell Damian Bell Infrastructure Engineer | Support | H Clarkson & Co Ltd Email: Damian.Bell@clarksons.com<mailto:Damian.Bell@clarksons.com> Group Email: infrastructure@clarksons.com Clarksons Platou TM Commodity Quay, St. Katharine Docks | London E1W 1BF | United Kingdom www.clarksons.com<http://www.clarksons.com> Please consider the environment before printing this e-mail ________________________________ This message is private and confidential. If you have received it in error, you are on notice of its status. Please notify us immediately by reply email and then delete this message from your system. Please do not copy it or use it for any purposes, or disclose its contents to any other person: to do so could be a breach of confidence. Emails may be monitored. Details of Clarkson group companies and their regulators (where applicable) can be found at this url: Disclosure<http://www.clarksons.com/disclosure/> ________________________________
Hi Damian, You need to specify the location to your elasticsearch installation, i.e. where the .jar files are installed. If you're using the official packages from elastic.co, they are most likely located here: /usr/share/elasticsearch/lib/ So your config ought to look like the following instead: source s_syslog { udp(ip(0.0.0.0) port(514)); }; destination d_elastic { elasticsearch2( client-lib-dir("/usr/share/elasticsearch/lib/") index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("test") cluster("someserver") client-mode("transport") template("$(format-json --scope rfc5424 --scope nv-pairs --exclude DATE --key ISODATE)") time-zone("UTC") ); }; Moreover, you might want to set the destination's timezone to UTC too, or you'll have surprises in kibana around midnight UTC: time-zone("UTC")
Is client-mode("transport") now supported with ES 5.1? I thought it was only http mode for ES 5. I got pipeline error then switched to http thinking it was the transport mode. http worked fine. On Tue, Jan 17, 2017 at 9:58 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi Damian,
You need to specify the location to your elasticsearch installation, i.e. where the .jar files are installed. If you're using the official packages from elastic.co, they are most likely located here: /usr/share/elasticsearch/lib/
So your config ought to look like the following instead:
source s_syslog { udp(ip(0.0.0.0) port(514)); };
destination d_elastic { elasticsearch2( client-lib-dir("/usr/share/elasticsearch/lib/") index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("test") cluster("someserver") client-mode("transport") template("$(format-json --scope rfc5424 --scope nv-pairs --exclude DATE --key ISODATE)") time-zone("UTC") ); };
Moreover, you might want to set the destination's timezone to UTC too, or you'll have surprises in kibana around midnight UTC: time-zone("UTC")
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
hi, we support ES5.x only via http mode. regards, Laszlo Budai _____________________________ From: Scot <scotrn@gmail.com<mailto:scotrn@gmail.com>> Sent: Wednesday, January 18, 2017 3:33 AM Subject: Re: [syslog-ng] Error initializing message pipeline; To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>>, Fabien Wernli <wernli@in2p3.fr<mailto:wernli@in2p3.fr>> Is client-mode("transport") now supported with ES 5.1? I thought it was only http mode for ES 5. I got pipeline error then switched to http thinking it was the transport mode. http worked fine. On Tue, Jan 17, 2017 at 9:58 AM, Fabien Wernli <wernli@in2p3.fr<mailto:wernli@in2p3.fr>> wrote: Hi Damian, You need to specify the location to your elasticsearch installation, i.e. where the .jar files are installed. If you're using the official packages from elastic.co<http://elastic.co>, they are most likely located here: /usr/share/elasticsearch/lib/ So your config ought to look like the following instead: source s_syslog { udp(ip(0.0.0.0) port(514)); }; destination d_elastic { elasticsearch2( client-lib-dir("/usr/share/elasticsearch/lib/") index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("test") cluster("someserver") client-mode("transport") template("$(format-json --scope rfc5424 --scope nv-pairs --exclude DATE --key ISODATE)") time-zone("UTC") ); }; Moreover, you might want to set the destination's timezone to UTC too, or you'll have surprises in kibana around midnight UTC: time-zone("UTC") ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Worked out http mode only supported not long after receiving the mail from Fabien, and now all working (thanks!). A more “generalised” question – I’ve used syslog-ng for years as a network engineer to receive Cisco network device input and output it to file, both as individual host data and also a collected “all” file on which I’ve used a very simple “swatch” implementation to both screen out noise, and also highlight interesting network events (routing convergence etc), as per config below. I’d like to replicate this somewhat with the syslog-ng/ES/Kibana build I now have, but I’m wondering the best way of doing it – should I filter “non-interesting” traffic at the syslog-ng level (if so, what is the best practice?) or do so at the Kibana level? In terms of transportation from syslog-ng into ES, does anyone have any tips or pointers as to the best way of formatting Cisco switch/firewall/router logs to best be utilised within ES/Kibana? Thank you very much in advance. Damian Bell Infrastructure Engineer | Support | H Clarkson & Co Ltd Email: Damian.Bell@clarksons.com<mailto:Damian.Bell@clarksons.com> From: syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Laszlo Budai Sent: 18 January 2017 05:40 To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu>; Fabien Wernli <wernli@in2p3.fr> Subject: Re: [syslog-ng] Error initializing message pipeline; hi, we support ES5.x only via http mode. regards, Laszlo Budai _____________________________ From: Scot <scotrn@gmail.com<mailto:scotrn@gmail.com>> Sent: Wednesday, January 18, 2017 3:33 AM Subject: Re: [syslog-ng] Error initializing message pipeline; To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>>, Fabien Wernli <wernli@in2p3.fr<mailto:wernli@in2p3.fr>> Is client-mode("transport") now supported with ES 5.1? I thought it was only http mode for ES 5. I got pipeline error then switched to http thinking it was the transport mode. http worked fine. On Tue, Jan 17, 2017 at 9:58 AM, Fabien Wernli <wernli@in2p3.fr<mailto:wernli@in2p3.fr>> wrote: Hi Damian, You need to specify the location to your elasticsearch installation, i.e. where the .jar files are installed. If you're using the official packages from elastic.co<http://elastic.co>, they are most likely located here: /usr/share/elasticsearch/lib/ So your config ought to look like the following instead: source s_syslog { udp(ip(0.0.0.0) port(514)); }; destination d_elastic { elasticsearch2( client-lib-dir("/usr/share/elasticsearch/lib/") index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("test") cluster("someserver") client-mode("transport") template("$(format-json --scope rfc5424 --scope nv-pairs --exclude DATE --key ISODATE)") time-zone("UTC") ); }; Moreover, you might want to set the destination's timezone to UTC too, or you'll have surprises in kibana around midnight UTC: time-zone("UTC") ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ________________________________ This message is private and confidential. If you have received it in error, you are on notice of its status. Please notify us immediately by reply email and then delete this message from your system. Please do not copy it or use it for any purposes, or disclose its contents to any other person: to do so could be a breach of confidence. Emails may be monitored. Details of Clarkson group companies and their regulators (where applicable) can be found at this url: Disclosure<http://www.clarksons.com/disclosure/> ________________________________
Hi Damian, On Fri, Jan 20, 2017 at 02:02:53PM +0000, Damian Bell wrote:
“non-interesting” traffic at the syslog-ng level (if so, what is the best practice?) or do so at the Kibana level? In terms of transportation from syslog-ng into ES, does anyone have any tips or pointers as to the best way of formatting Cisco switch/firewall/router logs to best be utilised within ES/Kibana?
I'd say it depends on your ability to store the full monty. If you can afford it, then you'll be happy to use the REST API to filter out relevant results. If you can't, you'll have to filter out stuff upstream using syslog-ng, and use ES to query what's left. That being said, in any case you'll want the data to be in a nice structured format, so that you can easily and efficiently filter it using either syslog-ng or kibana. And for that to work, you'll have to parse the lot, because you can't rely on regexp or lucene full-text searches for everything (unless you're extremely rich and have exabytes of SSDs lying around). So my suggested plan would be: 1. write parsers in syslog-ng to structure your logs into name/value pairs - patterndb - csv - kv-parser - python - … 2. use kibana AND/OR syslog_ng to filter using the key/value pairs you created 3. goto 1. We started out using patterndb and are extremely happy with it. But today you have a large choice of available parsers to extract keys with. Today we use a combination of many, and to choose one for a given use-case we usually try with patterndb, csv or kv-parser, and for anything more complex we use python. I'd be happy to share any particular use-case with you. Cheers
Fabian, I would be interested in seeing your patterndb file for the cisco logs. Our problem is that the cisco devices don't really log a program name, which makes using patterndb quite difficult. Can you share in or out of band? Evan On 01/20/2017 07:04 AM, Fabien Wernli wrote:
Hi Damian,
On Fri, Jan 20, 2017 at 02:02:53PM +0000, Damian Bell wrote:
“non-interesting” traffic at the syslog-ng level (if so, what is the best practice?) or do so at the Kibana level? In terms of transportation from syslog-ng into ES, does anyone have any tips or pointers as to the best way of formatting Cisco switch/firewall/router logs to best be utilised within ES/Kibana? I'd say it depends on your ability to store the full monty. If you can afford it, then you'll be happy to use the REST API to filter out relevant results. If you can't, you'll have to filter out stuff upstream using syslog-ng, and use ES to query what's left.
That being said, in any case you'll want the data to be in a nice structured format, so that you can easily and efficiently filter it using either syslog-ng or kibana. And for that to work, you'll have to parse the lot, because you can't rely on regexp or lucene full-text searches for everything (unless you're extremely rich and have exabytes of SSDs lying around).
So my suggested plan would be:
1. write parsers in syslog-ng to structure your logs into name/value pairs - patterndb - csv - kv-parser - python - … 2. use kibana AND/OR syslog_ng to filter using the key/value pairs you created 3. goto 1.
We started out using patterndb and are extremely happy with it. But today you have a large choice of available parsers to extract keys with. Today we use a combination of many, and to choose one for a given use-case we usually try with patterndb, csv or kv-parser, and for anything more complex we use python.
I'd be happy to share any particular use-case with you.
Cheers
Hi Evan, On Fri, Jan 20, 2017 at 07:07:53AM -0800, Evan Rempel wrote:
Fabian, I would be interested in seeing your patterndb file for the cisco logs. Our problem is that the cisco devices don't really log a program name, which makes using patterndb quite difficult.
Can you share in or out of band?
No I can't, as we don't really collect cisco logs :-) My answer was more targetting a general approach, sorry for raising your hopes in vain! That being said, I'd love to know more about the specific problem you're having, as we might well collect cisco logs in the future and I'd be glad to help. Cheers
participants (5)
-
Damian Bell
-
Evan Rempel
-
Fabien Wernli
-
Laszlo Budai
-
Scot