- syslog-ng - lists.balabit.hu

Error initializing message pipeline;
by Damian Bell 20 Jan '17

20 Jan '17

Hello, First time poster, go easy! :) I’m using syslog-ng for network logging and have been fine running it thus far with no problems. Having decided to go that step further and use Elasticsearch 5.1.1 and Kibana for visualisation, I’ve managed to get both those (along with nginx) working, but currently have an issue with starting syslog-ng. I think it’s java and associated libraries, but according to documentation I’ve read so far I can’t seem to work out what isn’t working correctly/where I should be pathing correct .jar files from? Yum installed from czanik-syslog-ng39-epel-7.repo [root@SOMESERVER syslog-ng]# syslog-ng -V syslog-ng 3.9.1 Installer-Version: 3.9.1 Revision: Module-Directory: //usr/lib64/syslog-ng Module-Path: //usr/lib64/syslog-ng Available-Modules: add-contextual-data,afamqp,affile,afprog,afsocket,afstomp,afuser,basicfuncs,cef,confgen,cryptofuncs,csvparser,date,dbparser,disk-buffer,graphite,json-plugin,kvformat,linux-kmsg-format,pseudofile,sdjournal,syslogformat,system-source,mod-java Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on [root@SOMESERVER syslog-ng]# java -version java version "1.8.0_112" Java(TM) SE Runtime Environment (build 1.8.0_112-b15) Java HotSpot(TM) 64-Bit Server VM (build 25.112-b15, mixed mode) [root@SOMESERVER syslog-ng]# set | grep LD LD_LIBRARY_PATH=/usr/java/jre1.8.0_112/lib/amd64/server Config:- @version:3.9 @include "scl.conf" @module mod-java options { threaded(yes); }; source s_syslog { udp(ip(0.0.0.0) port(514)); }; destination d_elastic { elasticsearch2( index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("test") cluster("someserver") client-mode("transport") template("$(format-json --scope rfc5424 --scope nv-pairs --exclude DATE --key ISODATE)") ); }; log { source(s_syslog); destination(d_elastic); flags(flow-control); }; Output:- [root@SOMESERVER syslog-ng]# syslog-ng -Fevd -f /etc/syslog-ng/syslog-ng.conf [2017-01-17T14:43:33.704051] Systemd is detected as the running init system; [2017-01-17T14:43:33.704988] Starting to read include file; filename='/etc/syslog-ng/scl.conf', depth='1' [2017-01-17T14:43:33.705231] Adding include file; filename='/usr/share/syslog-ng/include/scl/apache/apache.conf', depth='2' [2017-01-17T14:43:33.705240] Adding include file; filename='/usr/share/syslog-ng/include/scl/cim/template.conf', depth='2' [2017-01-17T14:43:33.705244] Adding include file; filename='/usr/share/syslog-ng/include/scl/elasticsearch/plugin.conf', depth='2' [2017-01-17T14:43:33.705249] Adding include file; filename='/usr/share/syslog-ng/include/scl/graphite/plugin.conf', depth='2' [2017-01-17T14:43:33.705253] Adding include file; filename='/usr/share/syslog-ng/include/scl/hdfs/plugin.conf', depth='2' [2017-01-17T14:43:33.705257] Adding include file; filename='/usr/share/syslog-ng/include/scl/kafka/plugin.conf', depth='2' [2017-01-17T14:43:33.705261] Adding include file; filename='/usr/share/syslog-ng/include/scl/loggly/loggly.conf', depth='2' [2017-01-17T14:43:33.705266] Adding include file; filename='/usr/share/syslog-ng/include/scl/logmatic/logmatic.conf', depth='2' [2017-01-17T14:43:33.705270] Adding include file; filename='/usr/share/syslog-ng/include/scl/mbox/mbox.conf', depth='2' [2017-01-17T14:43:33.705274] Adding include file; filename='/usr/share/syslog-ng/include/scl/nodejs/plugin.conf', depth='2' [2017-01-17T14:43:33.705278] Adding include file; filename='/usr/share/syslog-ng/include/scl/pacct/plugin.conf', depth='2' [2017-01-17T14:43:33.705282] Adding include file; filename='/usr/share/syslog-ng/include/scl/rewrite/cc-mask.conf', depth='2' [2017-01-17T14:43:33.705286] Adding include file; filename='/usr/share/syslog-ng/include/scl/solaris/plugin.conf', depth='2' [2017-01-17T14:43:33.705290] Adding include file; filename='/usr/share/syslog-ng/include/scl/syslogconf/plugin.conf', depth='2' [2017-01-17T14:43:33.705294] Adding include file; filename='/usr/share/syslog-ng/include/scl/system/plugin.conf', depth='2' [2017-01-17T14:43:33.705302] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/apache/apache.conf', depth='2' [2017-01-17T14:43:33.705357] Reading path for candidate modules; path='//usr/lib64/syslog-ng' [2017-01-17T14:43:33.705387] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='add-contextual-data.so', module='add-contextual-data' [2017-01-17T14:43:33.705607] Registering candidate plugin; module='add-contextual-data', context='parser', name='add_contextual_data', preference='0' [2017-01-17T14:43:33.705643] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='afamqp.so', module='afamqp' [2017-01-17T14:43:33.705806] Registering candidate plugin; module='afamqp', context='destination', name='amqp', preference='0' [2017-01-17T14:43:33.705836] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='affile.so', module='affile' [2017-01-17T14:43:33.706009] Registering candidate plugin; module='affile', context='source', name='file', preference='0' [2017-01-17T14:43:33.706019] Registering candidate plugin; module='affile', context='source', name='pipe', preference='0' [2017-01-17T14:43:33.706024] Registering candidate plugin; module='affile', context='destination', name='file', preference='0' [2017-01-17T14:43:33.706029] Registering candidate plugin; module='affile', context='destination', name='pipe', preference='0' [2017-01-17T14:43:33.706056] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='afprog.so', module='afprog' [2017-01-17T14:43:33.706207] Registering candidate plugin; module='afprog', context='source', name='program', preference='0' [2017-01-17T14:43:33.706216] Registering candidate plugin; module='afprog', context='destination', name='program', preference='0' [2017-01-17T14:43:33.706240] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='afsocket.so', module='afsocket' [2017-01-17T14:43:33.706560] Registering candidate plugin; module='afsocket', context='source', name='unix-stream', preference='100' [2017-01-17T14:43:33.706570] Registering candidate plugin; module='afsocket', context='destination', name='unix-stream', preference='100' [2017-01-17T14:43:33.706575] Registering candidate plugin; module='afsocket', context='source', name='unix-dgram', preference='100' [2017-01-17T14:43:33.706580] Registering candidate plugin; module='afsocket', context='destination', name='unix-dgram', preference='100' [2017-01-17T14:43:33.706585] Registering candidate plugin; module='afsocket', context='source', name='tcp', preference='100' [2017-01-17T14:43:33.706590] Registering candidate plugin; module='afsocket', context='destination', name='tcp', preference='100' [2017-01-17T14:43:33.706596] Registering candidate plugin; module='afsocket', context='source', name='tcp6', preference='100' [2017-01-17T14:43:33.706601] Registering candidate plugin; module='afsocket', context='destination', name='tcp6', preference='100' [2017-01-17T14:43:33.706605] Registering candidate plugin; module='afsocket', context='source', name='udp', preference='100' [2017-01-17T14:43:33.706610] Registering candidate plugin; module='afsocket', context='destination', name='udp', preference='100' [2017-01-17T14:43:33.706614] Registering candidate plugin; module='afsocket', context='source', name='udp6', preference='100' [2017-01-17T14:43:33.706619] Registering candidate plugin; module='afsocket', context='destination', name='udp6', preference='100' [2017-01-17T14:43:33.706624] Registering candidate plugin; module='afsocket', context='source', name='syslog', preference='100' [2017-01-17T14:43:33.706628] Registering candidate plugin; module='afsocket', context='destination', name='syslog', preference='100' [2017-01-17T14:43:33.706807] Registering candidate plugin; module='afsocket', context='source', name='network', preference='100' [2017-01-17T14:43:33.706815] Registering candidate plugin; module='afsocket', context='destination', name='network', preference='100' [2017-01-17T14:43:33.706819] Registering candidate plugin; module='afsocket', context='source', name='systemd-syslog', preference='100' [2017-01-17T14:43:33.706927] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='afstomp.so', module='afstomp' [2017-01-17T14:43:33.707133] Registering candidate plugin; module='afstomp', context='destination', name='stomp', preference='0' [2017-01-17T14:43:33.707195] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='afuser.so', module='afuser' [2017-01-17T14:43:33.707350] Registering candidate plugin; module='afuser', context='destination', name='usertty', preference='0' [2017-01-17T14:43:33.707379] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='basicfuncs.so', module='basicfuncs' [2017-01-17T14:43:33.707552] Registering candidate plugin; module='basicfuncs', context='template-func', name='grep', preference='0' [2017-01-17T14:43:33.707561] Registering candidate plugin; module='basicfuncs', context='template-func', name='if', preference='0' [2017-01-17T14:43:33.707566] Registering candidate plugin; module='basicfuncs', context='template-func', name='or', preference='0' [2017-01-17T14:43:33.707570] Registering candidate plugin; module='basicfuncs', context='template-func', name='echo', preference='0' [2017-01-17T14:43:33.707575] Registering candidate plugin; module='basicfuncs', context='template-func', name='length', preference='0' [2017-01-17T14:43:33.707584] Registering candidate plugin; module='basicfuncs', context='template-func', name='substr', preference='0' [2017-01-17T14:43:33.707589] Registering candidate plugin; module='basicfuncs', context='template-func', name='strip', preference='0' [2017-01-17T14:43:33.707594] Registering candidate plugin; module='basicfuncs', context='template-func', name='sanitize', preference='0' [2017-01-17T14:43:33.707598] Registering candidate plugin; module='basicfuncs', context='template-func', name='lowercase', preference='0' [2017-01-17T14:43:33.707603] Registering candidate plugin; module='basicfuncs', context='template-func', name='uppercase', preference='0' [2017-01-17T14:43:33.707608] Registering candidate plugin; module='basicfuncs', context='template-func', name='replace-delimiter', preference='0' [2017-01-17T14:43:33.707612] Registering candidate plugin; module='basicfuncs', context='template-func', name='padding', preference='0' [2017-01-17T14:43:33.707617] Registering candidate plugin; module='basicfuncs', context='template-func', name='+', preference='0' [2017-01-17T14:43:33.707622] Registering candidate plugin; module='basicfuncs', context='template-func', name='-', preference='0' [2017-01-17T14:43:33.707626] Registering candidate plugin; module='basicfuncs', context='template-func', name='*', preference='0' [2017-01-17T14:43:33.707631] Registering candidate plugin; module='basicfuncs', context='template-func', name='/', preference='0' [2017-01-17T14:43:33.707635] Registering candidate plugin; module='basicfuncs', context='template-func', name='%', preference='0' [2017-01-17T14:43:33.707639] Registering candidate plugin; module='basicfuncs', context='template-func', name='sum', preference='0' [2017-01-17T14:43:33.707644] Registering candidate plugin; module='basicfuncs', context='template-func', name='min', preference='0' [2017-01-17T14:43:33.707648] Registering candidate plugin; module='basicfuncs', context='template-func', name='max', preference='0' [2017-01-17T14:43:33.707653] Registering candidate plugin; module='basicfuncs', context='template-func', name='average', preference='0' [2017-01-17T14:43:33.707657] Registering candidate plugin; module='basicfuncs', context='template-func', name='ipv4-to-int', preference='0' [2017-01-17T14:43:33.707662] Registering candidate plugin; module='basicfuncs', context='template-func', name='indent-multi-line', preference='0' [2017-01-17T14:43:33.707794] Registering candidate plugin; module='basicfuncs', context='template-func', name='context-length', preference='0' [2017-01-17T14:43:33.707803] Registering candidate plugin; module='basicfuncs', context='template-func', name='env', preference='0' [2017-01-17T14:43:33.707808] Registering candidate plugin; module='basicfuncs', context='template-func', name='template', preference='0' [2017-01-17T14:43:33.707836] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='cef.so', module='cef' [2017-01-17T14:43:33.707990] Registering candidate plugin; module='cef', context='template-func', name='format-cef-extension', preference='0' [2017-01-17T14:43:33.708035] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='confgen.so', module='confgen' [2017-01-17T14:43:33.708183] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='cryptofuncs.so', module='cryptofuncs' [2017-01-17T14:43:33.708327] Registering candidate plugin; module='cryptofuncs', context='template-func', name='uuid', preference='0' [2017-01-17T14:43:33.708336] Registering candidate plugin; module='cryptofuncs', context='template-func', name='hash', preference='0' [2017-01-17T14:43:33.708341] Registering candidate plugin; module='cryptofuncs', context='template-func', name='sha1', preference='0' [2017-01-17T14:43:33.708346] Registering candidate plugin; module='cryptofuncs', context='template-func', name='sha256', preference='0' [2017-01-17T14:43:33.708350] Registering candidate plugin; module='cryptofuncs', context='template-func', name='sha512', preference='0' [2017-01-17T14:43:33.708355] Registering candidate plugin; module='cryptofuncs', context='template-func', name='md4', preference='0' [2017-01-17T14:43:33.708359] Registering candidate plugin; module='cryptofuncs', context='template-func', name='md5', preference='0' [2017-01-17T14:43:33.708381] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='csvparser.so', module='csvparser' [2017-01-17T14:43:33.708530] Registering candidate plugin; module='csvparser', context='parser', name='csv-parser', preference='0' [2017-01-17T14:43:33.708557] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='date.so', module='date' [2017-01-17T14:43:33.708697] Registering candidate plugin; module='date', context='parser', name='date-parser', preference='0' [2017-01-17T14:43:33.708724] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='dbparser.so', module='dbparser' [2017-01-17T14:43:33.708920] Registering candidate plugin; module='dbparser', context='parser', name='db-parser', preference='0' [2017-01-17T14:43:33.708929] Registering candidate plugin; module='dbparser', context='parser', name='grouping-by', preference='0' [2017-01-17T14:43:33.708965] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='disk-buffer.so', module='disk-buffer' [2017-01-17T14:43:33.709124] Registering candidate plugin; module='disk-buffer', context='inner-dest', name='disk_buffer', preference='0' [2017-01-17T14:43:33.709157] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='graphite.so', module='graphite' [2017-01-17T14:43:33.709293] Registering candidate plugin; module='graphite', context='template-func', name='graphite_output', preference='0' [2017-01-17T14:43:33.709318] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='json-plugin.so', module='json-plugin' [2017-01-17T14:43:33.709526] Registering candidate plugin; module='json-plugin', context='parser', name='json-parser', preference='0' [2017-01-17T14:43:33.709537] Registering candidate plugin; module='json-plugin', context='template-func', name='format_json', preference='0' [2017-01-17T14:43:33.709568] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='kvformat.so', module='kvformat' [2017-01-17T14:43:33.709711] Registering candidate plugin; module='kvformat', context='parser', name='kv-parser', preference='0' [2017-01-17T14:43:33.709720] Registering candidate plugin; module='kvformat', context='parser', name='linux-audit-parser', preference='0' [2017-01-17T14:43:33.709822] Registering candidate plugin; module='kvformat', context='template-func', name='format-welf', preference='0' [2017-01-17T14:43:33.709886] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='linux-kmsg-format.so', module='linux-kmsg-format' [2017-01-17T14:43:33.710060] Registering candidate plugin; module='linux-kmsg-format', context='format', name='linux-kmsg', preference='0' [2017-01-17T14:43:33.710088] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='pseudofile.so', module='pseudofile' [2017-01-17T14:43:33.710231] Registering candidate plugin; module='pseudofile', context='destination', name='pseudofile', preference='0' [2017-01-17T14:43:33.710256] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='sdjournal.so', module='sdjournal' [2017-01-17T14:43:33.710396] Registering candidate plugin; module='sdjournal', context='source', name='systemd-journal', preference='0' [2017-01-17T14:43:33.710422] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='syslogformat.so', module='syslogformat' [2017-01-17T14:43:33.710578] Registering candidate plugin; module='syslogformat', context='format', name='syslog', preference='0' [2017-01-17T14:43:33.710591] Registering candidate plugin; module='syslogformat', context='parser', name='syslog-parser', preference='0' [2017-01-17T14:43:33.710625] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='system-source.so', module='system-source' [2017-01-17T14:43:33.710783] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='mod-java.so', module='mod-java' [2017-01-17T14:43:33.712715] Registering candidate plugin; module='mod-java', context='destination', name='java', preference='0' [2017-01-17T14:43:33.712978] Finishing include; filename='/usr/share/syslog-ng/include/scl/apache/apache.conf', depth='2' [2017-01-17T14:43:33.713009] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/cim/template.conf', depth='2' [2017-01-17T14:43:33.713309] Module loaded and initialized successfully; module='json-plugin' [2017-01-17T14:43:33.713473] Finishing include; filename='/usr/share/syslog-ng/include/scl/cim/template.conf', depth='2' [2017-01-17T14:43:33.713496] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/elasticsearch/plugin.conf', depth='2' [2017-01-17T14:43:33.713674] Finishing include; filename='/usr/share/syslog-ng/include/scl/elasticsearch/plugin.conf', depth='2' [2017-01-17T14:43:33.713693] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/graphite/plugin.conf', depth='2' [2017-01-17T14:43:33.713773] Finishing include; filename='/usr/share/syslog-ng/include/scl/graphite/plugin.conf', depth='2' [2017-01-17T14:43:33.713793] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/hdfs/plugin.conf', depth='2' [2017-01-17T14:43:33.713880] Finishing include; filename='/usr/share/syslog-ng/include/scl/hdfs/plugin.conf', depth='2' [2017-01-17T14:43:33.713898] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/kafka/plugin.conf', depth='2' [2017-01-17T14:43:33.713997] Finishing include; filename='/usr/share/syslog-ng/include/scl/kafka/plugin.conf', depth='2' [2017-01-17T14:43:33.714017] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/loggly/loggly.conf', depth='2' [2017-01-17T14:43:33.714133] Finishing include; filename='/usr/share/syslog-ng/include/scl/loggly/loggly.conf', depth='2' [2017-01-17T14:43:33.714151] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/logmatic/logmatic.conf', depth='2' [2017-01-17T14:43:33.714267] Finishing include; filename='/usr/share/syslog-ng/include/scl/logmatic/logmatic.conf', depth='2' [2017-01-17T14:43:33.714286] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/mbox/mbox.conf', depth='2' [2017-01-17T14:43:33.714350] Finishing include; filename='/usr/share/syslog-ng/include/scl/mbox/mbox.conf', depth='2' [2017-01-17T14:43:33.714369] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/nodejs/plugin.conf', depth='2' [2017-01-17T14:43:33.714440] Finishing include; filename='/usr/share/syslog-ng/include/scl/nodejs/plugin.conf', depth='2' [2017-01-17T14:43:33.714458] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/pacct/plugin.conf', depth='2' [2017-01-17T14:43:33.714521] Finishing include; filename='/usr/share/syslog-ng/include/scl/pacct/plugin.conf', depth='2' [2017-01-17T14:43:33.714538] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/rewrite/cc-mask.conf', depth='2' [2017-01-17T14:43:33.714611] Global value changed; define='balabit.credit-card-regexp', value='(?P<1>:4[0-9]{12}(?:[0-9]{3})?|5[1-5][0-9]{14}|6(?:011|5[0-9][0-9])[0-9]{12}|3[47][0-9]{13}|3(?:0[0-5]|[68][0-9])[0-9]{11}|(?:2131|1800|35d{3})d{11})' [2017-01-17T14:43:33.714655] Finishing include; filename='/usr/share/syslog-ng/include/scl/rewrite/cc-mask.conf', depth='2' [2017-01-17T14:43:33.714673] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/solaris/plugin.conf', depth='2' [2017-01-17T14:43:33.714994] Finishing include; filename='/usr/share/syslog-ng/include/scl/solaris/plugin.conf', depth='2' [2017-01-17T14:43:33.715018] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/syslogconf/plugin.conf', depth='2' [2017-01-17T14:43:33.715254] Module loaded and initialized successfully; module='confgen' [2017-01-17T14:43:33.715268] Finishing include; filename='/usr/share/syslog-ng/include/scl/syslogconf/plugin.conf', depth='2' [2017-01-17T14:43:33.715287] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/system/plugin.conf', depth='2' [2017-01-17T14:43:33.715490] Module loaded and initialized successfully; module='system-source' [2017-01-17T14:43:33.715503] Finishing include; filename='/usr/share/syslog-ng/include/scl/system/plugin.conf', depth='2' [2017-01-17T14:43:33.715521] Finishing include; filename='/etc/syslog-ng/scl.conf', depth='1' [2017-01-17T14:43:33.716821] Module loaded and initialized successfully; module='mod-java' [2017-01-17T14:43:33.717202] Module loaded and initialized successfully; module='afsocket' [2017-01-17T14:43:33.717529] Finishing include; content='destination block elasticsearch2', depth='1' [2017-01-17T14:43:33.717869] Compiling #unnamed sequence [log] at [/etc/syslog-ng/syslog-ng.conf:23:3] [2017-01-17T14:43:33.717877] Compiling s_syslog reference [source] at [/etc/syslog-ng/syslog-ng.conf:23:3] [2017-01-17T14:43:33.717881] Compiling s_syslog sequence [source] at [/etc/syslog-ng/syslog-ng.conf:10:1] [2017-01-17T14:43:33.717884] Compiling #unnamed junction [log] at [/etc/syslog-ng/syslog-ng.conf:10:18] [2017-01-17T14:43:33.717887] Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf:10:19] [2017-01-17T14:43:33.717895] Compiling d_elastic reference [destination] at [/etc/syslog-ng/syslog-ng.conf:24:3] [2017-01-17T14:43:33.717899] Compiling d_elastic sequence [destination] at [/etc/syslog-ng/syslog-ng.conf:12:1] [2017-01-17T14:43:33.717902] Compiling #unnamed junction [log] at [/etc/syslog-ng/syslog-ng.conf:12:24] [2017-01-17T14:43:33.717905] Compiling #unnamed single [log] at [#buffer:2:3] [2017-01-17T14:43:33.718096] Module loaded and initialized successfully; module='syslogformat' [2017-01-17T14:43:33.824194] Add path to classpath: //usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar; [2017-01-17T14:43:33.824696] Add path to classpath: /usr/lib64/syslog-ng/java-modules/dummy.jar; [2017-01-17T14:43:33.824868] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-v2.jar; [2017-01-17T14:43:33.825032] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic.jar; [2017-01-17T14:43:33.825169] Add path to classpath: /usr/lib64/syslog-ng/java-modules/hdfs.jar; [2017-01-17T14:43:33.825322] Add path to classpath: /usr/lib64/syslog-ng/java-modules/http.jar; [2017-01-17T14:43:33.825470] Add path to classpath: /usr/lib64/syslog-ng/java-modules/kafka.jar; [2017-01-17T14:43:33.825629] Add path to classpath: /usr/lib64/syslog-ng/java-modules/log4j-1.2.16.jar; [2017-01-17T14:43:33.825781] Add path to classpath: /usr/lib64/syslog-ng/java-modules/syslog-ng-common.jar; [2017-01-17T14:43:33.825936] Add path to classpath: /usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar; [2017-01-17T14:43:33.826108] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-jest-client/commons-codec-1.9.jar; [2017-01-17T14:43:33.826286] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-jest-client/commons-lang3-3.4.jar; [2017-01-17T14:43:33.826459] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-jest-client/commons-logging-1.2.jar; [2017-01-17T14:43:33.826633] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-jest-client/gson-2.6.2.jar; [2017-01-17T14:43:33.826801] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-jest-client/guava-19.0.jar; [2017-01-17T14:43:33.826981] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-jest-client/httpasyncclient-4.1.1.jar; [2017-01-17T14:43:33.827156] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-jest-client/httpclient-4.5.2.jar; [2017-01-17T14:43:33.827343] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-jest-client/httpcore-4.4.4.jar; [2017-01-17T14:43:33.827520] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-jest-client/httpcore-nio-4.4.4.jar; [2017-01-17T14:43:33.827694] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-jest-client/jest-2.0.2.jar; [2017-01-17T14:43:33.827851] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-jest-client/jest-common-2.0.2.jar; [2017-01-17T14:43:33.828061] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-jest-client/slf4j-api-1.7.13.jar; [2017-01-17T14:43:33.828225] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-jest-client/slf4j-simple-1.7.13.jar; [2017-01-17T14:43:33.901817] Add path to classpath: //usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar; [2017-01-17T14:43:33.906737] Error initializing message pipeline; I’m sure I’m missing something basic – perhaps correct pathing of the ES .jar files? Many thanks for your help in advance. Damian Bell Damian Bell Infrastructure Engineer | Support | H Clarkson & Co Ltd Email: Damian.Bell(a)clarksons.com<mailto:Damian.Bell@clarksons.com> Group Email: infrastructure(a)clarksons.com Clarksons Platou TM Commodity Quay, St. Katharine Docks | London E1W 1BF | United Kingdom www.clarksons.com<http://www.clarksons.com> Please consider the environment before printing this e-mail ________________________________ This message is private and confidential. If you have received it in error, you are on notice of its status. Please notify us immediately by reply email and then delete this message from your system. Please do not copy it or use it for any purposes, or disclose its contents to any other person: to do so could be a breach of confidence. Emails may be monitored. Details of Clarkson group companies and their regulators (where applicable) can be found at this url: Disclosure<http://www.clarksons.com/disclosure/> ________________________________

5 7

Hitting g_assert when using sanitize-utf8 enabled!
by James Elstone 13 Jan '17

13 Jan '17

Hi Balabit et al, When using the sanitize-utf8 flag I am hitting a g_assert in modules/syslogformat/syslog-format.c; what could be causing this? Any advice welcome!! Kr, James -- Sent from my Android device with K-9 Mail. Please excuse my brevity.

3 13

unofficial syslog-ng 3.9.1 RPM packages
by Czanik, Péter 13 Jan '17

13 Jan '17

Hi, My unofficial syslog-ng 3.9.1 RPM packages are ready. They are now available from my "githead" repositories. This way I did not have to go through setting up separate 3.9 repositories over a very slow Internet connection and 3.9.1 is git HEAD right now anyway :) Supported distributions are: openSUSE Leap 42.1 and 42.2 SLES 11 and 12 Fedora 24, 25, 26 (rawhide) RHEL / CentOS 7 Instructions are available at https://www.balabit.com/blog/rpm-packages-from-syslog-ng-git-head/ Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 1

Insider 2017-01: syslog-ng 3.9.1; SearchGuard; FOSDEM & SCALE;
by Czanik, Péter 12 Jan '17

12 Jan '17

Dear syslog-ng users, This is the 54th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS syslog-ng 3.9.1 has been released ------------------------ Right before Christmas syslog-ng version 3.9.1 has been released, four months after the 3.8.1 release. It contains tons of bugfixes and many small incremental changes compared to the previous version. Performance has been improved at multiple places, Big Data drivers were updated, and secure logging to Elasticsearch is now possible using SearchGuard. For a complete list, read https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.9.1 where you can also find the source code .tgz file. For binary packages check https://syslog-ng.org/3rd-party-binaries/ Securing Elasticsearch access with SearchGuard ---------------------------------------------- Using syslog-ng to index events to your Elasticsearch cluster has been possible natively for some time now. Readabout using Search Guard support in syslog-ng 3.9 to encrypt and authenticate your connections to Elasticsearch. https://www.balabit.com/blog/securing-your-esk-stack-for-free-using-search-… Installing the latest syslog-ng DEB packages -------------------------------------------- The syslog-ng application is part of all major Linux distributions, and you can usually install syslog-ng from the official repositories. However, if you want to use the features of newer syslog-ng versions (for example, send log messages to Elasticsearch or Apache Kafka), you have to either compile the syslog-ng from source, or install it from unofficial repositories. This post explains you how to do that on DEB packages based systems: https://www.balabit.com/blog/installing-the-latest-syslog-ng-on-ubuntu-and-… RPM packages from syslog-ng git HEAD ------------------------------------ There are some situations, when even the latest stable release is not good enough as a bug was found and fixed after release. Or maybe you want to check the latest new features that are not yet available in a release. If you use an RPM-based distribution, packages generated from Git HEAD are right for you: https://www.balabit.com/blog/rpm-packages-from-syslog-ng-git-head/ UPCOMING EVENTS Peter Czanik will give two talks at FOSDEM (http://fosdem.org/) - Scaling your logging infrastructure: https://fosdem.org/2017/schedule/event/syslog_ng_scaling/ - Getting the most out of your security logs: https://fosdem.org/2017/schedule/event/syslog_ng_using/ Balabit will participate the Southern California Linux Expo this year again: http://www.socallinuxexpo.org/scale/15x You can talk to our engineers at the Balabit booth and listen to Peter Czanik about syslog-ng. NEW RELEASES - syslog-ng 3.9.1: https://www.balabit.com/blog/syslog-ng-3-9-1-released/ Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com. To read this newsletter online, visit: https://syslog-ng.org/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

is it possible to normalize the jSON keys to lowercase?
by Jorge Pereira 11 Jan '17

11 Jan '17

Hi, I am using the 'format-json' as my below snip. template t_raw2json { template("$(format-json --scope selected_macros \ --exclude TAGS \ --exclude DATE \ --exclude PRIORITY \ --exclude FACILITY \ --exclude PROGRAM )\n"); }; but, the problem is because sometimes I receive a jSON packet as { "Key": "value" } or {"key": "value}".... this has been a problem to my filter. e.g: destination d_nginx_modsec_log { file("/var/logs/myapp//${j.key:-invalid_key}.log" template(t_raw2json) ); }; Question: is it possible to normalize the jSON keys to lowercase? or someone has other approaches? -- Jorge Pereira

4 5

udp tuning - where to look.
by Jim Hendrick 11 Jan '17

11 Jan '17

Hi, I am working an issue with UDP drops in a high EPS rate environment (several thousand steady) and struggling with making any changes in the loss rate. (according to netstat -su | grep error). This is happening on two nodes, (one running an older syslog-ng 3.2 OSE, the other running a splunk forwarder) but I don't actually think it is making it out of the kernel to either application. On the syslog-ng side syslog-ng-ctl stats shows *no* drops at all. Increasing net.core.rmem_max and so_rcvbuf together all the way to 64 MB did not seem to make any significant difference. This is a RHEL 6 box with 16 GB and 4 cores (virtual - running in an ESX environment) Are there other parameters, things I should be looking at? Thanks, Jim

4 4

CentOS 7.2 // Syslog-ng (syslog-ng_3.6.4)
by vijay amruth 10 Jan '17

10 Jan '17

Hello guys, Happy new year!! Hope you are doing great. I've installed (compiled ) syslog-ng on CentOS Linux 7 (Core) but after I got through the install which is all good, I cant get it to start, all it says is : systemctl status syslog-ng ● syslog-ng.service Loaded: not-found (Reason: No such file or directory) Active: inactive (dead) [vch@slc ~]$ sudo systemctl enable syslog-ng Failed to execute operation: No such file or directory [vch@slc ~]$ sudo systemctl start syslog-ng Failed to start syslog-ng.service: Unit syslog-ng.service failed to load: No such file or directory. Rsyslog is not running, I can't think of any, I am I missing some simple ?????? Any help is appreciated. -- Thanks, Vijay Amrut.

5 6

Enable TLS encryption
by Sathish Sundaravel 09 Jan '17

09 Jan '17

Hi Team I have a Syslog Ng server running on Solaris and receiving events from native Solaris syslog clients. How do I enable UDP traffic pass through via TLS and how to configure client certificate for native Solaris syslog agent. We don't have rsyslog.

6 10

Adding specific information from Syslog-ng to Redis
by johnsc301＠gmail.com 07 Jan '17

07 Jan '17

I am trying to send information from Syslog-ng to Redis. In /etc/syslog-ng/syslog-ng.conf I added this: destination d_redis { redis( host("127.0.0.1") port(6379) command("RPUSH", "sensor_name", "${sensor_name}")); }; I am trying to create a list of variables, hopefully one being timestamp. Here is an example of my syslog that I am trying to pull: http://pastebin.com/Hx5vW4VA Here's is syslog-ng.conf, for reference: http://pastebin.com/2VQFBNmK Those are logs being sent from Snort to Syslog-ng through Snort. I want to connect to Redis. I saw that the command parameters are: comma-separated list of strings ("<redis-command>", "<first-command-parameter>", "<second-command-parameter>", "<third-command-parameter>") from: https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-… I'm assuming I'd say RPUSH <something>... However, I am unsure of how to find the correct parameters. Specifically, for now, I want to create a list (RPUSH) of timestamps, IP addresses (to and from), and event type (ICMP, for example). I did find this list of parameters: https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-… A good first try, I'd like to make a list of timestamps. How can I set the d_redis(command()) within syslog-ng.conf to do this? Sent from Mail for Windows 10

4 3

v3.9: Core dump on python module
by Clayton Dukes 04 Jan '17

04 Jan '17

Hey guys, just want you to know Ubuntu16 is dumping core on loading the python module. Ubuntu14 does not do this. [2017-01-01T19:54:52.576749] Module loaded and initialized successfully; module='mod-python' Segmentation fault (core dumped) -Clayton Dukes

4 16