January 2017 - syslog-ng - lists.balabit.hu

[request for feedback] syslog fundamentals post
by Radu Gheorghe 01 Feb '17

01 Feb '17

Hello, I just wrote a blog post that tries to disambiguate the word "syslog" and clarify the myths I heard around it (as I'm sure you know, there's a lot of vagueness): https://sematext.com/blog/2017/01/30/what-is-syslog-daemons-message-formats… Obviously, syslog-ng is an important chunk of it, so I wanted to ask for any feedback you may have (did I miss something important regarding syslog-ng? is there something wrong or something else you'd want to see there?) whichever way you prefer to send it (here, on the blog post comments, personal Email or Twitter...). Thanks in advance and best regards, Radu -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/

2 2

Stupid E-S-K Question
by Scot 31 Jan '17

31 Jan '17

*E*lastic, *S*yslog-ng *K*ibana Upgraded to latest of ES Stack, Kibana 5 and syslog-ng 3.9.1 I had a Kibana dashboard with a bar chart of unique count of systems that had sent a syslog heartbeat. So I could see any missed heartbeats for any host in the last 24 hours. Post upgrade of syslog-ng the host_from, host fields do not seem to come into ES as usable fields because they are not indexed. So visualizations "bar charts by unique 'host" is broken. Has anyone seen this? client-mode("http") index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("syslog") # Description: The type of the index. For example, type("test") template("$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n")

3 4

Convert rewrite rules (regex) to patterndb
by Clayton Dukes 27 Jan '17

27 Jan '17

Hi All, Are there any tools/scripts anyone has written that will convert regex to patterndb types of matches? So, given some rewrite rule from the "old" way of doing it, convert that rule to a pdb file. Like: filter f_asa { match( '(.*?):?(\d{1,3}+\.\d{1,3}+\.\d{1,3}+\.\d{1,3}+)\/(\d+)(.*?)(\).*?)?:?(\d{1,3}+\.\d{1,3}+\.\d{1,3}+\.\d{1,3}+)\/(\d+)(.*?)(\d{1,3}+\.\d{1,3}+\.\d{1,3}+\.\d{1,3}+)\/(\d+)(.*)' value("MESSAGE") type("pcre") flags("store-matches" "ignore-case") ); rewrite rw_cisco { set( "$1 SourceIP: $2 SourcePort:$3 $4 $5DestIP: $6 DestPort:$7 $8 SourceIP: $9 SourcePort:$10 $11" , value("MESSAGE") condition(filter(f_asa))); }; Convert to something like: <pattern>%ASA-6-305011: Built dynamic TCP translation from inside:@IPv4:src_ip@/@NUMBER:src_port@ to Outside:@IPv4:dst_ip@/@NUMBER:dst_port@</pattern> Note: the above is just an example, the regex may not be the same as the actual <pattern> - I just copy pasted from different files to show an example. [cid:image001.png@01D27730.889CAAF0]

3 3

RPM packages from git HEAD
by Czanik, Péter 23 Jan '17

23 Jan '17

Hi, Recently I started to create RPM packages from git HEAD. Obviously this is not for production, but helps to check, if a problem reported for 3.8.1 or earlier still exists. It is available for openSUSE, Fedora, RHEL 7 and SLES 11/12. I don't create RHEL 6 packages, as that would require maintaining a third spec file and repo, but if there are too many people trying to convince me, I can do it next year :) And the good news is, that I learned right after publishing my blog, that 3.8.2 is nearing and expected to arrive this year! Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

5 6

https to elasticsearch
by Czanik, Péter 20 Jan '17

20 Jan '17

Hi, Secure connection to Elasticsearch was already supported through Shield (now X-pack) for a while and SearchGuard support was also added recently. Both used transport mode for connection. Today encrypted connection using HTTPS became available thanks to Faxm0dem: https://github.com/balabit/syslog-ng/pull/1321 Before leaving for the weekend I prepared RPM packages from this PR in my 'githead' repositories. You can read about how to use those in my blog at https://www.balabit.com/blog/rpm-packages-from-syslog-ng-git-head/ Would be great if someone tested this against X-pack! Have a nice weekend! Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

Error initializing message pipeline;
by Damian Bell 20 Jan '17

20 Jan '17

Hello, First time poster, go easy! :) I’m using syslog-ng for network logging and have been fine running it thus far with no problems. Having decided to go that step further and use Elasticsearch 5.1.1 and Kibana for visualisation, I’ve managed to get both those (along with nginx) working, but currently have an issue with starting syslog-ng. I think it’s java and associated libraries, but according to documentation I’ve read so far I can’t seem to work out what isn’t working correctly/where I should be pathing correct .jar files from? Yum installed from czanik-syslog-ng39-epel-7.repo [root@SOMESERVER syslog-ng]# syslog-ng -V syslog-ng 3.9.1 Installer-Version: 3.9.1 Revision: Module-Directory: //usr/lib64/syslog-ng Module-Path: //usr/lib64/syslog-ng Available-Modules: add-contextual-data,afamqp,affile,afprog,afsocket,afstomp,afuser,basicfuncs,cef,confgen,cryptofuncs,csvparser,date,dbparser,disk-buffer,graphite,json-plugin,kvformat,linux-kmsg-format,pseudofile,sdjournal,syslogformat,system-source,mod-java Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on [root@SOMESERVER syslog-ng]# java -version java version "1.8.0_112" Java(TM) SE Runtime Environment (build 1.8.0_112-b15) Java HotSpot(TM) 64-Bit Server VM (build 25.112-b15, mixed mode) [root@SOMESERVER syslog-ng]# set | grep LD LD_LIBRARY_PATH=/usr/java/jre1.8.0_112/lib/amd64/server Config:- @version:3.9 @include "scl.conf" @module mod-java options { threaded(yes); }; source s_syslog { udp(ip(0.0.0.0) port(514)); }; destination d_elastic { elasticsearch2( index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("test") cluster("someserver") client-mode("transport") template("$(format-json --scope rfc5424 --scope nv-pairs --exclude DATE --key ISODATE)") ); }; log { source(s_syslog); destination(d_elastic); flags(flow-control); }; Output:- [root@SOMESERVER syslog-ng]# syslog-ng -Fevd -f /etc/syslog-ng/syslog-ng.conf [2017-01-17T14:43:33.704051] Systemd is detected as the running init system; [2017-01-17T14:43:33.704988] Starting to read include file; filename='/etc/syslog-ng/scl.conf', depth='1' [2017-01-17T14:43:33.705231] Adding include file; filename='/usr/share/syslog-ng/include/scl/apache/apache.conf', depth='2' [2017-01-17T14:43:33.705240] Adding include file; filename='/usr/share/syslog-ng/include/scl/cim/template.conf', depth='2' [2017-01-17T14:43:33.705244] Adding include file; filename='/usr/share/syslog-ng/include/scl/elasticsearch/plugin.conf', depth='2' [2017-01-17T14:43:33.705249] Adding include file; filename='/usr/share/syslog-ng/include/scl/graphite/plugin.conf', depth='2' [2017-01-17T14:43:33.705253] Adding include file; filename='/usr/share/syslog-ng/include/scl/hdfs/plugin.conf', depth='2' [2017-01-17T14:43:33.705257] Adding include file; filename='/usr/share/syslog-ng/include/scl/kafka/plugin.conf', depth='2' [2017-01-17T14:43:33.705261] Adding include file; filename='/usr/share/syslog-ng/include/scl/loggly/loggly.conf', depth='2' [2017-01-17T14:43:33.705266] Adding include file; filename='/usr/share/syslog-ng/include/scl/logmatic/logmatic.conf', depth='2' [2017-01-17T14:43:33.705270] Adding include file; filename='/usr/share/syslog-ng/include/scl/mbox/mbox.conf', depth='2' [2017-01-17T14:43:33.705274] Adding include file; filename='/usr/share/syslog-ng/include/scl/nodejs/plugin.conf', depth='2' [2017-01-17T14:43:33.705278] Adding include file; filename='/usr/share/syslog-ng/include/scl/pacct/plugin.conf', depth='2' [2017-01-17T14:43:33.705282] Adding include file; filename='/usr/share/syslog-ng/include/scl/rewrite/cc-mask.conf', depth='2' [2017-01-17T14:43:33.705286] Adding include file; filename='/usr/share/syslog-ng/include/scl/solaris/plugin.conf', depth='2' [2017-01-17T14:43:33.705290] Adding include file; filename='/usr/share/syslog-ng/include/scl/syslogconf/plugin.conf', depth='2' [2017-01-17T14:43:33.705294] Adding include file; filename='/usr/share/syslog-ng/include/scl/system/plugin.conf', depth='2' [2017-01-17T14:43:33.705302] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/apache/apache.conf', depth='2' [2017-01-17T14:43:33.705357] Reading path for candidate modules; path='//usr/lib64/syslog-ng' [2017-01-17T14:43:33.705387] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='add-contextual-data.so', module='add-contextual-data' [2017-01-17T14:43:33.705607] Registering candidate plugin; module='add-contextual-data', context='parser', name='add_contextual_data', preference='0' [2017-01-17T14:43:33.705643] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='afamqp.so', module='afamqp' [2017-01-17T14:43:33.705806] Registering candidate plugin; module='afamqp', context='destination', name='amqp', preference='0' [2017-01-17T14:43:33.705836] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='affile.so', module='affile' [2017-01-17T14:43:33.706009] Registering candidate plugin; module='affile', context='source', name='file', preference='0' [2017-01-17T14:43:33.706019] Registering candidate plugin; module='affile', context='source', name='pipe', preference='0' [2017-01-17T14:43:33.706024] Registering candidate plugin; module='affile', context='destination', name='file', preference='0' [2017-01-17T14:43:33.706029] Registering candidate plugin; module='affile', context='destination', name='pipe', preference='0' [2017-01-17T14:43:33.706056] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='afprog.so', module='afprog' [2017-01-17T14:43:33.706207] Registering candidate plugin; module='afprog', context='source', name='program', preference='0' [2017-01-17T14:43:33.706216] Registering candidate plugin; module='afprog', context='destination', name='program', preference='0' [2017-01-17T14:43:33.706240] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='afsocket.so', module='afsocket' [2017-01-17T14:43:33.706560] Registering candidate plugin; module='afsocket', context='source', name='unix-stream', preference='100' [2017-01-17T14:43:33.706570] Registering candidate plugin; module='afsocket', context='destination', name='unix-stream', preference='100' [2017-01-17T14:43:33.706575] Registering candidate plugin; module='afsocket', context='source', name='unix-dgram', preference='100' [2017-01-17T14:43:33.706580] Registering candidate plugin; module='afsocket', context='destination', name='unix-dgram', preference='100' [2017-01-17T14:43:33.706585] Registering candidate plugin; module='afsocket', context='source', name='tcp', preference='100' [2017-01-17T14:43:33.706590] Registering candidate plugin; module='afsocket', context='destination', name='tcp', preference='100' [2017-01-17T14:43:33.706596] Registering candidate plugin; module='afsocket', context='source', name='tcp6', preference='100' [2017-01-17T14:43:33.706601] Registering candidate plugin; module='afsocket', context='destination', name='tcp6', preference='100' [2017-01-17T14:43:33.706605] Registering candidate plugin; module='afsocket', context='source', name='udp', preference='100' [2017-01-17T14:43:33.706610] Registering candidate plugin; module='afsocket', context='destination', name='udp', preference='100' [2017-01-17T14:43:33.706614] Registering candidate plugin; module='afsocket', context='source', name='udp6', preference='100' [2017-01-17T14:43:33.706619] Registering candidate plugin; module='afsocket', context='destination', name='udp6', preference='100' [2017-01-17T14:43:33.706624] Registering candidate plugin; module='afsocket', context='source', name='syslog', preference='100' [2017-01-17T14:43:33.706628] Registering candidate plugin; module='afsocket', context='destination', name='syslog', preference='100' [2017-01-17T14:43:33.706807] Registering candidate plugin; module='afsocket', context='source', name='network', preference='100' [2017-01-17T14:43:33.706815] Registering candidate plugin; module='afsocket', context='destination', name='network', preference='100' [2017-01-17T14:43:33.706819] Registering candidate plugin; module='afsocket', context='source', name='systemd-syslog', preference='100' [2017-01-17T14:43:33.706927] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='afstomp.so', module='afstomp' [2017-01-17T14:43:33.707133] Registering candidate plugin; module='afstomp', context='destination', name='stomp', preference='0' [2017-01-17T14:43:33.707195] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='afuser.so', module='afuser' [2017-01-17T14:43:33.707350] Registering candidate plugin; module='afuser', context='destination', name='usertty', preference='0' [2017-01-17T14:43:33.707379] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='basicfuncs.so', module='basicfuncs' [2017-01-17T14:43:33.707552] Registering candidate plugin; module='basicfuncs', context='template-func', name='grep', preference='0' [2017-01-17T14:43:33.707561] Registering candidate plugin; module='basicfuncs', context='template-func', name='if', preference='0' [2017-01-17T14:43:33.707566] Registering candidate plugin; module='basicfuncs', context='template-func', name='or', preference='0' [2017-01-17T14:43:33.707570] Registering candidate plugin; module='basicfuncs', context='template-func', name='echo', preference='0' [2017-01-17T14:43:33.707575] Registering candidate plugin; module='basicfuncs', context='template-func', name='length', preference='0' [2017-01-17T14:43:33.707584] Registering candidate plugin; module='basicfuncs', context='template-func', name='substr', preference='0' [2017-01-17T14:43:33.707589] Registering candidate plugin; module='basicfuncs', context='template-func', name='strip', preference='0' [2017-01-17T14:43:33.707594] Registering candidate plugin; module='basicfuncs', context='template-func', name='sanitize', preference='0' [2017-01-17T14:43:33.707598] Registering candidate plugin; module='basicfuncs', context='template-func', name='lowercase', preference='0' [2017-01-17T14:43:33.707603] Registering candidate plugin; module='basicfuncs', context='template-func', name='uppercase', preference='0' [2017-01-17T14:43:33.707608] Registering candidate plugin; module='basicfuncs', context='template-func', name='replace-delimiter', preference='0' [2017-01-17T14:43:33.707612] Registering candidate plugin; module='basicfuncs', context='template-func', name='padding', preference='0' [2017-01-17T14:43:33.707617] Registering candidate plugin; module='basicfuncs', context='template-func', name='+', preference='0' [2017-01-17T14:43:33.707622] Registering candidate plugin; module='basicfuncs', context='template-func', name='-', preference='0' [2017-01-17T14:43:33.707626] Registering candidate plugin; module='basicfuncs', context='template-func', name='*', preference='0' [2017-01-17T14:43:33.707631] Registering candidate plugin; module='basicfuncs', context='template-func', name='/', preference='0' [2017-01-17T14:43:33.707635] Registering candidate plugin; module='basicfuncs', context='template-func', name='%', preference='0' [2017-01-17T14:43:33.707639] Registering candidate plugin; module='basicfuncs', context='template-func', name='sum', preference='0' [2017-01-17T14:43:33.707644] Registering candidate plugin; module='basicfuncs', context='template-func', name='min', preference='0' [2017-01-17T14:43:33.707648] Registering candidate plugin; module='basicfuncs', context='template-func', name='max', preference='0' [2017-01-17T14:43:33.707653] Registering candidate plugin; module='basicfuncs', context='template-func', name='average', preference='0' [2017-01-17T14:43:33.707657] Registering candidate plugin; module='basicfuncs', context='template-func', name='ipv4-to-int', preference='0' [2017-01-17T14:43:33.707662] Registering candidate plugin; module='basicfuncs', context='template-func', name='indent-multi-line', preference='0' [2017-01-17T14:43:33.707794] Registering candidate plugin; module='basicfuncs', context='template-func', name='context-length', preference='0' [2017-01-17T14:43:33.707803] Registering candidate plugin; module='basicfuncs', context='template-func', name='env', preference='0' [2017-01-17T14:43:33.707808] Registering candidate plugin; module='basicfuncs', context='template-func', name='template', preference='0' [2017-01-17T14:43:33.707836] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='cef.so', module='cef' [2017-01-17T14:43:33.707990] Registering candidate plugin; module='cef', context='template-func', name='format-cef-extension', preference='0' [2017-01-17T14:43:33.708035] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='confgen.so', module='confgen' [2017-01-17T14:43:33.708183] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='cryptofuncs.so', module='cryptofuncs' [2017-01-17T14:43:33.708327] Registering candidate plugin; module='cryptofuncs', context='template-func', name='uuid', preference='0' [2017-01-17T14:43:33.708336] Registering candidate plugin; module='cryptofuncs', context='template-func', name='hash', preference='0' [2017-01-17T14:43:33.708341] Registering candidate plugin; module='cryptofuncs', context='template-func', name='sha1', preference='0' [2017-01-17T14:43:33.708346] Registering candidate plugin; module='cryptofuncs', context='template-func', name='sha256', preference='0' [2017-01-17T14:43:33.708350] Registering candidate plugin; module='cryptofuncs', context='template-func', name='sha512', preference='0' [2017-01-17T14:43:33.708355] Registering candidate plugin; module='cryptofuncs', context='template-func', name='md4', preference='0' [2017-01-17T14:43:33.708359] Registering candidate plugin; module='cryptofuncs', context='template-func', name='md5', preference='0' [2017-01-17T14:43:33.708381] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='csvparser.so', module='csvparser' [2017-01-17T14:43:33.708530] Registering candidate plugin; module='csvparser', context='parser', name='csv-parser', preference='0' [2017-01-17T14:43:33.708557] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='date.so', module='date' [2017-01-17T14:43:33.708697] Registering candidate plugin; module='date', context='parser', name='date-parser', preference='0' [2017-01-17T14:43:33.708724] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='dbparser.so', module='dbparser' [2017-01-17T14:43:33.708920] Registering candidate plugin; module='dbparser', context='parser', name='db-parser', preference='0' [2017-01-17T14:43:33.708929] Registering candidate plugin; module='dbparser', context='parser', name='grouping-by', preference='0' [2017-01-17T14:43:33.708965] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='disk-buffer.so', module='disk-buffer' [2017-01-17T14:43:33.709124] Registering candidate plugin; module='disk-buffer', context='inner-dest', name='disk_buffer', preference='0' [2017-01-17T14:43:33.709157] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='graphite.so', module='graphite' [2017-01-17T14:43:33.709293] Registering candidate plugin; module='graphite', context='template-func', name='graphite_output', preference='0' [2017-01-17T14:43:33.709318] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='json-plugin.so', module='json-plugin' [2017-01-17T14:43:33.709526] Registering candidate plugin; module='json-plugin', context='parser', name='json-parser', preference='0' [2017-01-17T14:43:33.709537] Registering candidate plugin; module='json-plugin', context='template-func', name='format_json', preference='0' [2017-01-17T14:43:33.709568] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='kvformat.so', module='kvformat' [2017-01-17T14:43:33.709711] Registering candidate plugin; module='kvformat', context='parser', name='kv-parser', preference='0' [2017-01-17T14:43:33.709720] Registering candidate plugin; module='kvformat', context='parser', name='linux-audit-parser', preference='0' [2017-01-17T14:43:33.709822] Registering candidate plugin; module='kvformat', context='template-func', name='format-welf', preference='0' [2017-01-17T14:43:33.709886] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='linux-kmsg-format.so', module='linux-kmsg-format' [2017-01-17T14:43:33.710060] Registering candidate plugin; module='linux-kmsg-format', context='format', name='linux-kmsg', preference='0' [2017-01-17T14:43:33.710088] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='pseudofile.so', module='pseudofile' [2017-01-17T14:43:33.710231] Registering candidate plugin; module='pseudofile', context='destination', name='pseudofile', preference='0' [2017-01-17T14:43:33.710256] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='sdjournal.so', module='sdjournal' [2017-01-17T14:43:33.710396] Registering candidate plugin; module='sdjournal', context='source', name='systemd-journal', preference='0' [2017-01-17T14:43:33.710422] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='syslogformat.so', module='syslogformat' [2017-01-17T14:43:33.710578] Registering candidate plugin; module='syslogformat', context='format', name='syslog', preference='0' [2017-01-17T14:43:33.710591] Registering candidate plugin; module='syslogformat', context='parser', name='syslog-parser', preference='0' [2017-01-17T14:43:33.710625] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='system-source.so', module='system-source' [2017-01-17T14:43:33.710783] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='mod-java.so', module='mod-java' [2017-01-17T14:43:33.712715] Registering candidate plugin; module='mod-java', context='destination', name='java', preference='0' [2017-01-17T14:43:33.712978] Finishing include; filename='/usr/share/syslog-ng/include/scl/apache/apache.conf', depth='2' [2017-01-17T14:43:33.713009] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/cim/template.conf', depth='2' [2017-01-17T14:43:33.713309] Module loaded and initialized successfully; module='json-plugin' [2017-01-17T14:43:33.713473] Finishing include; filename='/usr/share/syslog-ng/include/scl/cim/template.conf', depth='2' [2017-01-17T14:43:33.713496] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/elasticsearch/plugin.conf', depth='2' [2017-01-17T14:43:33.713674] Finishing include; filename='/usr/share/syslog-ng/include/scl/elasticsearch/plugin.conf', depth='2' [2017-01-17T14:43:33.713693] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/graphite/plugin.conf', depth='2' [2017-01-17T14:43:33.713773] Finishing include; filename='/usr/share/syslog-ng/include/scl/graphite/plugin.conf', depth='2' [2017-01-17T14:43:33.713793] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/hdfs/plugin.conf', depth='2' [2017-01-17T14:43:33.713880] Finishing include; filename='/usr/share/syslog-ng/include/scl/hdfs/plugin.conf', depth='2' [2017-01-17T14:43:33.713898] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/kafka/plugin.conf', depth='2' [2017-01-17T14:43:33.713997] Finishing include; filename='/usr/share/syslog-ng/include/scl/kafka/plugin.conf', depth='2' [2017-01-17T14:43:33.714017] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/loggly/loggly.conf', depth='2' [2017-01-17T14:43:33.714133] Finishing include; filename='/usr/share/syslog-ng/include/scl/loggly/loggly.conf', depth='2' [2017-01-17T14:43:33.714151] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/logmatic/logmatic.conf', depth='2' [2017-01-17T14:43:33.714267] Finishing include; filename='/usr/share/syslog-ng/include/scl/logmatic/logmatic.conf', depth='2' [2017-01-17T14:43:33.714286] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/mbox/mbox.conf', depth='2' [2017-01-17T14:43:33.714350] Finishing include; filename='/usr/share/syslog-ng/include/scl/mbox/mbox.conf', depth='2' [2017-01-17T14:43:33.714369] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/nodejs/plugin.conf', depth='2' [2017-01-17T14:43:33.714440] Finishing include; filename='/usr/share/syslog-ng/include/scl/nodejs/plugin.conf', depth='2' [2017-01-17T14:43:33.714458] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/pacct/plugin.conf', depth='2' [2017-01-17T14:43:33.714521] Finishing include; filename='/usr/share/syslog-ng/include/scl/pacct/plugin.conf', depth='2' [2017-01-17T14:43:33.714538] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/rewrite/cc-mask.conf', depth='2' [2017-01-17T14:43:33.714611] Global value changed; define='balabit.credit-card-regexp', value='(?P<1>:4[0-9]{12}(?:[0-9]{3})?|5[1-5][0-9]{14}|6(?:011|5[0-9][0-9])[0-9]{12}|3[47][0-9]{13}|3(?:0[0-5]|[68][0-9])[0-9]{11}|(?:2131|1800|35d{3})d{11})' [2017-01-17T14:43:33.714655] Finishing include; filename='/usr/share/syslog-ng/include/scl/rewrite/cc-mask.conf', depth='2' [2017-01-17T14:43:33.714673] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/solaris/plugin.conf', depth='2' [2017-01-17T14:43:33.714994] Finishing include; filename='/usr/share/syslog-ng/include/scl/solaris/plugin.conf', depth='2' [2017-01-17T14:43:33.715018] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/syslogconf/plugin.conf', depth='2' [2017-01-17T14:43:33.715254] Module loaded and initialized successfully; module='confgen' [2017-01-17T14:43:33.715268] Finishing include; filename='/usr/share/syslog-ng/include/scl/syslogconf/plugin.conf', depth='2' [2017-01-17T14:43:33.715287] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/system/plugin.conf', depth='2' [2017-01-17T14:43:33.715490] Module loaded and initialized successfully; module='system-source' [2017-01-17T14:43:33.715503] Finishing include; filename='/usr/share/syslog-ng/include/scl/system/plugin.conf', depth='2' [2017-01-17T14:43:33.715521] Finishing include; filename='/etc/syslog-ng/scl.conf', depth='1' [2017-01-17T14:43:33.716821] Module loaded and initialized successfully; module='mod-java' [2017-01-17T14:43:33.717202] Module loaded and initialized successfully; module='afsocket' [2017-01-17T14:43:33.717529] Finishing include; content='destination block elasticsearch2', depth='1' [2017-01-17T14:43:33.717869] Compiling #unnamed sequence [log] at [/etc/syslog-ng/syslog-ng.conf:23:3] [2017-01-17T14:43:33.717877] Compiling s_syslog reference [source] at [/etc/syslog-ng/syslog-ng.conf:23:3] [2017-01-17T14:43:33.717881] Compiling s_syslog sequence [source] at [/etc/syslog-ng/syslog-ng.conf:10:1] [2017-01-17T14:43:33.717884] Compiling #unnamed junction [log] at [/etc/syslog-ng/syslog-ng.conf:10:18] [2017-01-17T14:43:33.717887] Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf:10:19] [2017-01-17T14:43:33.717895] Compiling d_elastic reference [destination] at [/etc/syslog-ng/syslog-ng.conf:24:3] [2017-01-17T14:43:33.717899] Compiling d_elastic sequence [destination] at [/etc/syslog-ng/syslog-ng.conf:12:1] [2017-01-17T14:43:33.717902] Compiling #unnamed junction [log] at [/etc/syslog-ng/syslog-ng.conf:12:24] [2017-01-17T14:43:33.717905] Compiling #unnamed single [log] at [#buffer:2:3] [2017-01-17T14:43:33.718096] Module loaded and initialized successfully; module='syslogformat' [2017-01-17T14:43:33.824194] Add path to classpath: //usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar; [2017-01-17T14:43:33.824696] Add path to classpath: /usr/lib64/syslog-ng/java-modules/dummy.jar; [2017-01-17T14:43:33.824868] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-v2.jar; [2017-01-17T14:43:33.825032] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic.jar; [2017-01-17T14:43:33.825169] Add path to classpath: /usr/lib64/syslog-ng/java-modules/hdfs.jar; [2017-01-17T14:43:33.825322] Add path to classpath: /usr/lib64/syslog-ng/java-modules/http.jar; [2017-01-17T14:43:33.825470] Add path to classpath: /usr/lib64/syslog-ng/java-modules/kafka.jar; [2017-01-17T14:43:33.825629] Add path to classpath: /usr/lib64/syslog-ng/java-modules/log4j-1.2.16.jar; [2017-01-17T14:43:33.825781] Add path to classpath: /usr/lib64/syslog-ng/java-modules/syslog-ng-common.jar; [2017-01-17T14:43:33.825936] Add path to classpath: /usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar; [2017-01-17T14:43:33.826108] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-jest-client/commons-codec-1.9.jar; [2017-01-17T14:43:33.826286] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-jest-client/commons-lang3-3.4.jar; [2017-01-17T14:43:33.826459] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-jest-client/commons-logging-1.2.jar; [2017-01-17T14:43:33.826633] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-jest-client/gson-2.6.2.jar; [2017-01-17T14:43:33.826801] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-jest-client/guava-19.0.jar; [2017-01-17T14:43:33.826981] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-jest-client/httpasyncclient-4.1.1.jar; [2017-01-17T14:43:33.827156] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-jest-client/httpclient-4.5.2.jar; [2017-01-17T14:43:33.827343] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-jest-client/httpcore-4.4.4.jar; [2017-01-17T14:43:33.827520] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-jest-client/httpcore-nio-4.4.4.jar; [2017-01-17T14:43:33.827694] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-jest-client/jest-2.0.2.jar; [2017-01-17T14:43:33.827851] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-jest-client/jest-common-2.0.2.jar; [2017-01-17T14:43:33.828061] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-jest-client/slf4j-api-1.7.13.jar; [2017-01-17T14:43:33.828225] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-jest-client/slf4j-simple-1.7.13.jar; [2017-01-17T14:43:33.901817] Add path to classpath: //usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar; [2017-01-17T14:43:33.906737] Error initializing message pipeline; I’m sure I’m missing something basic – perhaps correct pathing of the ES .jar files? Many thanks for your help in advance. Damian Bell Damian Bell Infrastructure Engineer | Support | H Clarkson & Co Ltd Email: Damian.Bell(a)clarksons.com<mailto:Damian.Bell@clarksons.com> Group Email: infrastructure(a)clarksons.com Clarksons Platou TM Commodity Quay, St. Katharine Docks | London E1W 1BF | United Kingdom www.clarksons.com<http://www.clarksons.com> Please consider the environment before printing this e-mail ________________________________ This message is private and confidential. If you have received it in error, you are on notice of its status. Please notify us immediately by reply email and then delete this message from your system. Please do not copy it or use it for any purposes, or disclose its contents to any other person: to do so could be a breach of confidence. Emails may be monitored. Details of Clarkson group companies and their regulators (where applicable) can be found at this url: Disclosure<http://www.clarksons.com/disclosure/> ________________________________

5 7

Hitting g_assert when using sanitize-utf8 enabled!
by James Elstone 13 Jan '17

13 Jan '17

Hi Balabit et al, When using the sanitize-utf8 flag I am hitting a g_assert in modules/syslogformat/syslog-format.c; what could be causing this? Any advice welcome!! Kr, James -- Sent from my Android device with K-9 Mail. Please excuse my brevity.

3 13

unofficial syslog-ng 3.9.1 RPM packages
by Czanik, Péter 13 Jan '17

13 Jan '17

Hi, My unofficial syslog-ng 3.9.1 RPM packages are ready. They are now available from my "githead" repositories. This way I did not have to go through setting up separate 3.9 repositories over a very slow Internet connection and 3.9.1 is git HEAD right now anyway :) Supported distributions are: openSUSE Leap 42.1 and 42.2 SLES 11 and 12 Fedora 24, 25, 26 (rawhide) RHEL / CentOS 7 Instructions are available at https://www.balabit.com/blog/rpm-packages-from-syslog-ng-git-head/ Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 1

Insider 2017-01: syslog-ng 3.9.1; SearchGuard; FOSDEM & SCALE;
by Czanik, Péter 12 Jan '17

12 Jan '17

Dear syslog-ng users, This is the 54th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS syslog-ng 3.9.1 has been released ------------------------ Right before Christmas syslog-ng version 3.9.1 has been released, four months after the 3.8.1 release. It contains tons of bugfixes and many small incremental changes compared to the previous version. Performance has been improved at multiple places, Big Data drivers were updated, and secure logging to Elasticsearch is now possible using SearchGuard. For a complete list, read https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.9.1 where you can also find the source code .tgz file. For binary packages check https://syslog-ng.org/3rd-party-binaries/ Securing Elasticsearch access with SearchGuard ---------------------------------------------- Using syslog-ng to index events to your Elasticsearch cluster has been possible natively for some time now. Readabout using Search Guard support in syslog-ng 3.9 to encrypt and authenticate your connections to Elasticsearch. https://www.balabit.com/blog/securing-your-esk-stack-for-free-using-search-… Installing the latest syslog-ng DEB packages -------------------------------------------- The syslog-ng application is part of all major Linux distributions, and you can usually install syslog-ng from the official repositories. However, if you want to use the features of newer syslog-ng versions (for example, send log messages to Elasticsearch or Apache Kafka), you have to either compile the syslog-ng from source, or install it from unofficial repositories. This post explains you how to do that on DEB packages based systems: https://www.balabit.com/blog/installing-the-latest-syslog-ng-on-ubuntu-and-… RPM packages from syslog-ng git HEAD ------------------------------------ There are some situations, when even the latest stable release is not good enough as a bug was found and fixed after release. Or maybe you want to check the latest new features that are not yet available in a release. If you use an RPM-based distribution, packages generated from Git HEAD are right for you: https://www.balabit.com/blog/rpm-packages-from-syslog-ng-git-head/ UPCOMING EVENTS Peter Czanik will give two talks at FOSDEM (http://fosdem.org/) - Scaling your logging infrastructure: https://fosdem.org/2017/schedule/event/syslog_ng_scaling/ - Getting the most out of your security logs: https://fosdem.org/2017/schedule/event/syslog_ng_using/ Balabit will participate the Southern California Linux Expo this year again: http://www.socallinuxexpo.org/scale/15x You can talk to our engineers at the Balabit booth and listen to Peter Czanik about syslog-ng. NEW RELEASES - syslog-ng 3.9.1: https://www.balabit.com/blog/syslog-ng-3-9-1-released/ Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com. To read this newsletter online, visit: https://syslog-ng.org/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

is it possible to normalize the jSON keys to lowercase?
by Jorge Pereira 11 Jan '17

11 Jan '17

Hi, I am using the 'format-json' as my below snip. template t_raw2json { template("$(format-json --scope selected_macros \ --exclude TAGS \ --exclude DATE \ --exclude PRIORITY \ --exclude FACILITY \ --exclude PROGRAM )\n"); }; but, the problem is because sometimes I receive a jSON packet as { "Key": "value" } or {"key": "value}".... this has been a problem to my filter. e.g: destination d_nginx_modsec_log { file("/var/logs/myapp//${j.key:-invalid_key}.log" template(t_raw2json) ); }; Question: is it possible to normalize the jSON keys to lowercase? or someone has other approaches? -- Jorge Pereira

4 5