- syslog-ng - lists.balabit.hu

Re: [syslog-ng] extracting jSON from $MESSAGE
by Jorge Pereira 05 Jul '16

05 Jul '16

Hi Robert, On Sun, Jul 3, 2016 at 2:26 PM, Fekete, Róbert <robert.fekete(a)balabit.com> wrote: > Hi Jorge, > > This seems to be a bit tricky message. If I see it correctly, after the > syslog header, you have: > > * some additional info that ends with [captcha] (is that literally > [captcha], or it changes with every message?) > this value is dinamic. > * then you have some JSON > * "while logging request," string > * and finally some key:value pairs. > > exactly! > As a first try, I would use the csv-parser() to separate the message to > the four blocks I listed above. Use syslog-ng OSE 3.7, because then you can > use strings as delimiters (see > https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-…) > For example, you can use the ] character and the "while logging request," > string as delimiters. > Then you can run the json-parser on the macro containing the JSON-part of > the message, and a kv-parser with the : delimiter to parse the last block > if needed. (Note that for kv-parser part you need a recent development > version of syslog-ng). > > Currently, I'm using the version 3.5.6. I will update. > Make sure to use the prefix() option in the json-parser, because as I see > the key:value block seems to have some of the same keys that the json block. > > BTW, is this a publicly available application that emits such a log > message? It seems ideal to showcase the wide range of available syslog-ng > parsers :) > > yep! these are errors generated by OpenResty/Nginx. today I noticed that we have another kind of error like the below sample. 2016/07/04 08:40:56 [error] 14011#0: *410 lua entry thread aborted: runtime error: /usr/local/openresty/lua-my <http://www.kipras.com/>/share/src/my <http://www.kipras.com/>/util.lua:42: attempt to get length of local 'str' (a nil value) stack traceback: coroutine 0: /usr/local/openresty/lua-my <http://www.kipras.com/>/share/src/my <http://www.kipras.com/>/util.lua: in function 'string_split' /usr/local/openresty/lua-my <http://www.kipras.com/>/my <http://www.kipras.com/>/edge/upstream.lua:235: in function 'extract_peer' /usr/local/openresty/lua-my <http://www.kipras.com/>/my <http://www.kipras.com/>/edge/upstream.lua:264: in function 'connect' /usr/local/openresty/lua-my <http://www.kipras.com/>/my <http://www.kipras.com/>/edge/upstream.lua:147: in function 'request' /usr/local/openresty/lua-my <http://www.kipras.com/>/my <http://www.kipras.com/>/edge/init.lua:85: in function </usr/local/openresty/lua-my <http://www.kipras.com/>/my <http://www.kipras.com/>/edge/init.lua:75> /usr/local/openresty/lua-my <http://www.kipras.com/>/my <http://www.kipras.com/>/edge/init.lua:153: in function 'handler' /usr/local/openresty/lua-my <http://www.kipras.com/>/my <http://www.kipras.com/>/edge/init.lua:166: in function 'content' .../openresty/nginx/../lua-my <http://www.kipras.com/>/handlers/src/content.lua:1: in function <.../openresty/nginx/../lua-my <http://www.kipras.com/>/handlers/src/content.lua:1>, client: 192.168.33.1, server: www. <http://www.kipras.com/>my <http://www.kipras.com/>.com <http://www.kipras.com/>, request: "GET /favicon.ico HTTP/1.1", host: "www.my.com <http://www.kipras.com/>" Now for me looks to be impossible to handler this... do you have some idea? because, I receive this from nginx it over unix-stream and need to send to a remote syslog-ng (collector) saving like: /var/log/syslog-ng/www.my.com_error.log > Regards, > Robert > > On Sun, Jul 3, 2016 at 3:32 PM, Jorge Pereira <jpereiran(a)gmail.com> wrote: > >> Hi, >> >> I am not sure about the best approach and way to fix my problem, >> below more information. >> >> 1) I receive the below packet sent from a nginx/openresty instance. >> >> 2016/07/02 01:17:04 [emerg] 19081#0: *13163 [lua] init.lua:115: [captcha] >> {"fail_count":"","response_code":200,"client_ip":"192.168.1.22","hostname":"server-lab01","request_id":"2016-07-02T01:17:03Z|9175f93c0c||i0Xb3BuBWV","host":" >> www.mytest.com","http_request":{"verb":"GET","url":"\/","user-agent":"Mozilla\/5.0 >> (pc-x86_64-linux-gnu) Siege\/3.0.8","http_version":"1.1","all":"{\"host\":\" >> www.mytest.com\",\"x-country-code\":\"US\",\"connection\":\"close\",\"accept\":\"*\\\/*\",\"x-client-ip\":\"192.168.1.22\",\"user-agent\":\"Mozilla\\\/5.0 >> (pc-x86_64-linux-gnu) >> Siege\\\/3.0.8\",\"accept-encoding\":\"gzip\"}"},"geoip":{"location":"-90.5334,38.6500","city_name":"Chesterfield","country_name":"United >> States","longitude":-90.5334,"area_code":314,"latitude":38.65,"country_code2":"US","country_code3":"USA"},"got":"","action":"show","expected":"h1szmM","webapp_domain":" >> www.mytest.com"} while logging request, client: 192.168.1.22, server: >> www.mytest.com, request: "GET / HTTP/1.1", host: "www.mytest.com" >> >> 2) In my server side, I need to save the logs following a value of *host: >> "www.mytest.com <http://www.mytest.com>"* like: >> >> /var/log/syslog-ng/www.mytest.com.log >> >> 3) The problem is because the packet received has a part being a jSON, >> but I can't use the *json-parser().* >> >> 4) What is the best approach? I have used: >> >> # Extracting only the jSON payload >> rewrite p_nginx_wb_error_log_clean { >> subst(".*captcha] ", "", value("MESSAGE"), flags("global")); >> subst(" while logging request.*$", "", value("MESSAGE"), >> flags("global")); >> }; >> >> parser p_nginx_wb_error_log_json { >> json-parser( >> marker("") >> prefix("j.") >> ); >> }; >> >> destination d_nginx_wb_error_log { >> >> file("/var/log/syslog-ng/nginx/${j.webapp_domain:-unknow-payload}_error.log" >> >> >> >> create_dirs(yes) >> owner("root") >> group("root") >> perm(0644) >> dir_perm(0755) >> template("${MSG}\n") >> ); >> }; >> >> -- >> Jorge Pereira >> >> >> ______________________________________________________________________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: >> http://www.balabit.com/support/documentation/?product=syslog-ng >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> >> >> > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > >

2 1

Re: [syslog-ng] extracting jSON from $MESSAGE
by jrhendri 03 Jul '16

03 Jul '16

Hanes you looked at using patterndb? That should let you assign the specific parts of the message to variables you choose and then use them as you like.It should be possible to do it with json parser but I have not used it personally. Jim Sent from my Verizon, Samsung Galaxy smartphone -------- Original message --------From: "Fekete, Róbert" <robert.fekete(a)balabit.com> Date: 7/3/16 1:26 PM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] extracting jSON from $MESSAGE Hi Jorge, This seems to be a bit tricky message. If I see it correctly, after the syslog header, you have: * some additional info that ends with [captcha] (is that literally [captcha], or it changes with every message?) * then you have some JSON * "while logging request," string * and finally some key:value pairs. As a first try, I would use the csv-parser() to separate the message to the four blocks I listed above. Use syslog-ng OSE 3.7, because then you can use strings as delimiters (see https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-…) For example, you can use the ] character and the "while logging request," string as delimiters. Then you can run the json-parser on the macro containing the JSON-part of the message, and a kv-parser with the : delimiter to parse the last block if needed. (Note that for kv-parser part you need a recent development version of syslog-ng). Make sure to use the prefix() option in the json-parser, because as I see the key:value block seems to have some of the same keys that the json block. BTW, is this a publicly available application that emits such a log message? It seems ideal to showcase the wide range of available syslog-ng parsers :) Regards, Robert On Sun, Jul 3, 2016 at 3:32 PM, Jorge Pereira <jpereiran(a)gmail.com> wrote: Hi, I am not sure about the best approach and way to fix my problem, below more information. 1) I receive the below packet sent from a nginx/openresty instance. 2016/07/02 01:17:04 [emerg] 19081#0: *13163 [lua] init.lua:115: [captcha] {"fail_count":"","response_code":200,"client_ip":"192.168.1.22","hostname":"server-lab01","request_id":"2016-07-02T01:17:03Z|9175f93c0c||i0Xb3BuBWV","host":"www.mytest.com","http_request":{"verb":"GET","url":"\/","user-agent":"Mozilla\/5.0 (pc-x86_64-linux-gnu) Siege\/3.0.8","http_version":"1.1","all":"{\"host\":\"www.mytest.com\",\"x-country-code\":\"US\",\"connection\":\"close\",\"accept\":\"*\\\/*\",\"x-client-ip\":\"192.168.1.22\",\"user-agent\":\"Mozilla\\\/5.0 (pc-x86_64-linux-gnu) Siege\\\/3.0.8\",\"accept-encoding\":\"gzip\"}"},"geoip":{"location":"-90.5334,38.6500","city_name":"Chesterfield","country_name":"United States","longitude":-90.5334,"area_code":314,"latitude":38.65,"country_code2":"US","country_code3":"USA"},"got":"","action":"show","expected":"h1szmM","webapp_domain":"www.mytest.com"} while logging request, client: 192.168.1.22, server: www.mytest.com, request: "GET / HTTP/1.1", host: "www.mytest.com" 2) In my server side, I need to save the logs following a value of host: "www.mytest.com" like: /var/log/syslog-ng/www.mytest.com.log 3) The problem is because the packet received has a part being a jSON, but I can't use the json-parser(). 4) What is the best approach? I have used: # Extracting only the jSON payloadrewrite p_nginx_wb_error_log_clean { subst(".*captcha] ", "", value("MESSAGE"), flags("global")); subst(" while logging request.*$", "", value("MESSAGE"), flags("global"));}; parser p_nginx_wb_error_log_json { json-parser( marker("") prefix("j.") ); }; destination d_nginx_wb_error_log { file("/var/log/syslog-ng/nginx/${j.webapp_domain:-unknow-payload}_error.log" create_dirs(yes) owner("root") group("root") perm(0644) dir_perm(0755) template("${MSG}\n") ); }; -- Jorge Pereira ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

extracting jSON from $MESSAGE
by Jorge Pereira 03 Jul '16

03 Jul '16

Hi, I am not sure about the best approach and way to fix my problem, below more information. 1) I receive the below packet sent from a nginx/openresty instance. 2016/07/02 01:17:04 [emerg] 19081#0: *13163 [lua] init.lua:115: [captcha] {"fail_count":"","response_code":200,"client_ip":"192.168.1.22","hostname":"server-lab01","request_id":"2016-07-02T01:17:03Z|9175f93c0c||i0Xb3BuBWV","host":" www.mytest.com","http_request":{"verb":"GET","url":"\/","user-agent":"Mozilla\/5.0 (pc-x86_64-linux-gnu) Siege\/3.0.8","http_version":"1.1","all":"{\"host\":\" www.mytest.com\",\"x-country-code\":\"US\",\"connection\":\"close\",\"accept\":\"*\\\/*\",\"x-client-ip\":\"192.168.1.22\",\"user-agent\":\"Mozilla\\\/5.0 (pc-x86_64-linux-gnu) Siege\\\/3.0.8\",\"accept-encoding\":\"gzip\"}"},"geoip":{"location":"-90.5334,38.6500","city_name":"Chesterfield","country_name":"United States","longitude":-90.5334,"area_code":314,"latitude":38.65,"country_code2":"US","country_code3":"USA"},"got":"","action":"show","expected":"h1szmM","webapp_domain":" www.mytest.com"} while logging request, client: 192.168.1.22, server: www.mytest.com, request: "GET / HTTP/1.1", host: "www.mytest.com" 2) In my server side, I need to save the logs following a value of *host: "www.mytest.com <http://www.mytest.com>"* like: /var/log/syslog-ng/www.mytest.com.log 3) The problem is because the packet received has a part being a jSON, but I can't use the *json-parser().* 4) What is the best approach? I have used: # Extracting only the jSON payload rewrite p_nginx_wb_error_log_clean { subst(".*captcha] ", "", value("MESSAGE"), flags("global")); subst(" while logging request.*$", "", value("MESSAGE"), flags("global")); }; parser p_nginx_wb_error_log_json { json-parser( marker("") prefix("j.") ); }; destination d_nginx_wb_error_log { file("/var/log/syslog-ng/nginx/${j.webapp_domain:-unknow-payload}_error.log" create_dirs(yes) owner("root") group("root") perm(0644) dir_perm(0755) template("${MSG}\n") ); }; -- Jorge Pereira

2 1

Re: [syslog-ng] Syslog-ng Multiple Instances
by David Campeau 01 Jul '16

01 Jul '16

Hello, I've been using syslog-ng to filter syslog before forwarding on to a log collector. However, I need to spin up a second instance for testing purposes. I've found a little bit of information on-line, but it hasn't completed the entire picture. This is the command used to start up the 2nd instance. I'm pointing to separate .conf .persist .pid and .ctl files -- However, it's still not working. I suspect the issue is due to OS log sources. How do a change log sources? syslog-ng --cfgfile=/usr/local/bin/syslog-ng-Second-Instance/syslog-ng.conf --persist-file=/usr/local/bin/syslog-ng-Second-Instance/syslog-ng.persist --pidfile=/usr/local/bin/syslog-ng-Second-Instance/syslog-ng.pid --control=/usr/local/bin/syslog-ng-Second-Instance/syslog-ng.ctl & This is the upper part of the syslog-ng.conf file for the 2nd instance I wish to run. @version: 3.3.4 @include "scl.conf" options { time-reap(30); mark-freq(10); keep-hostname(yes); chain-hostnames(no); use-dns(no); ## log-fifo-size(500000); ## Tuning Options ## flush_lines(10000); ## Tuning Options ## flush_timeout(10000); ## Tuning Options }; source s_second_instance { syslog(transport("udp") port("518")); #### Will receive test syslog on port 518 }; destination d_syslog_udp { syslog("10.X.X.X" transport("udp") port("514") throttle(4000) ); }; I'm hoping someone has experience or has seen information on how to run a 2nd instance on the same box. Best Regards, David

3 7

getting some information about the sender process
by Attila Szalai 30 Jun '16

30 Jun '16

Hi, Maybe I'm totally wrong, but I have/had the impression that syslog-ng collects some information about the client process if that connects through a unix domain socket. But actually I could not find anything related to this neither in the documentation nor the code. Is this something that only exists in my mind or I just missed something?

2 2

a log message is output in the two line
by Hirose, Shinsaku 30 Jun '16

30 Jun '16

Hello, all, I use syslog-ng-3.2.5-4.el6.x86_64 on Centos6. I got it from eple epel repository. I am troubled in how to use the source file driver. The touble is that a log message is output in the two lines on remote syslog server. How to reproduce is followings. 1. Prepare two hosts running syslog-ng. Host_A configuration is followings. ---------------------------------------- source test { file("/tmp/a.log"); }; destination d_remote { udp("192.168.0.2"); }; log { source(test); destination(d_remote); }; ---------------------------------------- Host_B(192.168.0.2) configuration is defaults. 2.Execute following command on Host_A. $ seq 8193 | (xargs -i echo -n "a";echo "") >> /tmp/a.log 3.Check the log on Host_B. As the result, a log message is output in the two lines on Host_B. One line is following. The num of "a" is 8192. aaaaaaaaaaaaa....... The other line is following. The num of "a" is 1. a I hope a log message is output in the one line on Host_B. Is my hope readily achievable? Please advise me.

5 7

CzP in Nuremberg
by Czanik, Péter 21 Jun '16

21 Jun '16

Hi, For the next couple of days I'll be in Nuremberg for the openSUSE conference: https://events.opensuse.org/conference/oSC16 I'll give a presentation on syslog-ng Friday in the afternoon at 13:30 in the Roter Salon. I'll wear my syslog-ng superhero t-shirt, so I'll be easy to spot and ready to answer syslog-ng questions any time. If you are in Nuremberg, but not at the conference, I'm happy to talk to you about syslog-ng also outside of the conference floor. You can reach me by e-mail or on twitter as @pczanik See you soon in Nuremberg, Peter

1 0

Syslog-ng 3.7.3 + Elasticsearch V2.3 - Error init pipeline
by Marco Mignone 16 Jun '16

16 Jun '16

Hi All, I am trying to setup syslog-ng to use elasticsearch as its destination on Ubuntu 14.04. This the version of syslog (the unofficial versions installed from laszlo_budai rep: syslog-ng 3.7.3 Installer-Version: 3.7.3 Revision: 3.7.3-8 Compile-Date: Jun 1 2016 16:33:00 Available-Modules: basicfuncs,linux-kmsg-format,riemann,afuser,afstomp,afprog,json-plugin,afsmtp,affile,csvparser,mod-java,pseudofile,confgen,afsocket,afamqp,redis,sdjournal,kvformat,syslogformat,afsql,system-source,mod-python,graphite,dbparser,geoip-plugin,afmongodb,cryptofuncs Elastic search is: "name" : "Theresa Cassidy", "cluster_name" : "elasticsearch", "version" : { "number" : "2.3.3", "build_hash" : "218bdf10790eef486ff2c41a3df5cfa32dadcfde", "build_timestamp" : "2016-05-17T15:40:04Z", "build_snapshot" : false, "lucene_version" : “5.5.0" And my custom configuration in /etc/syslog-ng/conf.d/test.conf which is: @module mod-java source s_net { udp(); tcp(); }; destination d_elastic { elasticsearch( index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("test") client_lib_dir("/usr/share/elasticsearch/lib") ); }; log { source(s_net); destination(d_elastic); flags(flow-control); }; When I try to launch syslog in debug mode this is what I get: [2016-06-16T15:54:29.378356] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/syslog-ng-core.jar; [2016-06-16T15:54:29.382446] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/syslog-ng-core.jar; [2016-06-16T15:54:29.382660] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/kafka.jar; [2016-06-16T15:54:29.382862] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/http.jar; [2016-06-16T15:54:29.383052] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/hdfs.jar; [2016-06-16T15:54:29.383258] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/elastic.jar; [2016-06-16T15:54:29.383479] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/syslog-ng-common.jar; [2016-06-16T15:54:29.383670] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/log4j-1.2.16.jar; [2016-06-16T15:54:29.383917] Add path to classpath: /usr/share/elasticsearch/lib/guava-18.0.jar; [2016-06-16T15:54:29.384098] Add path to classpath: /usr/share/elasticsearch/lib/jna-4.1.0.jar; [2016-06-16T15:54:29.384293] Add path to classpath: /usr/share/elasticsearch/lib/spatial4j-0.5.jar; [2016-06-16T15:54:29.384494] Add path to classpath: /usr/share/elasticsearch/lib/compress-lzf-1.0.2.jar; [2016-06-16T15:54:29.386104] Add path to classpath: /usr/share/elasticsearch/lib/lucene-spatial-5.5.0.jar; [2016-06-16T15:54:29.386342] Add path to classpath: /usr/share/elasticsearch/lib/lucene-backward-codecs-5.5.0.jar; [2016-06-16T15:54:29.386507] Add path to classpath: /usr/share/elasticsearch/lib/t-digest-3.0.jar; [2016-06-16T15:54:29.386677] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.6.6.jar; [2016-06-16T15:54:29.386865] Add path to classpath: /usr/share/elasticsearch/lib/lucene-memory-5.5.0.jar; [2016-06-16T15:54:29.387044] Add path to classpath: /usr/share/elasticsearch/lib/jackson-core-2.6.6.jar; [2016-06-16T15:54:29.387216] Add path to classpath: /usr/share/elasticsearch/lib/joda-time-2.8.2.jar; [2016-06-16T15:54:29.387394] Add path to classpath: /usr/share/elasticsearch/lib/lucene-join-5.5.0.jar; [2016-06-16T15:54:29.387673] Add path to classpath: /usr/share/elasticsearch/lib/lucene-grouping-5.5.0.jar; [2016-06-16T15:54:29.388476] Add path to classpath: /usr/share/elasticsearch/lib/HdrHistogram-2.1.6.jar; [2016-06-16T15:54:29.388647] Add path to classpath: /usr/share/elasticsearch/lib/compiler-0.8.13.jar; [2016-06-16T15:54:29.388818] Add path to classpath: /usr/share/elasticsearch/lib/netty-3.10.5.Final.jar; [2016-06-16T15:54:29.388972] Add path to classpath: /usr/share/elasticsearch/lib/lucene-misc-5.5.0.jar; [2016-06-16T15:54:29.389518] Add path to classpath: /usr/share/elasticsearch/lib/lucene-analyzers-common-5.5.0.jar; [2016-06-16T15:54:29.389711] Add path to classpath: /usr/share/elasticsearch/lib/lucene-sandbox-5.5.0.jar; [2016-06-16T15:54:29.390094] Add path to classpath: /usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar; [2016-06-16T15:54:29.390283] Add path to classpath: /usr/share/elasticsearch/lib/commons-cli-1.3.1.jar; [2016-06-16T15:54:29.390488] Add path to classpath: /usr/share/elasticsearch/lib/snakeyaml-1.15.jar; [2016-06-16T15:54:29.390659] Add path to classpath: /usr/share/elasticsearch/lib/jsr166e-1.1.0.jar; [2016-06-16T15:54:29.390935] Add path to classpath: /usr/share/elasticsearch/lib/lucene-core-5.5.0.jar; [2016-06-16T15:54:29.391176] Add path to classpath: /usr/share/elasticsearch/lib/lucene-suggest-5.5.0.jar; [2016-06-16T15:54:29.394616] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.6.6.jar; [2016-06-16T15:54:29.395279] Add path to classpath: /usr/share/elasticsearch/lib/log4j-1.2.17.jar; [2016-06-16T15:54:29.395458] Add path to classpath: /usr/share/elasticsearch/lib/joda-convert-1.2.jar; [2016-06-16T15:54:29.395970] Add path to classpath: /usr/share/elasticsearch/lib/hppc-0.7.1.jar; [2016-06-16T15:54:29.396734] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-smile-2.6.6.jar; [2016-06-16T15:54:29.397919] Add path to classpath: /usr/share/elasticsearch/lib/lucene-queryparser-5.5.0.jar; [2016-06-16T15:54:29.398106] Add path to classpath: /usr/share/elasticsearch/lib/lucene-spatial3d-5.5.0.jar; [2016-06-16T15:54:29.398281] Add path to classpath: /usr/share/elasticsearch/lib/lucene-queries-5.5.0.jar; [2016-06-16T15:54:29.398440] Add path to classpath: /usr/share/elasticsearch/lib/apache-log4j-extras-1.2.17.jar; [2016-06-16T15:54:29.398610] Add path to classpath: /usr/share/elasticsearch/lib/lucene-highlighter-5.5.0.jar; [2016-06-16T15:54:29.398784] Add path to classpath: /usr/share/elasticsearch/lib/jts-1.13.jar; [2016-06-16T15:54:29.398925] Add path to classpath: /usr/share/elasticsearch/lib/securesm-1.0.jar; [2016-06-16T15:54:29.501879] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/syslog-ng-core.jar; [2016-06-16T15:54:29.519443] Error initializing message pipeline; Was anyone able to make this work or suggest a way to fix this? I hope I am on the right place and if not apologies in advance. Thanks, Marco

3 3

Re: [syslog-ng] Syslog-ng 3.7.3 + Elasticsearch V2.3 - Error init pipeline
by Laszlo Budai 16 Jun '16

16 Jun '16

Hi, it should work with the latest 1.x. I'm working on my 3.8 OBS repo, but having some problems. When I have the alpha packages, I'll send a mail. Regards, Laszlo Budai Sent from my HTC ----- Reply message ----- From: "Marco Mignone" <info(a)marcomignone.com> To: "Syslog-ng users' and developers' mailing list" <syslog-ng(a)lists.balabit.hu> Subject: [syslog-ng] Syslog-ng 3.7.3 + Elasticsearch V2.3 - Error init pipeline Date: Thu, Jun 16, 2016 17:34 Hi Peter, Thanks for the quick reply. I thought there was some hack or way around it. I will then try with ElasticSearch1.X. Will the latest version of 1.x work or is there a specific version? Thanks again for your time. Marco > On 16 Jun 2016, at 16:26, Czanik, Péter <peter.czanik(a)balabit.com> wrote: > > Hi, > > syslog-ng 3.7 only works with Elasticsearch 1.X. For Elasticsearch 2.X > you need syslog-ng 3.8, which is still under development (alpha). > > Bye, > Peter Czanik (CzP) <peter.czanik(a)balabit.com> > Balabit / syslog-ng upstream > http://czanik.blogs.balabit.com/ > https://twitter.com/PCzanik > > > On Thu, Jun 16, 2016 at 5:11 PM, Marco Mignone <info(a)marcomignone.com> wrote: >> Hi All, >> I am trying to setup syslog-ng to use elasticsearch as its destination on >> Ubuntu 14.04. >> This the version of syslog (the unofficial versions installed from >> laszlo_budai rep: >> >> syslog-ng 3.7.3 >> Installer-Version: 3.7.3 >> Revision: 3.7.3-8 >> Compile-Date: Jun 1 2016 16:33:00 >> Available-Modules: >> basicfuncs,linux-kmsg-format,riemann,afuser,afstomp,afprog,json-plugin,afsmtp,affile,csvparser,mod-java,pseudofile,confgen,afsocket,afamqp,redis,sdjournal,kvformat,syslogformat,afsql,system-source,mod-python,graphite,dbparser,geoip-plugin,afmongodb,cryptofuncs >> >> Elastic search is: >> >> "name" : "Theresa Cassidy", >> "cluster_name" : "elasticsearch", >> "version" : { >> "number" : "2.3.3", >> "build_hash" : "218bdf10790eef486ff2c41a3df5cfa32dadcfde", >> "build_timestamp" : "2016-05-17T15:40:04Z", >> "build_snapshot" : false, >> "lucene_version" : “5.5.0" >> >> >> And my custom configuration in /etc/syslog-ng/conf.d/test.conf which is: >> >> @module mod-java >> >> source s_net { >> udp(); >> tcp(); >> }; >> >> destination d_elastic { >> elasticsearch( >> index("syslog-ng_${YEAR}.${MONTH}.${DAY}") >> type("test") >> client_lib_dir("/usr/share/elasticsearch/lib") >> ); >> }; >> >> log { >> source(s_net); >> destination(d_elastic); >> flags(flow-control); >> }; >> >> >> When I try to launch syslog in debug mode this is what I get: >> >> [2016-06-16T15:54:29.378356] Add path to classpath: >> /usr/lib/syslog-ng/3.7/java-modules/syslog-ng-core.jar; >> [2016-06-16T15:54:29.382446] Add path to classpath: >> /usr/lib/syslog-ng/3.7/java-modules/syslog-ng-core.jar; >> [2016-06-16T15:54:29.382660] Add path to classpath: >> /usr/lib/syslog-ng/3.7/java-modules/kafka.jar; >> [2016-06-16T15:54:29.382862] Add path to classpath: >> /usr/lib/syslog-ng/3.7/java-modules/http.jar; >> [2016-06-16T15:54:29.383052] Add path to classpath: >> /usr/lib/syslog-ng/3.7/java-modules/hdfs.jar; >> [2016-06-16T15:54:29.383258] Add path to classpath: >> /usr/lib/syslog-ng/3.7/java-modules/elastic.jar; >> [2016-06-16T15:54:29.383479] Add path to classpath: >> /usr/lib/syslog-ng/3.7/java-modules/syslog-ng-common.jar; >> [2016-06-16T15:54:29.383670] Add path to classpath: >> /usr/lib/syslog-ng/3.7/java-modules/log4j-1.2.16.jar; >> [2016-06-16T15:54:29.383917] Add path to classpath: >> /usr/share/elasticsearch/lib/guava-18.0.jar; >> [2016-06-16T15:54:29.384098] Add path to classpath: >> /usr/share/elasticsearch/lib/jna-4.1.0.jar; >> [2016-06-16T15:54:29.384293] Add path to classpath: >> /usr/share/elasticsearch/lib/spatial4j-0.5.jar; >> [2016-06-16T15:54:29.384494] Add path to classpath: >> /usr/share/elasticsearch/lib/compress-lzf-1.0.2.jar; >> [2016-06-16T15:54:29.386104] Add path to classpath: >> /usr/share/elasticsearch/lib/lucene-spatial-5.5.0.jar; >> [2016-06-16T15:54:29.386342] Add path to classpath: >> /usr/share/elasticsearch/lib/lucene-backward-codecs-5.5.0.jar; >> [2016-06-16T15:54:29.386507] Add path to classpath: >> /usr/share/elasticsearch/lib/t-digest-3.0.jar; >> [2016-06-16T15:54:29.386677] Add path to classpath: >> /usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.6.6.jar; >> [2016-06-16T15:54:29.386865] Add path to classpath: >> /usr/share/elasticsearch/lib/lucene-memory-5.5.0.jar; >> [2016-06-16T15:54:29.387044] Add path to classpath: >> /usr/share/elasticsearch/lib/jackson-core-2.6.6.jar; >> [2016-06-16T15:54:29.387216] Add path to classpath: >> /usr/share/elasticsearch/lib/joda-time-2.8.2.jar; >> [2016-06-16T15:54:29.387394] Add path to classpath: >> /usr/share/elasticsearch/lib/lucene-join-5.5.0.jar; >> [2016-06-16T15:54:29.387673] Add path to classpath: >> /usr/share/elasticsearch/lib/lucene-grouping-5.5.0.jar; >> [2016-06-16T15:54:29.388476] Add path to classpath: >> /usr/share/elasticsearch/lib/HdrHistogram-2.1.6.jar; >> [2016-06-16T15:54:29.388647] Add path to classpath: >> /usr/share/elasticsearch/lib/compiler-0.8.13.jar; >> [2016-06-16T15:54:29.388818] Add path to classpath: >> /usr/share/elasticsearch/lib/netty-3.10.5.Final.jar; >> [2016-06-16T15:54:29.388972] Add path to classpath: >> /usr/share/elasticsearch/lib/lucene-misc-5.5.0.jar; >> [2016-06-16T15:54:29.389518] Add path to classpath: >> /usr/share/elasticsearch/lib/lucene-analyzers-common-5.5.0.jar; >> [2016-06-16T15:54:29.389711] Add path to classpath: >> /usr/share/elasticsearch/lib/lucene-sandbox-5.5.0.jar; >> [2016-06-16T15:54:29.390094] Add path to classpath: >> /usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar; >> [2016-06-16T15:54:29.390283] Add path to classpath: >> /usr/share/elasticsearch/lib/commons-cli-1.3.1.jar; >> [2016-06-16T15:54:29.390488] Add path to classpath: >> /usr/share/elasticsearch/lib/snakeyaml-1.15.jar; >> [2016-06-16T15:54:29.390659] Add path to classpath: >> /usr/share/elasticsearch/lib/jsr166e-1.1.0.jar; >> [2016-06-16T15:54:29.390935] Add path to classpath: >> /usr/share/elasticsearch/lib/lucene-core-5.5.0.jar; >> [2016-06-16T15:54:29.391176] Add path to classpath: >> /usr/share/elasticsearch/lib/lucene-suggest-5.5.0.jar; >> [2016-06-16T15:54:29.394616] Add path to classpath: >> /usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.6.6.jar; >> [2016-06-16T15:54:29.395279] Add path to classpath: >> /usr/share/elasticsearch/lib/log4j-1.2.17.jar; >> [2016-06-16T15:54:29.395458] Add path to classpath: >> /usr/share/elasticsearch/lib/joda-convert-1.2.jar; >> [2016-06-16T15:54:29.395970] Add path to classpath: >> /usr/share/elasticsearch/lib/hppc-0.7.1.jar; >> [2016-06-16T15:54:29.396734] Add path to classpath: >> /usr/share/elasticsearch/lib/jackson-dataformat-smile-2.6.6.jar; >> [2016-06-16T15:54:29.397919] Add path to classpath: >> /usr/share/elasticsearch/lib/lucene-queryparser-5.5.0.jar; >> [2016-06-16T15:54:29.398106] Add path to classpath: >> /usr/share/elasticsearch/lib/lucene-spatial3d-5.5.0.jar; >> [2016-06-16T15:54:29.398281] Add path to classpath: >> /usr/share/elasticsearch/lib/lucene-queries-5.5.0.jar; >> [2016-06-16T15:54:29.398440] Add path to classpath: >> /usr/share/elasticsearch/lib/apache-log4j-extras-1.2.17.jar; >> [2016-06-16T15:54:29.398610] Add path to classpath: >> /usr/share/elasticsearch/lib/lucene-highlighter-5.5.0.jar; >> [2016-06-16T15:54:29.398784] Add path to classpath: >> /usr/share/elasticsearch/lib/jts-1.13.jar; >> [2016-06-16T15:54:29.398925] Add path to classpath: >> /usr/share/elasticsearch/lib/securesm-1.0.jar; >> [2016-06-16T15:54:29.501879] Add path to classpath: >> /usr/lib/syslog-ng/3.7/java-modules/syslog-ng-core.jar; >> [2016-06-16T15:54:29.519443] Error initializing message pipeline; >> >> >> Was anyone able to make this work or suggest a way to fix this? >> I hope I am on the right place and if not apologies in advance. >> >> Thanks, >> Marco >> >> >> ______________________________________________________________________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: >> http://www.balabit.com/support/documentation/?product=syslog-ng >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> >> > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

2 1

Is there a standard for naming tag/value pairs when parsing
by Evan Rempel 16 Jun '16

16 Jun '16

There was a project by Mitre (https://www.mitre.org/) called the Common Event Expression (https://cee.mitre.org/) that was going to be the official standard for metadata names for events, but that project has been stopped. Other than the two references that the CEE project has for logging standardization efforts, does anyone know of any major efforts by any group to define a standard for metadata naming? Evan.

3 6