- syslog-ng - lists.balabit.hu

NG Rewrite Capabilities on a Relay
by Schoonover, Mark E HHHH 18 Jul '16

18 Jul '16

Hello, I've run into a situation where I need to rewrite messages before sending them to our central NG server. When NG is used as a relay, is the rewrite functionality available? I've been through the manual but there isn't a mention of what functionality is available - other than writing logs to disk. Regards, Mark Schoonover Infrastructure Engineering Manager ENE : Tools, Instrumentation and Common Services Team Office: 32.8697° N, 116.9711° W Phone : 770-261-7934 Email : mark.schoonover(a)cigna.com<mailto:mark.schoonover@cigna.com> Confidential, unpublished property of Cigna Health & Life Insurance Co.. Do not duplicate or distribute. Use and distribution limited solely to authorized personnel. ©2016 CIGNA. ------------------------------------------------------------------------------ CONFIDENTIALITY NOTICE: If you have received this email in error, please immediately notify the sender by e-mail at the address shown. This email transmission may contain confidential information. This information is intended only for the use of the individual(s) or entity to whom it is intended even if addressed incorrectly. Please delete it from your files if you are not the intended recipient. Thank you for your compliance. Copyright (c) 2016 Cigna ==============================================================================

2 2

New ETPLC project with Syslog-NG for checking >9000 Threats on your logs!
by rmkml 13 Jul '16

13 Jul '16

Hello, I am pround to announce the new http://etplc.org open source project update with Syslog-NG for checking more than 9000 Threats on your webserver/proxy logs! It's a open source project, all feedbacks / informations are welcome. Easy to use since 3 years ago ;) 1) add ETPLC on your Syslog-NG configuration like that: (of course check before perl+etplc PATH and source/filter/destination configurations...) destination d_prog { program("/usr/bin/perl /var/tmp/etplc_12jul2016a.pl -f /var/tmp/emergingall_sigs11jul2016a_snort290b.rules -s"); }; log { source(s_src); destination(d_prog); }; 2) ETPLC send alert to localhost:514/udp with "-s" option 3) See All options with "-h" 4) Already supported format is Squid, Apache, Nginx, ForeFront, BlueCoat, McAfee Web Gateway, IIS logs... 5) ETPLC exist on Perl and Python versions ETPLC available on: -main http://etplc.org -http://sourceforge.net/projects/etplc/ -https://github.com/rmkml/etplc -https://hub.docker.com/r/rmkml/etplc/ -http://twitter.com/rmkml Special THX to InfoSec community and @EmergingThreats team! Best Regards @Rmkml

2 1

Need mvn repository info for syslog-ng-core.jar
by Kiran Doddi 12 Jul '16

12 Jul '16

Hi, We are using syslog-ng server for receiving logs from different sources & we need to configure Java Destination for writing those logs into local file. For this, While implementing, I found syslog-ng-core.jar file, but I could not found the same anywhere in maven repo, so that I can add its dependency in my pom.xml & avoid putting it manually in jar bundling. Kindly need your assistance to solve my issues. Thanks & Regards, Kiran

3 2

Re: [syslog-ng] Transport aux data overflow
by Scheidler, Balázs 07 Jul '16

07 Jul '16

We get various data points on the process connecting via /dev/log via recvmsg() that returns this in an aux buffer, statically sized to 1024. It's interesting that you have more. If you can reproduce it, can you strace syslog-ng to see what is coming aux data? This can be resized but that needs a recompilation. On Jul 6, 2016 11:18 PM, "Noémi Ványi" <sitbackandwait(a)gmail.com> wrote: Hi! It seems to me that aux buffer size is a hard-coded value. So it cannot be configured using the config file. I guess the size could be increased, but I am curious what the devs of syslog-ng would say. :) On 6 July 2016 at 22:09, Evan Rempel <erempel(a)uvic.ca> wrote: > I got this log line out of syslog-ng, but I'm not sure what it really > means. > > Can anyone shed some slight on it and let me know how I can increase the > aux buffer size? > > syslog-ng[29712]: Transport aux data overflow, some fields may not be > associated with the message, please increase aux buffer size; > aux_size='1024' > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

Transport aux data overflow
by Evan Rempel 06 Jul '16

06 Jul '16

I got this log line out of syslog-ng, but I'm not sure what it really means. Can anyone shed some slight on it and let me know how I can increase the aux buffer size? syslog-ng[29712]: Transport aux data overflow, some fields may not be associated with the message, please increase aux buffer size; aux_size='1024'

2 1

Re: [syslog-ng] extracting jSON from $MESSAGE
by Jorge Pereira 05 Jul '16

05 Jul '16

Hi Robert, On Sun, Jul 3, 2016 at 2:26 PM, Fekete, Róbert <robert.fekete(a)balabit.com> wrote: > Hi Jorge, > > This seems to be a bit tricky message. If I see it correctly, after the > syslog header, you have: > > * some additional info that ends with [captcha] (is that literally > [captcha], or it changes with every message?) > this value is dinamic. > * then you have some JSON > * "while logging request," string > * and finally some key:value pairs. > > exactly! > As a first try, I would use the csv-parser() to separate the message to > the four blocks I listed above. Use syslog-ng OSE 3.7, because then you can > use strings as delimiters (see > https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-…) > For example, you can use the ] character and the "while logging request," > string as delimiters. > Then you can run the json-parser on the macro containing the JSON-part of > the message, and a kv-parser with the : delimiter to parse the last block > if needed. (Note that for kv-parser part you need a recent development > version of syslog-ng). > > Currently, I'm using the version 3.5.6. I will update. > Make sure to use the prefix() option in the json-parser, because as I see > the key:value block seems to have some of the same keys that the json block. > > BTW, is this a publicly available application that emits such a log > message? It seems ideal to showcase the wide range of available syslog-ng > parsers :) > > yep! these are errors generated by OpenResty/Nginx. today I noticed that we have another kind of error like the below sample. 2016/07/04 08:40:56 [error] 14011#0: *410 lua entry thread aborted: runtime error: /usr/local/openresty/lua-my <http://www.kipras.com/>/share/src/my <http://www.kipras.com/>/util.lua:42: attempt to get length of local 'str' (a nil value) stack traceback: coroutine 0: /usr/local/openresty/lua-my <http://www.kipras.com/>/share/src/my <http://www.kipras.com/>/util.lua: in function 'string_split' /usr/local/openresty/lua-my <http://www.kipras.com/>/my <http://www.kipras.com/>/edge/upstream.lua:235: in function 'extract_peer' /usr/local/openresty/lua-my <http://www.kipras.com/>/my <http://www.kipras.com/>/edge/upstream.lua:264: in function 'connect' /usr/local/openresty/lua-my <http://www.kipras.com/>/my <http://www.kipras.com/>/edge/upstream.lua:147: in function 'request' /usr/local/openresty/lua-my <http://www.kipras.com/>/my <http://www.kipras.com/>/edge/init.lua:85: in function </usr/local/openresty/lua-my <http://www.kipras.com/>/my <http://www.kipras.com/>/edge/init.lua:75> /usr/local/openresty/lua-my <http://www.kipras.com/>/my <http://www.kipras.com/>/edge/init.lua:153: in function 'handler' /usr/local/openresty/lua-my <http://www.kipras.com/>/my <http://www.kipras.com/>/edge/init.lua:166: in function 'content' .../openresty/nginx/../lua-my <http://www.kipras.com/>/handlers/src/content.lua:1: in function <.../openresty/nginx/../lua-my <http://www.kipras.com/>/handlers/src/content.lua:1>, client: 192.168.33.1, server: www. <http://www.kipras.com/>my <http://www.kipras.com/>.com <http://www.kipras.com/>, request: "GET /favicon.ico HTTP/1.1", host: "www.my.com <http://www.kipras.com/>" Now for me looks to be impossible to handler this... do you have some idea? because, I receive this from nginx it over unix-stream and need to send to a remote syslog-ng (collector) saving like: /var/log/syslog-ng/www.my.com_error.log > Regards, > Robert > > On Sun, Jul 3, 2016 at 3:32 PM, Jorge Pereira <jpereiran(a)gmail.com> wrote: > >> Hi, >> >> I am not sure about the best approach and way to fix my problem, >> below more information. >> >> 1) I receive the below packet sent from a nginx/openresty instance. >> >> 2016/07/02 01:17:04 [emerg] 19081#0: *13163 [lua] init.lua:115: [captcha] >> {"fail_count":"","response_code":200,"client_ip":"192.168.1.22","hostname":"server-lab01","request_id":"2016-07-02T01:17:03Z|9175f93c0c||i0Xb3BuBWV","host":" >> www.mytest.com","http_request":{"verb":"GET","url":"\/","user-agent":"Mozilla\/5.0 >> (pc-x86_64-linux-gnu) Siege\/3.0.8","http_version":"1.1","all":"{\"host\":\" >> www.mytest.com\",\"x-country-code\":\"US\",\"connection\":\"close\",\"accept\":\"*\\\/*\",\"x-client-ip\":\"192.168.1.22\",\"user-agent\":\"Mozilla\\\/5.0 >> (pc-x86_64-linux-gnu) >> Siege\\\/3.0.8\",\"accept-encoding\":\"gzip\"}"},"geoip":{"location":"-90.5334,38.6500","city_name":"Chesterfield","country_name":"United >> States","longitude":-90.5334,"area_code":314,"latitude":38.65,"country_code2":"US","country_code3":"USA"},"got":"","action":"show","expected":"h1szmM","webapp_domain":" >> www.mytest.com"} while logging request, client: 192.168.1.22, server: >> www.mytest.com, request: "GET / HTTP/1.1", host: "www.mytest.com" >> >> 2) In my server side, I need to save the logs following a value of *host: >> "www.mytest.com <http://www.mytest.com>"* like: >> >> /var/log/syslog-ng/www.mytest.com.log >> >> 3) The problem is because the packet received has a part being a jSON, >> but I can't use the *json-parser().* >> >> 4) What is the best approach? I have used: >> >> # Extracting only the jSON payload >> rewrite p_nginx_wb_error_log_clean { >> subst(".*captcha] ", "", value("MESSAGE"), flags("global")); >> subst(" while logging request.*$", "", value("MESSAGE"), >> flags("global")); >> }; >> >> parser p_nginx_wb_error_log_json { >> json-parser( >> marker("") >> prefix("j.") >> ); >> }; >> >> destination d_nginx_wb_error_log { >> >> file("/var/log/syslog-ng/nginx/${j.webapp_domain:-unknow-payload}_error.log" >> >> >> >> create_dirs(yes) >> owner("root") >> group("root") >> perm(0644) >> dir_perm(0755) >> template("${MSG}\n") >> ); >> }; >> >> -- >> Jorge Pereira >> >> >> ______________________________________________________________________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: >> http://www.balabit.com/support/documentation/?product=syslog-ng >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> >> >> > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > >

2 1

Re: [syslog-ng] extracting jSON from $MESSAGE
by jrhendri 03 Jul '16

03 Jul '16

Hanes you looked at using patterndb? That should let you assign the specific parts of the message to variables you choose and then use them as you like.It should be possible to do it with json parser but I have not used it personally. Jim Sent from my Verizon, Samsung Galaxy smartphone -------- Original message --------From: "Fekete, Róbert" <robert.fekete(a)balabit.com> Date: 7/3/16 1:26 PM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] extracting jSON from $MESSAGE Hi Jorge, This seems to be a bit tricky message. If I see it correctly, after the syslog header, you have: * some additional info that ends with [captcha] (is that literally [captcha], or it changes with every message?) * then you have some JSON * "while logging request," string * and finally some key:value pairs. As a first try, I would use the csv-parser() to separate the message to the four blocks I listed above. Use syslog-ng OSE 3.7, because then you can use strings as delimiters (see https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-…) For example, you can use the ] character and the "while logging request," string as delimiters. Then you can run the json-parser on the macro containing the JSON-part of the message, and a kv-parser with the : delimiter to parse the last block if needed. (Note that for kv-parser part you need a recent development version of syslog-ng). Make sure to use the prefix() option in the json-parser, because as I see the key:value block seems to have some of the same keys that the json block. BTW, is this a publicly available application that emits such a log message? It seems ideal to showcase the wide range of available syslog-ng parsers :) Regards, Robert On Sun, Jul 3, 2016 at 3:32 PM, Jorge Pereira <jpereiran(a)gmail.com> wrote: Hi, I am not sure about the best approach and way to fix my problem, below more information. 1) I receive the below packet sent from a nginx/openresty instance. 2016/07/02 01:17:04 [emerg] 19081#0: *13163 [lua] init.lua:115: [captcha] {"fail_count":"","response_code":200,"client_ip":"192.168.1.22","hostname":"server-lab01","request_id":"2016-07-02T01:17:03Z|9175f93c0c||i0Xb3BuBWV","host":"www.mytest.com","http_request":{"verb":"GET","url":"\/","user-agent":"Mozilla\/5.0 (pc-x86_64-linux-gnu) Siege\/3.0.8","http_version":"1.1","all":"{\"host\":\"www.mytest.com\",\"x-country-code\":\"US\",\"connection\":\"close\",\"accept\":\"*\\\/*\",\"x-client-ip\":\"192.168.1.22\",\"user-agent\":\"Mozilla\\\/5.0 (pc-x86_64-linux-gnu) Siege\\\/3.0.8\",\"accept-encoding\":\"gzip\"}"},"geoip":{"location":"-90.5334,38.6500","city_name":"Chesterfield","country_name":"United States","longitude":-90.5334,"area_code":314,"latitude":38.65,"country_code2":"US","country_code3":"USA"},"got":"","action":"show","expected":"h1szmM","webapp_domain":"www.mytest.com"} while logging request, client: 192.168.1.22, server: www.mytest.com, request: "GET / HTTP/1.1", host: "www.mytest.com" 2) In my server side, I need to save the logs following a value of host: "www.mytest.com" like: /var/log/syslog-ng/www.mytest.com.log 3) The problem is because the packet received has a part being a jSON, but I can't use the json-parser(). 4) What is the best approach? I have used: # Extracting only the jSON payloadrewrite p_nginx_wb_error_log_clean { subst(".*captcha] ", "", value("MESSAGE"), flags("global")); subst(" while logging request.*$", "", value("MESSAGE"), flags("global"));}; parser p_nginx_wb_error_log_json { json-parser( marker("") prefix("j.") ); }; destination d_nginx_wb_error_log { file("/var/log/syslog-ng/nginx/${j.webapp_domain:-unknow-payload}_error.log" create_dirs(yes) owner("root") group("root") perm(0644) dir_perm(0755) template("${MSG}\n") ); }; -- Jorge Pereira ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

extracting jSON from $MESSAGE
by Jorge Pereira 03 Jul '16

03 Jul '16

Hi, I am not sure about the best approach and way to fix my problem, below more information. 1) I receive the below packet sent from a nginx/openresty instance. 2016/07/02 01:17:04 [emerg] 19081#0: *13163 [lua] init.lua:115: [captcha] {"fail_count":"","response_code":200,"client_ip":"192.168.1.22","hostname":"server-lab01","request_id":"2016-07-02T01:17:03Z|9175f93c0c||i0Xb3BuBWV","host":" www.mytest.com","http_request":{"verb":"GET","url":"\/","user-agent":"Mozilla\/5.0 (pc-x86_64-linux-gnu) Siege\/3.0.8","http_version":"1.1","all":"{\"host\":\" www.mytest.com\",\"x-country-code\":\"US\",\"connection\":\"close\",\"accept\":\"*\\\/*\",\"x-client-ip\":\"192.168.1.22\",\"user-agent\":\"Mozilla\\\/5.0 (pc-x86_64-linux-gnu) Siege\\\/3.0.8\",\"accept-encoding\":\"gzip\"}"},"geoip":{"location":"-90.5334,38.6500","city_name":"Chesterfield","country_name":"United States","longitude":-90.5334,"area_code":314,"latitude":38.65,"country_code2":"US","country_code3":"USA"},"got":"","action":"show","expected":"h1szmM","webapp_domain":" www.mytest.com"} while logging request, client: 192.168.1.22, server: www.mytest.com, request: "GET / HTTP/1.1", host: "www.mytest.com" 2) In my server side, I need to save the logs following a value of *host: "www.mytest.com <http://www.mytest.com>"* like: /var/log/syslog-ng/www.mytest.com.log 3) The problem is because the packet received has a part being a jSON, but I can't use the *json-parser().* 4) What is the best approach? I have used: # Extracting only the jSON payload rewrite p_nginx_wb_error_log_clean { subst(".*captcha] ", "", value("MESSAGE"), flags("global")); subst(" while logging request.*$", "", value("MESSAGE"), flags("global")); }; parser p_nginx_wb_error_log_json { json-parser( marker("") prefix("j.") ); }; destination d_nginx_wb_error_log { file("/var/log/syslog-ng/nginx/${j.webapp_domain:-unknow-payload}_error.log" create_dirs(yes) owner("root") group("root") perm(0644) dir_perm(0755) template("${MSG}\n") ); }; -- Jorge Pereira

2 1

Re: [syslog-ng] Syslog-ng Multiple Instances
by David Campeau 01 Jul '16

01 Jul '16

Hello, I've been using syslog-ng to filter syslog before forwarding on to a log collector. However, I need to spin up a second instance for testing purposes. I've found a little bit of information on-line, but it hasn't completed the entire picture. This is the command used to start up the 2nd instance. I'm pointing to separate .conf .persist .pid and .ctl files -- However, it's still not working. I suspect the issue is due to OS log sources. How do a change log sources? syslog-ng --cfgfile=/usr/local/bin/syslog-ng-Second-Instance/syslog-ng.conf --persist-file=/usr/local/bin/syslog-ng-Second-Instance/syslog-ng.persist --pidfile=/usr/local/bin/syslog-ng-Second-Instance/syslog-ng.pid --control=/usr/local/bin/syslog-ng-Second-Instance/syslog-ng.ctl & This is the upper part of the syslog-ng.conf file for the 2nd instance I wish to run. @version: 3.3.4 @include "scl.conf" options { time-reap(30); mark-freq(10); keep-hostname(yes); chain-hostnames(no); use-dns(no); ## log-fifo-size(500000); ## Tuning Options ## flush_lines(10000); ## Tuning Options ## flush_timeout(10000); ## Tuning Options }; source s_second_instance { syslog(transport("udp") port("518")); #### Will receive test syslog on port 518 }; destination d_syslog_udp { syslog("10.X.X.X" transport("udp") port("514") throttle(4000) ); }; I'm hoping someone has experience or has seen information on how to run a 2nd instance on the same box. Best Regards, David

3 7

getting some information about the sender process
by Attila Szalai 30 Jun '16

30 Jun '16

Hi, Maybe I'm totally wrong, but I have/had the impression that syslog-ng collects some information about the client process if that connects through a unix domain socket. But actually I could not find anything related to this neither in the documentation nor the code. Is this something that only exists in my mind or I just missed something?

2 2

a log message is output in the two line
by Hirose, Shinsaku 30 Jun '16

30 Jun '16

Hello, all, I use syslog-ng-3.2.5-4.el6.x86_64 on Centos6. I got it from eple epel repository. I am troubled in how to use the source file driver. The touble is that a log message is output in the two lines on remote syslog server. How to reproduce is followings. 1. Prepare two hosts running syslog-ng. Host_A configuration is followings. ---------------------------------------- source test { file("/tmp/a.log"); }; destination d_remote { udp("192.168.0.2"); }; log { source(test); destination(d_remote); }; ---------------------------------------- Host_B(192.168.0.2) configuration is defaults. 2.Execute following command on Host_A. $ seq 8193 | (xargs -i echo -n "a";echo "") >> /tmp/a.log 3.Check the log on Host_B. As the result, a log message is output in the two lines on Host_B. One line is following. The num of "a" is 8192. aaaaaaaaaaaaa....... The other line is following. The num of "a" is 1. a I hope a log message is output in the one line on Host_B. Is my hope readily achievable? Please advise me.

5 7

CzP in Nuremberg
by Czanik, Péter 21 Jun '16

21 Jun '16

Hi, For the next couple of days I'll be in Nuremberg for the openSUSE conference: https://events.opensuse.org/conference/oSC16 I'll give a presentation on syslog-ng Friday in the afternoon at 13:30 in the Roter Salon. I'll wear my syslog-ng superhero t-shirt, so I'll be easy to spot and ready to answer syslog-ng questions any time. If you are in Nuremberg, but not at the conference, I'm happy to talk to you about syslog-ng also outside of the conference floor. You can reach me by e-mail or on twitter as @pczanik See you soon in Nuremberg, Peter

1 0

Syslog-ng 3.7.3 + Elasticsearch V2.3 - Error init pipeline
by Marco Mignone 16 Jun '16

16 Jun '16

Hi All, I am trying to setup syslog-ng to use elasticsearch as its destination on Ubuntu 14.04. This the version of syslog (the unofficial versions installed from laszlo_budai rep: syslog-ng 3.7.3 Installer-Version: 3.7.3 Revision: 3.7.3-8 Compile-Date: Jun 1 2016 16:33:00 Available-Modules: basicfuncs,linux-kmsg-format,riemann,afuser,afstomp,afprog,json-plugin,afsmtp,affile,csvparser,mod-java,pseudofile,confgen,afsocket,afamqp,redis,sdjournal,kvformat,syslogformat,afsql,system-source,mod-python,graphite,dbparser,geoip-plugin,afmongodb,cryptofuncs Elastic search is: "name" : "Theresa Cassidy", "cluster_name" : "elasticsearch", "version" : { "number" : "2.3.3", "build_hash" : "218bdf10790eef486ff2c41a3df5cfa32dadcfde", "build_timestamp" : "2016-05-17T15:40:04Z", "build_snapshot" : false, "lucene_version" : “5.5.0" And my custom configuration in /etc/syslog-ng/conf.d/test.conf which is: @module mod-java source s_net { udp(); tcp(); }; destination d_elastic { elasticsearch( index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("test") client_lib_dir("/usr/share/elasticsearch/lib") ); }; log { source(s_net); destination(d_elastic); flags(flow-control); }; When I try to launch syslog in debug mode this is what I get: [2016-06-16T15:54:29.378356] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/syslog-ng-core.jar; [2016-06-16T15:54:29.382446] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/syslog-ng-core.jar; [2016-06-16T15:54:29.382660] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/kafka.jar; [2016-06-16T15:54:29.382862] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/http.jar; [2016-06-16T15:54:29.383052] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/hdfs.jar; [2016-06-16T15:54:29.383258] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/elastic.jar; [2016-06-16T15:54:29.383479] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/syslog-ng-common.jar; [2016-06-16T15:54:29.383670] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/log4j-1.2.16.jar; [2016-06-16T15:54:29.383917] Add path to classpath: /usr/share/elasticsearch/lib/guava-18.0.jar; [2016-06-16T15:54:29.384098] Add path to classpath: /usr/share/elasticsearch/lib/jna-4.1.0.jar; [2016-06-16T15:54:29.384293] Add path to classpath: /usr/share/elasticsearch/lib/spatial4j-0.5.jar; [2016-06-16T15:54:29.384494] Add path to classpath: /usr/share/elasticsearch/lib/compress-lzf-1.0.2.jar; [2016-06-16T15:54:29.386104] Add path to classpath: /usr/share/elasticsearch/lib/lucene-spatial-5.5.0.jar; [2016-06-16T15:54:29.386342] Add path to classpath: /usr/share/elasticsearch/lib/lucene-backward-codecs-5.5.0.jar; [2016-06-16T15:54:29.386507] Add path to classpath: /usr/share/elasticsearch/lib/t-digest-3.0.jar; [2016-06-16T15:54:29.386677] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.6.6.jar; [2016-06-16T15:54:29.386865] Add path to classpath: /usr/share/elasticsearch/lib/lucene-memory-5.5.0.jar; [2016-06-16T15:54:29.387044] Add path to classpath: /usr/share/elasticsearch/lib/jackson-core-2.6.6.jar; [2016-06-16T15:54:29.387216] Add path to classpath: /usr/share/elasticsearch/lib/joda-time-2.8.2.jar; [2016-06-16T15:54:29.387394] Add path to classpath: /usr/share/elasticsearch/lib/lucene-join-5.5.0.jar; [2016-06-16T15:54:29.387673] Add path to classpath: /usr/share/elasticsearch/lib/lucene-grouping-5.5.0.jar; [2016-06-16T15:54:29.388476] Add path to classpath: /usr/share/elasticsearch/lib/HdrHistogram-2.1.6.jar; [2016-06-16T15:54:29.388647] Add path to classpath: /usr/share/elasticsearch/lib/compiler-0.8.13.jar; [2016-06-16T15:54:29.388818] Add path to classpath: /usr/share/elasticsearch/lib/netty-3.10.5.Final.jar; [2016-06-16T15:54:29.388972] Add path to classpath: /usr/share/elasticsearch/lib/lucene-misc-5.5.0.jar; [2016-06-16T15:54:29.389518] Add path to classpath: /usr/share/elasticsearch/lib/lucene-analyzers-common-5.5.0.jar; [2016-06-16T15:54:29.389711] Add path to classpath: /usr/share/elasticsearch/lib/lucene-sandbox-5.5.0.jar; [2016-06-16T15:54:29.390094] Add path to classpath: /usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar; [2016-06-16T15:54:29.390283] Add path to classpath: /usr/share/elasticsearch/lib/commons-cli-1.3.1.jar; [2016-06-16T15:54:29.390488] Add path to classpath: /usr/share/elasticsearch/lib/snakeyaml-1.15.jar; [2016-06-16T15:54:29.390659] Add path to classpath: /usr/share/elasticsearch/lib/jsr166e-1.1.0.jar; [2016-06-16T15:54:29.390935] Add path to classpath: /usr/share/elasticsearch/lib/lucene-core-5.5.0.jar; [2016-06-16T15:54:29.391176] Add path to classpath: /usr/share/elasticsearch/lib/lucene-suggest-5.5.0.jar; [2016-06-16T15:54:29.394616] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.6.6.jar; [2016-06-16T15:54:29.395279] Add path to classpath: /usr/share/elasticsearch/lib/log4j-1.2.17.jar; [2016-06-16T15:54:29.395458] Add path to classpath: /usr/share/elasticsearch/lib/joda-convert-1.2.jar; [2016-06-16T15:54:29.395970] Add path to classpath: /usr/share/elasticsearch/lib/hppc-0.7.1.jar; [2016-06-16T15:54:29.396734] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-smile-2.6.6.jar; [2016-06-16T15:54:29.397919] Add path to classpath: /usr/share/elasticsearch/lib/lucene-queryparser-5.5.0.jar; [2016-06-16T15:54:29.398106] Add path to classpath: /usr/share/elasticsearch/lib/lucene-spatial3d-5.5.0.jar; [2016-06-16T15:54:29.398281] Add path to classpath: /usr/share/elasticsearch/lib/lucene-queries-5.5.0.jar; [2016-06-16T15:54:29.398440] Add path to classpath: /usr/share/elasticsearch/lib/apache-log4j-extras-1.2.17.jar; [2016-06-16T15:54:29.398610] Add path to classpath: /usr/share/elasticsearch/lib/lucene-highlighter-5.5.0.jar; [2016-06-16T15:54:29.398784] Add path to classpath: /usr/share/elasticsearch/lib/jts-1.13.jar; [2016-06-16T15:54:29.398925] Add path to classpath: /usr/share/elasticsearch/lib/securesm-1.0.jar; [2016-06-16T15:54:29.501879] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/syslog-ng-core.jar; [2016-06-16T15:54:29.519443] Error initializing message pipeline; Was anyone able to make this work or suggest a way to fix this? I hope I am on the right place and if not apologies in advance. Thanks, Marco

3 3

Re: [syslog-ng] Syslog-ng 3.7.3 + Elasticsearch V2.3 - Error init pipeline
by Laszlo Budai 16 Jun '16

16 Jun '16

Hi, it should work with the latest 1.x. I'm working on my 3.8 OBS repo, but having some problems. When I have the alpha packages, I'll send a mail. Regards, Laszlo Budai Sent from my HTC ----- Reply message ----- From: "Marco Mignone" <info(a)marcomignone.com> To: "Syslog-ng users' and developers' mailing list" <syslog-ng(a)lists.balabit.hu> Subject: [syslog-ng] Syslog-ng 3.7.3 + Elasticsearch V2.3 - Error init pipeline Date: Thu, Jun 16, 2016 17:34 Hi Peter, Thanks for the quick reply. I thought there was some hack or way around it. I will then try with ElasticSearch1.X. Will the latest version of 1.x work or is there a specific version? Thanks again for your time. Marco > On 16 Jun 2016, at 16:26, Czanik, Péter <peter.czanik(a)balabit.com> wrote: > > Hi, > > syslog-ng 3.7 only works with Elasticsearch 1.X. For Elasticsearch 2.X > you need syslog-ng 3.8, which is still under development (alpha). > > Bye, > Peter Czanik (CzP) <peter.czanik(a)balabit.com> > Balabit / syslog-ng upstream > http://czanik.blogs.balabit.com/ > https://twitter.com/PCzanik > > > On Thu, Jun 16, 2016 at 5:11 PM, Marco Mignone <info(a)marcomignone.com> wrote: >> Hi All, >> I am trying to setup syslog-ng to use elasticsearch as its destination on >> Ubuntu 14.04. >> This the version of syslog (the unofficial versions installed from >> laszlo_budai rep: >> >> syslog-ng 3.7.3 >> Installer-Version: 3.7.3 >> Revision: 3.7.3-8 >> Compile-Date: Jun 1 2016 16:33:00 >> Available-Modules: >> basicfuncs,linux-kmsg-format,riemann,afuser,afstomp,afprog,json-plugin,afsmtp,affile,csvparser,mod-java,pseudofile,confgen,afsocket,afamqp,redis,sdjournal,kvformat,syslogformat,afsql,system-source,mod-python,graphite,dbparser,geoip-plugin,afmongodb,cryptofuncs >> >> Elastic search is: >> >> "name" : "Theresa Cassidy", >> "cluster_name" : "elasticsearch", >> "version" : { >> "number" : "2.3.3", >> "build_hash" : "218bdf10790eef486ff2c41a3df5cfa32dadcfde", >> "build_timestamp" : "2016-05-17T15:40:04Z", >> "build_snapshot" : false, >> "lucene_version" : “5.5.0" >> >> >> And my custom configuration in /etc/syslog-ng/conf.d/test.conf which is: >> >> @module mod-java >> >> source s_net { >> udp(); >> tcp(); >> }; >> >> destination d_elastic { >> elasticsearch( >> index("syslog-ng_${YEAR}.${MONTH}.${DAY}") >> type("test") >> client_lib_dir("/usr/share/elasticsearch/lib") >> ); >> }; >> >> log { >> source(s_net); >> destination(d_elastic); >> flags(flow-control); >> }; >> >> >> When I try to launch syslog in debug mode this is what I get: >> >> [2016-06-16T15:54:29.378356] Add path to classpath: >> /usr/lib/syslog-ng/3.7/java-modules/syslog-ng-core.jar; >> [2016-06-16T15:54:29.382446] Add path to classpath: >> /usr/lib/syslog-ng/3.7/java-modules/syslog-ng-core.jar; >> [2016-06-16T15:54:29.382660] Add path to classpath: >> /usr/lib/syslog-ng/3.7/java-modules/kafka.jar; >> [2016-06-16T15:54:29.382862] Add path to classpath: >> /usr/lib/syslog-ng/3.7/java-modules/http.jar; >> [2016-06-16T15:54:29.383052] Add path to classpath: >> /usr/lib/syslog-ng/3.7/java-modules/hdfs.jar; >> [2016-06-16T15:54:29.383258] Add path to classpath: >> /usr/lib/syslog-ng/3.7/java-modules/elastic.jar; >> [2016-06-16T15:54:29.383479] Add path to classpath: >> /usr/lib/syslog-ng/3.7/java-modules/syslog-ng-common.jar; >> [2016-06-16T15:54:29.383670] Add path to classpath: >> /usr/lib/syslog-ng/3.7/java-modules/log4j-1.2.16.jar; >> [2016-06-16T15:54:29.383917] Add path to classpath: >> /usr/share/elasticsearch/lib/guava-18.0.jar; >> [2016-06-16T15:54:29.384098] Add path to classpath: >> /usr/share/elasticsearch/lib/jna-4.1.0.jar; >> [2016-06-16T15:54:29.384293] Add path to classpath: >> /usr/share/elasticsearch/lib/spatial4j-0.5.jar; >> [2016-06-16T15:54:29.384494] Add path to classpath: >> /usr/share/elasticsearch/lib/compress-lzf-1.0.2.jar; >> [2016-06-16T15:54:29.386104] Add path to classpath: >> /usr/share/elasticsearch/lib/lucene-spatial-5.5.0.jar; >> [2016-06-16T15:54:29.386342] Add path to classpath: >> /usr/share/elasticsearch/lib/lucene-backward-codecs-5.5.0.jar; >> [2016-06-16T15:54:29.386507] Add path to classpath: >> /usr/share/elasticsearch/lib/t-digest-3.0.jar; >> [2016-06-16T15:54:29.386677] Add path to classpath: >> /usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.6.6.jar; >> [2016-06-16T15:54:29.386865] Add path to classpath: >> /usr/share/elasticsearch/lib/lucene-memory-5.5.0.jar; >> [2016-06-16T15:54:29.387044] Add path to classpath: >> /usr/share/elasticsearch/lib/jackson-core-2.6.6.jar; >> [2016-06-16T15:54:29.387216] Add path to classpath: >> /usr/share/elasticsearch/lib/joda-time-2.8.2.jar; >> [2016-06-16T15:54:29.387394] Add path to classpath: >> /usr/share/elasticsearch/lib/lucene-join-5.5.0.jar; >> [2016-06-16T15:54:29.387673] Add path to classpath: >> /usr/share/elasticsearch/lib/lucene-grouping-5.5.0.jar; >> [2016-06-16T15:54:29.388476] Add path to classpath: >> /usr/share/elasticsearch/lib/HdrHistogram-2.1.6.jar; >> [2016-06-16T15:54:29.388647] Add path to classpath: >> /usr/share/elasticsearch/lib/compiler-0.8.13.jar; >> [2016-06-16T15:54:29.388818] Add path to classpath: >> /usr/share/elasticsearch/lib/netty-3.10.5.Final.jar; >> [2016-06-16T15:54:29.388972] Add path to classpath: >> /usr/share/elasticsearch/lib/lucene-misc-5.5.0.jar; >> [2016-06-16T15:54:29.389518] Add path to classpath: >> /usr/share/elasticsearch/lib/lucene-analyzers-common-5.5.0.jar; >> [2016-06-16T15:54:29.389711] Add path to classpath: >> /usr/share/elasticsearch/lib/lucene-sandbox-5.5.0.jar; >> [2016-06-16T15:54:29.390094] Add path to classpath: >> /usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar; >> [2016-06-16T15:54:29.390283] Add path to classpath: >> /usr/share/elasticsearch/lib/commons-cli-1.3.1.jar; >> [2016-06-16T15:54:29.390488] Add path to classpath: >> /usr/share/elasticsearch/lib/snakeyaml-1.15.jar; >> [2016-06-16T15:54:29.390659] Add path to classpath: >> /usr/share/elasticsearch/lib/jsr166e-1.1.0.jar; >> [2016-06-16T15:54:29.390935] Add path to classpath: >> /usr/share/elasticsearch/lib/lucene-core-5.5.0.jar; >> [2016-06-16T15:54:29.391176] Add path to classpath: >> /usr/share/elasticsearch/lib/lucene-suggest-5.5.0.jar; >> [2016-06-16T15:54:29.394616] Add path to classpath: >> /usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.6.6.jar; >> [2016-06-16T15:54:29.395279] Add path to classpath: >> /usr/share/elasticsearch/lib/log4j-1.2.17.jar; >> [2016-06-16T15:54:29.395458] Add path to classpath: >> /usr/share/elasticsearch/lib/joda-convert-1.2.jar; >> [2016-06-16T15:54:29.395970] Add path to classpath: >> /usr/share/elasticsearch/lib/hppc-0.7.1.jar; >> [2016-06-16T15:54:29.396734] Add path to classpath: >> /usr/share/elasticsearch/lib/jackson-dataformat-smile-2.6.6.jar; >> [2016-06-16T15:54:29.397919] Add path to classpath: >> /usr/share/elasticsearch/lib/lucene-queryparser-5.5.0.jar; >> [2016-06-16T15:54:29.398106] Add path to classpath: >> /usr/share/elasticsearch/lib/lucene-spatial3d-5.5.0.jar; >> [2016-06-16T15:54:29.398281] Add path to classpath: >> /usr/share/elasticsearch/lib/lucene-queries-5.5.0.jar; >> [2016-06-16T15:54:29.398440] Add path to classpath: >> /usr/share/elasticsearch/lib/apache-log4j-extras-1.2.17.jar; >> [2016-06-16T15:54:29.398610] Add path to classpath: >> /usr/share/elasticsearch/lib/lucene-highlighter-5.5.0.jar; >> [2016-06-16T15:54:29.398784] Add path to classpath: >> /usr/share/elasticsearch/lib/jts-1.13.jar; >> [2016-06-16T15:54:29.398925] Add path to classpath: >> /usr/share/elasticsearch/lib/securesm-1.0.jar; >> [2016-06-16T15:54:29.501879] Add path to classpath: >> /usr/lib/syslog-ng/3.7/java-modules/syslog-ng-core.jar; >> [2016-06-16T15:54:29.519443] Error initializing message pipeline; >> >> >> Was anyone able to make this work or suggest a way to fix this? >> I hope I am on the right place and if not apologies in advance. >> >> Thanks, >> Marco >> >> >> ______________________________________________________________________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: >> http://www.balabit.com/support/documentation/?product=syslog-ng >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> >> > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

2 1

Is there a standard for naming tag/value pairs when parsing
by Evan Rempel 16 Jun '16

16 Jun '16

There was a project by Mitre (https://www.mitre.org/) called the Common Event Expression (https://cee.mitre.org/) that was going to be the official standard for metadata names for events, but that project has been stopped. Other than the two references that the CEE project has for logging standardization efforts, does anyone know of any major efforts by any group to define a standard for metadata naming? Evan.

3 6

Insider 2016-06: Red Hat; 3.8 RPMs; BMW; MongoDB; FLARE; Java;
by Czanik, Péter 16 Jun '16

16 Jun '16

Dear syslog-ng users, This is the 50th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Meet syslog-ng at Red Hat Summit -------------------------------- This year Balabit is sponsoring the Red Hat Summit again. The event will be held in Moscone Center North in San Francisco, June 27-29. Visit Balabit’s booth (booth #918) and meet Balázs Scheidler, the author of syslog-ng. You can request an appointment at http://pages2.balabit.com/red-hat-summit-appointment-request/ If you want to meet him outside of the conference floor, there will also be a meetup in Café Prague on the 30th of June: https://www.eventbrite.com/e/techie-happy-hour-optimize-your-log-data-in-th… Preview syslog-ng 3.8 RPM packages and send us your feedback ------------------------------------------------------------ While syslog-ng 3.8 does not yet have an alpha release, it already has many interesting features. As it is still under heavy development, we can't recommend it for production use. On the other hand, any feedback is very welcome. New features include disk buffer, Elasticsearch 2 support, grouping-by, Rust-based parsers, and so on: https://czanik.blogs.balabit.com/2016/05/state-of-syslog-ng-3-8-rpm-packagi… syslog-ng in the BMW i3 ----------------------- While most people know syslog-ng as a central syslog server, there is another use, which is less known but most likely has a lot larger installed base. It is syslog-ng embedded. Read how BMW is utilizing syslog-ng at https://czanik.blogs.balabit.com/2016/06/embedded-syslog-ng-bmw-i3-all-elec… and let us know how you use syslog-ng! MongoDB changes in 3.8 ---------------------- Under the hood, the driver to access MongoDB databases was replaced in syslog-ng 3.8. Read about the advantages of the new driver and how it affects configuration at: https://syslog-ng.org/mongodb-destination-receives-face-lift/ FLARE: Filtered Log Alert and Reporting Engine ---------------------------------------------- FLARE staplies together syslog-ng, ELK, Alerting and Incident Response with metadata. The aim was to develop a solution that is able to provide a unified view of log events and incidents, helping the work of operations admins with actionable insight. It is an internal software developed at the University of Victoria by long-time members of the syslog-ng community. You can read the complete presentation about this syslog-ng & PatternDB-based solution at https://www.bc.net/flare-stapling-together-syslog-ng-elk-alerting-and-incid… Writing syslog-ng Java destination drivers ------------------------------------------ If a destination is not supported by syslog-ng out of the box, you can develop a new driver in C or one of the language bindings. Here are the first steps for creating a Java-based destination driver: https://vithulanmv.wordpress.com/2016/06/07/the-syslog-ng-java-destinations/ Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com . To read this newsletter online, visit: http://insider.blogs.balabit.com/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

Custom plugin/module
by Dmitry Shmulevich 16 Jun '16

16 Jun '16

Hello, I'm looking for an advice. I'm sending the logs from my machine to a third-party server. The protocol between the client (syslog-ng service) and the third-party server is quite simple: for each received log entry the server replies with a confirmation message on the same tcp socket. Currently I'm using a simple syslog-ng config to send logs via network driver to a specified ip/port. However, since syslog-ng service doesn't receive/read the messages from the server, the connection eventually dies. Is there any way to modify syslog-ng config to allow reads after writes? I didn't find anything similar in the documentation. An alternative way seems to be implementing a syslog-ng module or a plugin. Would that be the right approach? It seems that I can customize _LogProtoClient structure, and create a plugin. But I have no clue on how the overall API looks like, and what are the steps to build the module. Could you guys point me to the right resources? Is there any sample plugin, or a guidance for developing custom plugins? Thank you in advance, Dmitry

3 3

Embedded syslog-ng: BMW i3 all-electric car
by Czanik, Péter 13 Jun '16

13 Jun '16

Hi, A few years ago I wrote about syslog-ng running on all Kindle e-readers. Recently I found, that BMW also uses syslog-ng in the i3 all-electric car: https://czanik.blogs.balabit.com/2016/06/embedded-syslog-ng-bmw-i3-all-elec… If you have another interesting use case, either embedded or the more traditional central server, I'd be glad to add it to our "Powered by syslog-ng" page at https://syslog-ng.org/powered-by-syslog-ng/ Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

Re: [syslog-ng] Is there a standard for naming tag/value pairs when parsing
by jrhendri 12 Jun '16

12 Jun '16

There is something to be said for making it easy to use syslog-ng with the major players and right now that is splunk. I would hasten to add keeping json actively used and the native elasticsearch destination current also makes it easy to position syslog-ng as a standard collection and distribution layer. Within 2 to 3 years something will leapfrog splunk and if syslog-ng is ready, it will make that transition much easier. Jim Sent from my Verizon, Samsung Galaxy smartphone -------- Original message --------From: "Scheidler, Balázs" <balazs.scheidler(a)balabit.com> Date: 6/12/16 5:44 AM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] Is there a standard for naming tag/value pairs when parsing Well, CEE is pretty much dead, and I didn't see too much activity wrt lumberjack either. I would rather see consolidation instead of further fragmentation in this area. Cheers Bazsi You are the last person I thought would point me toward the splunk CIM. Given the support that Balabit has put behind CEE and then lumberjack and even the experimental patternDB schema (https://github.com/balabit/syslog-ng-patterndb/blob/master/SCHEMAS.txt) I was sure you would steer me toward lumberjack. At first glance the splunk CIM appears to be structured around and partially dependant on some of the data flows of the splunk product. I'll continue to review it but at this point I am still open to alternate suggestions. Evan. On 06/11/2016 11:45 AM, Scheidler, Balázs wrote: There's common information model at splunk or the field dictionary of CEF, of arcsight fame. I would probably use the splunk one, except if you plan to use arcsight at the end. On Jun 11, 2016 18:32, "Evan Rempel" <erempel(a)uvic.ca> wrote: There was a project by Mitre (https://www.mitre.org/) called the Common Event Expression (https://cee.mitre.org/) that was going to be the official standard for metadata names for events, but that project has been stopped. Other than the two references that the CEE project has for logging standardization efforts, does anyone know of any major efforts by any group to define a standard for metadata naming? Evan. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

UTF-8 handling
by Robin Blanchard 10 Jun '16

10 Jun '16

Hi, I've got syslog-ng-3.7.3-3.el7.centos.x86_64 feeding logstash-2.2.4-1.noarch. I've had flags(sanitize-utf8) enabled on the relevant syslog-ng listener, yet LS continues to complain with ':expected_charset=>"UTF-8"', to the point of LS ultimately choking. The underlying messages are indeed garbage, eg: $ pcregrep -n "[\x80-\xFF]" garbage.txt |tail -1 299:\\xc2\u0003\u0003\\xe0\u0001|\\x9d\\x9bX\\xa3wt}\\xe4\u001e\\xd6g\u0013j\\x9c!j\\xd9#*\\x95#\\xf8�� Should I instead be employing flags(validate-utf8) ? Ultimately, I'd like for syslog-ng to either re-encode messages into proper UTF8, or -- if not possible -- just squelch/drop them…. Thanks in advance -------------------------------------------------------------------------------------------------------------------------- This email has been sent to you on behalf of Nephila Advisors LLC (“Advisors”). Advisors provides consultancy services to Nephila Capital Ltd. (“Capital”), an investment advisor managed and carrying on business in Bermuda. Advisors and its employees do not act as agents for Capital or the funds it advises and do not have the authority to bind Capital or such funds to any transaction or agreement. The information in this e-mail, and any attachment therein, is confidential and for use by the addressee only. Any use, disclosure, reproduction, modification or distribution of the contents of this e-mail, or any part thereof, other than by the intended recipient, is strictly prohibited. If you are not the intended recipient, please return the e-mail to the sender and delete it from your computer. This email is for information purposes only, nothing contained herein constitutes an offer to sell or buy securities, as such an offer may only be made from a properly authorized offering document. Although Nephila attempts to sweep e-mail and attachments for viruses, it does not guarantee that either are virus-free and accepts no liability for any damage sustained as a result of viruses. --------------------------------------------------------------------------------------------------------------------------

1 0

Invalid parsing of syslog messages having timezone
by Nutan Shinde 10 Jun '16

10 Jun '16

Hi, Following is the syslog message received from Cisco router : *Mar 1 09:30:25.249 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel2, changed state to down As, you can see UTC is included in the above timestamp. That is why value of $PROGRAM is UTC and $MSGONLY is %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel2, changed state to down. What should I include in the syslog-ng.conf so that time zone is ignored?

3 2

grouping-by parser questions
by Fekete, Róbert 07 Jun '16

07 Jun '16

Hi Bazsi, I've started to document the grouping-by parser, and have a few questions/comments about it: * It seems that some of the grouping-by options are the same (or very similar) to the correlation-related attributes of the pattern database, but have different names. Could we name them consistently where they are the same? (I haven't checked the correlation module from Rust, but maybe we could align that as well.) For example: grouping-by | patterndb scope | context-scope timeout | context-timeout aggregate | message or action * In the original commit message, you mention three possible values for the 'scope' option, whereas the context-scope in the patterndb has four (program). Are these deliberately different, or they use the same code? * grouping-by doesn't look to me as an actual parser. From the existing objects, it resembles a filter more (IMHO), but I'd rather categorize it as something else that transforms/processes the incoming data, and should be therefore in a separate configuration object (along with the geoip parser). Robert

2 2

Problems to receive multiple/single log lines
by Jorge Pereira 04 Jun '16

04 Jun '16

Hi everyone, Question: Is it possible to send multiple/single log lines to the same destination? In this case, exist some way to differentiate the receiving of multiple/single lines? Below my client and server setup. # Client Side <MY CLIENT CONF> options { threaded(yes); flush_lines(0); use-dns(no); normalize-hostnames(yes); keep-hostname(yes); }; destination d_collector_waf { tcp("syslog-server.internal.net" port(514) keep-alive(on) flags(no-parse) ); }; source s_modsec_log { # This file is multiple-line, below the content. # # <SAMPLE CONTENT> #--bfd16c01-H-- #Message: Access denied with code 401 (phase 2). Pattern match "(?i:(?:[\\;\\|\\`]\\W*?\\bcc|\\b(wget|curl))\\b|\\/cc(?:[\\'\"\\|\\;\\`\\-\\s]|$))" at ARGS:a. [file "/usr/local/openresty/nginx/conf/waf/www.happystage.tk.conf"] [line "177"] [id "950907"] [rev "2"] [msg "System Command Injection"] [data "Matched Data: wget found within ARGS:a: wget"] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "8"] [tag "OWASP_CRS/WEB_ATTACK/COMMAND_INJECTION"] [tag "WASCTC/WASC-31"] [tag "OWASP_TOP_10/A1"] [tag "PCI/6.5.2"] #Action: Intercepted (phase 2) #Apache-Handler: IIS #Stopwatch: 1465000225000663 664069 (- - -) #Stopwatch2: 1465000225000663 664069; combined=239, p1=138, p2=80, p3=0, p4=0, p5=20, sr=10, sw=1, l=0, gc=0 #Producer: ModSecurity for nginx (STABLE)/2.8.0 (http://www.modsecurity.org/) 200911012341. #Server: ModSecurity Standalone #Engine-Mode: "ENABLED" # # </SAMPLE CONTENT> # The below regex in "multi-line-prefix()" works fine, transform the inputstream in a single line. # below debug output: # Incoming log entry; line='--3215e80c-Z--\x0a\x0a--4b80ac01-A--\x0a[04/Jun/2016:00:31:57 +0000] AiA7Ac8cA0AWAcAcAcA1@cYc 206.128.156.45 0 127.0.0.1 80\x0a--4b80ac01-B--\x0aGET /WAF-Testing/?a=wget HTTP/1.1\x0ahost: www.happystage.tk\x0arequest-id: 2016-06-04T00:31:57Z|51d35e61f4|206.128.156.45|EcWqiPItjA\x0aaccept: */*\x0auser-agent: curl/7.47.0\x0a\x0a--4b80ac01-H--\x0aMessage: Access denied with code 401 (phase 2). Pattern match "(?i:(?:[\\;\\|\\`]\\W*?\\bcc|\\b(wget|curl))\\b|\\/cc(?:[\\\'\"\\|\\;\\`\\-\\s]|$))" at ARGS:a. [file "/usr/local/openresty/nginx/conf/waf/www.happystage.tk.conf"] [line "177"] [id "950907"] [rev "2"] [msg "System Command Injection"] [data "Matched Data: wget found within ARGS:a: wget"] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "8"] [tag "OWASP_CRS/WEB_ATTACK/COMMAND_INJECTION"] [tag "WASCTC/WASC-31"] [tag "OWASP_TOP_10/A1"] [tag "PCI/6.5.2"]\x0aAction: Intercepted (phase 2)\x0aApache-Handler: IIS\x0aStopwatch: 1465000317000372 373530 (- - -)\x0aStopwatch2: 1465000317000372 373530; combined=221, p1=137, p2=63, p3=0, p4=0, p5=2' file("/usr/local/openresty/nginx/logs/waf/www.mydomain.com" follow_freq(1) flags(no-parse) multi-line-mode(regexp) override_program("ng_waf:www.mydomain.com") multi-line-prefix("--[a-fA-F0-9]{8}-Z--") ); }; source s_access_log { # The content of this file is a single-line delimited by a "\n" file("/usr/local/openresty/nginx/logs/access.log" program_override("ng_access") follow_freq(1) flags(no-parse) ); }; log { source(s_modsec_log); source(s_access_log); destination(d_collector); }; </MY CLIENT CONF> My setver setup: <SERVER CONF> options { flush-lines(100); log-fifo-size(1000); threaded(yes); use-dns(no); normalize-hostnames(yes); keep-hostname(yes); }; source s_collector { tcp(ip(0.0.0.0) port(514) keep-alive(on) flags(no-parse) ); }; filter f_nginx_waf { match("ng_modsec" value("PROGRAM")); }; rewrite r_nginx_waf { subst("ng_modsec:", "", value("PROGRAM")); }; destination d_nginx_waf { file("/var/log/syslog-ng/nginx/waf/${PROGRAM}_log" create_dirs(yes) owner("root") group("root") perm(0640) dir_perm(0750) flags(no-parse) template("${MSG}") ); }; log { source(s_collector); filter(f_nginx_waf); rewrite(r_nginx_waf); destination(d_nginx_waf); flags(flow-control); }; filter f_nginx_access { match("ng_access" value("PROGRAM")); }; destination d_nginx_access { file("/var/log/syslog-ng/nginx/${PROGRAM}_log" create_dirs(yes) owner("root") group("root") perm(0640) dir_perm(0750) flags(no-parse) template("${MSG}") ); }; log { source(s_collector); filter(f_nginx_access); destination(d_nginx_access); flags(flow-control); }; </SERVER CONF>

1 0

How to roll journald cursor in persist file
by Matwey V. Kornilov 02 Jun '16

02 Jun '16

Hello, I am running syslog-ng 3.7.1 and journald v210. The goal I want to reach is the following. I would like to replicate local system logs to remote endpoint using standard UDP protocol. So, I've just disabled all destinations except the following one: destination logserver { udp("10.0.0.47" port(514)); }; log { source(src); destination(logserver); }; The issue is the following. It seems that syslog-ng 3.7.1 uses journald natively for system() source. And I like it. The issue is that when I start syslog-ng it tries to sync all existing local journald data (about 5 GB of compressed journald data, 1.5 years of logs) from the beginning. I found in modules/systemd-journal/journal-reader.c that syslog-ng stores journald cursor in the persists file. I would like to point the cursor to the end of journald database in order to avoid unneeded log sync. The question is how could I do that in opensource syslog-ng version?

3 3

Elasticsearch with Syslog-ng
by Ivan Adji - Krstev 01 Jun '16

01 Jun '16

Hi all, I have the following error when i trying to run syslog-ng. [2016-05-31T11:17:31.298897] Seeking the journal to the last cursor position; cursor='s=6b605b6b981a471ea3d1abe52861ae82;i=21a8;b=42ab72c238874f7d8afc3c2cf0fea52d;m=38dbdc6d9;t=533be9608d083;x=140acfae276a2e55' [2016-05-31T11:17:31.299921] Module loaded and initialized successfully; module='syslogformat' [2016-05-31T11:17:31.299958] Failed to acquire /run/systemd/journal/syslog socket, disabling systemd-syslog source; [2016-05-31T11:17:31.796854] Add path to classpath: //usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar; [2016-05-31T11:17:31.797915] Add path to classpath: /usr/lib64/syslog-ng/java-modules/commons-codec-1.9.jar; [2016-05-31T11:17:31.798191] Add path to classpath: /usr/lib64/syslog-ng/java-modules/commons-lang3-3.4.jar; [2016-05-31T11:17:31.798466] Add path to classpath: /usr/lib64/syslog-ng/java-modules/commons-logging-1.2.jar; [2016-05-31T11:17:31.798821] Add path to classpath: /usr/lib64/syslog-ng/java-modules/dummy.jar; [2016-05-31T11:17:31.799058] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-v2.jar; [2016-05-31T11:17:31.799296] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic.jar; [2016-05-31T11:17:31.799503] Add path to classpath: /usr/lib64/syslog-ng/java-modules/gson-2.6.2.jar; [2016-05-31T11:17:31.799778] Add path to classpath: /usr/lib64/syslog-ng/java-modules/guava-19.0.jar; [2016-05-31T11:17:31.799988] Add path to classpath: /usr/lib64/syslog-ng/java-modules/hdfs.jar; [2016-05-31T11:17:31.800249] Add path to classpath: /usr/lib64/syslog-ng/java-modules/http.jar; [2016-05-31T11:17:31.800477] Add path to classpath: /usr/lib64/syslog-ng/java-modules/httpasyncclient-4.1.1.jar; [2016-05-31T11:17:31.800684] Add path to classpath: /usr/lib64/syslog-ng/java-modules/httpclient-4.5.2.jar; [2016-05-31T11:17:31.800940] Add path to classpath: /usr/lib64/syslog-ng/java-modules/httpcore-4.4.4.jar; [2016-05-31T11:17:31.801215] Add path to classpath: /usr/lib64/syslog-ng/java-modules/httpcore-nio-4.4.4.jar; [2016-05-31T11:17:31.801449] Add path to classpath: /usr/lib64/syslog-ng/java-modules/jest-2.0.2.jar; [2016-05-31T11:17:31.801667] Add path to classpath: /usr/lib64/syslog-ng/java-modules/jest-common-2.0.2.jar; [2016-05-31T11:17:31.801932] Add path to classpath: /usr/lib64/syslog-ng/java-modules/kafka.jar; [2016-05-31T11:17:31.802119] Add path to classpath: /usr/lib64/syslog-ng/java-modules/log4j-1.2.16.jar; [2016-05-31T11:17:31.802353] Add path to classpath: /usr/lib64/syslog-ng/java-modules/slf4j-api-1.7.13.jar; [2016-05-31T11:17:31.802569] Add path to classpath: /usr/lib64/syslog-ng/java-modules/syslog-ng-common.jar; [2016-05-31T11:17:31.802849] Add path to classpath: /usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar; [2016-05-31T11:17:31.803043] Add path to classpath: /usr/share/elasticsearch/lib/HdrHistogram-2.1.6.jar; [2016-05-31T11:17:31.803264] Add path to classpath: /usr/share/elasticsearch/lib/apache-log4j-extras-1.2.17.jar; [2016-05-31T11:17:31.803497] Add path to classpath: /usr/share/elasticsearch/lib/commons-cli-1.3.1.jar; [2016-05-31T11:17:31.803746] Add path to classpath: /usr/share/elasticsearch/lib/compiler-0.8.13.jar; [2016-05-31T11:17:31.803992] Add path to classpath: /usr/share/elasticsearch/lib/compress-lzf-1.0.2.jar; [2016-05-31T11:17:31.804249] Add path to classpath: /usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar; [2016-05-31T11:17:31.804436] Add path to classpath: /usr/share/elasticsearch/lib/guava-18.0.jar; [2016-05-31T11:17:31.804651] Add path to classpath: /usr/share/elasticsearch/lib/hppc-0.7.1.jar; [2016-05-31T11:17:31.804877] Add path to classpath: /usr/share/elasticsearch/lib/jackson-core-2.6.6.jar; [2016-05-31T11:17:31.805109] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.6.6.jar; [2016-05-31T11:17:31.805383] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-smile-2.6.6.jar; [2016-05-31T11:17:31.805663] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.6.6.jar; [2016-05-31T11:17:31.805908] Add path to classpath: /usr/share/elasticsearch/lib/jna-4.1.0.jar; [2016-05-31T11:17:31.806160] Add path to classpath: /usr/share/elasticsearch/lib/joda-convert-1.2.jar; [2016-05-31T11:17:31.806402] Add path to classpath: /usr/share/elasticsearch/lib/joda-time-2.8.2.jar; [2016-05-31T11:17:31.806654] Add path to classpath: /usr/share/elasticsearch/lib/jsr166e-1.1.0.jar; [2016-05-31T11:17:31.806889] Add path to classpath: /usr/share/elasticsearch/lib/jts-1.13.jar; [2016-05-31T11:17:31.807087] Add path to classpath: /usr/share/elasticsearch/lib/log4j-1.2.17.jar; [2016-05-31T11:17:31.807260] Add path to classpath: /usr/share/elasticsearch/lib/lucene-analyzers-common-5.5.0.jar; [2016-05-31T11:17:31.807476] Add path to classpath: /usr/share/elasticsearch/lib/lucene-backward-codecs-5.5.0.jar; [2016-05-31T11:17:31.807759] Add path to classpath: /usr/share/elasticsearch/lib/lucene-core-5.5.0.jar; [2016-05-31T11:17:31.808003] Add path to classpath: /usr/share/elasticsearch/lib/lucene-grouping-5.5.0.jar; [2016-05-31T11:17:31.808261] Add path to classpath: /usr/share/elasticsearch/lib/lucene-highlighter-5.5.0.jar; [2016-05-31T11:17:31.808476] Add path to classpath: /usr/share/elasticsearch/lib/lucene-join-5.5.0.jar; [2016-05-31T11:17:31.808653] Add path to classpath: /usr/share/elasticsearch/lib/lucene-memory-5.5.0.jar; [2016-05-31T11:17:31.808929] Add path to classpath: /usr/share/elasticsearch/lib/lucene-misc-5.5.0.jar; [2016-05-31T11:17:31.809140] Add path to classpath: /usr/share/elasticsearch/lib/lucene-queries-5.5.0.jar; [2016-05-31T11:17:31.809362] Add path to classpath: /usr/share/elasticsearch/lib/lucene-queryparser-5.5.0.jar; [2016-05-31T11:17:31.809595] Add path to classpath: /usr/share/elasticsearch/lib/lucene-sandbox-5.5.0.jar; [2016-05-31T11:17:31.809823] Add path to classpath: /usr/share/elasticsearch/lib/lucene-spatial-5.5.0.jar; [2016-05-31T11:17:31.810023] Add path to classpath: /usr/share/elasticsearch/lib/lucene-spatial3d-5.5.0.jar; [2016-05-31T11:17:31.810229] Add path to classpath: /usr/share/elasticsearch/lib/lucene-suggest-5.5.0.jar; [2016-05-31T11:17:31.810427] Add path to classpath: /usr/share/elasticsearch/lib/netty-3.10.5.Final.jar; [2016-05-31T11:17:31.810628] Add path to classpath: /usr/share/elasticsearch/lib/securesm-1.0.jar; [2016-05-31T11:17:31.810885] Add path to classpath: /usr/share/elasticsearch/lib/snakeyaml-1.15.jar; [2016-05-31T11:17:31.811065] Add path to classpath: /usr/share/elasticsearch/lib/spatial4j-0.5.jar; [2016-05-31T11:17:31.811279] Add path to classpath: /usr/share/elasticsearch/lib/t-digest-3.0.jar; [2016-05-31T11:17:33.037026] Add path to classpath: //usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar; [2016-05-31T11:17:33.073362] Error initializing message pipeline; Im running with Elasticsearch with the following configuration: destination d_es { java( class-path("/usr/lib64/syslog-ng/java-modules/*.jar:/usr/share/elasticsearch/lib/*.jar") class-name("org.syslog_ng.elasticsearch.ElasticSearchDestination") option("index", "syslog-ng_${YEAR}.${MONTH}.${DAY}") option("type", "test") option("cluster", "elasticsearch") # option("flush_limit", "100") option( "message_template", "$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n") ); }; Any hints on this ? Ivan

4 13