July 2016 - syslog-ng - lists.balabit.hu

New ETPLC project with Syslog-NG for checking >9000 Threats on your logs!
by rmkml 13 Jul '16

13 Jul '16

Hello, I am pround to announce the new http://etplc.org open source project update with Syslog-NG for checking more than 9000 Threats on your webserver/proxy logs! It's a open source project, all feedbacks / informations are welcome. Easy to use since 3 years ago ;) 1) add ETPLC on your Syslog-NG configuration like that: (of course check before perl+etplc PATH and source/filter/destination configurations...) destination d_prog { program("/usr/bin/perl /var/tmp/etplc_12jul2016a.pl -f /var/tmp/emergingall_sigs11jul2016a_snort290b.rules -s"); }; log { source(s_src); destination(d_prog); }; 2) ETPLC send alert to localhost:514/udp with "-s" option 3) See All options with "-h" 4) Already supported format is Squid, Apache, Nginx, ForeFront, BlueCoat, McAfee Web Gateway, IIS logs... 5) ETPLC exist on Perl and Python versions ETPLC available on: -main http://etplc.org -http://sourceforge.net/projects/etplc/ -https://github.com/rmkml/etplc -https://hub.docker.com/r/rmkml/etplc/ -http://twitter.com/rmkml Special THX to InfoSec community and @EmergingThreats team! Best Regards @Rmkml

2 1

Need mvn repository info for syslog-ng-core.jar
by Kiran Doddi 12 Jul '16

12 Jul '16

Hi, We are using syslog-ng server for receiving logs from different sources & we need to configure Java Destination for writing those logs into local file. For this, While implementing, I found syslog-ng-core.jar file, but I could not found the same anywhere in maven repo, so that I can add its dependency in my pom.xml & avoid putting it manually in jar bundling. Kindly need your assistance to solve my issues. Thanks & Regards, Kiran

3 2

Re: [syslog-ng] Transport aux data overflow
by Scheidler, Balázs 07 Jul '16

07 Jul '16

We get various data points on the process connecting via /dev/log via recvmsg() that returns this in an aux buffer, statically sized to 1024. It's interesting that you have more. If you can reproduce it, can you strace syslog-ng to see what is coming aux data? This can be resized but that needs a recompilation. On Jul 6, 2016 11:18 PM, "Noémi Ványi" <sitbackandwait(a)gmail.com> wrote: Hi! It seems to me that aux buffer size is a hard-coded value. So it cannot be configured using the config file. I guess the size could be increased, but I am curious what the devs of syslog-ng would say. :) On 6 July 2016 at 22:09, Evan Rempel <erempel(a)uvic.ca> wrote: > I got this log line out of syslog-ng, but I'm not sure what it really > means. > > Can anyone shed some slight on it and let me know how I can increase the > aux buffer size? > > syslog-ng[29712]: Transport aux data overflow, some fields may not be > associated with the message, please increase aux buffer size; > aux_size='1024' > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

Transport aux data overflow
by Evan Rempel 06 Jul '16

06 Jul '16

I got this log line out of syslog-ng, but I'm not sure what it really means. Can anyone shed some slight on it and let me know how I can increase the aux buffer size? syslog-ng[29712]: Transport aux data overflow, some fields may not be associated with the message, please increase aux buffer size; aux_size='1024'

2 1

Re: [syslog-ng] extracting jSON from $MESSAGE
by Jorge Pereira 05 Jul '16

05 Jul '16

Hi Robert, On Sun, Jul 3, 2016 at 2:26 PM, Fekete, Róbert <robert.fekete(a)balabit.com> wrote: > Hi Jorge, > > This seems to be a bit tricky message. If I see it correctly, after the > syslog header, you have: > > * some additional info that ends with [captcha] (is that literally > [captcha], or it changes with every message?) > this value is dinamic. > * then you have some JSON > * "while logging request," string > * and finally some key:value pairs. > > exactly! > As a first try, I would use the csv-parser() to separate the message to > the four blocks I listed above. Use syslog-ng OSE 3.7, because then you can > use strings as delimiters (see > https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-…) > For example, you can use the ] character and the "while logging request," > string as delimiters. > Then you can run the json-parser on the macro containing the JSON-part of > the message, and a kv-parser with the : delimiter to parse the last block > if needed. (Note that for kv-parser part you need a recent development > version of syslog-ng). > > Currently, I'm using the version 3.5.6. I will update. > Make sure to use the prefix() option in the json-parser, because as I see > the key:value block seems to have some of the same keys that the json block. > > BTW, is this a publicly available application that emits such a log > message? It seems ideal to showcase the wide range of available syslog-ng > parsers :) > > yep! these are errors generated by OpenResty/Nginx. today I noticed that we have another kind of error like the below sample. 2016/07/04 08:40:56 [error] 14011#0: *410 lua entry thread aborted: runtime error: /usr/local/openresty/lua-my <http://www.kipras.com/>/share/src/my <http://www.kipras.com/>/util.lua:42: attempt to get length of local 'str' (a nil value) stack traceback: coroutine 0: /usr/local/openresty/lua-my <http://www.kipras.com/>/share/src/my <http://www.kipras.com/>/util.lua: in function 'string_split' /usr/local/openresty/lua-my <http://www.kipras.com/>/my <http://www.kipras.com/>/edge/upstream.lua:235: in function 'extract_peer' /usr/local/openresty/lua-my <http://www.kipras.com/>/my <http://www.kipras.com/>/edge/upstream.lua:264: in function 'connect' /usr/local/openresty/lua-my <http://www.kipras.com/>/my <http://www.kipras.com/>/edge/upstream.lua:147: in function 'request' /usr/local/openresty/lua-my <http://www.kipras.com/>/my <http://www.kipras.com/>/edge/init.lua:85: in function </usr/local/openresty/lua-my <http://www.kipras.com/>/my <http://www.kipras.com/>/edge/init.lua:75> /usr/local/openresty/lua-my <http://www.kipras.com/>/my <http://www.kipras.com/>/edge/init.lua:153: in function 'handler' /usr/local/openresty/lua-my <http://www.kipras.com/>/my <http://www.kipras.com/>/edge/init.lua:166: in function 'content' .../openresty/nginx/../lua-my <http://www.kipras.com/>/handlers/src/content.lua:1: in function <.../openresty/nginx/../lua-my <http://www.kipras.com/>/handlers/src/content.lua:1>, client: 192.168.33.1, server: www. <http://www.kipras.com/>my <http://www.kipras.com/>.com <http://www.kipras.com/>, request: "GET /favicon.ico HTTP/1.1", host: "www.my.com <http://www.kipras.com/>" Now for me looks to be impossible to handler this... do you have some idea? because, I receive this from nginx it over unix-stream and need to send to a remote syslog-ng (collector) saving like: /var/log/syslog-ng/www.my.com_error.log > Regards, > Robert > > On Sun, Jul 3, 2016 at 3:32 PM, Jorge Pereira <jpereiran(a)gmail.com> wrote: > >> Hi, >> >> I am not sure about the best approach and way to fix my problem, >> below more information. >> >> 1) I receive the below packet sent from a nginx/openresty instance. >> >> 2016/07/02 01:17:04 [emerg] 19081#0: *13163 [lua] init.lua:115: [captcha] >> {"fail_count":"","response_code":200,"client_ip":"192.168.1.22","hostname":"server-lab01","request_id":"2016-07-02T01:17:03Z|9175f93c0c||i0Xb3BuBWV","host":" >> www.mytest.com","http_request":{"verb":"GET","url":"\/","user-agent":"Mozilla\/5.0 >> (pc-x86_64-linux-gnu) Siege\/3.0.8","http_version":"1.1","all":"{\"host\":\" >> www.mytest.com\",\"x-country-code\":\"US\",\"connection\":\"close\",\"accept\":\"*\\\/*\",\"x-client-ip\":\"192.168.1.22\",\"user-agent\":\"Mozilla\\\/5.0 >> (pc-x86_64-linux-gnu) >> Siege\\\/3.0.8\",\"accept-encoding\":\"gzip\"}"},"geoip":{"location":"-90.5334,38.6500","city_name":"Chesterfield","country_name":"United >> States","longitude":-90.5334,"area_code":314,"latitude":38.65,"country_code2":"US","country_code3":"USA"},"got":"","action":"show","expected":"h1szmM","webapp_domain":" >> www.mytest.com"} while logging request, client: 192.168.1.22, server: >> www.mytest.com, request: "GET / HTTP/1.1", host: "www.mytest.com" >> >> 2) In my server side, I need to save the logs following a value of *host: >> "www.mytest.com <http://www.mytest.com>"* like: >> >> /var/log/syslog-ng/www.mytest.com.log >> >> 3) The problem is because the packet received has a part being a jSON, >> but I can't use the *json-parser().* >> >> 4) What is the best approach? I have used: >> >> # Extracting only the jSON payload >> rewrite p_nginx_wb_error_log_clean { >> subst(".*captcha] ", "", value("MESSAGE"), flags("global")); >> subst(" while logging request.*$", "", value("MESSAGE"), >> flags("global")); >> }; >> >> parser p_nginx_wb_error_log_json { >> json-parser( >> marker("") >> prefix("j.") >> ); >> }; >> >> destination d_nginx_wb_error_log { >> >> file("/var/log/syslog-ng/nginx/${j.webapp_domain:-unknow-payload}_error.log" >> >> >> >> create_dirs(yes) >> owner("root") >> group("root") >> perm(0644) >> dir_perm(0755) >> template("${MSG}\n") >> ); >> }; >> >> -- >> Jorge Pereira >> >> >> ______________________________________________________________________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: >> http://www.balabit.com/support/documentation/?product=syslog-ng >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> >> >> > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > >

2 1

Re: [syslog-ng] extracting jSON from $MESSAGE
by jrhendri 03 Jul '16

03 Jul '16

Hanes you looked at using patterndb? That should let you assign the specific parts of the message to variables you choose and then use them as you like.It should be possible to do it with json parser but I have not used it personally. Jim Sent from my Verizon, Samsung Galaxy smartphone -------- Original message --------From: "Fekete, Róbert" <robert.fekete(a)balabit.com> Date: 7/3/16 1:26 PM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] extracting jSON from $MESSAGE Hi Jorge, This seems to be a bit tricky message. If I see it correctly, after the syslog header, you have: * some additional info that ends with [captcha] (is that literally [captcha], or it changes with every message?) * then you have some JSON * "while logging request," string * and finally some key:value pairs. As a first try, I would use the csv-parser() to separate the message to the four blocks I listed above. Use syslog-ng OSE 3.7, because then you can use strings as delimiters (see https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-…) For example, you can use the ] character and the "while logging request," string as delimiters. Then you can run the json-parser on the macro containing the JSON-part of the message, and a kv-parser with the : delimiter to parse the last block if needed. (Note that for kv-parser part you need a recent development version of syslog-ng). Make sure to use the prefix() option in the json-parser, because as I see the key:value block seems to have some of the same keys that the json block. BTW, is this a publicly available application that emits such a log message? It seems ideal to showcase the wide range of available syslog-ng parsers :) Regards, Robert On Sun, Jul 3, 2016 at 3:32 PM, Jorge Pereira <jpereiran(a)gmail.com> wrote: Hi, I am not sure about the best approach and way to fix my problem, below more information. 1) I receive the below packet sent from a nginx/openresty instance. 2016/07/02 01:17:04 [emerg] 19081#0: *13163 [lua] init.lua:115: [captcha] {"fail_count":"","response_code":200,"client_ip":"192.168.1.22","hostname":"server-lab01","request_id":"2016-07-02T01:17:03Z|9175f93c0c||i0Xb3BuBWV","host":"www.mytest.com","http_request":{"verb":"GET","url":"\/","user-agent":"Mozilla\/5.0 (pc-x86_64-linux-gnu) Siege\/3.0.8","http_version":"1.1","all":"{\"host\":\"www.mytest.com\",\"x-country-code\":\"US\",\"connection\":\"close\",\"accept\":\"*\\\/*\",\"x-client-ip\":\"192.168.1.22\",\"user-agent\":\"Mozilla\\\/5.0 (pc-x86_64-linux-gnu) Siege\\\/3.0.8\",\"accept-encoding\":\"gzip\"}"},"geoip":{"location":"-90.5334,38.6500","city_name":"Chesterfield","country_name":"United States","longitude":-90.5334,"area_code":314,"latitude":38.65,"country_code2":"US","country_code3":"USA"},"got":"","action":"show","expected":"h1szmM","webapp_domain":"www.mytest.com"} while logging request, client: 192.168.1.22, server: www.mytest.com, request: "GET / HTTP/1.1", host: "www.mytest.com" 2) In my server side, I need to save the logs following a value of host: "www.mytest.com" like: /var/log/syslog-ng/www.mytest.com.log 3) The problem is because the packet received has a part being a jSON, but I can't use the json-parser(). 4) What is the best approach? I have used: # Extracting only the jSON payloadrewrite p_nginx_wb_error_log_clean { subst(".*captcha] ", "", value("MESSAGE"), flags("global")); subst(" while logging request.*$", "", value("MESSAGE"), flags("global"));}; parser p_nginx_wb_error_log_json { json-parser( marker("") prefix("j.") ); }; destination d_nginx_wb_error_log { file("/var/log/syslog-ng/nginx/${j.webapp_domain:-unknow-payload}_error.log" create_dirs(yes) owner("root") group("root") perm(0644) dir_perm(0755) template("${MSG}\n") ); }; -- Jorge Pereira ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

extracting jSON from $MESSAGE
by Jorge Pereira 03 Jul '16

03 Jul '16

Hi, I am not sure about the best approach and way to fix my problem, below more information. 1) I receive the below packet sent from a nginx/openresty instance. 2016/07/02 01:17:04 [emerg] 19081#0: *13163 [lua] init.lua:115: [captcha] {"fail_count":"","response_code":200,"client_ip":"192.168.1.22","hostname":"server-lab01","request_id":"2016-07-02T01:17:03Z|9175f93c0c||i0Xb3BuBWV","host":" www.mytest.com","http_request":{"verb":"GET","url":"\/","user-agent":"Mozilla\/5.0 (pc-x86_64-linux-gnu) Siege\/3.0.8","http_version":"1.1","all":"{\"host\":\" www.mytest.com\",\"x-country-code\":\"US\",\"connection\":\"close\",\"accept\":\"*\\\/*\",\"x-client-ip\":\"192.168.1.22\",\"user-agent\":\"Mozilla\\\/5.0 (pc-x86_64-linux-gnu) Siege\\\/3.0.8\",\"accept-encoding\":\"gzip\"}"},"geoip":{"location":"-90.5334,38.6500","city_name":"Chesterfield","country_name":"United States","longitude":-90.5334,"area_code":314,"latitude":38.65,"country_code2":"US","country_code3":"USA"},"got":"","action":"show","expected":"h1szmM","webapp_domain":" www.mytest.com"} while logging request, client: 192.168.1.22, server: www.mytest.com, request: "GET / HTTP/1.1", host: "www.mytest.com" 2) In my server side, I need to save the logs following a value of *host: "www.mytest.com <http://www.mytest.com>"* like: /var/log/syslog-ng/www.mytest.com.log 3) The problem is because the packet received has a part being a jSON, but I can't use the *json-parser().* 4) What is the best approach? I have used: # Extracting only the jSON payload rewrite p_nginx_wb_error_log_clean { subst(".*captcha] ", "", value("MESSAGE"), flags("global")); subst(" while logging request.*$", "", value("MESSAGE"), flags("global")); }; parser p_nginx_wb_error_log_json { json-parser( marker("") prefix("j.") ); }; destination d_nginx_wb_error_log { file("/var/log/syslog-ng/nginx/${j.webapp_domain:-unknow-payload}_error.log" create_dirs(yes) owner("root") group("root") perm(0644) dir_perm(0755) template("${MSG}\n") ); }; -- Jorge Pereira

2 1

Re: [syslog-ng] Syslog-ng Multiple Instances
by David Campeau 01 Jul '16

01 Jul '16

Hello, I've been using syslog-ng to filter syslog before forwarding on to a log collector. However, I need to spin up a second instance for testing purposes. I've found a little bit of information on-line, but it hasn't completed the entire picture. This is the command used to start up the 2nd instance. I'm pointing to separate .conf .persist .pid and .ctl files -- However, it's still not working. I suspect the issue is due to OS log sources. How do a change log sources? syslog-ng --cfgfile=/usr/local/bin/syslog-ng-Second-Instance/syslog-ng.conf --persist-file=/usr/local/bin/syslog-ng-Second-Instance/syslog-ng.persist --pidfile=/usr/local/bin/syslog-ng-Second-Instance/syslog-ng.pid --control=/usr/local/bin/syslog-ng-Second-Instance/syslog-ng.ctl & This is the upper part of the syslog-ng.conf file for the 2nd instance I wish to run. @version: 3.3.4 @include "scl.conf" options { time-reap(30); mark-freq(10); keep-hostname(yes); chain-hostnames(no); use-dns(no); ## log-fifo-size(500000); ## Tuning Options ## flush_lines(10000); ## Tuning Options ## flush_timeout(10000); ## Tuning Options }; source s_second_instance { syslog(transport("udp") port("518")); #### Will receive test syslog on port 518 }; destination d_syslog_udp { syslog("10.X.X.X" transport("udp") port("514") throttle(4000) ); }; I'm hoping someone has experience or has seen information on how to run a 2nd instance on the same box. Best Regards, David

3 7