<div dir="ltr"><div class="gmail_extra"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Hi Robert,</div><div dir="ltr"><br></div><div dir="ltr">On Sun, Jul 3, 2016 at 2:26 PM, Fekete, Róbert <span dir="ltr"><<a href="mailto:robert.fekete@balabit.com" target="_blank">robert.fekete@balabit.com</a>></span> wrote:<br></div></div></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div><div><div>Hi Jorge, <br><br></div>This seems to be a bit tricky message. If I see it correctly, after the syslog header, you have: <br><br> * some additional info that ends with [captcha] (is that literally [captcha], or it changes with every message?)<br></div></div></div></div></div></div></div></div></blockquote><div><br></div><div>this value is dinamic.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div><div></div> * then you have some JSON<br> * "while logging request," string<br></div> * and finally some key:value pairs.<br><br></div></div></div></div></div></div></blockquote><div><br></div><div>exactly!</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div><div><div></div>As a first try, I would use the csv-parser() to separate the message to the four blocks I listed above. Use syslog-ng OSE 3.7, because then you can use strings as delimiters (see <a href="https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/reference-parsers-csv.html#csv-parser-delimiter" target="_blank">https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/reference-parsers-csv.html#csv-parser-delimiter</a>). For example, you can use the ] character and the "while logging request," string as delimiters. <br></div>Then you can run the json-parser on the macro containing the JSON-part of the message, and a kv-parser with the : delimiter to parse the last block if needed. (Note that for kv-parser part you need a recent development version of syslog-ng).<br><br></div></div></div></div></blockquote><div><br></div><div>Currently, I'm using the version 3.5.6. I will update.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div></div><div>Make sure to use the prefix() option in the json-parser, because as I see the key:value block seems to have some of the same keys that the json block.<br></div><div><br></div>BTW, is this a publicly available application that emits such a log message? It seems ideal to showcase the wide range of available syslog-ng parsers :)<br><br></div></div></div></blockquote><div><br></div><div>yep! these are errors generated by OpenResty/Nginx. today I noticed that we have another kind of error like the below sample.</div><div><br></div><div><pre class="" style="padding:0.5rem;font-family:Monaco,Menlo,Consolas,"Courier New",monospace;font-size:0.75rem;color:rgb(51,51,51);border-radius:4px;margin-top:0.5rem;margin-bottom:0.2rem;line-height:1.15rem;word-wrap:break-word;white-space:pre-wrap;border:1px solid rgba(0,0,0,0.14902);background:rgb(251,250,248)">2016/07/04 08:40:56 [error] 14011#0: *410 lua entry thread aborted: runtime error: /usr/local/openresty/lua-<a href="http://www.kipras.com/" rel="noreferrer" target="_blank" style="font-size:0.75rem;line-height:1.15rem;color:inherit;text-decoration:none">my</a><span style="font-size:0.75rem;line-height:1.15rem">/share/src/</span><a href="http://www.kipras.com/" rel="noreferrer" target="_blank" style="font-size:0.75rem;line-height:1.15rem;color:inherit;text-decoration:none">my</a><span style="font-size:0.75rem;line-height:1.15rem">/util.lua:42: attempt to get length of local 'str' (a nil value)</span><br>stack traceback:<br style="">coroutine 0:<br style=""> /usr/local/openresty/lua-<a href="http://www.kipras.com/" rel="noreferrer" target="_blank" style="font-size:0.75rem;line-height:1.15rem;color:inherit;text-decoration:none">my</a><span style="font-size:0.75rem;line-height:1.15rem">/share/src/</span><a href="http://www.kipras.com/" rel="noreferrer" target="_blank" style="font-size:0.75rem;line-height:1.15rem;color:inherit;text-decoration:none">my</a><span style="font-size:0.75rem;line-height:1.15rem">/util.lua: in function 'string_split'</span><br> /usr/local/openresty/lua-<a href="http://www.kipras.com/" rel="noreferrer" target="_blank" style="font-size:0.75rem;line-height:1.15rem;color:inherit;text-decoration:none">my</a><span style="font-size:0.75rem;line-height:1.15rem">/</span><a href="http://www.kipras.com/" rel="noreferrer" target="_blank" style="font-size:0.75rem;line-height:1.15rem;color:inherit;text-decoration:none">my</a><span style="font-size:0.75rem;line-height:1.15rem">/edge/upstream.lua:235: in function 'extract_peer'</span><br> /usr/local/openresty/lua-<a href="http://www.kipras.com/" rel="noreferrer" target="_blank" style="font-size:0.75rem;line-height:1.15rem;color:inherit;text-decoration:none">my</a><span style="font-size:0.75rem;line-height:1.15rem">/</span><a href="http://www.kipras.com/" rel="noreferrer" target="_blank" style="font-size:0.75rem;line-height:1.15rem;color:inherit;text-decoration:none">my</a><span style="font-size:0.75rem;line-height:1.15rem">/edge/upstream.lua:264: in function 'connect'</span><br> /usr/local/openresty/lua-<a href="http://www.kipras.com/" rel="noreferrer" target="_blank" style="font-size:0.75rem;line-height:1.15rem;color:inherit;text-decoration:none">my</a><span style="font-size:0.75rem;line-height:1.15rem">/</span><a href="http://www.kipras.com/" rel="noreferrer" target="_blank" style="font-size:0.75rem;line-height:1.15rem;color:inherit;text-decoration:none">my</a><span style="font-size:0.75rem;line-height:1.15rem">/edge/upstream.lua:147: in function 'request'</span><br> /usr/local/openresty/lua-<a href="http://www.kipras.com/" rel="noreferrer" target="_blank" style="font-size:0.75rem;line-height:1.15rem;color:inherit;text-decoration:none">my</a><span style="font-size:0.75rem;line-height:1.15rem">/</span><a href="http://www.kipras.com/" rel="noreferrer" target="_blank" style="font-size:0.75rem;line-height:1.15rem;color:inherit;text-decoration:none">my</a><span style="font-size:0.75rem;line-height:1.15rem">/edge/init.lua:85: in function </usr/local/openresty/lua-</span><a href="http://www.kipras.com/" rel="noreferrer" target="_blank" style="font-size:0.75rem;line-height:1.15rem;color:inherit;text-decoration:none">my</a><span style="font-size:0.75rem;line-height:1.15rem">/</span><a href="http://www.kipras.com/" rel="noreferrer" target="_blank" style="font-size:0.75rem;line-height:1.15rem;color:inherit;text-decoration:none">my</a><span style="font-size:0.75rem;line-height:1.15rem">/edge/init.lua:75></span><br> /usr/local/openresty/lua-<a href="http://www.kipras.com/" rel="noreferrer" target="_blank" style="font-size:0.75rem;line-height:1.15rem;color:inherit;text-decoration:none">my</a><span style="font-size:0.75rem;line-height:1.15rem">/</span><a href="http://www.kipras.com/" rel="noreferrer" target="_blank" style="font-size:0.75rem;line-height:1.15rem;color:inherit;text-decoration:none">my</a><span style="font-size:0.75rem;line-height:1.15rem">/edge/init.lua:153: in function 'handler'</span><br> /usr/local/openresty/lua-<a href="http://www.kipras.com/" rel="noreferrer" target="_blank" style="font-size:0.75rem;line-height:1.15rem;color:inherit;text-decoration:none">my</a><span style="font-size:0.75rem;line-height:1.15rem">/</span><a href="http://www.kipras.com/" rel="noreferrer" target="_blank" style="font-size:0.75rem;line-height:1.15rem;color:inherit;text-decoration:none">my</a><span style="font-size:0.75rem;line-height:1.15rem">/edge/init.lua:166: in function 'content'</span><br> .../openresty/nginx/../lua-<a href="http://www.kipras.com/" rel="noreferrer" target="_blank" style="font-size:0.75rem;line-height:1.15rem;color:inherit;text-decoration:none">my</a><span style="font-size:0.75rem;line-height:1.15rem">/handlers/src/content.lua:1: in function <.../openresty/nginx/../lua-</span><a href="http://www.kipras.com/" rel="noreferrer" target="_blank" style="font-size:0.75rem;line-height:1.15rem;color:inherit;text-decoration:none">my</a><span style="font-size:0.75rem;line-height:1.15rem">/handlers/src/content.lua:1>, client: 192.168.33.1, server: </span><a href="http://www.kipras.com/" rel="noreferrer" target="_blank" style="font-size:0.75rem;line-height:1.15rem;color:inherit;text-decoration:none">www.</a><a href="http://www.kipras.com/" rel="noreferrer" target="_blank" style="font-size:0.75rem;line-height:1.15rem;color:inherit;text-decoration:none">my</a><a href="http://www.kipras.com/" rel="noreferrer" target="_blank" style="font-size:0.75rem;line-height:1.15rem;color:inherit;text-decoration:none">.com</a><span style="font-size:0.75rem;line-height:1.15rem">, request: "GET /favicon.ico HTTP/1.1", host: "</span><a href="http://www.kipras.com/" rel="noreferrer" target="_blank" style="font-size:0.75rem;line-height:1.15rem;color:inherit;text-decoration:none">www.my.com</a><span style="font-size:0.75rem;line-height:1.15rem">"</span></pre></div><div><br></div><div>Now for me looks to be impossible to handler this... do you have some idea? because, I receive this from nginx it over unix-stream and need to send to a remote syslog-ng (collector) saving like:</div><div><br></div><div>/var/log/syslog-ng/www.my.com_error.log</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div></div>Regards, <br></div>Robert<br></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On Sun, Jul 3, 2016 at 3:32 PM, Jorge Pereira <span dir="ltr"><<a href="mailto:jpereiran@gmail.com" target="_blank">jpereiran@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div><div class="h5"><div dir="ltr">Hi,<br><br> I am not sure about the best approach and way to fix my problem, below more information.<br><br>1) I receive the below packet sent from a nginx/openresty instance.<br><br>2016/07/02 01:17:04 [emerg] 19081#0: *13163 [lua] init.lua:115: [captcha] {"fail_count":"","response_code":200,"client_ip":"192.168.1.22","hostname":"server-lab01","request_id":"2016-07-02T01:17:03Z|9175f93c0c||i0Xb3BuBWV","host":"<a href="http://www.mytest.com" target="_blank">www.mytest.com</a>","http_request":{"verb":"GET","url":"\/","user-agent":"Mozilla\/5.0 (pc-x86_64-linux-gnu) Siege\/3.0.8","http_version":"1.1","all":"{\"host\":\"<a href="http://www.mytest.com" target="_blank">www.mytest.com</a>\",\"x-country-code\":\"US\",\"connection\":\"close\",\"accept\":\"*\\\/*\",\"x-client-ip\":\"192.168.1.22\",\"user-agent\":\"Mozilla\\\/5.0 (pc-x86_64-linux-gnu) Siege\\\/3.0.8\",\"accept-encoding\":\"gzip\"}"},"geoip":{"location":"-90.5334,38.6500","city_name":"Chesterfield","country_name":"United States","longitude":-90.5334,"area_code":314,"latitude":38.65,"country_code2":"US","country_code3":"USA"},"got":"","action":"show","expected":"h1szmM","webapp_domain":"<a href="http://www.mytest.com" target="_blank">www.mytest.com</a>"} while logging request, client: 192.168.1.22, server: <a href="http://www.mytest.com" target="_blank">www.mytest.com</a>, request: "GET / HTTP/1.1", host: "<a href="http://www.mytest.com" target="_blank">www.mytest.com</a>"<br><br>2) In my server side, I need to save the logs following a value of <i>host: "<a href="http://www.mytest.com" target="_blank">www.mytest.com</a>"</i> like:<div><br></div><div>/var/log/syslog-ng/www.mytest.com.log</div><div><br></div><div>3) The problem is because the packet received has a part being a jSON, but I can't use the <i>json-parser().</i></div><div><i><br></i></div><div>4) What is the best approach? I have used:</div><div><br></div><div><div># Extracting only the jSON payload</div><div>rewrite p_nginx_wb_error_log_clean {</div><div> subst(".*captcha] ", "", value("MESSAGE"), flags("global"));</div><div> subst(" while logging request.*$", "", value("MESSAGE"), flags("global"));</div><div>};</div><div><br></div><div>parser p_nginx_wb_error_log_json {</div><div> json-parser(</div><div> marker("")</div><div> prefix("j.")</div><div> ); </div><div>};</div><div><br></div><div>destination d_nginx_wb_error_log {</div><div> file("/var/log/syslog-ng/nginx/${j.webapp_domain:-unknow-payload}_error.log" </div><div> create_dirs(yes)</div><div> owner("root")</div><div> group("root")</div><div> perm(0644)</div><div> dir_perm(0755)</div><div> template("${MSG}\n")</div><div> ); </div><div>};</div><div><br></div>--<br>Jorge Pereira</div></div>
<br></div></div>______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div>
<br>______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div></div>