[syslog-ng] extracting jSON from $MESSAGE

Tue Jul 5 12:14:57 CEST 2016

Hi,

theoretically, the csv-parser might handle multi-line messages as well, so
you could try to use the ">," characters as delimiter to split the message
into two parts, and then use the kv-parsers on the the key:value pairs that
go into the second part of the message.

But I'm not sure if csv-parser handles multiline messages perfectly, so let
us know if something is not working.

Regards,
Robert

On Mon, Jul 4, 2016 at 3:02 PM, Jorge Pereira <jpereiran at gmail.com> wrote:

> Hi Robert,
>
> On Sun, Jul 3, 2016 at 2:26 PM, Fekete, Róbert <robert.fekete at balabit.com>
> wrote:
>
>> Hi Jorge,
>>
>> This seems to be a bit tricky message. If I see it correctly, after the
>> syslog header, you have:
>>
>>  * some additional info that ends with [captcha] (is that literally
>> [captcha], or it changes with every message?)
>>
>
> this value is dinamic.
>
>
>>  * then you have some JSON
>>  * "while logging request," string
>>  * and finally some key:value pairs.
>>
>>
> exactly!
>
>
>> As a first try, I would use the csv-parser() to separate the message to
>> the four blocks I listed above. Use syslog-ng OSE 3.7, because then you can
>> use strings as delimiters (see
>> https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/reference-parsers-csv.html#csv-parser-delimiter).
>> For example, you can use the ] character and the "while logging request,"
>> string as delimiters.
>> Then you can run the json-parser on the macro containing the JSON-part of
>> the message, and a kv-parser with the : delimiter to parse the last block
>> if needed. (Note that for kv-parser part you need a recent development
>> version of syslog-ng).
>>
>>
> Currently, I'm using the version 3.5.6. I will update.
>
>
>> Make sure to use the prefix() option in the json-parser, because as I see
>> the key:value block seems to have some of the same keys that the json block.
>>
>> BTW, is this a publicly available application that emits such a log
>> message? It seems ideal to showcase the wide range of available syslog-ng
>> parsers :)
>>
>>
> yep! these are errors generated by OpenResty/Nginx. today I noticed that
> we have another kind of error like the below sample.
>
> 2016/07/04 08:40:56 [error] 14011#0: *410 lua entry thread aborted: runtime error: /usr/local/openresty/lua-my <http://www.kipras.com/>/share/src/my <http://www.kipras.com/>/util.lua:42: attempt to get length of local 'str' (a nil value)
> stack traceback:
> coroutine 0:
>     /usr/local/openresty/lua-my <http://www.kipras.com/>/share/src/my <http://www.kipras.com/>/util.lua: in function 'string_split'
>     /usr/local/openresty/lua-my <http://www.kipras.com/>/my <http://www.kipras.com/>/edge/upstream.lua:235: in function 'extract_peer'
>     /usr/local/openresty/lua-my <http://www.kipras.com/>/my <http://www.kipras.com/>/edge/upstream.lua:264: in function 'connect'
>     /usr/local/openresty/lua-my <http://www.kipras.com/>/my <http://www.kipras.com/>/edge/upstream.lua:147: in function 'request'
>     /usr/local/openresty/lua-my <http://www.kipras.com/>/my <http://www.kipras.com/>/edge/init.lua:85: in function </usr/local/openresty/lua-my <http://www.kipras.com/>/my <http://www.kipras.com/>/edge/init.lua:75>
>     /usr/local/openresty/lua-my <http://www.kipras.com/>/my <http://www.kipras.com/>/edge/init.lua:153: in function 'handler'
>     /usr/local/openresty/lua-my <http://www.kipras.com/>/my <http://www.kipras.com/>/edge/init.lua:166: in function 'content'
>     .../openresty/nginx/../lua-my <http://www.kipras.com/>/handlers/src/content.lua:1: in function <.../openresty/nginx/../lua-my <http://www.kipras.com/>/handlers/src/content.lua:1>, client: 192.168.33.1, server: www. <http://www.kipras.com/>my <http://www.kipras.com/>.com <http://www.kipras.com/>, request: "GET /favicon.ico HTTP/1.1", host: "www.my.com <http://www.kipras.com/>"
>
>
> Now for me looks to be impossible to handler this... do you have some
> idea? because, I receive this from nginx it over unix-stream and need to
> send to a remote syslog-ng (collector) saving like:
>
> /var/log/syslog-ng/www.my.com_error.log
>
>
>> Regards,
>> Robert
>>
>> On Sun, Jul 3, 2016 at 3:32 PM, Jorge Pereira <jpereiran at gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>>     I am not sure about the best approach and way to fix my problem,
>>> below more information.
>>>
>>> 1) I receive the below packet sent from a nginx/openresty instance.
>>>
>>> 2016/07/02 01:17:04 [emerg] 19081#0: *13163 [lua] init.lua:115:
>>> [captcha]
>>> {"fail_count":"","response_code":200,"client_ip":"192.168.1.22","hostname":"server-lab01","request_id":"2016-07-02T01:17:03Z|9175f93c0c||i0Xb3BuBWV","host":"
>>> www.mytest.com","http_request":{"verb":"GET","url":"\/","user-agent":"Mozilla\/5.0
>>> (pc-x86_64-linux-gnu) Siege\/3.0.8","http_version":"1.1","all":"{\"host\":\"
>>> www.mytest.com\",\"x-country-code\":\"US\",\"connection\":\"close\",\"accept\":\"*\\\/*\",\"x-client-ip\":\"192.168.1.22\",\"user-agent\":\"Mozilla\\\/5.0
>>> (pc-x86_64-linux-gnu)
>>> Siege\\\/3.0.8\",\"accept-encoding\":\"gzip\"}"},"geoip":{"location":"-90.5334,38.6500","city_name":"Chesterfield","country_name":"United
>>> States","longitude":-90.5334,"area_code":314,"latitude":38.65,"country_code2":"US","country_code3":"USA"},"got":"","action":"show","expected":"h1szmM","webapp_domain":"
>>> www.mytest.com"} while logging request, client: 192.168.1.22, server:
>>> www.mytest.com, request: "GET / HTTP/1.1", host: "www.mytest.com"
>>>
>>> 2) In my server side, I need to save the logs following a value of *host:
>>> "www.mytest.com <http://www.mytest.com>"* like:
>>>
>>> /var/log/syslog-ng/www.mytest.com.log
>>>
>>> 3) The problem is because the packet received has a part being a jSON,
>>> but I can't use the *json-parser().*
>>>
>>> 4) What is the best approach? I have used:
>>>
>>> # Extracting only the jSON payload
>>> rewrite p_nginx_wb_error_log_clean {
>>>     subst(".*captcha] ", "", value("MESSAGE"), flags("global"));
>>>     subst(" while logging request.*$", "", value("MESSAGE"),
>>> flags("global"));
>>> };
>>>
>>> parser p_nginx_wb_error_log_json {
>>>     json-parser(
>>>         marker("")
>>>         prefix("j.")
>>>     );
>>> };
>>>
>>> destination d_nginx_wb_error_log {
>>>
>>> file("/var/log/syslog-ng/nginx/${j.webapp_domain:-unknow-payload}_error.log"
>>>
>>>
>>>
>>>          create_dirs(yes)
>>>          owner("root")
>>>          group("root")
>>>          perm(0644)
>>>          dir_perm(0755)
>>>          template("${MSG}\n")
>>>     );
>>> };
>>>
>>> --
>>> Jorge Pereira
>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20160705/98f9fb03/attachment.htm 