[syslog-ng] I need some help with Syslog-ng and the new json parser

[Gergely Nagy]

Sebastien Pasche <braoru at gmail.com> writes:

> I will present to you what I want to do and what I actually have.
>
> I would like to extract a field from a json log arriving in this source :
>
> source s_collector_tcp_json {
>     tcp(ip(0.0.0.0) port(514) flags(no-multi-line) flags(no-parse));
> };
>
> And replacing the Program field I use in my destination
[...]
> from the field @type of this json log :
>
> {
>     "@source": "tcp://127.0.0.1:9999/client/127.0.0.1:57530",
>     "@type": "tomcat_logstash_raw_json",
>     "@tags": [
>         "tomcat_site"
>     ],
>     "@fields": {
>         "priority": "INFO",
>         "logger_name": "com.zzz.user.UserData",
>         "thread": "TP-Processor7",
>         "class":
> "org.apache.jsp.WEB_002dINF.jsp.user.ViewInvoiceDetail_jsp",
>         "file": "ViewInvoiceDetail_jsp.java:162",
>         "method": "_jspService",
>         "prop_userIp": "192.168.215.50",
>         "prop_userId": "1440704"
>     },
>     "@source_host": "127.0.0.1:57530",
>     "@source_path": "com.leshop.user.UserData",
>     "@message": "order : {WAREHOUSE_TYPE=drive, OID=5693367,
> ORDER_DATE=2012-10-03 08:49:17.41, SHIPPING_FRESH=0.0,
> FROZEN_DEPOSIT=0.0, WAREHOUSE_ID=5, DUE_AMOUNT=0.0, TOTAL_CREDITS=0.0,
> ADDRESS_NUMBER=, DELIV_HELPFUL_INDICATION=, DELIVERY_MODE=20:00,
> DELIVERY_DATE=2012-10-03 00:00:00.0, TOTAL=134.75, ACTION_TOTAL=0.0,
> ORDER_NUMBER=abc-014085706-xyz, TRACK_TRACE=, RETAILER_GROUP=0, ZIP=,
> ORDER_STATE=3, PAYMENT_TYPE=7, DELIV_DOORCODE=, FROZEN_FEES=0.0,
> ENV_CO2=0.0, NAME= , ENV_CO2_RETAIL=0.0, HIDE_BVR=false, ADDRESS=,
> TOTAL_CREDIT=0.0, MODIFICATION_STATE=1, REMINDER_LEVEL=0,
> SUBTOTAL=134.75, GRAND_TOTAL=134.75, BVR_REFERENCE=, CITY=,
> DELIV_PHONE=, SHIPPING_FIXED=0.0}",
>     "@timestamp": "2012-10-03T06:49:23.373000Z"
> }
[...]

Assuming that the JSON arrives on a single line, something along these
lines should do the trick:

parser p_tomcat_json {
       json-parser(prefix("json."));
};

rewrite rw_tomcat_site_logstash_json_program_name {
       set("${json.type}", value("$PROGRAM"));
};

And then chain it together:

log {
 source(s_collector_tcp_json);
 parser(p_tomcat_json);
 rewrite(rw_tomcat_site_logstash_json_program_name);
 destination(d_file_normal_r);
};

Hope that helps!

-- 
|8]