[syslog-ng] I need some help with Syslog-ng and the new json parser

[Sebastien Pasche]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Thank a lot for the help.

I tested with syslog-ng 3.4.0 alpha3 and its working but I use 3.3 :-(

So I will use regex and move to json parser when 3.4 will be stable :)

Thanks again !

Sébastien

On 10/05/2012 12:00 PM, syslog-ng-request at lists.balabit.hu wrote:
> Sebastien Pasche <braoru at gmail.com> writes:
>
>> > I will present to you what I want to do and what I actually have.
>> >
>> > I would like to extract a field from a json log arriving in this
source :
>> >
>> > source s_collector_tcp_json {
>> > tcp(ip(0.0.0.0) port(514) flags(no-multi-line) flags(no-parse));
>> > };
>> >
>> > And replacing the Program field I use in my destination
> [...]
>> > from the field @type of this json log :
>> >
>> > {
>> > "@source": "tcp://127.0.0.1:9999/client/127.0.0.1:57530",
>> > "@type": "tomcat_logstash_raw_json",
>> > "@tags": [
>> > "tomcat_site"
>> > ],
>> > "@fields": {
>> > "priority": "INFO",
>> > "logger_name": "com.zzz.user.UserData",
>> > "thread": "TP-Processor7",
>> > "class":
>> > "org.apache.jsp.WEB_002dINF.jsp.user.ViewInvoiceDetail_jsp",
>> > "file": "ViewInvoiceDetail_jsp.java:162",
>> > "method": "_jspService",
>> > "prop_userIp": "192.168.215.50",
>> > "prop_userId": "1440704"
>> > },
>> > "@source_host": "127.0.0.1:57530",
>> > "@source_path": "com.leshop.user.UserData",
>> > "@message": "order : {WAREHOUSE_TYPE=drive, OID=5693367,
>> > ORDER_DATE=2012-10-03 08:49:17.41, SHIPPING_FRESH=0.0,
>> > FROZEN_DEPOSIT=0.0, WAREHOUSE_ID=5, DUE_AMOUNT=0.0, TOTAL_CREDITS=0.0,
>> > ADDRESS_NUMBER=, DELIV_HELPFUL_INDICATION=, DELIVERY_MODE=20:00,
>> > DELIVERY_DATE=2012-10-03 00:00:00.0, TOTAL=134.75, ACTION_TOTAL=0.0,
>> > ORDER_NUMBER=abc-014085706-xyz, TRACK_TRACE=, RETAILER_GROUP=0, ZIP=,
>> > ORDER_STATE=3, PAYMENT_TYPE=7, DELIV_DOORCODE=, FROZEN_FEES=0.0,
>> > ENV_CO2=0.0, NAME= , ENV_CO2_RETAIL=0.0, HIDE_BVR=false, ADDRESS=,
>> > TOTAL_CREDIT=0.0, MODIFICATION_STATE=1, REMINDER_LEVEL=0,
>> > SUBTOTAL=134.75, GRAND_TOTAL=134.75, BVR_REFERENCE=, CITY=,
>> > DELIV_PHONE=, SHIPPING_FIXED=0.0}",
>> > "@timestamp": "2012-10-03T06:49:23.373000Z"
>> > }
> [...]
>
> Assuming that the JSON arrives on a single line, something along these
> lines should do the trick:
>
> parser p_tomcat_json {
> json-parser(prefix("json."));
> };
>
> rewrite rw_tomcat_site_logstash_json_program_name {
> set("${json.type}", value("$PROGRAM"));
> };
>
> And then chain it together:
>
> log {
> source(s_collector_tcp_json);
> parser(p_tomcat_json);
> rewrite(rw_tomcat_site_logstash_json_program_name);
> destination(d_file_normal_r);
> };
>
> Hope that helps!

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQIcBAEBAgAGBQJQbrE/AAoJEE3IBph3MKVPidkQALooeGZRm2FEIGPN7uqJVrRc
2yeQ0L68Nny0WLnR2mZU4nOy7EHMKLPufrYINAp1FyNM7U4xDOkLTxoVhkPtejis
hKce2Z/djJX1cABlPl0qf3ClBsSnH7oe745BvSdXUnBXFGc7mBV7E10GF4tb4Ce/
ElUUEx2UyIfCByOcETOJkvZrY1WSky1MqJZLI5dX2BzjKsdBML5Bmi/pj1VVRCaa
WUC6lkSnGUdoyAdVCF7hHzNwQbm1txTUjsdo1oUvdaly4ASDkHWuHhRXsNw9gugA
3TRNugj/69oeOzxZkt/eVukpcK2JbeJ6UGVWpYe8Vo3x5slHXT5tObOcIni4eCYU
/DnGFs7UT1nGmQ8HYTkytgnDUa3dpAcVU60xdyLsjkx7WPUmLjKJcKD//cXdJXH/
uzI3WvEEqnPqb3oRJfNqrR0Ikn1tF0jOHpJxeW3lQIYMaadahdTUQ1dhccycUWrB
ISEoy1tFSUJepOHcZf+RGRTopxJVxvpK7JondR3oV0BMDf00Vgu8SLmuWl/ok3aZ
KFAZ019kMYX1Bq0sCAtOyJwWzIXUFEtMz8rEkc1V0fbAI032WY6IzUecWuLGM783
uZ/C2bDBR2Xgbs/oDMhZImKJzjouN8IGuNK9ZaRMH0Qx0dx4wjc8QORD9hB2kmmF
lEjQIEQ8D8ADAavqwURp
=pUIX
-----END PGP SIGNATURE-----