[syslog-ng] I need some help with Syslog-ng and the new json parser

[Sebastien Pasche]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello :)

I will present to you what I want to do and what I actually have.


I would like to extract a field from a json log arriving in this source :

source s_collector_tcp_json {
    tcp(ip(0.0.0.0) port(514) flags(no-multi-line) flags(no-parse));
};

And replacing the Program field I use in my destination :

#Destination that uses received time as timestamp for logs
destination d_file_normal_r
{file("/var/log/leshop/leshop_log/$R_YEAR/$HOST/$PROGRAM/$R_YEAR-$R_MONTH-$R_DAY.log"
template(t_d_default_r) group(users) dir_group(users) perm(0640)
dir_perm(0750) flags(no-multi-line) frac_digits(6));};

#Templates for destination that uses received time as timestamp for logs
template t_d_default_r { template("$R_ISODATE $HOST LEVEL=$LEVEL $MSGHDR
$MSG\n"); };

from the field @type of this json log :

{
    "@source": "tcp://127.0.0.1:9999/client/127.0.0.1:57530",
    "@type": "tomcat_logstash_raw_json",
    "@tags": [
        "tomcat_site"
    ],
    "@fields": {
        "priority": "INFO",
        "logger_name": "com.zzz.user.UserData",
        "thread": "TP-Processor7",
        "class":
"org.apache.jsp.WEB_002dINF.jsp.user.ViewInvoiceDetail_jsp",
        "file": "ViewInvoiceDetail_jsp.java:162",
        "method": "_jspService",
        "prop_userIp": "192.168.215.50",
        "prop_userId": "1440704"
    },
    "@source_host": "127.0.0.1:57530",
    "@source_path": "com.leshop.user.UserData",
    "@message": "order : {WAREHOUSE_TYPE=drive, OID=5693367,
ORDER_DATE=2012-10-03 08:49:17.41, SHIPPING_FRESH=0.0,
FROZEN_DEPOSIT=0.0, WAREHOUSE_ID=5, DUE_AMOUNT=0.0, TOTAL_CREDITS=0.0,
ADDRESS_NUMBER=, DELIV_HELPFUL_INDICATION=, DELIVERY_MODE=20:00,
DELIVERY_DATE=2012-10-03 00:00:00.0, TOTAL=134.75, ACTION_TOTAL=0.0,
ORDER_NUMBER=abc-014085706-xyz, TRACK_TRACE=, RETAILER_GROUP=0, ZIP=,
ORDER_STATE=3, PAYMENT_TYPE=7, DELIV_DOORCODE=, FROZEN_FEES=0.0,
ENV_CO2=0.0, NAME= , ENV_CO2_RETAIL=0.0, HIDE_BVR=false, ADDRESS=,
TOTAL_CREDIT=0.0, MODIFICATION_STATE=1, REMINDER_LEVEL=0,
SUBTOTAL=134.75, GRAND_TOTAL=134.75, BVR_REFERENCE=, CITY=,
DELIV_PHONE=, SHIPPING_FIXED=0.0}",
    "@timestamp": "2012-10-03T06:49:23.373000Z"
}

I know I can do it with patterndb or directly with a regex like .

#match and create a group with le type value
filter f_bigip_http_vs_extract { match('"@type": "([^\"]+)",'
value("MESSAGE") type("pcre") flags("store-matches" "ignore-case")); };

#replace program field with extracted value from le log line
rewrite rw_tomcat_site_logstash_json_program_name { subst('.*', "${1}",
value("PROGRAM"));};

But I would like to use the new json parser to keep a configuration as
clean as possible.

Anyone can help me to know where I need to start ? (not found anyting
into the admin guide :/)

My version of syslog-ng :

[root at mgblcof01 192.168.217.205]# syslog-ng --version
syslog-ng 3.3.6.90
Installer-Version: 3.3.6.90
Revision:
Compile-Date: Sep 20 2012 13:34:34
Default-Modules:
affile,afprog,afsocket,afuser,basicfuncs,csvparser,dbparser,syslogformat,afsql
Available-Modules:
afsocket-tls,dbparser,afuser,syslogformat,afprog,confgen,csvparser,affile,dummy,basicfuncs,afsocket,afmongodb,tfjson,afsocket-notls,afsql,convertfuncs
Enable-Debug: off
Enable-GProf: off
Enable-Memtrace: off
Enable-IPv6: on
Enable-Spoof-Source: on
Enable-TCP-Wrapper: on
Enable-Linux-Caps: on
Enable-Pcre: on


Thank !

Seb
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQIcBAEBAgAGBQJQbStAAAoJEE3IBph3MKVPpZgP/0fyydXA3dBwBm4/A66DsJCJ
DfsGtQYVEXMgtQhit+nh47ruBvl6O+E6XPTnL2vKjObKr8J7A1gy3Zsf6CAzkBod
Wxs74wPHOvy8UIRRmzDl2rU+xv0Ak84lN7ngFjgJZpqlqajlPyfj48dKPwINV80X
3hZLpFHXD0y0T8JoL30rm35nbDMVfRBqFnLd5U4J8dMBZ6O8Y0bnYvPZ65GR3M0S
L8x3cc4suOBSg/PEF4FAL/j/QMCTAnGvntIFN4pYOU9qkXixqtwheJ2EenzI+W2M
mcs8VPWjRAfgg2w3gzy3UOGMgogHS2yR9guqHowGOeTuVfKEO1D7ZaP1MRHEfATj
AdJVNGuDilquJwuLCLUwinvp68BpcJ1cM5E2t5P7FwZJQxfX8MJrGwQ+4bF2pvhI
YzIRGHDmB888CRNYoyyrFCoXkjZ3/Nd8FxSx2BBhb3PNRgEkNWgfbkuAr422W+Ni
ZzzDrD5Rpl7E7Fzz7Xb2XAxJJS7E+tj5d/ukD3PGFFfr6kXNHwfQyxCCgQOt93+2
zuSYGg0QPpztyK4y83NXaM+L7dYsP75pG24FhnUQFvA164sOKqT/i4/4mgu6u8pF
ZX1Ji//fa2CfqEw0nhmrZxYYwXi4HYZ8uUd6MpQSfE04J5ymyOcrWa7wVBN+jlev
Ts/ZAjTBDJku7JYLBmSf
=VpL2
-----END PGP SIGNATURE-----

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20121004/0fab7585/attachment.htm