<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=ISO-8859-1">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<br>
-----BEGIN PGP SIGNED MESSAGE-----<br>
Hash: SHA1<br>
<br>
Hello :)<br>
<br>
I will present to you what I want to do and what I actually have.<br>
<br>
<br>
I would like to extract a field from a json log arriving in this
source :<br>
<br>
source s_collector_tcp_json {<br>
tcp(ip(0.0.0.0) port(514) flags(no-multi-line) flags(no-parse));<br>
};<br>
<br>
And replacing the Program field I use in my destination :<br>
<br>
#Destination that uses received time as timestamp for logs<br>
destination d_file_normal_r
{file("/var/log/leshop/leshop_log/$R_YEAR/$HOST/$PROGRAM/$R_YEAR-$R_MONTH-$R_DAY.log"
template(t_d_default_r) group(users) dir_group(users) perm(0640)
dir_perm(0750) flags(no-multi-line) frac_digits(6));};<br>
<br>
#Templates for destination that uses received time as timestamp for
logs<br>
template t_d_default_r { template("$R_ISODATE $HOST LEVEL=$LEVEL
$MSGHDR $MSG\n"); };<br>
<br>
from the field @type of this json log :<br>
<br>
{<br>
"@source": "tcp://127.0.0.1:9999/client/127.0.0.1:57530",<br>
"@type": "tomcat_logstash_raw_json",<br>
"@tags": [<br>
"tomcat_site"<br>
],<br>
"@fields": {<br>
"priority": "INFO",<br>
"logger_name": "com.zzz.user.UserData",<br>
"thread": "TP-Processor7",<br>
"class":
"org.apache.jsp.WEB_002dINF.jsp.user.ViewInvoiceDetail_jsp",<br>
"file": "ViewInvoiceDetail_jsp.java:162",<br>
"method": "_jspService",<br>
"prop_userIp": "192.168.215.50",<br>
"prop_userId": "1440704"<br>
},<br>
"@source_host": "127.0.0.1:57530",<br>
"@source_path": "com.leshop.user.UserData",<br>
"@message": "order : {WAREHOUSE_TYPE=drive, OID=5693367,
ORDER_DATE=2012-10-03 08:49:17.41, SHIPPING_FRESH=0.0,
FROZEN_DEPOSIT=0.0, WAREHOUSE_ID=5, DUE_AMOUNT=0.0,
TOTAL_CREDITS=0.0, ADDRESS_NUMBER=, DELIV_HELPFUL_INDICATION=,
DELIVERY_MODE=20:00, DELIVERY_DATE=2012-10-03 00:00:00.0,
TOTAL=134.75, ACTION_TOTAL=0.0, ORDER_NUMBER=abc-014085706-xyz,
TRACK_TRACE=, RETAILER_GROUP=0, ZIP=, ORDER_STATE=3, PAYMENT_TYPE=7,
DELIV_DOORCODE=, FROZEN_FEES=0.0, ENV_CO2=0.0, NAME= ,
ENV_CO2_RETAIL=0.0, HIDE_BVR=false, ADDRESS=, TOTAL_CREDIT=0.0,
MODIFICATION_STATE=1, REMINDER_LEVEL=0, SUBTOTAL=134.75,
GRAND_TOTAL=134.75, BVR_REFERENCE=, CITY=, DELIV_PHONE=,
SHIPPING_FIXED=0.0}",<br>
"@timestamp": "2012-10-03T06:49:23.373000Z"<br>
}<br>
<br>
I know I can do it with patterndb or directly with a regex like .<br>
<br>
#match and create a group with le type value<br>
filter f_bigip_http_vs_extract { match('"@type": "([^\"]+)",'
value("MESSAGE") type("pcre") flags("store-matches" "ignore-case"));
};<br>
<br>
#replace program field with extracted value from le log line<br>
rewrite rw_tomcat_site_logstash_json_program_name { subst('.*',
"${1}", value("PROGRAM"));};<br>
<br>
But I would like to use the new json parser to keep a configuration
as clean as possible.<br>
<br>
Anyone can help me to know where I need to start ? (not found
anyting into the admin guide :/)<br>
<br>
My version of syslog-ng :<br>
<br>
[root@mgblcof01 192.168.217.205]# syslog-ng --version<br>
syslog-ng 3.3.6.90<br>
Installer-Version: 3.3.6.90<br>
Revision:<br>
Compile-Date: Sep 20 2012 13:34:34<br>
Default-Modules:
affile,afprog,afsocket,afuser,basicfuncs,csvparser,dbparser,syslogformat,afsql<br>
Available-Modules:
afsocket-tls,dbparser,afuser,syslogformat,afprog,confgen,csvparser,affile,dummy,basicfuncs,afsocket,afmongodb,tfjson,afsocket-notls,afsql,convertfuncs<br>
Enable-Debug: off<br>
Enable-GProf: off<br>
Enable-Memtrace: off<br>
Enable-IPv6: on<br>
Enable-Spoof-Source: on<br>
Enable-TCP-Wrapper: on<br>
Enable-Linux-Caps: on<br>
Enable-Pcre: on<br>
<br>
<br>
Thank !<br>
<br>
Seb<br>
-----BEGIN PGP SIGNATURE-----<br>
Version: GnuPG v1.4.12 (GNU/Linux)<br>
Comment: Using GnuPG with Mozilla - <a class="moz-txt-link-freetext" href="http://www.enigmail.net/">http://www.enigmail.net/</a><br>
<br>
iQIcBAEBAgAGBQJQbStAAAoJEE3IBph3MKVPpZgP/0fyydXA3dBwBm4/A66DsJCJ<br>
DfsGtQYVEXMgtQhit+nh47ruBvl6O+E6XPTnL2vKjObKr8J7A1gy3Zsf6CAzkBod<br>
Wxs74wPHOvy8UIRRmzDl2rU+xv0Ak84lN7ngFjgJZpqlqajlPyfj48dKPwINV80X<br>
3hZLpFHXD0y0T8JoL30rm35nbDMVfRBqFnLd5U4J8dMBZ6O8Y0bnYvPZ65GR3M0S<br>
L8x3cc4suOBSg/PEF4FAL/j/QMCTAnGvntIFN4pYOU9qkXixqtwheJ2EenzI+W2M<br>
mcs8VPWjRAfgg2w3gzy3UOGMgogHS2yR9guqHowGOeTuVfKEO1D7ZaP1MRHEfATj<br>
AdJVNGuDilquJwuLCLUwinvp68BpcJ1cM5E2t5P7FwZJQxfX8MJrGwQ+4bF2pvhI<br>
YzIRGHDmB888CRNYoyyrFCoXkjZ3/Nd8FxSx2BBhb3PNRgEkNWgfbkuAr422W+Ni<br>
ZzzDrD5Rpl7E7Fzz7Xb2XAxJJS7E+tj5d/ukD3PGFFfr6kXNHwfQyxCCgQOt93+2<br>
zuSYGg0QPpztyK4y83NXaM+L7dYsP75pG24FhnUQFvA164sOKqT/i4/4mgu6u8pF<br>
ZX1Ji//fa2CfqEw0nhmrZxYYwXi4HYZ8uUd6MpQSfE04J5ymyOcrWa7wVBN+jlev<br>
Ts/ZAjTBDJku7JYLBmSf<br>
=VpL2<br>
-----END PGP SIGNATURE-----<br>
<br>
</body>
</html>