Hi, I've tried this with a simple json based output and it does seem to work for me in 3.3. Here is my configuration (that mimics the SDATA with a set() rewrite rule): @version: 3.3 @module tfjson destination d_json { file("foo" template("$(format-json foo=bar reason='${.SDATA.junos@2636.1.1.1.2.36.reason}' foobar=barfoo)\n")); }; source s_all { unix-stream("log"); }; rewrite junos { set("junos-reason" value('.SDATA.junos@2636.1.1.1.2.36.reason')); }; log { source(s_all); rewrite(junos); destination(d_json); }; On Fri, 2012-02-10 at 10:04 -0800, Chris Johnson wrote:
Thanks for the reply. I am following your recommendation to create my own pair from the SDATA. With that in mind, here is my current configuration
#####Sources##### source s_network { udp(tags("firewall")); };
#####filter##### filter f_session_close {message("RT_FLOW_SESSION_CLOSE"); };
#####Destinations##### destination d_mongodb { mongodb( value-pairs( pair("reason","${.SDATA.junos@2636.1.1.1.2.36.reason}") scope("base") ) ); };
#####Log##### log { source(s_network); filter(f_session_close); destination(d_mongodb); };
But what I am seeing in mongo is the following which does not have the pair defined in the destination statement: { "DATE" : "Feb 10 09:48:25", "FACILITY" : "user", "HOST" : "X.X.X.X", "MESSAGE" : "2012-02-10T09:48:41.668 RT_FLOW - RT_FLOW_SESSION_CLOSE [junos@2636.1.1.1.2.36 reason=\"unset\" source-address=\"X.X.X.x\" source-port=\"123\" destination-address=\"X.X.X.X\" destination-port=\"123\" service-name=\"junos-ntp\" nat-source-address=\"X.X.X.X\" nat-source-port=\"13773\" nat-destination-address=\"X.X.X.X\" nat-destination-port=\"123\" src-nat-rule-name=\"source-nat-rule\" dst-nat-rule-name=\"None\" protocol-id=\"17\" policy-name=\"trust-to-untrust\" source-zone-name=\"trust\" destination-zone-name=\"untrust\" session-id-32=\"30391\" packets-from-client=\"1\" bytes-from-client=\"76\" packets-from-server=\"1\" bytes-from-server=\"76\" elapsed-time=\"60\" application=\"N/A\" nested-application=\"N/A\" username=\"N/A\" role=\"N/A\" packet-incoming-interface=\"vlan.0\"]", "PRIORITY" : "info", "PROGRAM" : "1", "_id" : ObjectId("4f355869aa8f27451a000001") }
If there is any additional information I can provide, please let me know, I will be happy to do so. Thanks again for all of your help Chris
On Feb 10, 2012, at 5:11 AM, Gergely Nagy wrote:
Chris Johnson <masterof0@gmail.com> writes:
Based on the following structured syslog, I am trying to extract the reason from the SDATA portion of the log.
2012-02-05T16:24:45.368 RT_FLOW - RT_FLOW_SESSION_CLOSE [junos@2636.1.1.1.2.36 reason=\”unset\” ]
Problem I think I am running into is the value-pair that I am trying to extract includes the dots so the parser thinks its nested information
destination d_mongodb { mongodb( value-pairs( scope("everything") key(".SDATA.junos@2636.1.1.1.2.36.reason") ) ); };
Thoughts? Any help is greatly appreciated.
I just tried this with syslog-ng 3.4, without the key() (because scope("everything") already includes the key), and the reason ended up like this:
db.messages.find({}, {_SDATA: 1, _id: 0}) { "_SDATA" : { "junos@2636" : { "1" : { "1" : { "1" : { "2" : { "36" : { "reason" : "unset" } } } } } } } }
This does look awkward, indeed. If you just want the reason, with a better key name, then the best you can do at the moment is to use something like the following:
value-pairs ( pair("SDATA.junos@2636-1-1-1-2-36.reason", "${.SDATA.junos@2636.1.1.1.2.36.reason}") scope("base") );
The same should work in 3.3, too. If it doesn't, then I screwed something up terribly.
Nevertheless, an easy way to replace recurring patterns in a value-pair key name would be neat, and would fit into the key rewrite feature already in 3.4... I'll think about this a little more.
-- |8]
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Bazsi