Based on the following structured syslog, I am trying to extract the reason from the SDATA portion of the log. 2012-02-05T16:24:45.368 RT_FLOW - RT_FLOW_SESSION_CLOSE [junos@2636.1.1.1.2.36 reason=\”unset\” ] Problem I think I am running into is the value-pair that I am trying to extract includes the dots so the parser thinks its nested information destination d_mongodb { mongodb( value-pairs( scope("everything") key(".SDATA.junos@2636.1.1.1.2.36.reason") ) ); }; Thoughts? Any help is greatly appreciated. Chris
Chris Johnson <masterof0@gmail.com> writes:
Based on the following structured syslog, I am trying to extract the reason from the SDATA portion of the log.
2012-02-05T16:24:45.368 RT_FLOW - RT_FLOW_SESSION_CLOSE [junos@2636.1.1.1.2.36 reason=\”unset\” ]
Problem I think I am running into is the value-pair that I am trying to extract includes the dots so the parser thinks its nested information
destination d_mongodb { mongodb( value-pairs( scope("everything") key(".SDATA.junos@2636.1.1.1.2.36.reason") ) ); };
Thoughts? Any help is greatly appreciated.
I just tried this with syslog-ng 3.4, without the key() (because scope("everything") already includes the key), and the reason ended up like this:
db.messages.find({}, {_SDATA: 1, _id: 0}) { "_SDATA" : { "junos@2636" : { "1" : { "1" : { "1" : { "2" : { "36" : { "reason" : "unset" } } } } } } } }
This does look awkward, indeed. If you just want the reason, with a better key name, then the best you can do at the moment is to use something like the following: value-pairs ( pair("SDATA.junos@2636-1-1-1-2-36.reason", "${.SDATA.junos@2636.1.1.1.2.36.reason}") scope("base") ); The same should work in 3.3, too. If it doesn't, then I screwed something up terribly. Nevertheless, an easy way to replace recurring patterns in a value-pair key name would be neat, and would fit into the key rewrite feature already in 3.4... I'll think about this a little more. -- |8]
Thanks for the reply. I am following your recommendation to create my own pair from the SDATA. With that in mind, here is my current configuration #####Sources##### source s_network { udp(tags("firewall")); }; #####filter##### filter f_session_close {message("RT_FLOW_SESSION_CLOSE"); }; #####Destinations##### destination d_mongodb { mongodb( value-pairs( pair("reason","${.SDATA.junos@2636.1.1.1.2.36.reason}") scope("base") ) ); }; #####Log##### log { source(s_network); filter(f_session_close); destination(d_mongodb); }; But what I am seeing in mongo is the following which does not have the pair defined in the destination statement: { "DATE" : "Feb 10 09:48:25", "FACILITY" : "user", "HOST" : "X.X.X.X", "MESSAGE" : "2012-02-10T09:48:41.668 RT_FLOW - RT_FLOW_SESSION_CLOSE [junos@2636.1.1.1.2.36 reason=\"unset\" source-address=\"X.X.X.x\" source-port=\"123\" destination-address=\"X.X.X.X\" destination-port=\"123\" service-name=\"junos-ntp\" nat-source-address=\"X.X.X.X\" nat-source-port=\"13773\" nat-destination-address=\"X.X.X.X\" nat-destination-port=\"123\" src-nat-rule-name=\"source-nat-rule\" dst-nat-rule-name=\"None\" protocol-id=\"17\" policy-name=\"trust-to-untrust\" source-zone-name=\"trust\" destination-zone-name=\"untrust\" session-id-32=\"30391\" packets-from-client=\"1\" bytes-from-client=\"76\" packets-from-server=\"1\" bytes-from-server=\"76\" elapsed-time=\"60\" application=\"N/A\" nested-application=\"N/A\" username=\"N/A\" role=\"N/A\" packet-incoming-interface=\"vlan.0\"]", "PRIORITY" : "info", "PROGRAM" : "1", "_id" : ObjectId("4f355869aa8f27451a000001") } If there is any additional information I can provide, please let me know, I will be happy to do so. Thanks again for all of your help Chris On Feb 10, 2012, at 5:11 AM, Gergely Nagy wrote:
Chris Johnson <masterof0@gmail.com> writes:
Based on the following structured syslog, I am trying to extract the reason from the SDATA portion of the log.
2012-02-05T16:24:45.368 RT_FLOW - RT_FLOW_SESSION_CLOSE [junos@2636.1.1.1.2.36 reason=\”unset\” ]
Problem I think I am running into is the value-pair that I am trying to extract includes the dots so the parser thinks its nested information
destination d_mongodb { mongodb( value-pairs( scope("everything") key(".SDATA.junos@2636.1.1.1.2.36.reason") ) ); };
Thoughts? Any help is greatly appreciated.
I just tried this with syslog-ng 3.4, without the key() (because scope("everything") already includes the key), and the reason ended up like this:
db.messages.find({}, {_SDATA: 1, _id: 0}) { "_SDATA" : { "junos@2636" : { "1" : { "1" : { "1" : { "2" : { "36" : { "reason" : "unset" } } } } } } } }
This does look awkward, indeed. If you just want the reason, with a better key name, then the best you can do at the moment is to use something like the following:
value-pairs ( pair("SDATA.junos@2636-1-1-1-2-36.reason", "${.SDATA.junos@2636.1.1.1.2.36.reason}") scope("base") );
The same should work in 3.3, too. If it doesn't, then I screwed something up terribly.
Nevertheless, an easy way to replace recurring patterns in a value-pair key name would be neat, and would fit into the key rewrite feature already in 3.4... I'll think about this a little more.
-- |8]
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Chris Johnson <masterof0@gmail.com> writes:
Thanks for the reply. I am following your recommendation to create my own pair from the SDATA. With that in mind, here is my current configuration
Thanks, I'll try this with 3.3, and see what happens. I'll try to get back to you as soon as possible, but might take a day or two 'till I find time to investigate. -- |8]
Hi, I've tried this with a simple json based output and it does seem to work for me in 3.3. Here is my configuration (that mimics the SDATA with a set() rewrite rule): @version: 3.3 @module tfjson destination d_json { file("foo" template("$(format-json foo=bar reason='${.SDATA.junos@2636.1.1.1.2.36.reason}' foobar=barfoo)\n")); }; source s_all { unix-stream("log"); }; rewrite junos { set("junos-reason" value('.SDATA.junos@2636.1.1.1.2.36.reason')); }; log { source(s_all); rewrite(junos); destination(d_json); }; On Fri, 2012-02-10 at 10:04 -0800, Chris Johnson wrote:
Thanks for the reply. I am following your recommendation to create my own pair from the SDATA. With that in mind, here is my current configuration
#####Sources##### source s_network { udp(tags("firewall")); };
#####filter##### filter f_session_close {message("RT_FLOW_SESSION_CLOSE"); };
#####Destinations##### destination d_mongodb { mongodb( value-pairs( pair("reason","${.SDATA.junos@2636.1.1.1.2.36.reason}") scope("base") ) ); };
#####Log##### log { source(s_network); filter(f_session_close); destination(d_mongodb); };
But what I am seeing in mongo is the following which does not have the pair defined in the destination statement: { "DATE" : "Feb 10 09:48:25", "FACILITY" : "user", "HOST" : "X.X.X.X", "MESSAGE" : "2012-02-10T09:48:41.668 RT_FLOW - RT_FLOW_SESSION_CLOSE [junos@2636.1.1.1.2.36 reason=\"unset\" source-address=\"X.X.X.x\" source-port=\"123\" destination-address=\"X.X.X.X\" destination-port=\"123\" service-name=\"junos-ntp\" nat-source-address=\"X.X.X.X\" nat-source-port=\"13773\" nat-destination-address=\"X.X.X.X\" nat-destination-port=\"123\" src-nat-rule-name=\"source-nat-rule\" dst-nat-rule-name=\"None\" protocol-id=\"17\" policy-name=\"trust-to-untrust\" source-zone-name=\"trust\" destination-zone-name=\"untrust\" session-id-32=\"30391\" packets-from-client=\"1\" bytes-from-client=\"76\" packets-from-server=\"1\" bytes-from-server=\"76\" elapsed-time=\"60\" application=\"N/A\" nested-application=\"N/A\" username=\"N/A\" role=\"N/A\" packet-incoming-interface=\"vlan.0\"]", "PRIORITY" : "info", "PROGRAM" : "1", "_id" : ObjectId("4f355869aa8f27451a000001") }
If there is any additional information I can provide, please let me know, I will be happy to do so. Thanks again for all of your help Chris
On Feb 10, 2012, at 5:11 AM, Gergely Nagy wrote:
Chris Johnson <masterof0@gmail.com> writes:
Based on the following structured syslog, I am trying to extract the reason from the SDATA portion of the log.
2012-02-05T16:24:45.368 RT_FLOW - RT_FLOW_SESSION_CLOSE [junos@2636.1.1.1.2.36 reason=\”unset\” ]
Problem I think I am running into is the value-pair that I am trying to extract includes the dots so the parser thinks its nested information
destination d_mongodb { mongodb( value-pairs( scope("everything") key(".SDATA.junos@2636.1.1.1.2.36.reason") ) ); };
Thoughts? Any help is greatly appreciated.
I just tried this with syslog-ng 3.4, without the key() (because scope("everything") already includes the key), and the reason ended up like this:
db.messages.find({}, {_SDATA: 1, _id: 0}) { "_SDATA" : { "junos@2636" : { "1" : { "1" : { "1" : { "2" : { "36" : { "reason" : "unset" } } } } } } } }
This does look awkward, indeed. If you just want the reason, with a better key name, then the best you can do at the moment is to use something like the following:
value-pairs ( pair("SDATA.junos@2636-1-1-1-2-36.reason", "${.SDATA.junos@2636.1.1.1.2.36.reason}") scope("base") );
The same should work in 3.3, too. If it doesn't, then I screwed something up terribly.
Nevertheless, an easy way to replace recurring patterns in a value-pair key name would be neat, and would fit into the key rewrite feature already in 3.4... I'll think about this a little more.
-- |8]
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Bazsi
participants (3)
-
Balazs Scheidler
-
Chris Johnson
-
Gergely Nagy