[syslog-ng] help with SDATA

[Balazs Scheidler]

Hi,

I've tried this with a simple json based output and it does seem to work
for me in 3.3. Here is my configuration (that mimics the SDATA with a
set() rewrite rule):

@version: 3.3
@module tfjson

destination d_json {
	file("foo" template("$(format-json foo=bar reason='${.SDATA.junos at 2636.1.1.1.2.36.reason}' foobar=barfoo)\n"));
};

source s_all {
	unix-stream("log");
};

rewrite junos {
	set("junos-reason" value('.SDATA.junos at 2636.1.1.1.2.36.reason'));
};

log { source(s_all); rewrite(junos); destination(d_json); };


On Fri, 2012-02-10 at 10:04 -0800, Chris Johnson wrote:
> Thanks for the reply. I am following your recommendation to create my own pair from the SDATA. With that in mind, here is my current configuration
> 
> #####Sources#####
> source s_network {
>         udp(tags("firewall"));
> };
> 
> #####filter#####
> filter f_session_close {message("RT_FLOW_SESSION_CLOSE"); };
> 
> #####Destinations#####
> destination d_mongodb {
>         mongodb(
>                 value-pairs(
>                         pair("reason","${.SDATA.junos at 2636.1.1.1.2.36.reason}")
>                         scope("base")
>                 )
>         );
> };
> 
> #####Log#####
> log {
>         source(s_network);
>         filter(f_session_close);
>         destination(d_mongodb);
> };
> 
> 
> But what I am seeing in mongo is the following which does not have the pair defined in the destination statement:
> {
> 	"DATE" : "Feb 10 09:48:25",
> 	"FACILITY" : "user",
> 	"HOST" : "X.X.X.X",
> 	"MESSAGE" : "2012-02-10T09:48:41.668  RT_FLOW - RT_FLOW_SESSION_CLOSE [junos at 2636.1.1.1.2.36 reason=\"unset\" source-address=\"X.X.X.x\" source-port=\"123\" destination-address=\"X.X.X.X\" destination-port=\"123\" service-name=\"junos-ntp\" nat-source-address=\"X.X.X.X\" nat-source-port=\"13773\" nat-destination-address=\"X.X.X.X\" nat-destination-port=\"123\" src-nat-rule-name=\"source-nat-rule\" dst-nat-rule-name=\"None\" protocol-id=\"17\" policy-name=\"trust-to-untrust\" source-zone-name=\"trust\" destination-zone-name=\"untrust\" session-id-32=\"30391\" packets-from-client=\"1\" bytes-from-client=\"76\" packets-from-server=\"1\" bytes-from-server=\"76\" elapsed-time=\"60\" application=\"N/A\" nested-application=\"N/A\" username=\"N/A\" role=\"N/A\" packet-incoming-interface=\"vlan.0\"]",
> 	"PRIORITY" : "info",
> 	"PROGRAM" : "1",
> 	"_id" : ObjectId("4f355869aa8f27451a000001")
> }
> 
> If there is any additional information I can provide, please let me know, I will be happy to do so.
> Thanks again for all of your help
> Chris
> 
> 
> On Feb 10, 2012, at 5:11 AM, Gergely Nagy wrote:
> 
> > Chris Johnson <masterof0 at gmail.com> writes:
> > 
> >> Based on the following structured syslog, I am trying to extract the reason from the SDATA portion of the log.
> >> 
> >> 2012-02-05T16:24:45.368  RT_FLOW - RT_FLOW_SESSION_CLOSE [junos at 2636.1.1.1.2.36 reason=\”unset\” ]
> >> 
> >> Problem I think I am running into is the value-pair that I am trying to extract includes the dots so the parser thinks its nested information
> >> 
> >> destination d_mongodb {
> >>        mongodb(
> >>                value-pairs(
> >> 			scope("everything")
> >>                        key(".SDATA.junos at 2636.1.1.1.2.36.reason")
> >>                )
> >>        );
> >> };
> >> 
> >> 
> >> Thoughts? Any help is greatly appreciated.
> > 
> > I just tried this with syslog-ng 3.4, without the key() (because
> > scope("everything") already includes the key), and the reason ended up
> > like this:
> > 
> >> db.messages.find({}, {_SDATA: 1, _id: 0})
> > { "_SDATA" : { "junos at 2636" : { "1" : { "1" : { "1" : { "2" : { "36" : { "reason" : "unset" } } } } } } } }
> > 
> > This does look awkward, indeed. If you just want the reason, with a
> > better key name, then the best you can do at the moment is to use
> > something like the following:
> > 
> >  value-pairs (
> >    pair("SDATA.junos at 2636-1-1-1-2-36.reason",
> >         "${.SDATA.junos at 2636.1.1.1.2.36.reason}")
> >    scope("base")
> >  );
> > 
> > The same should work in 3.3, too. If it doesn't, then I screwed
> > something up terribly.
> > 
> > Nevertheless, an easy way to replace recurring patterns in a value-pair
> > key name would be neat, and would fit into the key rewrite feature
> > already in 3.4... I'll think about this a little more.
> > 
> > -- 
> > |8]
> > 
> > ______________________________________________________________________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> > 
> 
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> 
> 

-- 
Bazsi