[syslog-ng] advice/assistance with parsing attempt requested

[syslog-ng2010 at hushmail.com]

i've spent the better part of the past week reading and trying to 
understand both the documentation and list posts trying to sort 
this out, if anyone can offer some advice as to whether this is 
possible or not and if so, what i'm doing wrong; i would really 
appreciate it! …

i have a simple enough task, or so i thought! i've got a syslog 
stream being received by syslog-ng with too much data. what i'd 
like to do is parse out pieces of the stream and write only those 
to a file. the tricky part is that the order of the stream is very 
variable so that sometimes the desired named parser preceding 
strings and associated values are present and sometimes not. 
furthermore, the extra data is also quite variable. can this 
challenge even be addressed with syslog-ng ose? if so, can it be 
done with patterned without creating a pattern for EVERY variation 
of possible streams? for clarification, we've tried to leverage an 
external perl script which does this using regexs but, it seems 
that it can't keep up with the stream, we only receive 10% of the 
original events in the output. if this (external parsing script) 
only way this can be done, we will continue our efforts to enhance 
the external script but, if this is possible to be done natively 
within syslog-ng, i'd rather do that.

with the following configuration, i am able to generate output log 
entries which correctly contain the global macros of $DATE 
$FULLHOST $PROGRAM as well as the strings preceding the named 
parser variables but, not the named parser macros. my output looks 
like this:

Dec 2 11:11:11 127.0.0.1 ABC: 0 namedparser1= namedparser2= 
namedparser3= namedparser4= namedparser5=


*****examples of entries in source stream*****

Dec 2 11:11:11 127.0.0.1 ABC: 0 namedparser1=namedparser1value 
extra1=extravalue1 namedparser2=namedparser2value 
namedparser3=namedparser3value extra2=extravalue2 
namedparser4=namedparser4value namedparser5=namedparser5value 
extra3=extravalue3

Dec 2 11:11:11 127.0.0.1 ABC: 0 extra1=extravalue1 
namedparser3=namedparser3value extra2=extravalue2 
namedparser4=namedparser4value namedparser5=namedparser5value 
extra3=extravalue3 extra4=extravalue4

Dec 2 11:11:11 127.0.0.1 ABC: 0 namedparser1=namedparser1value 
extra1=extravalue1 namedparser2=namedparser2value 
namedparser3=namedparser3value extra2=extravalue2 
namedparser4=namedparser4value extra3=extravalue3

*****examples of desired output*****

Dec 2 11:11:11 127.0.0.1 ABC: 0 namedparser1=namedparser1value 
namedparser2=namedparser2value namedparser3=namedparser3value 
namedparser4=namedparser4value namedparser5=namedparser5value

Dec 2 11:11:11 127.0.0.1 ABC: 0  namedparser3=namedparser3value 
namedparser4=namedparser4value namedparser5=namedparser5value

Dec 2 11:11:11 127.0.0.1 ABC: 0 namedparser1=namedparser1value 
namedparser2=namedparser2value namedparser3=namedparser3value 
namedparser4=namedparser4value


*****included in conf file*****

parser pattern_db {
	db_parser(file("/opt/syslog-ng/config/patterndb.xml")
};

template reduced {
	template("$DATE $FULLHOST $PROGRAM: 0 namedparser1=$NAMEDPARSER1 
namedparser2=$NAMEDPARSER2 namedparser3=$NAMEDPARSER3 
namedparser4=$NAMEDPARSER4 namedparser5=$NAMEDPARSER5 \n");
	template_escape(no);
}

destination d_logfile {
	file("/opt/syslog-ng/logs/logfile" template(reduced));
}

log {
	source(source);
	parser(pattern_db);
	destination(d_logfile);
};


*****patterndb.xml contents*****

<patterndb version='3' pub_date=''>
	<ruleset name='globe' id='1234567890'>
		<pattern>ABC</pattern>
			<rules>
				<rule provider='someone' id='123' class='system'>
					<patterns>
						<pattern>ABC namedparser1=@ESTRING:NAMEDPARSER1:\  @ 
namedparser2=@ESTRING:NAMEDPARSER2:\ @ 
namedparser3=@ESTRING:NAMEDPARSER3:\ @ 
namedparser4=@ESTRING:NAMEDPARSER4:\ @ 
namedparser5=@ESTRING:NAMEDPARSER5:\ @</pattern>
					<patterns>
				</rule>
			<rules>
	</ruleset>
</patterndb>


MANY thanks in advance!