Hi Joe, Just wanted to mention that I added coverage for this feature in a unit test program, and now integrated to the master branch. It should be available in tomorrow's snapshot. On Sun, 2009-02-22 at 20:13 +0100, Balazs Scheidler wrote:
On Wed, 2009-02-18 at 14:48 -0500, Joe Shaw wrote:
Hi,
What happens if a parser fails to parse a message? Let's say I have a parser:
csv_parser(columns("COL1", "COL2", "COL3") delimiters(","));
If I pass in "foo", does COL1 get set to "foo" and COL2 and COL3 are undefined?
Yes.
If I pass in "foo,bar" is just COL3 undefined?
Yes.
If I pass in "foo,bar,baz,quux" is COL3 "baz,quux" ?
No. I was planning to create a separate flag for this case, so that the last column is greedy in case further info is available, but I have not gotten there yet.
I am wondering if I can set up a filter for messages which don't meet the parsing rules, so that I can use the catch-all directive for logging rules and filter out messages that don't match the format (in lieu of the tagging that was discussed in a different thread).
Currently parsers always match (but they could drop messages just like filters), as one way or the other it always tries to understand the current message. So currently, neither csv-parser nor db-parser reports ill-formatted messages.
However db-parser can assign a "classification" tag to the message and you can filter based on that. It is also possible to check all fields after a csv-parser() if all got values, but I see that this could be difficult.
Hmm.. here's an experimental patch that implements two flags: greedy and drop-invalid:
commit d9b53cc4ae268109dc095c17dbc98bcf4df73bf9 Author: Balazs Scheidler <bazsi@balabit.hu> Date: Sun Feb 22 20:11:13 2009 +0100
[csv-parser] implement greedy and drop-invalid flags
greedy:: the new greedy flag makes the last column of a csv-parser() greedy, e.g. it gets all the remaining characters at the end of the parsed line.
drop-invalid:: the new drop-invalid flag makes the parser drop all messages that do no have enough columns to fill the required number of fields.
I haven't added this to the "master" branch, but I've pushed a csv-parser-flags branch to the public git repository, because I can't test this right now:
http://git.balabit.hu/?p=bazsi/syslog-ng-3.0.git;a=shortlog;h=csv-parser-fla...
-- Bazsi