[syslog-ng] question about message parsing behavior

[Balazs Scheidler]

On Wed, 2009-02-18 at 14:48 -0500, Joe Shaw wrote:
> Hi,
> 
> What happens if a parser fails to parse a message?  Let's say I have a parser:
> 
> csv_parser(columns("COL1", "COL2", "COL3") delimiters(","));
> 
> If I pass in "foo", does COL1 get set to "foo" and COL2 and COL3 are undefined?

Yes.

> 
> If I pass in "foo,bar" is just COL3 undefined?

Yes.

> 
> If I pass in "foo,bar,baz,quux" is COL3 "baz,quux" ?

No. I was planning to create a separate flag for this case, so that the
last column is greedy in case further info is available, but I have not
gotten there yet.

> 
> I am wondering if I can set up a filter for messages which don't meet
> the parsing rules, so that I can use the catch-all directive for
> logging rules and filter out messages that don't match the format (in
> lieu of the tagging that was discussed in a different thread).
> 

Currently parsers always match (but they could drop messages just like
filters), as one way or the other it always tries to understand the
current message. So currently, neither csv-parser nor db-parser reports
ill-formatted messages. 

However db-parser can assign a "classification" tag to the message and
you can filter based on that. It is also possible to check all fields
after a csv-parser() if all got values, but I see that this could be
difficult.

Hmm.. here's an experimental patch that implements two flags: greedy and
drop-invalid:

commit d9b53cc4ae268109dc095c17dbc98bcf4df73bf9
Author: Balazs Scheidler <bazsi at balabit.hu>
Date:   Sun Feb 22 20:11:13 2009 +0100

    [csv-parser] implement greedy and drop-invalid flags
    
    greedy:: the new greedy flag makes the last column of a csv-parser() greedy,
    e.g. it gets all the remaining characters at the end of the parsed line.
    
    drop-invalid:: the new drop-invalid flag makes the parser drop all messages
    that do no have enough columns to fill the required number of fields.


I haven't added this to the "master" branch, but I've pushed a 
csv-parser-flags branch  to the public git repository, because I can't 
test this right now:

http://git.balabit.hu/?p=bazsi/syslog-ng-3.0.git;a=shortlog;h=csv-parser-flags



-- 
Bazsi