[syslog-ng] question about message parsing behavior

[Balazs Scheidler]

Hi Joe,

Just wanted to mention that I added coverage for this feature in a unit
test program, and now integrated to the master branch. It should be
available in tomorrow's snapshot.

On Sun, 2009-02-22 at 20:13 +0100, Balazs Scheidler wrote:
> On Wed, 2009-02-18 at 14:48 -0500, Joe Shaw wrote:
> > Hi,
> > 
> > What happens if a parser fails to parse a message?  Let's say I have a parser:
> > 
> > csv_parser(columns("COL1", "COL2", "COL3") delimiters(","));
> > 
> > If I pass in "foo", does COL1 get set to "foo" and COL2 and COL3 are undefined?
> 
> Yes.
> 
> > 
> > If I pass in "foo,bar" is just COL3 undefined?
> 
> Yes.
> 
> > 
> > If I pass in "foo,bar,baz,quux" is COL3 "baz,quux" ?
> 
> No. I was planning to create a separate flag for this case, so that the
> last column is greedy in case further info is available, but I have not
> gotten there yet.
> 
> > 
> > I am wondering if I can set up a filter for messages which don't meet
> > the parsing rules, so that I can use the catch-all directive for
> > logging rules and filter out messages that don't match the format (in
> > lieu of the tagging that was discussed in a different thread).
> > 
> 
> Currently parsers always match (but they could drop messages just like
> filters), as one way or the other it always tries to understand the
> current message. So currently, neither csv-parser nor db-parser reports
> ill-formatted messages. 
> 
> However db-parser can assign a "classification" tag to the message and
> you can filter based on that. It is also possible to check all fields
> after a csv-parser() if all got values, but I see that this could be
> difficult.
> 
> Hmm.. here's an experimental patch that implements two flags: greedy and
> drop-invalid:
> 
> commit d9b53cc4ae268109dc095c17dbc98bcf4df73bf9
> Author: Balazs Scheidler <bazsi at balabit.hu>
> Date:   Sun Feb 22 20:11:13 2009 +0100
> 
>     [csv-parser] implement greedy and drop-invalid flags
>     
>     greedy:: the new greedy flag makes the last column of a csv-parser() greedy,
>     e.g. it gets all the remaining characters at the end of the parsed line.
>     
>     drop-invalid:: the new drop-invalid flag makes the parser drop all messages
>     that do no have enough columns to fill the required number of fields.
> 
> 
> I haven't added this to the "master" branch, but I've pushed a 
> csv-parser-flags branch  to the public git repository, because I can't 
> test this right now:
> 
> http://git.balabit.hu/?p=bazsi/syslog-ng-3.0.git;a=shortlog;h=csv-parser-flags
> 
> 
> 
-- 
Bazsi