[Bug 87] New: csv-parser: specifying drop-invalid and greedy flags always fails.
https://bugzilla.balabit.com/show_bug.cgi?id=87 Summary: csv-parser: specifying drop-invalid and greedy flags always fails. Product: syslog-ng Version: 3.0.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi@balabit.hu ReportedBy: alex-pub.balabit@reflexion.net Type of the Report: bug Estimated Hours: 0.0 In this example, I am trying to parse the postfix log statement. Postfix specifies $PROGRAM as a pair separated by slash, e.g "postfix/master" or "postfix/postfix-script". The "postfix" part is configurable, and I would like to parse it out from the $PROGRAM macro. I tried it first with this parser parser p_rfx_postfix_program_parser { csv-parser( columns( "POSTFIX.SERVICE_NAME", "POSTFIX.PROGRAM" ) delimiters( "/" ) flags( drop-invalid, escape-none, greedy ) template( "${PROGRAM}" ) ); }; The intention here is that ${PROGRAM} should always contain at least 1 forward slash, and I only want to parse the part before the first slash. However this parser always fails. Removing greedy flag makes this parser succeed, however it does change the meaning of the action that I am trying to perform. Looking at the logparser.c: log_csv_parser_process, I've noticed that when LOG_CSV_PARSER_GREEDY flag is on, then `cur_column` variable is reset to NULL. Then right before the function ends it checks if `cur_column` is null and LOG_CSV_PARSER_DROP_INVALID flag is on, then fails the function. I think the check at the end of the function should take into account whether GREEDY operation has been performed or not. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=87 Balazs Scheidler <bazsi@balabit.hu> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution| |FIXED Status|NEW |RESOLVED --- Comment #1 from Balazs Scheidler <bazsi@balabit.hu> 2010-06-29 17:42:27 --- this patch in 3.0 should fix the issue: commit b6ff55b0955cfe285ac7afe2063f832daed078a9 Author: Balazs Scheidler <bazsi@balabit.hu> Date: Tue Jun 29 17:41:36 2010 +0200 csv-parser: fixed greedy + drop-invalid combination greedy & drop-invalid never really worked well in combination, if they both were specified, the message always got rejected. This patch fixes this issue. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=87 jsu2@emory.edu changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jsu2@emory.edu --- Comment #2 from jsu2@emory.edu 2010-07-28 01:59:25 --- I have a similar problem with 3.0.3. Sample syslog message: Jul 27 19:55:33 myhost ZabbixConnector.log : 19:55:32,782 INFO [Thread-2834] - [ZabbixEventSyncCommand] Processing message <?xml version="1.0" encoding="UTF-8"?> The following template and parser combination: template t_esb { template("${ESB.LOG}, ${ESB.MSG}\n"); }; parser p_app_esb { csv-parser(columns("ESB.LOG","ESB.MSG") delimiters(" ") flags(greedy)); }; Yields (yes, just the comma): , If I remove the greedy flag: ZabbixConnector.log, : So the greedy flag breaks the parser and does not pick up the rest of the message as described. Is this fixed in 3.0.6? -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=87 jsu2@emory.edu changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|FIXED | Status|RESOLVED |REOPENED --- Comment #3 from jsu2@emory.edu 2010-07-28 22:43:51 --- So I upgraded to 3.0.8 which in the release notes says "* Fixed the use of the greedy and drop-invalid flags for csv-parser." But, I'm still getting the same results as I did with 3.0.3. The greedy flag breaks the parser completely. Without the greedy flag, the first column gets parsed. The second column gets parsed up to the next delimiter. This would be the expected behavior. With the greedy flag, I would have expected the second column to capture from the first delimiter to the end of the line. However, with the greedy flag on, even the first column is not properly parsed. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=87 jsu2@emory.edu changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution| |FIXED Status|REOPENED |RESOLVED --- Comment #4 from jsu2@emory.edu 2010-08-04 05:26:49 --- Using the "escape-none" in conjunction with "greedy" seems to have fixed my problem. I'm now getting the behavior I was expecting. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=87 --- Comment #5 from Balazs Scheidler <bazsi@balabit.hu> 2010-08-14 13:12:21 --- the requirement to specify an escaping mode and use no default wasn't created on purpose, so this was a genuine bug, which was fixed with this patch (in 3.0, 3.1 and 3.2 too) commit f8a8ae5ba67a4add7ed8905fba7c2118d4349b5d Author: Balazs Scheidler <bazsi@balabit.hu> Date: Fri Aug 6 13:49:59 2010 +0200 csvparser: remove the need to explicitly specify an escaping mode csvparser will not default to escape-none if it is not specified explicitly. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
participants (1)
-
bugzilla＠bugzilla.balabit.com