CSV parser ignoring template()?
I'm upgrading from 3.1.3 to 3.3.3. My configuration has a CSV parser: parser p_find_role { csv-parser( columns("ROLE", "REMAINDER") delimiters("0123456789.") template("${HOST}") ); }; With 3.1.3, this takes the first word of the machine's hostname and stores it in $ROLE, but 3.3.3 seems to ignore template() and operates on the log message instead, storing the first field of the message in $ROLE. $HOST expands correctly in a log template, so that token is still available in 3.3.x. Am I missing something else related to the upgrade? john -- John Morrissey _o /\ ---- __o jwm@horde.net _-< \_ / \ ---- < \, www.horde.net/ __(_)/_(_)________/ \_______(_) /_(_)__
John Morrissey <jwm@horde.net> writes:
I'm upgrading from 3.1.3 to 3.3.3. My configuration has a CSV parser:
parser p_find_role { csv-parser( columns("ROLE", "REMAINDER") delimiters("0123456789.") template("${HOST}") ); };
With 3.1.3, this takes the first word of the machine's hostname and stores it in $ROLE, but 3.3.3 seems to ignore template() and operates on the log message instead, storing the first field of the message in $ROLE.
$HOST expands correctly in a log template, so that token is still available in 3.3.x. Am I missing something else related to the upgrade?
Are you using the same csv-parser in multiple log paths? If so, that's a known bug in 3.3.x, and has since been fixed in git. I would suggest giving a try to the latest git snapshot (also available as a tarball[1], ready to ./configure && make && make install), and see if that works. [1]: http://packages.madhouse-project.org/syslog-ng/3.3/syslog-ng-3.3-HEAD.tar.gz -- |8]
On Mon, Jan 09, 2012 at 10:29:24AM +0100, Gergely Nagy wrote:
John Morrissey <jwm@horde.net> writes:
I'm upgrading from 3.1.3 to 3.3.3. My configuration has a CSV parser:
parser p_find_role { csv-parser( columns("ROLE", "REMAINDER") delimiters("0123456789.") template("${HOST}") ); };
With 3.1.3, this takes the first word of the machine's hostname and stores it in $ROLE, but 3.3.3 seems to ignore template() and operates on the log message instead, storing the first field of the message in $ROLE.
$HOST expands correctly in a log template, so that token is still available in 3.3.x. Am I missing something else related to the upgrade?
Are you using the same csv-parser in multiple log paths? If so, that's a known bug in 3.3.x, and has since been fixed in git.
nod, I forgot to mention I had already cherry-picked 79fc32703615e045fca4c9aaf85b7ab2915e7ed4 and applied it to our local syslog-ng build. That bug caused the token to be empty, but the current problem (after applying 79fc32) is that the token is being filled, just based on the wrong data. john -- John Morrissey _o /\ ---- __o jwm@horde.net _-< \_ / \ ---- < \, www.horde.net/ __(_)/_(_)________/ \_______(_) /_(_)__
On Mon, Jan 09, 2012 at 08:57:31AM -0500, John Morrissey wrote:
On Mon, Jan 09, 2012 at 10:29:24AM +0100, Gergely Nagy wrote:
John Morrissey <jwm@horde.net> writes:
I'm upgrading from 3.1.3 to 3.3.3. My configuration has a CSV parser:
parser p_find_role { csv-parser( columns("ROLE", "REMAINDER") delimiters("0123456789.") template("${HOST}") ); };
With 3.1.3, this takes the first word of the machine's hostname and stores it in $ROLE, but 3.3.3 seems to ignore template() and operates on the log message instead, storing the first field of the message in $ROLE.
$HOST expands correctly in a log template, so that token is still available in 3.3.x. Am I missing something else related to the upgrade?
Are you using the same csv-parser in multiple log paths? If so, that's a known bug in 3.3.x, and has since been fixed in git.
nod, I forgot to mention I had already cherry-picked 79fc32703615e045fca4c9aaf85b7ab2915e7ed4 and applied it to our local syslog-ng build.
That bug caused the token to be empty, but the current problem (after applying 79fc32) is that the token is being filled, just based on the wrong data.
Just tried the latest git HEAD too. It doesn't make any difference. The CSV parser is still operating on $MSG instead of (as configured) $HOST. john -- John Morrissey _o /\ ---- __o jwm@horde.net _-< \_ / \ ---- < \, www.horde.net/ __(_)/_(_)________/ \_______(_) /_(_)__
On Sun, 2012-01-08 at 11:11 -0500, John Morrissey wrote:
I'm upgrading from 3.1.3 to 3.3.3. My configuration has a CSV parser:
parser p_find_role { csv-parser( columns("ROLE", "REMAINDER") delimiters("0123456789.") template("${HOST}") ); };
With 3.1.3, this takes the first word of the machine's hostname and stores it in $ROLE, but 3.3.3 seems to ignore template() and operates on the log message instead, storing the first field of the message in $ROLE.
Here is a quickfix: diff --git a/modules/csvparser/csvparser.c b/modules/csvparser/csvparser.c index 1be19fd..57f2ad0 100644 --- a/modules/csvparser/csvparser.c +++ b/modules/csvparser/csvparser.c @@ -350,6 +350,8 @@ log_csv_parser_clone(LogProcessPipe *s) cloned->quotes_end = g_strdup(self->quotes_end); cloned->null_value = self->null_value ? g_strdup(self->null_value) : NULL; cloned->flags = self->flags; + + cloned->super.super.template = log_template_ref(self->super.super.template); for (l = self->super.columns; l; l = l->next) { cloned->super.columns = g_list_append(cloned->super.columns, g_strdup(l->data)); This should be properly implemented a different way, by delegating the clone operation to the superclass, however I got to go now, just wanted to let you know about the fix. Thanks for reporting this. -- Bazsi
participants (3)
-
Balazs Scheidler
-
Gergely Nagy
-
John Morrissey