On Tue, 2011-02-01 at 08:29 -0800, Evan Rempel wrote:
It is really nothing more than a power-select or power-exclude but it does not honour the order requirement of the select/exclude options.
Yep, and that's by design. There's a priority among the selectors: explicit selects ("$HOST", "$MESSAGE" and key-value pairs) are the highest, followed by builtins() and select()/exclude() on the lowest priority.
Thus, if one turns builtins() off, one can still explicitly add key-value pairs that use builtin stuff. Likewise, if any builtins are excluded, they can still be explicitly added, however, since builtins() has higher priority than select()/exclude(), if they're turned off, select()/exclude() will not see them at all.
This is a good example of why things are so confusing. The paragraph above is contradictory within itself.
On one hand you state that if "one builtins() off, once can still explictitly add key-valued pairs" and on the other hand you state "if they're turned off, select()/exclude() will not see them at all".
Yep, it is a little confusing if you look at it that way. But I don't see the contradiction: select()/exclude() are not the same as explicit specification (either in the (current) form of "$HOST" or ("host" "$HOST")). The former will not see them, but one can add them back explicitly. Even if we find a way to get rid of builtins(), this priority will remain nevertheless, because I do need a way to include value pairs that were formerly excluded - if for nothing else, for convenience's sake. (That, and the implementation is a lot simpler this way :P)
I realize that you need to add the builtins using something other than select, but that seems confusing too.
Hopefully documentation will be able to help with that :) (Myself, I'm terrible at documentation, sadly)
It is also unclear where the $UNIXTIME came from since it was not shown at all in the example that apparently incleded everything.
Yep, unfortunately the way macros and builtins are handled in syslog-ng is a bit... unclear, and chaotic. I'm trying to figure out an easy way to fix this, and make builtins() include all of the built-in macros, including $UNIXTIME and the rest.
Perhaps just relying on the select/exclude (which should probably be renamed to include/exclude) would be sufficient since in most cases at least some of the builtin macros will be desired and like in your example where you included the $HOST and $MESSAGE it would have been almost as easy to merely exclude the others by name and not use the builtin option at all.
The problem with that, is that there's no other easy way to exclude all of the builtin macros, which might be preferable in some cases.
I am not so much concerned with making the process of excluding all of the built in macros easy, but am much more concerned with making the syntax consistent, deterministic and obvious so that when writing, on probably more importantly for trouble shooting, when reading the configuration file.
Creating a system where there is a priority of options, and then having the order define the priority for some of the options makes a system that is more easily misunderstood.
True enough, but at the moment, I don't really see a better solution.
destination d_mongo { mongodb( value_pairs(builtins(no) select("*") exclude(".classifier.rule_id") "$HOST" "$MESSAGE" ("PROGRAM" "$PROGRAM[$PID]") ("TIMESTAMP" "$UNIXTIME")) ); };
would seem to be very different from
destination d_mongo { mongodb( value_pairs(select("*") exclude(".classifier.rule_id") "$HOST" "$MESSAGE" ("PROGRAM" "$PROGRAM[$PID]") ("TIMESTAMP" "$UNIXTIME")) builtins(no) ); };
but given the proposal that "builtins(no)" has highest priority, these two examples actually have the same semantics.
Yeah, I see your point. Hmm... perhaps something like: exclude(builtins) (and similarly, select(builtins), for what it's worth) could be added, that has the same priority as select()/exclude()? Or, come to think of it, I might be able to beat some sense into builtins() and have it on the same priority as select() and exclude()... We'd still have the "$HOST" and ("key" "value") stuff on another level, though, and mixing those in would be quite a challenge to implement.
Thanks a lot for the detailed feedback by the way, it's most appreciated!
That's nice. Sometimes I am told I am argumentative :-(
Sometimes it's good to be argumentative >;) -- |8]