On Sat, 2011-01-29 at 07:40 -0800, Evan Rempel wrote:
destination d_mongo { mongodb( value_pairs(builtins(no) select("*") exclude(".classifier.rule_id") "$HOST" "$MESSAGE" ("PROGRAM" "$PROGRAM[$PID]") ("TIMESTAMP" "$UNIXTIME")) ); };
And this will do exactly what it says: skip builtins, select everything that is left, and exclude ".classifier.rule_id" from that, and then add a few extra stuff on our own.
I think that the "builtin(no)" option should be abandon in favour of something else.
In my opinion, it'd be better to clarify what builtin() is for. At the moment, there's a short list of builtin macros: HOST, HOST_FROM, MESSAGE, PROGRAM, PID, MSGID, SOURCE, LEGACY_MSGHDR (defined in lib/logmsg.c), and there's a few standard macros, like $UNIXTIME. By default, the standard macros that are not part of the builtins, will not be included unless explicitly requested, which is a shame, and that's what makes builtins() confusing, imo. If builtins() dealt with the standard macros, it'd be much easier - and I plan to figure out how to do just that. That will also affect select() and exclude() too. Perhaps it can be renamed to builtin-macros() then?
It is really nothing more than a power-select or power-exclude but it does not honour the order requirement of the select/exclude options.
Yep, and that's by design. There's a priority among the selectors: explicit selects ("$HOST", "$MESSAGE" and key-value pairs) are the highest, followed by builtins() and select()/exclude() on the lowest priority. Thus, if one turns builtins() off, one can still explicitly add key-value pairs that use builtin stuff. Likewise, if any builtins are excluded, they can still be explicitly added, however, since builtins() has higher priority than select()/exclude(), if they're turned off, select()/exclude() will not see them at all.
In the above example you have excluded the built in macros but then used a select("*") which implies adding everything back in. If you had done these in the oposite order, what semantic would be intended.
That's due to the explicit > builtins() > select/exclude priority order.
It is unclear to me what is defined as a builtin macro and which ones are not.
Indeed, it is unclear - even to me. I plan to fix that, though (see above).
It is also unclear where the $UNIXTIME came from since it was not shown at all in the example that apparently incleded everything.
Yep, unfortunately the way macros and builtins are handled in syslog-ng is a bit... unclear, and chaotic. I'm trying to figure out an easy way to fix this, and make builtins() include all of the built-in macros, including $UNIXTIME and the rest.
Perhaps just relying on the select/exclude (which should probably be renamed to include/exclude) would be sufficient since in most cases at least some of the builtin macros will be desired and like in your example where you included the $HOST and $MESSAGE it would have been almost as easy to merely exclude the others by name and not use the builtin option at all.
The problem with that, is that there's no other easy way to exclude all of the builtin macros, which might be preferable in some cases. Thanks a lot for the detailed feedback by the way, it's most appreciated! -- |8]