[syslog-ng] [RFC]: value_pairs() demo

[Gergely Nagy]

On Sat, 2011-01-29 at 07:40 -0800, Evan Rempel wrote:
> > destination d_mongo {
> >   mongodb(
> >     value_pairs(builtins(no) select("*") exclude(".classifier.rule_id")
> >                 "$HOST" "$MESSAGE"
> >                 ("PROGRAM" "$PROGRAM[$PID]") ("TIMESTAMP" "$UNIXTIME"))
> >   );
> > };
> > 
> > And this will do exactly what it says: skip builtins, select everything
> > that is left, and exclude ".classifier.rule_id" from that, and then add
> > a few extra stuff on our own.
> 
> I think that the "builtin(no)" option should be abandon in favour of something else.

In my opinion, it'd be better to clarify what builtin() is for. At the
moment, there's a short list of builtin macros:

HOST, HOST_FROM, MESSAGE, PROGRAM, PID, MSGID, SOURCE, LEGACY_MSGHDR
(defined in lib/logmsg.c), and there's a few standard macros, like
$UNIXTIME.

By default, the standard macros that are not part of the builtins, will
not be included unless explicitly requested, which is a shame, and
that's what makes builtins() confusing, imo.

If builtins() dealt with the standard macros, it'd be much easier - and
I plan to figure out how to do just that. That will also affect select()
and exclude() too.

Perhaps it can be renamed to builtin-macros() then?

> It is really nothing more than a power-select or power-exclude but it does not
> honour the order requirement of the select/exclude options.

Yep, and that's by design. There's a priority among the selectors:
explicit selects ("$HOST", "$MESSAGE" and key-value pairs) are the
highest, followed by builtins() and select()/exclude() on the lowest
priority.

Thus, if one turns builtins() off, one can still explicitly add
key-value pairs that use builtin stuff. Likewise, if any builtins are
excluded, they can still be explicitly added, however, since builtins()
has higher priority than select()/exclude(), if they're turned off,
select()/exclude() will not see them at all.

> In the above example
> you have excluded the built in macros but then used a select("*") which implies adding
> everything back in. If you had done these in the oposite order, what semantic would 
> be intended.

That's due to the explicit > builtins() > select/exclude priority order.

> It is unclear to me what is defined as a builtin macro and which ones are not.

Indeed, it is unclear - even to me. I plan to fix that, though (see
above).

> It is also unclear where the $UNIXTIME came from since it was not shown at all
> in the example that apparently incleded everything.

Yep, unfortunately the way macros and builtins are handled in syslog-ng
is a bit... unclear, and chaotic. I'm trying to figure out an easy way
to fix this, and make builtins() include all of the built-in macros,
including $UNIXTIME and the rest.

> Perhaps just relying on the select/exclude (which should probably be renamed to include/exclude)
> would be sufficient since in most cases at least some of the builtin macros will be desired and
> like in your example where you included the $HOST and $MESSAGE it would have been almost
> as easy to merely exclude the others by name and not use the builtin option at all.

The problem with that, is that there's no other easy way to exclude all
of the builtin macros, which might be preferable in some cases.

Thanks a lot for the detailed feedback by the way, it's most
appreciated!

-- 
|8]