[syslog-ng] [RFC]: value_pairs() demo

[Gergely Nagy]

On Tue, 2011-02-01 at 08:29 -0800, Evan Rempel wrote:
> >> It is really nothing more than a power-select or power-exclude but it does not
> >> honour the order requirement of the select/exclude options.
> > 
> > Yep, and that's by design. There's a priority among the selectors:
> > explicit selects ("$HOST", "$MESSAGE" and key-value pairs) are the
> > highest, followed by builtins() and select()/exclude() on the lowest
> > priority.
> > 
> > Thus, if one turns builtins() off, one can still explicitly add
> > key-value pairs that use builtin stuff. Likewise, if any builtins are
> > excluded, they can still be explicitly added, however, since builtins()
> > has higher priority than select()/exclude(), if they're turned off,
> > select()/exclude() will not see them at all.
> 
> This is a good example of why things are so confusing.
> The paragraph above is contradictory within itself.
> 
> On one hand you state that if "one builtins() off, once can still explictitly add
> key-valued pairs" and on the other hand you state "if they're turned off,
> select()/exclude() will not see them at all".

Yep, it is a little confusing if you look at it that way. But I don't
see the contradiction: select()/exclude() are not the same as explicit
specification (either in the (current) form of "$HOST" or ("host"
"$HOST")).

The former will not see them, but one can add them back explicitly. Even
if we find a way to get rid of builtins(), this priority will remain
nevertheless, because I do need a way to include value pairs that were
formerly excluded - if for nothing else, for convenience's sake.

(That, and the implementation is a lot simpler this way :P)

> I realize that you need to add the builtins using something other than select, but
> that seems confusing too.

Hopefully documentation will be able to help with that :)

(Myself, I'm terrible at documentation, sadly)

> >> It is also unclear where the $UNIXTIME came from since it was not shown at all
> >> in the example that apparently incleded everything.
> > 
> > Yep, unfortunately the way macros and builtins are handled in syslog-ng
> > is a bit... unclear, and chaotic. I'm trying to figure out an easy way
> > to fix this, and make builtins() include all of the built-in macros,
> > including $UNIXTIME and the rest.
> > 
> >> Perhaps just relying on the select/exclude (which should probably be renamed to include/exclude)
> >> would be sufficient since in most cases at least some of the builtin macros will be desired and
> >> like in your example where you included the $HOST and $MESSAGE it would have been almost
> >> as easy to merely exclude the others by name and not use the builtin option at all.
> > 
> > The problem with that, is that there's no other easy way to exclude all
> > of the builtin macros, which might be preferable in some cases.
> 
> I am not so much concerned with making the process of excluding all of the
> built in macros easy, but am much more concerned with making the syntax consistent,
> deterministic and obvious so that when writing, on probably more importantly for trouble
> shooting, when reading the configuration file.
> 
> Creating a system where there is a priority of options, and then having
> the order define the priority for some of the options makes a system that is more easily
> misunderstood.

True enough, but at the moment, I don't really see a better solution.

> destination d_mongo {
>    mongodb(
>      value_pairs(builtins(no) select("*") exclude(".classifier.rule_id")
>                  "$HOST" "$MESSAGE"
>                  ("PROGRAM" "$PROGRAM[$PID]") ("TIMESTAMP" "$UNIXTIME"))
>    );
> };
> 
> would seem to be very different from
> 
> destination d_mongo {
>    mongodb(
>      value_pairs(select("*") exclude(".classifier.rule_id")
>                  "$HOST" "$MESSAGE"
>                  ("PROGRAM" "$PROGRAM[$PID]") ("TIMESTAMP" "$UNIXTIME"))
>                  builtins(no)
>    );
> };
> 
> but given the proposal that "builtins(no)" has highest priority,
> these two examples actually have the same semantics.

Yeah, I see your point.

Hmm... perhaps something like: exclude(builtins) (and similarly,
select(builtins), for what it's worth) could be added, that has the same
priority as select()/exclude()?

Or, come to think of it, I might be able to beat some sense into
builtins() and have it on the same priority as select() and exclude()...

We'd still have the "$HOST" and ("key" "value") stuff on another level,
though, and mixing those in would be quite a challenge to implement.

> > Thanks a lot for the detailed feedback by the way, it's most
> > appreciated!
> 
> That's nice. Sometimes I am told I am argumentative :-(

Sometimes it's good to be argumentative >;)

-- 
|8]