[syslog-ng] [RFC]: value_pairs() demo

[Evan Rempel]

Gergely Nagy wrote:
> On Sat, 2011-01-29 at 07:40 -0800, Evan Rempel wrote:
>>> destination d_mongo {
>>>   mongodb(
>>>     value_pairs(builtins(no) select("*") exclude(".classifier.rule_id")
>>>                 "$HOST" "$MESSAGE"
>>>                 ("PROGRAM" "$PROGRAM[$PID]") ("TIMESTAMP" "$UNIXTIME"))
>>>   );
>>> };
>>>
>>> And this will do exactly what it says: skip builtins, select everything
>>> that is left, and exclude ".classifier.rule_id" from that, and then add
>>> a few extra stuff on our own.
>> I think that the "builtin(no)" option should be abandon in favour of something else.
> 
> In my opinion, it'd be better to clarify what builtin() is for. At the
> moment, there's a short list of builtin macros:
> 
> HOST, HOST_FROM, MESSAGE, PROGRAM, PID, MSGID, SOURCE, LEGACY_MSGHDR
> (defined in lib/logmsg.c), and there's a few standard macros, like
> $UNIXTIME.
> 
> By default, the standard macros that are not part of the builtins, will
> not be included unless explicitly requested, which is a shame, and
> that's what makes builtins() confusing, imo.
> 
> If builtins() dealt with the standard macros, it'd be much easier - and
> I plan to figure out how to do just that. That will also affect select()
> and exclude() too.
> 
> Perhaps it can be renamed to builtin-macros() then?
> 
>> It is really nothing more than a power-select or power-exclude but it does not
>> honour the order requirement of the select/exclude options.
> 
> Yep, and that's by design. There's a priority among the selectors:
> explicit selects ("$HOST", "$MESSAGE" and key-value pairs) are the
> highest, followed by builtins() and select()/exclude() on the lowest
> priority.
> 
> Thus, if one turns builtins() off, one can still explicitly add
> key-value pairs that use builtin stuff. Likewise, if any builtins are
> excluded, they can still be explicitly added, however, since builtins()
> has higher priority than select()/exclude(), if they're turned off,
> select()/exclude() will not see them at all.

This is a good example of why things are so confusing.
The paragraph above is contradictory within itself.

On one hand you state that if "one builtins() off, once can still explictitly add
key-valued pairs" and on the other hand you state "if they're turned off,
select()/exclude() will not see them at all".

I realize that you need to add the builtins using something other than select, but
that seems confusing too.


>> In the above example
>> you have excluded the built in macros but then used a select("*") which implies adding
>> everything back in. If you had done these in the oposite order, what semantic would 
>> be intended.
> 
> That's due to the explicit > builtins() > select/exclude priority order.
> 
>> It is unclear to me what is defined as a builtin macro and which ones are not.
> 
> Indeed, it is unclear - even to me. I plan to fix that, though (see
> above).
> 
>> It is also unclear where the $UNIXTIME came from since it was not shown at all
>> in the example that apparently incleded everything.
> 
> Yep, unfortunately the way macros and builtins are handled in syslog-ng
> is a bit... unclear, and chaotic. I'm trying to figure out an easy way
> to fix this, and make builtins() include all of the built-in macros,
> including $UNIXTIME and the rest.
> 
>> Perhaps just relying on the select/exclude (which should probably be renamed to include/exclude)
>> would be sufficient since in most cases at least some of the builtin macros will be desired and
>> like in your example where you included the $HOST and $MESSAGE it would have been almost
>> as easy to merely exclude the others by name and not use the builtin option at all.
> 
> The problem with that, is that there's no other easy way to exclude all
> of the builtin macros, which might be preferable in some cases.

I am not so much concerned with making the process of excluding all of the
built in macros easy, but am much more concerned with making the syntax consistent,
deterministic and obvious so that when writing, on probably more importantly for trouble
shooting, when reading the configuration file.

Creating a system where there is a priority of options, and then having
the order define the priority for some of the options makes a system that is more easily
misunderstood.

destination d_mongo {
   mongodb(
     value_pairs(builtins(no) select("*") exclude(".classifier.rule_id")
                 "$HOST" "$MESSAGE"
                 ("PROGRAM" "$PROGRAM[$PID]") ("TIMESTAMP" "$UNIXTIME"))
   );
};

would seem to be very different from

destination d_mongo {
   mongodb(
     value_pairs(select("*") exclude(".classifier.rule_id")
                 "$HOST" "$MESSAGE"
                 ("PROGRAM" "$PROGRAM[$PID]") ("TIMESTAMP" "$UNIXTIME"))
                 builtins(no)
   );
};

but given the proposal that "builtins(no)" has highest priority,
these two examples actually have the same semantics.

And this just won't do what you expect either.

destination d_mongo {
   mongodb(
     value_pairs(select("*") exclude(".classifier.rule_id")
                 select("HOST") select("MESSAGE")
                 ("PROGRAM" "$PROGRAM[$PID]") ("TIMESTAMP" "$UNIXTIME"))
                 builtins(no)
   );
};


> Thanks a lot for the detailed feedback by the way, it's most
> appreciated!

That's nice. Sometimes I am told I am argumentative :-(

-- 
Evan Rempel