[syslog-ng] [RFC]: value-pairs()

[Gergely Nagy]

Hi!

The idea of a generic framework to set up value pairs came up first
during the afmongodb discussion (see Bazsi's ideas at
http://article.gmane.org/gmane.comp.syslog-ng/10432), and while
working on the tfjson driver, it came up again.

I've learnt a lot since the dynamic-variables implementation in
afmongodb, and following Bazsi's comments on the list earlier, we
managed to talk a little about the value-pairs() feature too. That,
and another quick chat with Balint, followed by working on tfjson
helped form the proposal below.

I'll divide this into two sections: the first part, mostly aimed at
people using syslog-ng in production would be a request for comments
regarding the syntax of the feature.

The second part will be a few notes on the planned implementation.

The Syntax
==========

value-pairs (
  glob-select ("usracct.*")
  glob-exclude ("*.*id")
  builtins (no)
  $HOST $MESSAGE
  "program_n_pid" = "$PROGRAM[$PID]"
)

What we have here, is basically three possible ways to define - or
control - value pairs:

* Using a function, like glob-select and glob-exclude or builtins
  (more about these later)
* Macro names, with the '$' sign.
* "key" = "value" pairs.

The way this is supposed to work is that the base set is all the
various key=value pairs syslog-ng collected about a particular log
message: the built-in macros ($HOST, $MESSAGE, etc), sdata, patterndb
- you name it.

This is the set we can select or exclude from, the set we can add new
pairs to (different values for each value-pairs() instance, of
course).

With glob-select(), we can mark a subset of the orignal set for
inclusion. Subsequent glob-exclude()s can override this, and remove
parts of it from the set.

Specifying builtins(no) removes every built-in key from the set.

Listing known keys, unquoted, '$' prefixed, will add that value to the
set, with the key set to the macro name without the dollar sign. For
example, $HOST is exactly the same as "HOST" = "$HOST".

Finally, "key" = "value" pairs do the obvious thing: add custom pairs
to our set.

The order of functions does matter, so glob-select("*")
glob-exclude("*id") glob-select("usracct.id") will select usracct.id,
but no other key ending with "id".

This is partly due to me finding this logical, and because if/when we
add other functions later on (say, ltrim() or add-prefix() or similar
that modify the keys), the order becomes essential.

But, back to the purpose of this all: this would be templating on
steroids, so to say. Drivers should be able to easily implement
support for value-pairs(), which would make system administrators to
tweak the way syslog-ng discovered data will appear in the output.

Particularly, this functionality is what will make the afmongodb
driver shine. This is a much more advanced version of the
dynamic-variables() functionality. This will also come in handy for
the JSON template function.

I leave the rest up to your imagination. Feedback on the syntax would
be most appreciated.

The Design
==========

Value pairs would be stored in a struct like the following:

typedef struct
{
  GPatternSpec *pattern;
  gboolean exclude;
} ValuePairPattern;

typedef struct
{
  ValuePairPattern **patterns;
  gboolean builtins;
  NVRegistry extras_registry;
  NVTable extras;
} ValuePairs;

ValuePairs->patterns would be a NULL terminated array (we can allow
for a lot of reallocs during init, imo - otherwise we can make it a
GPtrArray or something similar to save time on that), where each
pattern is either an exclude or a select pattern. The ->builtins
boolean would flag whether we should include the builtin variables
($HOST, etc), and would default to TRUE. And ->extras are the extra
keys specified explicitly: both in $MACRO and "key"="value" form (the
former translated to "MACRO"="$MACRO" form).

We'd have the following support API:

ValuePairs *value_pairs_new ();
void value_pairs_add_pattern (ValuePairs *vp, const gchar *pattern,
           gboolean exclude);
void value_pairs_set_builtins (ValuePairs *vp, gboolean state);
void value_pairs_add_extra (ValuePairs *vp, const gchar *key,
         const gchar *value);
void value_pairs_foreach (ValuePairs *vp, VPForeachFunc *func,
       gpointer user_data);

typedef gboolean (*VPForeachFunc) (const gchar *name,
   const gchar *value,
   gssize value_len,
   gpointer user_data);

We'd also have a VALUE_PAIRS keyword in the parser (if it is possible,
which I think it is), which drivers could use. The basic VALUE_PAIRS
stuff would set up a ValuePairs structure, and store it in a global
variable, which the driver could then either copy, or just assign it
to it's own (since the main parser won't free or touch it anymore
anyway).

Then, all the drivers have to do, is call value_pairs_foreach(), which
would firs iterate over logmsg_registry's keys, taking the specified
patterns into account.

Then it would iterate over it's own extras, and pass the results to
the foreach function one by one, similar to how nv_table_foreach does.

That's about it. It's a bit vague, mostly because I did not research
if this is possible the way I imagine it yet.

Hopefully this all made at least some sense.

-- 
|8]