Hi! The idea of a generic framework to set up value pairs came up first during the afmongodb discussion (see Bazsi's ideas at http://article.gmane.org/gmane.comp.syslog-ng/10432), and while working on the tfjson driver, it came up again. I've learnt a lot since the dynamic-variables implementation in afmongodb, and following Bazsi's comments on the list earlier, we managed to talk a little about the value-pairs() feature too. That, and another quick chat with Balint, followed by working on tfjson helped form the proposal below. I'll divide this into two sections: the first part, mostly aimed at people using syslog-ng in production would be a request for comments regarding the syntax of the feature. The second part will be a few notes on the planned implementation. The Syntax ========== value-pairs ( glob-select ("usracct.*") glob-exclude ("*.*id") builtins (no) $HOST $MESSAGE "program_n_pid" = "$PROGRAM[$PID]" ) What we have here, is basically three possible ways to define - or control - value pairs: * Using a function, like glob-select and glob-exclude or builtins (more about these later) * Macro names, with the '$' sign. * "key" = "value" pairs. The way this is supposed to work is that the base set is all the various key=value pairs syslog-ng collected about a particular log message: the built-in macros ($HOST, $MESSAGE, etc), sdata, patterndb - you name it. This is the set we can select or exclude from, the set we can add new pairs to (different values for each value-pairs() instance, of course). With glob-select(), we can mark a subset of the orignal set for inclusion. Subsequent glob-exclude()s can override this, and remove parts of it from the set. Specifying builtins(no) removes every built-in key from the set. Listing known keys, unquoted, '$' prefixed, will add that value to the set, with the key set to the macro name without the dollar sign. For example, $HOST is exactly the same as "HOST" = "$HOST". Finally, "key" = "value" pairs do the obvious thing: add custom pairs to our set. The order of functions does matter, so glob-select("*") glob-exclude("*id") glob-select("usracct.id") will select usracct.id, but no other key ending with "id". This is partly due to me finding this logical, and because if/when we add other functions later on (say, ltrim() or add-prefix() or similar that modify the keys), the order becomes essential. But, back to the purpose of this all: this would be templating on steroids, so to say. Drivers should be able to easily implement support for value-pairs(), which would make system administrators to tweak the way syslog-ng discovered data will appear in the output. Particularly, this functionality is what will make the afmongodb driver shine. This is a much more advanced version of the dynamic-variables() functionality. This will also come in handy for the JSON template function. I leave the rest up to your imagination. Feedback on the syntax would be most appreciated. The Design ========== Value pairs would be stored in a struct like the following: typedef struct { GPatternSpec *pattern; gboolean exclude; } ValuePairPattern; typedef struct { ValuePairPattern **patterns; gboolean builtins; NVRegistry extras_registry; NVTable extras; } ValuePairs; ValuePairs->patterns would be a NULL terminated array (we can allow for a lot of reallocs during init, imo - otherwise we can make it a GPtrArray or something similar to save time on that), where each pattern is either an exclude or a select pattern. The ->builtins boolean would flag whether we should include the builtin variables ($HOST, etc), and would default to TRUE. And ->extras are the extra keys specified explicitly: both in $MACRO and "key"="value" form (the former translated to "MACRO"="$MACRO" form). We'd have the following support API: ValuePairs *value_pairs_new (); void value_pairs_add_pattern (ValuePairs *vp, const gchar *pattern, gboolean exclude); void value_pairs_set_builtins (ValuePairs *vp, gboolean state); void value_pairs_add_extra (ValuePairs *vp, const gchar *key, const gchar *value); void value_pairs_foreach (ValuePairs *vp, VPForeachFunc *func, gpointer user_data); typedef gboolean (*VPForeachFunc) (const gchar *name, const gchar *value, gssize value_len, gpointer user_data); We'd also have a VALUE_PAIRS keyword in the parser (if it is possible, which I think it is), which drivers could use. The basic VALUE_PAIRS stuff would set up a ValuePairs structure, and store it in a global variable, which the driver could then either copy, or just assign it to it's own (since the main parser won't free or touch it anymore anyway). Then, all the drivers have to do, is call value_pairs_foreach(), which would firs iterate over logmsg_registry's keys, taking the specified patterns into account. Then it would iterate over it's own extras, and pass the results to the foreach function one by one, similar to how nv_table_foreach does. That's about it. It's a bit vague, mostly because I did not research if this is possible the way I imagine it yet. Hopefully this all made at least some sense. -- |8]