[PATCH 0/3]: value-pairs() & related patches
Hi! For a multitude of reasons, ranging from the feature freeze looming over my head to hope that discrete patches are easier to review and merge than my awfully chaotic git tree, I'll be posting a couple of patches in a short while, as soon as I figure out what magic arguments I need to give to git send-email to make it send the batch the way I imagined. value-pairs() ============= The first one will be the value-pairs() implementation (also available on my integration/value-pairs branch), which is the base of the two others that will follow. While the value-pairs() implementation could still use some work, especially in the performance department, I believe it's good enough for initial use, and the benefits of having it (so that the mongodb driver can make use of it) far outweight the downsides. The implementation itself is reasonably light, it hardly touches anything outside it's own two files: the grammar and the makefile only, to hook itself in. If not used by a driver, then it does not change functionality at all. Therefore, I believe this could go in before the feature freeze. afmongodb value-pairs() port ============================ The next patch will be the port of the already existing mongodb destination driver to use value-pairs(). Also available on my integration/afmongodb-vp branch. This means that the driver will finally be able to do what it's good at: store arbitrary information in the database: whether that information comes via SDATA or patterndb, it doesn't matter, it will be stored in a structured manner. In my opinion, this is the major selling point of the mongodb destination driver at the moment (I have some other grand plans with it in the future, but that's more invasive, less important, and can easily wait 'till 3.4 ;). Without value-pairs(), afmongodb is pretty much useless. tfson() ======= For extra fun & profit, Balint Kovacs' format_json has been ported to use value-pairs(), and this will be the third patch in the series, and is also available on my integration/tfjson branch. This one's the least important, though it's dead simple, therefore I'd be very happy if it made past the feature freeze aswell. -- |8]
This is the implementation of the value-pairs() framework discussed on the list in the past. The syntax is as follows: destination d_driver { driver( value-pairs( scope($SCOPE) exclude("S_*") # shell-style glob to exclude keys pair("my-own-key" "$SOURCE/$HOST") key("HOST_FROM") ) ); }; Of course, this is not only for destination drivers: there is support for parsing arguments passed to template functions. The above set would look like this in a template function: $(function --scope $SCOPE --exclude S_* --pair my-own-key=$SOURCE/$HOST --key HOST_FROM) The idea is that we select a scope (see later for the list of scopes currently supported), from which we can exclude certain keys using a shell-style glob pattern, or add custom key-value pairs of our own, or add back pre-defined keys that we do want, but the scope doesn't have by default. The available scopes at the moment are: * rfc3164: Key-value pairs based on the old syslog protocol, also available as "core" and "base". * rfc5424: Key-value pairs based on the new syslog protocol, also available as "syslog_proto". * all_macros: All macros known to syslog-ng (this only includes the pre-defined macros!) * selected_macros: A smaller set of selected macros, which the developers deemed important (see the source for the list, for now). * nv_pairs: Name-value pairs, the results of SDATA or patterndb parsing. * everything: All of the above, combined into one big set. On the driver side, supporting value-pairs is reasonably easy: driver_option : value_pair_stmt { driver_dd_set_value_pairs(last_driver, last_value_pairs); } | <etc...> ; And in the driver itself, make a foreach callback: static gboolean driver_vp_foreach (const gchar *name, const gchar *value, gpointer user_data) { /* Do something here with the name-value pairs */ return FALSE; } And whenever you want to work with the set, call value_pairs_foreach(): value_pairs_foreach (self->value_pairs, driver_vp_foreach, msg, self->seq_num, USER_DATA); And that's about it. Signed-off-by: Gergely Nagy <algernon@balabit.hu> --- lib/Makefile.am | 3 +- lib/cfg-grammar.y | 68 ++++++++- lib/cfg-parser.c | 17 ++ lib/cfg-parser.h | 1 + lib/syslog-ng.h | 2 + lib/value-pairs.c | 458 +++++++++++++++++++++++++++++++++++++++++++++++++++++ lib/value-pairs.h | 62 +++++++ 7 files changed, 608 insertions(+), 3 deletions(-) create mode 100644 lib/value-pairs.c create mode 100644 lib/value-pairs.h diff --git a/lib/Makefile.am b/lib/Makefile.am index 33a886e..16075b9 100644 --- a/lib/Makefile.am +++ b/lib/Makefile.am @@ -59,7 +59,8 @@ libsyslog_ng_la_SOURCES = \ block-ref-grammar.y block-ref-parser.c block-ref-parser.h \ pragma-grammar.y pragma-parser.h pragma-parser.c \ parser-expr-grammar.y parser-expr-parser.c parser-expr-parser.h \ - rewrite-expr-grammar.y rewrite-expr-parser.c rewrite-expr-parser.h + rewrite-expr-grammar.y rewrite-expr-parser.c rewrite-expr-parser.h \ + value-pairs.c value-pairs.h BUILT_SOURCES = cfg-lex.c cfg-lex.h cfg-grammar.c cfg-grammar.h \ filter-expr-grammar.y filter-expr-grammar.c filter-expr-grammar.h \ diff --git a/lib/cfg-grammar.y b/lib/cfg-grammar.y index 0ec55ad..e1d97b6 100644 --- a/lib/cfg-grammar.y +++ b/lib/cfg-grammar.y @@ -28,6 +28,7 @@ #include "cfg-lexer.h" #include "driver.h" +#include "value-pairs.h" } %name-prefix "main_" @@ -134,7 +135,6 @@ %token KW_SYSLOG 10060 - /* option items */ %token KW_MARK_FREQ 10071 %token KW_STATS_FREQ 10072 @@ -263,6 +263,20 @@ %token LL_EOL 10428 %token LL_ERROR 10429 +/* value pairs */ +%token KW_VALUE_PAIRS 10500 +%token KW_SELECT 10501 +%token KW_EXCLUDE 10502 +%token KW_PAIR 10503 +%token KW_KEY 10504 +%token KW_SCOPE 10505 +%token KW_SCOPE_NV_PAIRS 10506 +%token KW_SCOPE_RFC3164 10507 +%token KW_SCOPE_RFC5424 10508 +%token KW_SCOPE_ALL_MACROS 10509 +%token KW_SCOPE_SELECTED_MACROS 10510 +%token KW_SCOPE_EVERYTHING 10511 + /* END_DECLS */ %code { @@ -277,6 +291,7 @@ #include "logreader.h" #include "logparser.h" #include "logrewrite.h" +#include "value-pairs.h" #include "filter-expr-parser.h" #include "rewrite-expr-parser.h" #include "block-ref-parser.h" @@ -306,7 +321,7 @@ GList *last_rewrite_expr; GList *last_parser_expr; FilterExprNode *last_filter_expr; CfgArgs *last_block_args; - +ValuePairs *last_value_pairs; } @@ -318,6 +333,7 @@ CfgArgs *last_block_args; %type <ptr> log_stmt %type <ptr> options_stmt %type <ptr> template_stmt +%type <ptr> value_pair_stmt %type <ptr> source_items %type <ptr> source_item @@ -803,6 +819,54 @@ dest_writer_options_flags | { $$ = 0; } ; +value_pair_stmt + : KW_VALUE_PAIRS value_pairs_def { } + ; + +value_pairs_def + : { last_value_pairs = value_pairs_new(configuration); } + '(' value_pair_options ')' + { } + ; + +value_pair_options + : value_pair_option value_pair_options + | + ; + +value_pair_option + : KW_PAIR '(' string ':' string ')' { value_pairs_add_pair(last_value_pairs, $3, $5); free($3); free($5); } + | KW_PAIR '(' string string ')' { value_pairs_add_pair(last_value_pairs, $3, $4); free($3); free($4); } + | KW_KEY '(' string ')' { + gchar *k = g_strconcat("$", $3, NULL); + value_pairs_add_pair(last_value_pairs, $3, k); + g_free(k); + free($3); + } + | KW_EXCLUDE '(' string ')' { value_pairs_exclude_glob_pattern(last_value_pairs, $3); free($3); } + | KW_SCOPE '(' vp_scope_list ')' { } + ; + +vp_scope_list + : vp_scope vp_scope_list + | + ; + +vp_scope + : KW_SCOPE_NV_PAIRS + { value_pairs_add_scope(last_value_pairs, VALUE_PAIR_SCOPE_NV_PAIRS); } + | KW_SCOPE_RFC3164 + { value_pairs_add_scope(last_value_pairs, VALUE_PAIR_SCOPE_RFC3164); } + | KW_SCOPE_RFC5424 + { value_pairs_add_scope(last_value_pairs, VALUE_PAIR_SCOPE_RFC5424); } + | KW_SCOPE_ALL_MACROS + { value_pairs_add_scope(last_value_pairs, VALUE_PAIR_SCOPE_ALL_MACROS); } + | KW_SCOPE_SELECTED_MACROS + { value_pairs_add_scope(last_value_pairs, VALUE_PAIR_SCOPE_SELECTED_MACROS); } + | KW_SCOPE_EVERYTHING + { value_pairs_add_scope(last_value_pairs, VALUE_PAIR_SCOPE_EVERYTHING); } + ; + /* END_RULES */ diff --git a/lib/cfg-parser.c b/lib/cfg-parser.c index e53c58d..1123d87 100644 --- a/lib/cfg-parser.c +++ b/lib/cfg-parser.c @@ -53,6 +53,23 @@ static CfgLexerKeyword main_keywords[] = { { "quote_pairs", KW_QUOTE_PAIRS, 0x0300}, { "null", KW_NULL, 0x0300 }, + /* value pairs */ + { "value_pairs", KW_VALUE_PAIRS }, + { "select", KW_SELECT }, + { "exclude", KW_EXCLUDE }, + { "pair", KW_PAIR }, + { "key", KW_KEY }, + { "scope", KW_SCOPE }, + { "nv_pairs", KW_SCOPE_NV_PAIRS }, + { "rfc3164", KW_SCOPE_RFC3164 }, + { "core", KW_SCOPE_RFC3164 }, + { "base", KW_SCOPE_RFC3164 }, + { "rfc5424", KW_SCOPE_RFC5424 }, + { "syslog_proto", KW_SCOPE_RFC5424 }, + { "all_macros", KW_SCOPE_ALL_MACROS }, + { "selected_macros", KW_SCOPE_SELECTED_MACROS }, + { "everything", KW_SCOPE_EVERYTHING }, + /* option items */ { "flags", KW_FLAGS }, { "pad_size", KW_PAD_SIZE }, diff --git a/lib/cfg-parser.h b/lib/cfg-parser.h index 3527621..b874cd9 100644 --- a/lib/cfg-parser.h +++ b/lib/cfg-parser.h @@ -27,6 +27,7 @@ #include "syslog-ng.h" #include "cfg-lexer.h" +#include "value-pairs.h" /* high level interface to a configuration file parser, it encapsulates the * grammar/lexer pair. */ diff --git a/lib/syslog-ng.h b/lib/syslog-ng.h index ce4c017..83a9651 100644 --- a/lib/syslog-ng.h +++ b/lib/syslog-ng.h @@ -61,10 +61,12 @@ typedef struct _LogMessage LogMessage; typedef struct _GlobalConfig GlobalConfig; +typedef struct _ValuePairs TValuePairs; extern GlobalConfig *configuration; extern gboolean seed_rng; extern gchar *module_path; extern gchar *default_modules; +extern TValuePairs *last_value_pairs; #endif diff --git a/lib/value-pairs.c b/lib/value-pairs.c new file mode 100644 index 0000000..5dfbb6f --- /dev/null +++ b/lib/value-pairs.c @@ -0,0 +1,458 @@ +/* + * Copyright (c) 2002-2011 BalaBit IT Ltd, Budapest, Hungary + * Copyright (c) 2011 Gergely Nagy <algernon@balabit.hu> + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA + * + * As an additional exemption you are allowed to compile & link against the + * OpenSSL libraries as published by the OpenSSL project. See the file + * COPYING for details. + * + */ + +#include "logmsg.h" +#include "templates.h" + +#include "value-pairs.h" + +#include <string.h> + +typedef struct +{ + gchar *name; + LogTemplate *template; +} vp_template_set; + +struct _ValuePairs +{ + GlobalConfig *cfg; + + guint8 scopes; + + GPatternSpec **excludes; + gsize exclude_size; + GHashTable *vpairs; + + struct + { + vp_template_set *rfc3164; + vp_template_set *rfc5424; + vp_template_set *selected_macros; + vp_template_set *all_macros; + } templates; + + /* Temporary */ + GString *res; +}; + +static inline void +vp_add_scope_template (GlobalConfig *cfg, vp_template_set *target, + gint index, const gchar *name, const gchar *macro) +{ + LogTemplate *t = NULL; + gchar *m; + + if (macro) + m = g_strdup (macro); + else + m = g_strconcat ("$", name, NULL); + + t = log_template_new(cfg, NULL, m); + target[index].name = g_strdup(name); + target[index].template = t; + g_free (m); +} + +ValuePairs * +value_pairs_new (GlobalConfig *cfg) +{ + ValuePairs *vp; + gint i = 0; + + vp = g_try_new0(ValuePairs, 1); + if (!vp) + return NULL; + vp->res = g_string_sized_new(256); + if (!vp->res) + { + g_free(vp); + return NULL; + } + vp->vpairs = g_hash_table_new_full(g_str_hash, g_str_equal, g_free, + (GDestroyNotify) log_template_unref); + if (!vp->vpairs) + { + g_string_free(vp->res, TRUE); + g_free(vp); + return NULL; + } + + vp->cfg = cfg; + + vp->templates.rfc3164 = g_try_new0 (vp_template_set, 8); + vp->templates.rfc5424 = g_try_new0 (vp_template_set, 3); + vp->templates.selected_macros = g_try_new0 (vp_template_set, 5); + + while (macros[i].name) + i++; + + vp->templates.all_macros = g_try_new0 (vp_template_set, i + 1); + i = 0; + + vp_add_scope_template (vp->cfg, vp->templates.rfc3164, 0, "FACILITY", NULL); + vp_add_scope_template (vp->cfg, vp->templates.rfc3164, 1, "PRIORITY", NULL); + vp_add_scope_template (vp->cfg, vp->templates.rfc3164, 2, "DATE", "$S_DATE"); + vp_add_scope_template (vp->cfg, vp->templates.rfc3164, 3, "HOST", NULL); + vp_add_scope_template (vp->cfg, vp->templates.rfc3164, 4, "PROGRAM", NULL); + vp_add_scope_template (vp->cfg, vp->templates.rfc3164, 5, "PID", NULL); + vp_add_scope_template (vp->cfg, vp->templates.rfc3164, 6, "MSG", NULL); + + vp_add_scope_template (vp->cfg, vp->templates.rfc5424, 0, "MSGID", NULL); + vp_add_scope_template (vp->cfg, vp->templates.rfc5424, 1, "SDATA", NULL); + + vp_add_scope_template (vp->cfg, vp->templates.selected_macros, 0, "TAGS", NULL); + vp_add_scope_template (vp->cfg, vp->templates.selected_macros, 1, "SDATA", NULL); + vp_add_scope_template (vp->cfg, vp->templates.selected_macros, 2, "SOURCEIP", NULL); + vp_add_scope_template (vp->cfg, vp->templates.selected_macros, 3, "SEQNUM", NULL); + + while (macros[i].name) + { + vp_add_scope_template (vp->cfg, vp->templates.all_macros, i, macros[i].name, NULL); + i++; + } + + return vp; +} + +static gboolean +vp_cmdline_parse_scope (const gchar *option_name, const gchar *value, + gpointer data, GError **error) +{ + ValuePairs *vp = (ValuePairs *)data; + + if (g_ascii_strcasecmp (value, "nv_pairs") == 0) + value_pairs_add_scope (vp, VALUE_PAIR_SCOPE_NV_PAIRS); + else if (g_ascii_strcasecmp (value, "rfc3164") == 0 || + g_ascii_strcasecmp (value, "core") == 0 || + g_ascii_strcasecmp (value, "base")) + value_pairs_add_scope (vp, VALUE_PAIR_SCOPE_RFC3164); + else if (g_ascii_strcasecmp (value, "rfc5424") == 0 || + g_ascii_strcasecmp (value, "syslog_proto") == 0) + value_pairs_add_scope (vp, VALUE_PAIR_SCOPE_RFC5424); + else if (g_ascii_strcasecmp (value, "selected_macros") == 0) + value_pairs_add_scope (vp, VALUE_PAIR_SCOPE_SELECTED_MACROS); + else if (g_ascii_strcasecmp (value, "all_macros") == 0) + value_pairs_add_scope (vp, VALUE_PAIR_SCOPE_ALL_MACROS); + else if (g_ascii_strcasecmp (value, "everything") == 0) + value_pairs_add_scope (vp, VALUE_PAIR_SCOPE_EVERYTHING); + else + { + g_set_error (error, G_OPTION_ERROR, G_OPTION_ERROR_BAD_VALUE, + "Error parsing value-pairs' scope()"); + return FALSE; + } + return TRUE; +} + +static gboolean +vp_cmdline_parse_exclude (const gchar *option_name, const gchar *value, + gpointer data, GError **error) +{ + ValuePairs *vp = (ValuePairs *)data; + + value_pairs_exclude_glob_pattern (vp, value); + return TRUE; +} + +static gboolean +vp_cmdline_parse_key (const gchar *option_name, const gchar *value, + gpointer data, GError **error) +{ + ValuePairs *vp = (ValuePairs *)data; + gchar *k = g_strconcat ("$", value, NULL); + + value_pairs_add_pair (vp, value, k); + g_free (k); + + return TRUE; +} + +static gboolean +vp_cmdline_parse_pair (const gchar *option_name, const gchar *value, + gpointer data, GError **error) +{ + ValuePairs *vp = (ValuePairs *)data; + gchar **kv; + + if (!g_strstr_len (value, strlen (value), "=")) + { + g_set_error (error, G_OPTION_ERROR, G_OPTION_ERROR_BAD_VALUE, + "Error parsing value-pairs' key=value pair"); + return FALSE; + } + + kv = g_strsplit(value, "=", 2); + value_pairs_add_pair (vp, kv[0], kv[1]); + + g_free (kv[0]); + g_free (kv[1]); + g_free (kv); + + return TRUE; +} + +ValuePairs * +value_pairs_new_from_cmdline (GlobalConfig *cfg, + gint cargc, gchar **cargv, + GError **error) +{ + ValuePairs *vp; + GOptionContext *ctx; + GOptionEntry vp_options[] = { + { "scope", 's', 0, G_OPTION_ARG_CALLBACK, vp_cmdline_parse_scope, + NULL, NULL }, + { "exclude", 'x', 0, G_OPTION_ARG_CALLBACK, vp_cmdline_parse_exclude, + NULL, NULL }, + { "key", 'k', 0, G_OPTION_ARG_CALLBACK, vp_cmdline_parse_key, + NULL, NULL }, + { "pair", 'p', 0, G_OPTION_ARG_CALLBACK, vp_cmdline_parse_pair, + NULL, NULL }, + { G_OPTION_REMAINING, 0, 0, G_OPTION_ARG_CALLBACK, vp_cmdline_parse_pair, + NULL, NULL }, + { NULL } + }; + gchar **argv; + gint argc = cargc + 1; + gint i; + GOptionGroup *og; + + argv = g_new (gchar *, argc + 1); + for (i = 0; i < argc; i++) + argv[i + 1] = cargv[i]; + argv[0] = "value-pairs"; + argv[argc] = NULL; + + vp = value_pairs_new (cfg); + + ctx = g_option_context_new ("value-pairs"); + og = g_option_group_new (NULL, NULL, NULL, vp, NULL); + g_option_group_add_entries (og, vp_options); + g_option_context_set_main_group (ctx, og); + + if (!g_option_context_parse (ctx, &argc, &argv, error)) + { + value_pairs_free (vp); + g_option_context_free (ctx); + g_free (argv); + return NULL; + } + g_option_context_free (ctx); + + if (vp->scopes == 0) + value_pairs_add_scope (vp, VALUE_PAIR_SCOPE_RFC3164); + + return vp; +} + +void +value_pairs_free (ValuePairs *vp) +{ + gint i; + + if (!vp) + return; + + g_hash_table_destroy(vp->vpairs); + + for (i = 0; vp->templates.rfc3164[i].name; i++) + { + g_free (vp->templates.rfc3164[i].name); + log_template_unref (vp->templates.rfc3164[i].template); + } + g_free (vp->templates.rfc3164); + + for (i = 0; vp->templates.rfc5424[i].name; i++) + { + g_free (vp->templates.rfc5424[i].name); + log_template_unref (vp->templates.rfc5424[i].template); + } + g_free (vp->templates.rfc5424); + + for (i = 0; vp->templates.selected_macros[i].name; i++) + { + g_free (vp->templates.selected_macros[i].name); + log_template_unref (vp->templates.selected_macros[i].template); + } + g_free (vp->templates.selected_macros); + + for (i = 0; vp->templates.all_macros[i].name; i++) + { + g_free (vp->templates.all_macros[i].name); + log_template_unref (vp->templates.all_macros[i].template); + } + g_free (vp->templates.all_macros); + + for (i = 0; i < vp->exclude_size; i++) + g_pattern_spec_free(vp->excludes[i]); + g_free(vp->excludes); + g_string_free(vp->res, TRUE); + g_free(vp); +} + +void +value_pairs_add_scope (ValuePairs *vp, value_pair_scope_t scope) +{ + if (!vp) + return; + + vp->scopes |= scope; +} + +void +value_pairs_exclude_glob_pattern (ValuePairs *vp, const gchar *pattern) +{ + gint i; + + if (!vp || !pattern) + return; + + i = vp->exclude_size++; + vp->excludes = g_renew(GPatternSpec *, vp->excludes, vp->exclude_size); + vp->excludes[i] = g_pattern_spec_new(pattern); +} + +void +value_pairs_add_pair (ValuePairs *vp, const gchar *key, + const gchar *value) +{ + LogTemplate *t = NULL; + + if (!vp || !key || !value) + return; + + t = log_template_new(vp->cfg, NULL, value); + g_hash_table_insert(vp->vpairs, g_strdup(key), t); +} + +static void +vp_foreach_merge_vp(gpointer key, gpointer value, gpointer user_data) +{ + ValuePairs *vp = ((gpointer *)user_data)[0]; + LogMessage *msg = ((gpointer *)user_data)[2]; + gint32 seq_num = GPOINTER_TO_INT (((gpointer *)user_data)[3]); + GHashTable *scope_set = ((gpointer *)user_data)[5]; + + g_string_truncate(vp->res, 0); + log_template_format((LogTemplate *)value, msg, NULL, LTZ_LOCAL, + seq_num, NULL, vp->res); + + if (!vp->res->str || !vp->res->str[0]) + return; + + g_hash_table_insert(scope_set, key, g_strdup (vp->res->str)); +} + +static gboolean +vp_foreach_nv_add(NVHandle handle, gchar *name, + const gchar *value, gssize value_len, + gpointer user_data) +{ + ValuePairs *vp = ((gpointer *)user_data)[0]; + GHashTable *scope_set = ((gpointer *)user_data)[5]; + gint j; + + for (j = 0; j < vp->exclude_size; j++) + if (g_pattern_match_string(vp->excludes[j], name)) + return FALSE; + + g_hash_table_insert (scope_set, name, (gpointer)g_strndup (value, value_len)); + return FALSE; +} + +static void +vp_foreach_merge_template (ValuePairs *vp, LogMessage *msg, gint32 seq_num, + vp_template_set *set, GHashTable *dest) +{ + gint i; + + for (i = 0; set[i].name; i++) + { + gint j; + gboolean exclude = FALSE; + + for (j = 0; j < vp->exclude_size; j++) + if (g_pattern_match_string(vp->excludes[j], set[i].name)) + exclude = TRUE; + if (exclude) + continue; + + g_string_truncate(vp->res, 0); + log_template_format(set[i].template, msg, NULL, LTZ_LOCAL, seq_num, NULL, + vp->res); + + if (!vp->res->str || !vp->res->str[0]) + continue; + + g_hash_table_insert (dest, set[i].name, g_strdup (vp->res->str)); + } +} + +void +value_pairs_foreach (ValuePairs *vp, VPForeachFunc func, + LogMessage *msg, gint32 seq_num, gpointer user_data) +{ + gpointer args[] = { vp, func, msg, GINT_TO_POINTER (seq_num), user_data, NULL }; + GHashTable *scope_set; + + scope_set = g_hash_table_new_full(g_str_hash, g_str_equal, NULL, + (GDestroyNotify) g_free); + + args[5] = scope_set; + + /* + * Build up the base set + */ + if (vp->scopes & VALUE_PAIR_SCOPE_NV_PAIRS) + { + nv_table_foreach(msg->payload, logmsg_registry, + (NVTableForeachFunc)vp_foreach_nv_add, args); + } + + if (vp->scopes & VALUE_PAIR_SCOPE_RFC3164 || + vp->scopes & VALUE_PAIR_SCOPE_RFC5424 || + vp->scopes & VALUE_PAIR_SCOPE_SELECTED_MACROS) + vp_foreach_merge_template (vp, msg, seq_num, vp->templates.rfc3164, + scope_set); + + if (vp->scopes & VALUE_PAIR_SCOPE_RFC5424) + vp_foreach_merge_template (vp, msg, seq_num, vp->templates.rfc5424, + scope_set); + + if (vp->scopes & VALUE_PAIR_SCOPE_SELECTED_MACROS) + vp_foreach_merge_template (vp, msg, seq_num, vp->templates.selected_macros, + scope_set); + + if (vp->scopes & VALUE_PAIR_SCOPE_ALL_MACROS) + vp_foreach_merge_template (vp, msg, seq_num, vp->templates.all_macros, + scope_set); + + /* Merge the explicit key-value pairs too */ + g_hash_table_foreach (vp->vpairs, (GHFunc)vp_foreach_merge_vp, args); + + /* Aaand we run it through the callback! */ + g_hash_table_foreach (scope_set, (GHFunc)func, user_data); + + g_hash_table_destroy (scope_set); +} diff --git a/lib/value-pairs.h b/lib/value-pairs.h new file mode 100644 index 0000000..5642f28 --- /dev/null +++ b/lib/value-pairs.h @@ -0,0 +1,62 @@ +/* + * Copyright (c) 2002-2011 BalaBit IT Ltd, Budapest, Hungary + * Copyright (c) 2011 Gergely Nagy <algernon@balabit.hu> + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA + * + * As an additional exemption you are allowed to compile & link against the + * OpenSSL libraries as published by the OpenSSL project. See the file + * COPYING for details. + * + */ + +#ifndef VALUE_PAIRS_H_INCLUDED +#define VALUE_PAIRS_H_INCLUDED 1 + +#include "syslog-ng.h" +#include "nvtable.h" + +#include <glib.h> + +typedef enum + { + VALUE_PAIR_SCOPE_NV_PAIRS = 0x01, + VALUE_PAIR_SCOPE_RFC3164 = 0x02, + VALUE_PAIR_SCOPE_RFC5424 = 0x04, + VALUE_PAIR_SCOPE_ALL_MACROS = 0x08, + VALUE_PAIR_SCOPE_SELECTED_MACROS = 0x10, + VALUE_PAIR_SCOPE_EVERYTHING = 0xff, + } value_pair_scope_t; + +typedef struct _ValuePairs ValuePairs; +typedef gboolean (*VPForeachFunc) (const gchar *name, const gchar *value, + gpointer user_data); + +ValuePairs *value_pairs_new (GlobalConfig *cfg); +void value_pairs_free (ValuePairs *vp); + +void value_pairs_add_scope (ValuePairs *vp, value_pair_scope_t scope); +void value_pairs_exclude_glob_pattern (ValuePairs *vp, const gchar *pattern); +void value_pairs_add_pair (ValuePairs *vp, const gchar *key, const gchar *value); + +void value_pairs_foreach (ValuePairs *vp, VPForeachFunc func, + LogMessage *msg, gint32 seq_num, + gpointer user_data); + +ValuePairs *value_pairs_new_from_cmdline (GlobalConfig *cfg, + gint argc, gchar **argv, + GError **error); + +#endif -- 1.7.2.5
Ported from custom, explicit key/value pairs to the far more flexible value-pairs() solution. By default, the driver uses a custom scope: selected_macros and nv_pairs, with a few patterns excluded. The patch also turns the collection() parameter into a plain string (from a templatable string), to considerably reduce the work needed to insert messages into the database. Signed-off-by: Gergely Nagy <algernon@balabit.hu> --- modules/afmongodb/afmongodb-grammar.ym | 4 +- modules/afmongodb/afmongodb-parser.c | 2 - modules/afmongodb/afmongodb.c | 154 +++++++++---------------------- modules/afmongodb/afmongodb.h | 4 +- 4 files changed, 48 insertions(+), 116 deletions(-) diff --git a/modules/afmongodb/afmongodb-grammar.ym b/modules/afmongodb/afmongodb-grammar.ym index a19e2b4..7b5e680 100644 --- a/modules/afmongodb/afmongodb-grammar.ym +++ b/modules/afmongodb/afmongodb-grammar.ym @@ -48,7 +48,6 @@ extern LogDriver *last_driver; %token KW_MONGODB %token KW_COLLECTION -%token KW_KEYS %% @@ -70,10 +69,9 @@ afmongodb_option | KW_PORT '(' LL_NUMBER ')' { afmongodb_dd_set_port(last_driver, $3); } | KW_DATABASE '(' string ')' { afmongodb_dd_set_database(last_driver, $3); free($3); } | KW_COLLECTION '(' string ')' { afmongodb_dd_set_collection(last_driver, $3); free($3); } - | KW_VALUES '(' string_list ')' { afmongodb_dd_set_values(last_driver, $3); } - | KW_KEYS '(' string_list ')' { afmongodb_dd_set_keys(last_driver, $3); } | KW_USERNAME '(' string ')' { afmongodb_dd_set_user(last_driver, $3); free($3); } | KW_PASSWORD '(' string ')' { afmongodb_dd_set_password(last_driver, $3); free($3); } + | value_pair_stmt { afmongodb_dd_set_value_pairs(last_driver, last_value_pairs); } | dest_driver_option ; diff --git a/modules/afmongodb/afmongodb-parser.c b/modules/afmongodb/afmongodb-parser.c index 5f9161e..67a9b0b 100644 --- a/modules/afmongodb/afmongodb-parser.c +++ b/modules/afmongodb/afmongodb-parser.c @@ -34,8 +34,6 @@ static CfgLexerKeyword afmongodb_keywords[] = { { "port", KW_PORT }, { "database", KW_DATABASE }, { "collection", KW_COLLECTION }, - { "keys", KW_KEYS }, - { "values", KW_VALUES }, { "username", KW_USERNAME }, { "password", KW_PASSWORD }, { "log_fifo_size", KW_LOG_FIFO_SIZE }, diff --git a/modules/afmongodb/afmongodb.c b/modules/afmongodb/afmongodb.c index 284066a..94f0ce1 100644 --- a/modules/afmongodb/afmongodb.c +++ b/modules/afmongodb/afmongodb.c @@ -47,7 +47,7 @@ typedef struct /* Shared between main/writer; only read by the writer, never written */ gchar *db; - LogTemplate *coll; + gchar *coll; gchar *host; gint port; @@ -55,12 +55,6 @@ typedef struct gchar *user; gchar *password; - GList *keys; - GList *values; - - gint num_fields; - MongoDBField *fields; - time_t time_reopen; guint32 *dropped_messages; @@ -68,6 +62,8 @@ typedef struct time_t last_msg_stamp; + ValuePairs *vp; + /* Thread related stuff; shared */ GThread *writer_thread; GMutex *queue_mutex; @@ -84,11 +80,9 @@ typedef struct mongo_connection *conn; gint32 seq_num; - GString *current_namespace; - gint ns_prefix_len; + gchar *ns; GString *current_value; - bson *bson_sel, *bson_upd, *bson_set; } MongoDBDestDriver; @@ -145,26 +139,17 @@ afmongodb_dd_set_collection(LogDriver *d, const gchar *collection) { MongoDBDestDriver *self = (MongoDBDestDriver *)d; - log_template_unref(self->coll); - self->coll = log_template_new(log_pipe_get_config(&d->super), NULL, collection); + g_free(self->coll); + self->coll = g_strdup(collection); } void -afmongodb_dd_set_keys(LogDriver *d, GList *keys) +afmongodb_dd_set_value_pairs(LogDriver *d, ValuePairs *vp) { MongoDBDestDriver *self = (MongoDBDestDriver *)d; - string_list_free(self->keys); - self->keys = keys; -} - -void -afmongodb_dd_set_values(LogDriver *d, GList *values) -{ - MongoDBDestDriver *self = (MongoDBDestDriver *)d; - - string_list_free(self->values); - self->values = values; + value_pairs_free (self->vp); + self->vp = vp; } /* @@ -177,7 +162,7 @@ afmongodb_dd_format_stats_instance(MongoDBDestDriver *self) static gchar persist_name[1024]; g_snprintf(persist_name, sizeof(persist_name), - "mongodb,%s,%u,%s,%s", self->host, self->port, self->db, self->coll->template); + "mongodb,%s,%u,%s,%s", self->host, self->port, self->db, self->coll); return persist_name; } @@ -187,7 +172,7 @@ afmongodb_dd_format_persist_name(MongoDBDestDriver *self) static gchar persist_name[1024]; g_snprintf(persist_name, sizeof(persist_name), - "afmongodb(%s,%u,%s,%s)", self->host, self->port, self->db, self->coll->template); + "afmongodb(%s,%u,%s,%s)", self->host, self->port, self->db, self->coll); return persist_name; } @@ -244,11 +229,23 @@ afmongodb_dd_connect(MongoDBDestDriver *self, gboolean reconnect) /* * Worker thread */ +static gboolean +afmongodb_vp_foreach (const gchar *name, const gchar *value, + gpointer user_data) +{ + bson *bson_set = (bson *)user_data; + + if (name[0] == '.') + bson_append_string (bson_set, name + 1, value, -1); + else + bson_append_string (bson_set, name, value, -1); + + return FALSE; +} static gboolean afmongodb_worker_insert (MongoDBDestDriver *self) { - gint i; gboolean success; mongo_packet *p; guint8 *oid; @@ -274,25 +271,15 @@ afmongodb_worker_insert (MongoDBDestDriver *self) g_free (oid); bson_finish (self->bson_sel); - g_string_truncate(self->current_namespace, self->ns_prefix_len); - log_template_append_format(self->coll, msg, NULL, LTZ_LOCAL, - self->seq_num, NULL, self->current_namespace); - - for (i = 0; i < self->num_fields; i++) - { - log_template_format(self->fields[i].value, msg, NULL, LTZ_SEND, - self->seq_num, NULL, self->current_value); - if (self->current_value->len) - bson_append_string(self->bson_set, self->fields[i].name, - self->current_value->str, -1); - } + value_pairs_foreach (self->vp, afmongodb_vp_foreach, + msg, self->seq_num, self->bson_set); bson_finish (self->bson_set); bson_append_document (self->bson_upd, "$set", self->bson_set); bson_finish (self->bson_upd); - p = mongo_wire_cmd_update (1, self->current_namespace->str, 1, + p = mongo_wire_cmd_update (1, self->ns, 1, self->bson_sel, self->bson_upd); if (!mongo_packet_send (self->conn, p)) @@ -335,12 +322,8 @@ afmongodb_worker_thread (gpointer arg) NULL); success = afmongodb_dd_connect(self, FALSE); - self->current_namespace = g_string_sized_new(64); - self->ns_prefix_len = strlen (self->db) + 1; - self->current_namespace = - g_string_append_c (g_string_assign (self->current_namespace, - self->db), '.'); + self->ns = g_strconcat (self->db, ".", self->coll, NULL); self->current_value = g_string_sized_new(256); @@ -383,7 +366,7 @@ afmongodb_worker_thread (gpointer arg) afmongodb_dd_disconnect(self); - g_string_free (self->current_namespace, TRUE); + g_free (self->ns); g_string_free (self->current_value, TRUE); bson_free (self->bson_sel); @@ -421,15 +404,25 @@ afmongodb_dd_init(LogPipe *s) MongoDBDestDriver *self = (MongoDBDestDriver *)s; GlobalConfig *cfg = log_pipe_get_config(s); - gint num_keys, num_values, i; - GList *key, *value; - if (!log_dest_driver_init_method(s)) return FALSE; if (cfg) self->time_reopen = cfg->time_reopen; + if (!self->vp) + { + self->vp = value_pairs_new(cfg); + value_pairs_add_scope(self->vp, VALUE_PAIR_SCOPE_SELECTED_MACROS); + value_pairs_add_scope(self->vp, VALUE_PAIR_SCOPE_NV_PAIRS); + value_pairs_exclude_glob_pattern(self->vp, "R_*"); + value_pairs_exclude_glob_pattern(self->vp, "S_*"); + value_pairs_exclude_glob_pattern(self->vp, "HOST_FROM"); + value_pairs_exclude_glob_pattern(self->vp, "LEGACY_MSGHDR"); + value_pairs_exclude_glob_pattern(self->vp, "MSG"); + value_pairs_exclude_glob_pattern(self->vp, "SDATA"); + } + msg_verbose("Initializing MongoDB destination", evt_tag_str("host", self->host), evt_tag_int("port", self->port), @@ -438,29 +431,6 @@ afmongodb_dd_init(LogPipe *s) self->queue = log_dest_driver_acquire_queue(&self->super, afmongodb_dd_format_persist_name(self)); - if (!self->fields) - { - num_keys = g_list_length(self->keys); - num_values = g_list_length(self->values); - - if (num_keys != num_values) - { - msg_error("The number of keys and values do not match", - evt_tag_int("num_keys", num_keys), - evt_tag_int("num_values", num_values), - NULL); - return FALSE; - } - self->num_fields = num_keys; - self->fields = g_new0(MongoDBField, num_keys); - - for (i = 0, key = self->keys, value = self->values; key && value; i++, key = key->next, value = value->next) - { - self->fields[i].name = g_strdup(key->data); - self->fields[i].value = log_template_new(cfg, NULL, (gchar *)value->data); - } - } - stats_register_counter(0, SCS_MONGODB | SCS_DESTINATION, self->super.super.id, afmongodb_dd_format_stats_instance(self), SC_TYPE_STORED, &self->stored_messages); @@ -499,7 +469,6 @@ static void afmongodb_dd_free(LogPipe *d) { MongoDBDestDriver *self = (MongoDBDestDriver *)d; - gint i; g_mutex_free(self->suspend_mutex); g_mutex_free(self->queue_mutex); @@ -508,20 +477,13 @@ afmongodb_dd_free(LogPipe *d) if (self->queue) log_queue_unref(self->queue); - for (i = 0; i < self->num_fields; i++) - { - g_free(self->fields[i].name); - log_template_unref(self->fields[i].value); - } - - g_free(self->fields); g_free(self->db); - log_template_unref(self->coll); + g_free(self->coll); g_free(self->user); g_free(self->password); g_free(self->host); - string_list_free(self->keys); - string_list_free(self->values); + + value_pairs_free(self->vp); log_dest_driver_free(d); } @@ -548,30 +510,6 @@ afmongodb_dd_queue(LogPipe *s, LogMessage *msg, const LogPathOptions *path_optio * Plugin glue. */ -const gchar *default_keys[] = -{ - "date", - "facility", - "level", - "host", - "program", - "pid", - "message", - NULL -}; - -const gchar *default_values[] = -{ - "${R_YEAR}-${R_MONTH}-${R_DAY} ${R_HOUR}:${R_MIN}:${R_SEC}", - "$FACILITY", - "$LEVEL", - "$HOST", - "$PROGRAM", - "$PID", - "$MSGONLY", - NULL -}; - LogDriver * afmongodb_dd_new(void) { @@ -589,8 +527,6 @@ afmongodb_dd_new(void) afmongodb_dd_set_port((LogDriver *)self, 27017); afmongodb_dd_set_database((LogDriver *)self, "syslog"); afmongodb_dd_set_collection((LogDriver *)self, "messages"); - afmongodb_dd_set_keys((LogDriver *)self, string_array_to_list(default_keys)); - afmongodb_dd_set_values((LogDriver *)self, string_array_to_list(default_values)); init_sequence_number(&self->seq_num); diff --git a/modules/afmongodb/afmongodb.h b/modules/afmongodb/afmongodb.h index e54faf2..bd7b258 100644 --- a/modules/afmongodb/afmongodb.h +++ b/modules/afmongodb/afmongodb.h @@ -25,6 +25,7 @@ #define AFMONGODB_H_INCLUDED #include "driver.h" +#include "value-pairs.h" LogDriver *afmongodb_dd_new(void); @@ -32,9 +33,8 @@ void afmongodb_dd_set_host(LogDriver *d, const gchar *host); void afmongodb_dd_set_port(LogDriver *d, gint port); void afmongodb_dd_set_database(LogDriver *d, const gchar *database); void afmongodb_dd_set_collection(LogDriver *d, const gchar *collection); -void afmongodb_dd_set_values(LogDriver *d, GList *values); -void afmongodb_dd_set_keys(LogDriver *d, GList *keys); void afmongodb_dd_set_user(LogDriver *d, const gchar *user); void afmongodb_dd_set_password(LogDriver *d, const gchar *password); +void afmongodb_dd_set_value_pairs(LogDriver *d, ValuePairs *vp); #endif -- 1.7.2.5
On Sun, 2011-04-24 at 11:29 +0200, Gergely Nagy wrote:
Ported from custom, explicit key/value pairs to the far more flexible value-pairs() solution. By default, the driver uses a custom scope: selected_macros and nv_pairs, with a few patterns excluded.
The patch also turns the collection() parameter into a plain string (from a templatable string), to considerably reduce the work needed to insert messages into the database.
Signed-off-by: Gergely Nagy <algernon@balabit.hu>
I've rebased this patch and adapted it to my value-pairs changes. Can you please check if it works for you? Thanks. commit 7b9c8e2b12272c5454198e642c478c52e7859da0 Author: Balazs Scheidler <bazsi@balabit.hu> Date: Sat Apr 30 23:34:16 2011 +0200 afmongodb: Port to value-pairs(). Ported from custom, explicit key/value pairs to the far more flexible value-pairs() solution. By default, the driver uses a custom scope: selected_macros and nv_pairs, with a few patterns excluded. The patch also turns the collection() parameter into a plain string (from a templatable string), to considerably reduce the work needed to insert messages into the database. Signed-off-by: Gergely Nagy <algernon@balabit.hu> Signed-off-by: Balazs Scheidler <bazsi@balabit.hu> -- Bazsi
Balazs Scheidler <bazsi@balabit.hu> writes:
On Sun, 2011-04-24 at 11:29 +0200, Gergely Nagy wrote:
Ported from custom, explicit key/value pairs to the far more flexible value-pairs() solution. By default, the driver uses a custom scope: selected_macros and nv_pairs, with a few patterns excluded.
The patch also turns the collection() parameter into a plain string (from a templatable string), to considerably reduce the work needed to insert messages into the database.
Signed-off-by: Gergely Nagy <algernon@balabit.hu>
I've rebased this patch and adapted it to my value-pairs changes. Can you please check if it works for you?
It's not on your master branch, neither in 3.3 nor in 3.4, as far as I see. I can reach the patch on git.balabit.hu via gitweb, but a cloned repository does not have it. I looked through the patch on the web though, and I believe it's correct, and should work. One comment though, regarding the commit message: you moved the collection name un-templating to a separate patch, so you might want to delete that part of the commit message :) I'll download & apply the patch by hand later today, if time allows, and test it properly. But I'm fairly sure it'll work. -- |8]
On Sun, 2011-05-01 at 09:32 +0200, Gergely Nagy wrote:
Balazs Scheidler <bazsi@balabit.hu> writes:
On Sun, 2011-04-24 at 11:29 +0200, Gergely Nagy wrote:
Ported from custom, explicit key/value pairs to the far more flexible value-pairs() solution. By default, the driver uses a custom scope: selected_macros and nv_pairs, with a few patterns excluded.
The patch also turns the collection() parameter into a plain string (from a templatable string), to considerably reduce the work needed to insert messages into the database.
Signed-off-by: Gergely Nagy <algernon@balabit.hu>
I've rebased this patch and adapted it to my value-pairs changes. Can you please check if it works for you?
It's not on your master branch, neither in 3.3 nor in 3.4, as far as I see. I can reach the patch on git.balabit.hu via gitweb, but a cloned repository does not have it.
hmm, I don't understand, it is on the master branch, like all the others. Are you sure you properly cloned the repo? Can you try to explicitly check out origin/master ?
I looked through the patch on the web though, and I believe it's correct, and should work. One comment though, regarding the commit message: you moved the collection name un-templating to a separate patch, so you might want to delete that part of the commit message :)
Well, you submitted that portion in a separate email, so I've just committed them separately.
I'll download & apply the patch by hand later today, if time allows, and test it properly. But I'm fairly sure it'll work.
would be great. I just compile tested it. Can you point me to a simple description for the simplest ever usecase for the mongodb destination? So that I can test it when I apply a change. -- Bazsi
Balazs Scheidler <bazsi@balabit.hu> writes:
On Sun, 2011-05-01 at 09:32 +0200, Gergely Nagy wrote:
Balazs Scheidler <bazsi@balabit.hu> writes:
On Sun, 2011-04-24 at 11:29 +0200, Gergely Nagy wrote:
Ported from custom, explicit key/value pairs to the far more flexible value-pairs() solution. By default, the driver uses a custom scope: selected_macros and nv_pairs, with a few patterns excluded.
The patch also turns the collection() parameter into a plain string (from a templatable string), to considerably reduce the work needed to insert messages into the database.
Signed-off-by: Gergely Nagy <algernon@balabit.hu>
I've rebased this patch and adapted it to my value-pairs changes. Can you please check if it works for you?
It's not on your master branch, neither in 3.3 nor in 3.4, as far as I see. I can reach the patch on git.balabit.hu via gitweb, but a cloned repository does not have it.
hmm, I don't understand, it is on the master branch, like all the others. Are you sure you properly cloned the repo? Can you try to explicitly check out origin/master ?
Yep, I'm pretty sure: $ git clone git://git.balabit.hu/bazsi/syslog-ng-3.3.git Cloning into syslog-ng-3.3... remote: Generating pack... remote: Done counting 9638 objects. remote: Deltifying 9638 objects... remote: 100% (9638/9638) done remote: Total 9638 (delta 7421), reused 8274 (delta 6348) Receiving objects: 100% (9638/9638), 2.32 MiB | 1.67 MiB/s, done. Resolving deltas: 100% (7421/7421), done. $ cd syslog-ng-3.3/ $ git log --pretty=oneline | head -n 30 7efc9d72a74f78f7ae7ce426c36174301af2224c Merge remote branch '3.2/master' 9245389973dbc78f14f79c2e091e3234eb0a6652 preparations for 3.2.3 c844bef80554d2b4dacd3796e0522f90b3721702 make systemd support a conditional feature a8e298f300ad29442cd6f0aee7de5956438658e7 affile: don't attempt to remember the position for follow-freq(0) files 88a884a52bfc975335a05ffa0107684b6655ef29 tfjson: Template function for outputting JSON. ae01dddab39926c86c0651d4a3aac46284393ef1 value-pairs: fixed macro expansion 0c62650f7a2dffed3a110c6589c136806105d000 afmongodb: Make the collection name a simple string. dd65482f75270c40ac5f2f0cf948a9f772a2e9b7 affile: don't attempt to remember the position for follow-freq(0) files 2b65655028da7cc216aa747d0bb206fe28b93955 logwriter: initialize the queue_filled event earlier 2e243ea62396e6305ada53862ff4f0a3800b7443 libsyslog-ng-crypto.so: This patch introduces a new shared object where openssl dependent core goes adff96aa401c8ef398a617c14daf7dc45f85cab7 logreader: fixed parsing flags containing dashes or underscores be4f2fc5efdf637ef29e1216dd301db3a810f7bd The following options now can use with their default values (in file and in pipe destination): fe958c6a486b33996fca8e6595b2c2995ee943e2 test_msgparse: added testcase to check unescaped quote in structured data 451071b928805322aed742be6e6f813c1468fa80 loggen: fixed compile error caused by the ignored SIGPIPE signal dfdb9b17dff33e11019b46184e54420c35432a8f syslogformat: make sure all partially initialized NV pairs are removed if parsing the message fails d329a86c1b2b94d9b8df4853ca458a4dcec47b08 [tags] fix log_tags_deinit() to properly free all tags related variables 1ba9335884d2fe44dcd51ad40d0f8e4afa9314bf value-pairs: A framework to filter key=value pairs. 5a2274a4f3e392537505313b2035bab22cbd6c6e loggen: handle SIGPIPE be36e3c31a32cefce1fab0610b433f273f5b12a6 loggen: call shutdown on the socket 63f50e6abdeb6e859b22ff76a229688dd0b2cd5f [loggen] multithreaded loggen handles if file source is given as log source and more than one active-connection is used. and other improvments fe4548bb3d985a22b636910641d1a8b925dd623a loggen: Added dont-parse option and restructured help menu bf5bcb649c39130a869636373239052c32c5f7a3 loggen: added support for IPv6 3e9a0df3b6319c69d42a33c73f232838b762761f mainloop: fixed a possible use-after-free ebde08e1efd3a53b5c30161edfbac11950821b81 fixed possible segmentation fault at exit 379420e5b0a4add464c6d8b03667dc5c4a244075 fixed 100% CPU usage if SSL is enabled on a tcp() source 3abdd8773662f9d779429262262a1ed9229e98e6 logproto: Handle EAGAIN and EINTR correctly in _text_client_flush(). 78ea4d8aeba41a89a932e93b8b555480b2ebe130 fixed __thread keyword usage 8ce7ad823d450e3ceee93696b6ddbc491783c64f fixed refcache in consumer threads in case flow-control is not in use c892da8491e65803e54c7bb1fd7b5a61f8810ef8 SCL: implement uninstall target 745111ea007c8ffa9a3d083757a6c47c78ddf0cd LogWriter: handle epoll/kqueue failures better The collection name patch is there, but the value-pairs port isn't, as far as I can see.
I looked through the patch on the web though, and I believe it's correct, and should work. One comment though, regarding the commit message: you moved the collection name un-templating to a separate patch, so you might want to delete that part of the commit message :)
Well, you submitted that portion in a separate email, so I've just committed them separately.
It was in both patches: one separate, and it was included as part of the value-pairs() update too. (It's confusing, and I shouldn't have done that, but I was dead tired at the time).
I'll download & apply the patch by hand later today, if time allows, and test it properly. But I'm fairly sure it'll work.
would be great. I just compile tested it.
Can you point me to a simple description for the simplest ever usecase for the mongodb destination? So that I can test it when I apply a change.
The easiest way to test the driver is to install mongodb (for Debian & Ubuntu systems, see http://www.mongodb.org/display/DOCS/Ubuntu+and+Debian+packages), and once done, set up a mongodb destination: destination d_mongo { mongodb(); }; Then send a few messages to this destination, and from the shell, you can do the following: $ mongo localhost/syslog
db.messages.find()
This will dump all the messages in the syslog.messages collection (the default for the mongodb destination). The output should be reasonably straightforward, I believe: all the stuff set by the default scope (selected_macros + nvpairs, minus a few stuff like R_* and S_*), in JSON-ish format. -- |8]
The template function implementation below is a port of Balint Kovacs' format_json work, from syslog-ng 3.2 to 3.3, and to json-c and value-pairs(). The usage is as simple as: destination d_json { file("/tmp/messages.json" template("$(format_json --scope selected_macros --scope nv_pairs)")); }; Signed-off-by: Gergely Nagy <algernon@balabit.hu> --- configure.in | 26 ++++++++++ modules/Makefile.am | 2 +- modules/tfjson/Makefile.am | 11 ++++ modules/tfjson/tfjson.c | 109 ++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 147 insertions(+), 1 deletions(-) create mode 100644 modules/tfjson/Makefile.am create mode 100644 modules/tfjson/tfjson.c diff --git a/configure.in b/configure.in index 16f70c1..284b33d 100644 --- a/configure.in +++ b/configure.in @@ -26,6 +26,7 @@ OPENSSL_MIN_VERSION="0.9.8" LIBDBI_MIN_VERSION="0.8.0" IVYKIS_MIN_VERSION="0.18" PCRE_MIN_VERSION="6.1" +JSON_C_MIN_VERSION="0.9" dnl *************************************************************************** dnl Initial setup @@ -162,6 +163,10 @@ AC_ARG_WITH(ivykis, Link against the system supplied or the builting ivykis library.] ,,with_ivykis="internal") +AC_ARG_ENABLE(json, + [ --enable-json Enable support for JSON template formatting (default: auto)] + ,,enable_json="auto") + dnl *************************************************************************** dnl Checks for programs. AC_PROG_CC @@ -537,6 +542,23 @@ if test "x$linking_mode" != "xdynamic" -a "x$blb_cv_static_glib" = "xno"; then fi dnl *************************************************************************** +dnl json-glib headers/libraries +dnl *************************************************************************** +if test "x$enable_json" = "xyes" -o "x$enable_json" = "xauto"; then + PKG_CHECK_MODULES(JSON, json >= $JSON_C_MIN_VERSION,,JSON_LIBS="") + if test -z "$JSON_LIBS"; then + if test "x$enable_json" = "xyes"; then + AC_MSG_ERROR(Cannot find json >= $JSON_C_MIN_VERSION.) + else + AC_MSG_WARN(Cannot find json >= $JSON_C_MIN_VERSION.) + fi + enable_json="no" + else + enable_json="yes" + fi +fi + +dnl *************************************************************************** dnl pcre headers/libraries dnl *************************************************************************** @@ -893,6 +915,7 @@ AM_CONDITIONAL(ENABLE_SQL, [test "$enable_sql" = "yes"]) AM_CONDITIONAL(ENABLE_SUN_STREAMS, [test "$enable_sun_streams" = "yes"]) AM_CONDITIONAL(ENABLE_PACCT, [test "$enable_pacct" = "yes"]) AM_CONDITIONAL(ENABLE_MONGODB, [test "$enable_mongodb" = "yes"]) +AM_CONDITIONAL(ENABLE_JSON, [test "$enable_json" = "yes"]) # substitution into manual pages expanded_sysconfdir=[`patheval $sysconfdir | sed -e 's/-/\\\\-/g'`] @@ -953,6 +976,7 @@ AC_OUTPUT(dist.conf modules/pacctformat/Makefile modules/basicfuncs/Makefile modules/convertfuncs/Makefile + modules/tfjson/Makefile scripts/Makefile scripts/update-patterndb doc/Makefile @@ -1000,4 +1024,6 @@ echo " SSL support (module) : ${enable_ssl:=no}" echo " SQL support (module) : ${enable_sql:=no}" echo " PACCT module (EXPERIMENTAL) : ${enable_pacct:=no}" echo " MongoDB destination (module): ${enable_mongodb:=no}" +echo " JSON support (module) : ${enable_json:=no}" + diff --git a/modules/Makefile.am b/modules/Makefile.am index 38cde6b..46de2b8 100644 --- a/modules/Makefile.am +++ b/modules/Makefile.am @@ -1 +1 @@ -SUBDIRS = afsocket afsql afstreams affile afprog afuser afmongodb csvparser confgen syslogformat pacctformat basicfuncs convertfuncs dbparser dummy +SUBDIRS = afsocket afsql afstreams affile afprog afuser afmongodb csvparser confgen syslogformat pacctformat basicfuncs convertfuncs dbparser tfjson dummy diff --git a/modules/tfjson/Makefile.am b/modules/tfjson/Makefile.am new file mode 100644 index 0000000..8b30f83 --- /dev/null +++ b/modules/tfjson/Makefile.am @@ -0,0 +1,11 @@ +moduledir = @moduledir@ +export top_srcdir + +if ENABLE_JSON +AM_CPPFLAGS = -I$(top_srcdir)/lib -I../../lib @JSON_CFLAGS@ +module_LTLIBRARIES = libtfjson.la + +libtfjson_la_SOURCES = tfjson.c +libtfjson_la_LIBADD = ../../lib/libsyslog-ng.la @JSON_LIBS@ +libtfjson_la_LDFLAGS = -avoid-version +endif diff --git a/modules/tfjson/tfjson.c b/modules/tfjson/tfjson.c new file mode 100644 index 0000000..f23b2ac --- /dev/null +++ b/modules/tfjson/tfjson.c @@ -0,0 +1,109 @@ +/* + * Copyright (c) 2002-2011 BalaBit IT Ltd, Budapest, Hungary + * Copyright (c) 2011 Balint Kovacs <blint@balabit.hu> + * Copyright (c) 2011 Gergely Nagy <algernon@balabit.hu> + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License version 2 as published + * by the Free Software Foundation, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA + * + */ + +#include "plugin.h" +#include "templates.h" +#include "filter.h" +#include "filter-expr-parser.h" +#include "cfg.h" + +#include <json.h> + +static gboolean +tf_json_prepare(LogTemplateFunction *self, LogTemplate *parent, + gint argc, gchar *argv[], + gpointer *state, GDestroyNotify *state_destroy, + GError **error) +{ + ValuePairs *vp; + + vp = value_pairs_new_from_cmdline (parent->cfg, argc, argv, error); + if (!vp) + return FALSE; + + *state = vp; + *state_destroy = (GDestroyNotify) value_pairs_free; + + return TRUE; +} + +static gboolean +tf_json_foreach (const gchar *name, const gchar *value, gpointer user_data) +{ + json_object *root = (json_object *)user_data; + json_object *this; + + this = json_object_new_string (value); + json_object_object_add (root, name, this); + + return FALSE; +} + +static json_object * +tf_json_format_message(ValuePairs *vp, LogMessage *msg) +{ + json_object *root; + + root = json_object_new_object(); + value_pairs_foreach (vp, tf_json_foreach, msg, 0, root); + + return root; +} + +static void +tf_json_call(LogTemplateFunction *self, gpointer state, GPtrArray *arg_bufs, + LogMessage **messages, gint num_messages, LogTemplateOptions *opts, + gint tz, gint seq_num, const gchar *context_id, GString *result) +{ + gint i; + ValuePairs *vp = (ValuePairs *)state; + + for (i = 0; i < num_messages; i++) + { + LogMessage *msg = messages[i]; + json_object *json; + + json = tf_json_format_message(vp, msg); + g_string_append(result, json_object_to_json_string (json)); + json_object_put(json); + } +} + +static void +tf_json_eval (LogTemplateFunction *self, gpointer state, GPtrArray *arg_bufs, + LogMessage **messages, gint num_messages, LogTemplateOptions *opts, + gint tz, gint seq_num) +{ + return; +} + +TEMPLATE_FUNCTION(tf_json, tf_json_prepare, tf_json_eval, tf_json_call, NULL); + +static Plugin builtin_tmpl_func_plugins[] = + { + TEMPLATE_FUNCTION_PLUGIN(tf_json, "format_json"), + }; + +gboolean +tfjson_module_init(GlobalConfig *cfg, CfgArgs *args) +{ + plugin_register(cfg, builtin_tmpl_func_plugins, G_N_ELEMENTS(builtin_tmpl_func_plugins)); + return TRUE; +} -- 1.7.2.5
Hi, I've integrated this patch into 3.3 as agreed, it needed some slight modifications, but I've done that while integrating. Thanks. On Sun, 2011-04-24 at 11:30 +0200, Gergely Nagy wrote:
The template function implementation below is a port of Balint Kovacs' format_json work, from syslog-ng 3.2 to 3.3, and to json-c and value-pairs().
The usage is as simple as:
destination d_json { file("/tmp/messages.json" template("$(format_json --scope selected_macros --scope nv_pairs)")); };
Signed-off-by: Gergely Nagy <algernon@balabit.hu> --- configure.in | 26 ++++++++++ modules/Makefile.am | 2 +- modules/tfjson/Makefile.am | 11 ++++ modules/tfjson/tfjson.c | 109 ++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 147 insertions(+), 1 deletions(-) create mode 100644 modules/tfjson/Makefile.am create mode 100644 modules/tfjson/tfjson.c
diff --git a/configure.in b/configure.in index 16f70c1..284b33d 100644 --- a/configure.in +++ b/configure.in @@ -26,6 +26,7 @@ OPENSSL_MIN_VERSION="0.9.8" LIBDBI_MIN_VERSION="0.8.0" IVYKIS_MIN_VERSION="0.18" PCRE_MIN_VERSION="6.1" +JSON_C_MIN_VERSION="0.9"
dnl *************************************************************************** dnl Initial setup @@ -162,6 +163,10 @@ AC_ARG_WITH(ivykis, Link against the system supplied or the builting ivykis library.] ,,with_ivykis="internal")
+AC_ARG_ENABLE(json, + [ --enable-json Enable support for JSON template formatting (default: auto)] + ,,enable_json="auto") + dnl *************************************************************************** dnl Checks for programs. AC_PROG_CC @@ -537,6 +542,23 @@ if test "x$linking_mode" != "xdynamic" -a "x$blb_cv_static_glib" = "xno"; then fi
dnl *************************************************************************** +dnl json-glib headers/libraries +dnl *************************************************************************** +if test "x$enable_json" = "xyes" -o "x$enable_json" = "xauto"; then + PKG_CHECK_MODULES(JSON, json >= $JSON_C_MIN_VERSION,,JSON_LIBS="") + if test -z "$JSON_LIBS"; then + if test "x$enable_json" = "xyes"; then + AC_MSG_ERROR(Cannot find json >= $JSON_C_MIN_VERSION.) + else + AC_MSG_WARN(Cannot find json >= $JSON_C_MIN_VERSION.) + fi + enable_json="no" + else + enable_json="yes" + fi +fi + +dnl *************************************************************************** dnl pcre headers/libraries dnl ***************************************************************************
@@ -893,6 +915,7 @@ AM_CONDITIONAL(ENABLE_SQL, [test "$enable_sql" = "yes"]) AM_CONDITIONAL(ENABLE_SUN_STREAMS, [test "$enable_sun_streams" = "yes"]) AM_CONDITIONAL(ENABLE_PACCT, [test "$enable_pacct" = "yes"]) AM_CONDITIONAL(ENABLE_MONGODB, [test "$enable_mongodb" = "yes"]) +AM_CONDITIONAL(ENABLE_JSON, [test "$enable_json" = "yes"])
# substitution into manual pages expanded_sysconfdir=[`patheval $sysconfdir | sed -e 's/-/\\\\-/g'`] @@ -953,6 +976,7 @@ AC_OUTPUT(dist.conf modules/pacctformat/Makefile modules/basicfuncs/Makefile modules/convertfuncs/Makefile + modules/tfjson/Makefile scripts/Makefile scripts/update-patterndb doc/Makefile @@ -1000,4 +1024,6 @@ echo " SSL support (module) : ${enable_ssl:=no}" echo " SQL support (module) : ${enable_sql:=no}" echo " PACCT module (EXPERIMENTAL) : ${enable_pacct:=no}" echo " MongoDB destination (module): ${enable_mongodb:=no}" +echo " JSON support (module) : ${enable_json:=no}" +
diff --git a/modules/Makefile.am b/modules/Makefile.am index 38cde6b..46de2b8 100644 --- a/modules/Makefile.am +++ b/modules/Makefile.am @@ -1 +1 @@ -SUBDIRS = afsocket afsql afstreams affile afprog afuser afmongodb csvparser confgen syslogformat pacctformat basicfuncs convertfuncs dbparser dummy +SUBDIRS = afsocket afsql afstreams affile afprog afuser afmongodb csvparser confgen syslogformat pacctformat basicfuncs convertfuncs dbparser tfjson dummy diff --git a/modules/tfjson/Makefile.am b/modules/tfjson/Makefile.am new file mode 100644 index 0000000..8b30f83 --- /dev/null +++ b/modules/tfjson/Makefile.am @@ -0,0 +1,11 @@ +moduledir = @moduledir@ +export top_srcdir + +if ENABLE_JSON +AM_CPPFLAGS = -I$(top_srcdir)/lib -I../../lib @JSON_CFLAGS@ +module_LTLIBRARIES = libtfjson.la + +libtfjson_la_SOURCES = tfjson.c +libtfjson_la_LIBADD = ../../lib/libsyslog-ng.la @JSON_LIBS@ +libtfjson_la_LDFLAGS = -avoid-version +endif diff --git a/modules/tfjson/tfjson.c b/modules/tfjson/tfjson.c new file mode 100644 index 0000000..f23b2ac --- /dev/null +++ b/modules/tfjson/tfjson.c @@ -0,0 +1,109 @@ +/* + * Copyright (c) 2002-2011 BalaBit IT Ltd, Budapest, Hungary + * Copyright (c) 2011 Balint Kovacs <blint@balabit.hu> + * Copyright (c) 2011 Gergely Nagy <algernon@balabit.hu> + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License version 2 as published + * by the Free Software Foundation, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA + * + */ + +#include "plugin.h" +#include "templates.h" +#include "filter.h" +#include "filter-expr-parser.h" +#include "cfg.h" + +#include <json.h> + +static gboolean +tf_json_prepare(LogTemplateFunction *self, LogTemplate *parent, + gint argc, gchar *argv[], + gpointer *state, GDestroyNotify *state_destroy, + GError **error) +{ + ValuePairs *vp; + + vp = value_pairs_new_from_cmdline (parent->cfg, argc, argv, error); + if (!vp) + return FALSE; + + *state = vp; + *state_destroy = (GDestroyNotify) value_pairs_free; + + return TRUE; +} + +static gboolean +tf_json_foreach (const gchar *name, const gchar *value, gpointer user_data) +{ + json_object *root = (json_object *)user_data; + json_object *this; + + this = json_object_new_string (value); + json_object_object_add (root, name, this); + + return FALSE; +} + +static json_object * +tf_json_format_message(ValuePairs *vp, LogMessage *msg) +{ + json_object *root; + + root = json_object_new_object(); + value_pairs_foreach (vp, tf_json_foreach, msg, 0, root); + + return root; +} + +static void +tf_json_call(LogTemplateFunction *self, gpointer state, GPtrArray *arg_bufs, + LogMessage **messages, gint num_messages, LogTemplateOptions *opts, + gint tz, gint seq_num, const gchar *context_id, GString *result) +{ + gint i; + ValuePairs *vp = (ValuePairs *)state; + + for (i = 0; i < num_messages; i++) + { + LogMessage *msg = messages[i]; + json_object *json; + + json = tf_json_format_message(vp, msg); + g_string_append(result, json_object_to_json_string (json)); + json_object_put(json); + } +} + +static void +tf_json_eval (LogTemplateFunction *self, gpointer state, GPtrArray *arg_bufs, + LogMessage **messages, gint num_messages, LogTemplateOptions *opts, + gint tz, gint seq_num) +{ + return; +} + +TEMPLATE_FUNCTION(tf_json, tf_json_prepare, tf_json_eval, tf_json_call, NULL); + +static Plugin builtin_tmpl_func_plugins[] = + { + TEMPLATE_FUNCTION_PLUGIN(tf_json, "format_json"), + }; + +gboolean +tfjson_module_init(GlobalConfig *cfg, CfgArgs *args) +{ + plugin_register(cfg, builtin_tmpl_func_plugins, G_N_ELEMENTS(builtin_tmpl_func_plugins)); + return TRUE; +}
-- Bazsi
On Sun, 2011-04-24 at 11:18 +0200, Gergely Nagy wrote:
Hi!
For a multitude of reasons, ranging from the feature freeze looming over my head to hope that discrete patches are easier to review and merge than my awfully chaotic git tree, I'll be posting a couple of patches in a short while, as soon as I figure out what magic arguments I need to give to git send-email to make it send the batch the way I imagined.
value-pairs() =============
The first one will be the value-pairs() implementation (also available on my integration/value-pairs branch), which is the base of the two others that will follow.
While the value-pairs() implementation could still use some work, especially in the performance department, I believe it's good enough for initial use, and the benefits of having it (so that the mongodb driver can make use of it) far outweight the downsides.
The implementation itself is reasonably light, it hardly touches anything outside it's own two files: the grammar and the makefile only, to hook itself in. If not used by a driver, then it does not change functionality at all. Therefore, I believe this could go in before the feature freeze.
Ok, I've reviewed this patch and since it is really quite self-contained it can go in. I'm commenting on the patch in the separate mail.
afmongodb value-pairs() port ============================
The next patch will be the port of the already existing mongodb destination driver to use value-pairs(). Also available on my integration/afmongodb-vp branch.
This means that the driver will finally be able to do what it's good at: store arbitrary information in the database: whether that information comes via SDATA or patterndb, it doesn't matter, it will be stored in a structured manner.
In my opinion, this is the major selling point of the mongodb destination driver at the moment (I have some other grand plans with it in the future, but that's more invasive, less important, and can easily wait 'till 3.4 ;). Without value-pairs(), afmongodb is pretty much useless.
since mongodb is quite new, and probably not many people use it, this can also go in.
tfson() =======
For extra fun & profit, Balint Kovacs' format_json has been ported to use value-pairs(), and this will be the third patch in the series, and is also available on my integration/tfjson branch.
This one's the least important, though it's dead simple, therefore I'd be very happy if it made past the feature freeze aswell.
I'd integrate this into 3.4 instead. mongodb is already in, and the value-pairs stuff is a clear win there. tfjson and substr (anything I've missed) should go to 3.4. Hopefully that can be a faster release-cycle, as I don't intend to break everything as I did with 3.3. (you know converting a program from single to multi-threaded has a certain cost). -- Bazsi
Balazs Scheidler <bazsi@balabit.hu> writes: [...lots of lovely news skipped...] \o/
tfson() ======= [...snip...]
I'd integrate this into 3.4 instead. mongodb is already in, and the value-pairs stuff is a clear win there.
Aye, understandable. The important part (at least for me) was value-pairs() and the mongodb changes O:)
tfjson and substr (anything I've missed) should go to 3.4.
I'll go ahead and start a 3.4 tree myself aswell then. Regarding substr: there was talk of implementing ${MACRO:S:L}-style syntax aswell, which I still plan to do, just never got around to it yet. Would you prefer that on a separate branch, or can I do it on the same one as $(substr) ? -- |8]
tfson() =======
For extra fun & profit, Balint Kovacs' format_json has been ported to use value-pairs(), and this will be the third patch in the series, and is also available on my integration/tfjson branch.
This one's the least important, though it's dead simple, therefore I'd be very happy if it made past the feature freeze aswell.
I'd integrate this into 3.4 instead. mongodb is already in, and the value-pairs stuff is a clear win there.
tfjson and substr (anything I've missed) should go to 3.4. Hopefully that can be a faster release-cycle, as I don't intend to break everything as I did with 3.3. (you know converting a program from single to multi-threaded has a certain cost).
Having JSON support would be a big boost for those of us who want to do high speed integration with external apps without using stuff that involves slow regexes. I'd miss this feature if it didn't go in. Matthew.
Matthew Hall <mhall@mhcomputing.net> writes:
tfson() =======
For extra fun & profit, Balint Kovacs' format_json has been ported to use value-pairs(), and this will be the third patch in the series, and is also available on my integration/tfjson branch.
This one's the least important, though it's dead simple, therefore I'd be very happy if it made past the feature freeze aswell.
I'd integrate this into 3.4 instead. mongodb is already in, and the value-pairs stuff is a clear win there.
tfjson and substr (anything I've missed) should go to 3.4. Hopefully that can be a faster release-cycle, as I don't intend to break everything as I did with 3.3. (you know converting a program from single to multi-threaded has a certain cost).
Having JSON support would be a big boost for those of us who want to do high speed integration with external apps without using stuff that involves slow regexes.
I'd miss this feature if it didn't go in.
If it doesn't, that's not too bad, either, in my opinion: as long as the libsyslog-ng headers & whatnot are installed (which 3.3 is supposed to install, I think), I can easily turn tfjson into an external module until 3.4 is released as stable with tfjson included. Not as convenient as having it in the core, but with a little bit of work on my end, it can be made to compile without patching. I'd think that's an acceptable compromise. (Come to think of it, I could even collect a bunch of other stuff like the $(substr) macro into this external module, so that people who want to use them just have to download a tarball, do ./configure && make && make install and be done with it - no patching required) -- |8]
On Mon, 2011-04-25 at 19:08 +0200, Gergely Nagy wrote:
Matthew Hall <mhall@mhcomputing.net> writes:
tfson() =======
For extra fun & profit, Balint Kovacs' format_json has been ported to use value-pairs(), and this will be the third patch in the series, and is also available on my integration/tfjson branch.
This one's the least important, though it's dead simple, therefore I'd be very happy if it made past the feature freeze aswell.
I'd integrate this into 3.4 instead. mongodb is already in, and the value-pairs stuff is a clear win there.
tfjson and substr (anything I've missed) should go to 3.4. Hopefully that can be a faster release-cycle, as I don't intend to break everything as I did with 3.3. (you know converting a program from single to multi-threaded has a certain cost).
Having JSON support would be a big boost for those of us who want to do high speed integration with external apps without using stuff that involves slow regexes.
I'd miss this feature if it didn't go in.
If it doesn't, that's not too bad, either, in my opinion: as long as the libsyslog-ng headers & whatnot are installed (which 3.3 is supposed to install, I think), I can easily turn tfjson into an external module until 3.4 is released as stable with tfjson included.
Not as convenient as having it in the core, but with a little bit of work on my end, it can be made to compile without patching. I'd think that's an acceptable compromise.
(Come to think of it, I could even collect a bunch of other stuff like the $(substr) macro into this external module, so that people who want to use them just have to download a tarball, do ./configure && make && make install and be done with it - no patching required)
Yeah, I was thinking creating a syslog-ng-modules repository, which perhaps with submodules could make it possible to delegate maintenance for certain modules to 3rd parties. -- Bazsi
Balazs Scheidler <bazsi@balabit.hu> writes:
(Come to think of it, I could even collect a bunch of other stuff like the $(substr) macro into this external module, so that people who want to use them just have to download a tarball, do ./configure && make && make install and be done with it - no patching required)
Yeah, I was thinking creating a syslog-ng-modules repository, which perhaps with submodules could make it possible to delegate maintenance for certain modules to 3rd parties.
Sounds like a great idea! If you don't mind, I'll go ahead and set up something similar tonight, as a means to test how well this idea would work in practice (and because it sounds like something easy I can play with tonight :). -- |8]
Gergely Nagy <algernon@balabit.hu> writes:
Balazs Scheidler <bazsi@balabit.hu> writes:
(Come to think of it, I could even collect a bunch of other stuff like the $(substr) macro into this external module, so that people who want to use them just have to download a tarball, do ./configure && make && make install and be done with it - no patching required)
Yeah, I was thinking creating a syslog-ng-modules repository, which perhaps with submodules could make it possible to delegate maintenance for certain modules to 3rd parties.
Sounds like a great idea!
If you don't mind, I'll go ahead and set up something similar tonight, as a means to test how well this idea would work in practice (and because it sounds like something easy I can play with tonight :).
Well, that "tonight" turned into something that'll take a bit longer. I ran into a few problems with some of the modules I wanted to include in the collection (afsmtp being one of them, which for some reason, refuses to compile; and tfjson being the other, which needs value-pairs.h, which is currently not installed by syslog-ng's make install). However, as a preview, here's the list of stuff I plan to include: * afsmtp: A reasonably simple SMTP destination, based on libesmtp * tfjson: $(format-json) template function * afmongodb: My development version of afmongodb, which at the moment is the same as the patch I sent to this list a couple of days ago. * tf-strings: Various string related template functions, like $(substr) and $(length). * tf-numeric: Various random numeric functions, like $(+ A B), $(- A B), $(* A B), $(/ A B), and $(% A B). The names of these are up to debate, only $(%) is implemented so far (and it's called $(modN) on one of my branches). I was also thinking about adding a couple of more template functions, to split a string into tokens, $(foreach) and a couple of other things, perhaps. Maybe even $(defvar), just for kicks! But I've yet to find a use-case for those. -- |8]
participants (3)
-
Balazs Scheidler
-
Gergely Nagy
-
Matthew Hall