[PATCH 0/7] value-pairs dot notation walker
The patches that will follow this overview soon implement a 'walker' for value-pairs, a way to walk the internal flat structure syslog-ng uses as if it was a tree. This is important for two modules in particular: $(format-json) and the mongodb destination. The latter already supportes storing structured data, but it is using upserts (with all its downsides) to accomplish that. The former - up until now - was only able to format flat JSON. With these patches, both change: $(format-json) becomes able to emit structured data, and the mongodb destination starts to use inserts. The patches also merge the tfjson and jsonparser modules into a single json module, and implements our own JSON formatting for $(format-json). This means that json-glib support is completely removed, as it became unnecessary, and json-c is only used by the parser, but for practical reasons, required to build the plugin. That is, it is no longer possible to build $(format-json) only, but not json-parser, nor the other way around. The final patch is a bonus, that corrects a small error in jsonparser, and stops it from emitting an assertion warning on stderr in certain cases.
There is no point in supporting two json formatters anymore, so drop support for json-glib. Signed-off-by: Gergely Nagy <algernon@balabit.hu> --- configure.in | 56 +++++------------------------------------------ modules/tfjson/tfjson.c | 52 ------------------------------------------- 2 files changed, 6 insertions(+), 102 deletions(-) diff --git a/configure.in b/configure.in index 5c00e0f..8991835 100644 --- a/configure.in +++ b/configure.in @@ -26,7 +26,6 @@ OPENSSL_MIN_VERSION="0.9.8" LIBDBI_MIN_VERSION="0.8.0" IVYKIS_MIN_VERSION="0.30.1" JSON_C_MIN_VERSION="0.9" -JSON_GLIB_MIN_VERSION="0.12" PCRE_MIN_VERSION="6.1" LMC_MIN_VERSION="0.1.6" @@ -169,14 +168,9 @@ AC_ARG_WITH(ivykis, Link against the system supplied or the built-in ivykis library.] ,,with_ivykis="internal") -AC_ARG_WITH(json, - [ --with-json=[json-c/json-glib/auto] - Use the JSON implementation specified] - ,,with_json="auto") - AC_ARG_ENABLE(json, [ --disable-json Disable JSON support (default: auto)], - [case "${enableval}" in yes) with_json="auto";; *) with_json="${enableval}";; esac],with_json="auto") + ,,enable_json="auto") AC_ARG_ENABLE(smtp, [ --disable-smtp Disable SMTP support (default: auto)], @@ -616,12 +610,7 @@ fi dnl *************************************************************************** dnl json headers/libraries dnl *************************************************************************** -if test "x$with_json" = "xauto" || test "x$with_json" = "xjson-c"; then - PKG_CHECK_MODULES(JSON_C, json >= $JSON_C_MIN_VERSION,, JSON_C_LIBS="") -fi -if test "x$with_json" = "xauto" || test "x$with_json" = "xjson-glib"; then - PKG_CHECK_MODULES(JSON_GLIB, json-glib-1.0 >= $JSON_GLIB_MIN_VERSION,, JSON_GLIB_LIBS="") -fi +PKG_CHECK_MODULES(JSON_C, json >= $JSON_C_MIN_VERSION,, [JSON_C_LIBS=""; enable_json="no"]) dnl *************************************************************************** dnl pcre headers/libraries @@ -900,45 +889,12 @@ if test "x$enable_mongodb" = "xauto"; then AC_MSG_RESULT([$enable_mongodb]) fi -if test "x$with_json" != "xjson-glib" -a "x$with_json" != "xjson-c" -a "x$with_json" != "xauto" -a "x$with_json" != "xno"; then - # unknown --with-json argument - - AC_MSG_ERROR([Unknown JSON implementation ($with_json) specified, only json-glib, json-c and auto is known]) -elif test \( "x$with_json" = "xauto" -o "x$with_json" = "xjson-c" \) -a -n "$JSON_C_LIBS" ; then - # json-c found and user requested json-c or auto - +if test "x$enable_json" != "xno"; then JSON_LIBS=$JSON_C_LIBS JSON_CFLAGS=$JSON_C_CFLAGS - AC_DEFINE_UNQUOTED(HAVE_JSON_C, 1, [Have json-c]) - with_json="json-c" -elif test \( "x$with_json" = "xauto" -o "x$with_json" = "xjson-glib" \) -a -n "$JSON_GLIB_LIBS"; then - # json-glib found and user requested json-glib or auto - - JSON_LIBS=$JSON_GLIB_LIBS - JSON_CFLAGS=$JSON_GLIB_CFLAGS - AC_DEFINE_UNQUOTED(HAVE_JSON_GLIB, 1, [Have json-glib]) - with_json="json-glib" - -elif test "x$with_json" = "xauto" -o "x$with_json" = "xno"; then - # nothing found, but autodetect was selected - - JSON_LIBS="" - JSON_CFLAGS="" - with_json="no" -elif test "x$with_json" = "xjson-glib"; then - AC_MSG_ERROR([Cannot find json-glib-1.0 >= $JSON_GLIB_MIN_VERSION: is pkg-config in path?]) -elif test "x$with_json" = "xjson-c"; then - AC_MSG_ERROR([Cannot find json-c version >= $JSON_C_MIN_VERSION: is pkg-config in path?]) -fi - -if test "x$with_json" = "xjson-c"; then - enable_json_parse="yes" - enable_json_format="yes" -fi -if test "x$with_json" = "xjson-glib"; then - enable_json_parse="no" - enable_json_format="yes" + enable_json_parse="yes" + enable_json_format="yes" fi if test "x$enable_systemd" = "xauto"; then @@ -1237,6 +1193,6 @@ echo " SSL support (module) : ${enable_ssl:=no}" echo " SQL support (module) : ${enable_sql:=no}" echo " PACCT module (EXPERIMENTAL) : ${enable_pacct:=no}" echo " MongoDB destination (module): ${enable_mongodb:=no}" -echo " JSON support (module) : parser=${enable_json_parse:=no}, formatter=${enable_json_format:=no} (using ${with_json})" +echo " JSON support (module) : parser=${enable_json_parse:=no}, formatter=${enable_json_format:=no}" echo " SMTP support (module) : ${enable_smtp:=no}" diff --git a/modules/tfjson/tfjson.c b/modules/tfjson/tfjson.c index 16c9d4a..8eb1d42 100644 --- a/modules/tfjson/tfjson.c +++ b/modules/tfjson/tfjson.c @@ -25,17 +25,9 @@ #include "cfg.h" #include "value-pairs.h" -#include "config.h" - -#ifdef HAVE_JSON_C #include <printbuf.h> #include <json.h> #include <json_object_private.h> -#endif - -#ifdef HAVE_JSON_GLIB -#include <json-glib/json-glib.h> -#endif typedef struct _TFJsonState { @@ -57,7 +49,6 @@ tf_json_prepare(LogTemplateFunction *self, gpointer s, LogTemplate *parent, return TRUE; } -#if HAVE_JSON_C static int tf_json_object_to_string (struct json_object *jso, struct printbuf *pb) @@ -112,45 +103,6 @@ tf_json_append(GString *result, ValuePairs *vp, LogMessage *msg) g_string_append(result, json_object_to_json_string (json)); json_object_put(json); } -#endif - -#if HAVE_JSON_GLIB -static gboolean -tf_json_foreach (const gchar *name, const gchar *value, gpointer user_data) -{ - JsonBuilder *builder = (JsonBuilder *)user_data; - - json_builder_set_member_name(builder, name); - json_builder_add_string_value(builder, value); - - return FALSE; -} - -static void -tf_json_append(GString *result, ValuePairs *vp, LogMessage *msg) -{ - JsonBuilder *builder; - JsonGenerator *gen; - gchar *str; - - builder = json_builder_new(); - json_builder_begin_object(builder); - - value_pairs_foreach(vp, tf_json_foreach, msg, 0, builder); - - json_builder_end_object(builder); - - gen = json_generator_new(); - json_generator_set_root(gen, json_builder_get_root(builder)); - str = json_generator_to_data(gen, NULL); - - g_object_unref(gen); - g_object_unref(builder); - - g_string_append(result, str); - g_free(str); -} -#endif static void tf_json_call(LogTemplateFunction *self, gpointer s, @@ -184,10 +136,6 @@ static Plugin builtin_tmpl_func_plugins[] = gboolean tfjson_module_init(GlobalConfig *cfg, CfgArgs *args) { -#ifdef HAVE_JSON_GLIB - g_type_init (); -#endif - plugin_register(cfg, builtin_tmpl_func_plugins, G_N_ELEMENTS(builtin_tmpl_func_plugins)); return TRUE; } -- 1.7.10.4
The reason tfjson and jsonparser were separate is because the former supported two json libs, and they could be compiled separately. Now that tfjson dropped json-glib support, the separation became pointless, so this patch merges the two together. Signed-off-by: Gergely Nagy <algernon@balabit.hu> --- configure.in | 14 ++---- modules/Makefile.am | 2 +- modules/{jsonparser => json}/Makefile.am | 26 ++++++---- .../jsonparser.h => json/json-plugin.h} | 10 ++-- modules/{jsonparser => json}/jsonparser-grammar.ym | 2 +- modules/{jsonparser => json}/jsonparser-parser.c | 2 +- modules/{jsonparser => json}/jsonparser-parser.h | 0 modules/{jsonparser => json}/jsonparser.c | 2 +- modules/{tfjson => json}/tests/Makefile.am | 4 +- modules/{tfjson => json}/tests/test_json.c | 2 +- modules/{tfjson => json}/tfjson.c | 21 +++++--- modules/jsonparser/jsonparser-plugin.c | 53 -------------------- modules/tfjson/Makefile.am | 13 ----- 13 files changed, 47 insertions(+), 104 deletions(-) rename modules/{jsonparser => json}/Makefile.am (50%) rename modules/{jsonparser/jsonparser.h => json/json-plugin.h} (83%) rename modules/{jsonparser => json}/jsonparser-grammar.ym (98%) rename modules/{jsonparser => json}/jsonparser-parser.c (98%) rename modules/{jsonparser => json}/jsonparser-parser.h (100%) rename modules/{jsonparser => json}/jsonparser.c (99%) rename modules/{tfjson => json}/tests/Makefile.am (76%) rename modules/{tfjson => json}/tests/test_json.c (93%) rename modules/{tfjson => json}/tfjson.c (87%) delete mode 100644 modules/jsonparser/jsonparser-plugin.c delete mode 100644 modules/tfjson/Makefile.am diff --git a/configure.in b/configure.in index 8991835..a23c6e8 100644 --- a/configure.in +++ b/configure.in @@ -892,9 +892,7 @@ fi if test "x$enable_json" != "xno"; then JSON_LIBS=$JSON_C_LIBS JSON_CFLAGS=$JSON_C_CFLAGS - - enable_json_parse="yes" - enable_json_format="yes" + enable_json="yes" fi if test "x$enable_systemd" = "xauto"; then @@ -1070,8 +1068,7 @@ AM_CONDITIONAL(ENABLE_SUN_STREAMS, [test "$enable_sun_streams" = "yes"]) AM_CONDITIONAL(ENABLE_PACCT, [test "$enable_pacct" = "yes"]) AM_CONDITIONAL(ENABLE_MONGODB, [test "$enable_mongodb" = "yes"]) AM_CONDITIONAL(ENABLE_SMTP, [test "$enable_smtp" = "yes"]) -AM_CONDITIONAL(ENABLE_JSON_FORMAT, [test "$enable_json_format" = "yes"]) -AM_CONDITIONAL(ENABLE_JSON_PARSE, [test "$enable_json_parse" = "yes"]) +AM_CONDITIONAL(ENABLE_JSON, [test "$enable_json" = "yes"]) AM_CONDITIONAL(WITH_LIBSYSTEMD, [test "$with_libsystemd" = "yes"]) # substitution into manual pages @@ -1147,9 +1144,8 @@ AC_OUTPUT(dist.conf modules/basicfuncs/tests/Makefile modules/cryptofuncs/Makefile modules/cryptofuncs/tests/Makefile - modules/tfjson/Makefile - modules/tfjson/tests/Makefile - modules/jsonparser/Makefile + modules/json/Makefile + modules/json/tests/Makefile scripts/Makefile scripts/update-patterndb doc/Makefile @@ -1193,6 +1189,6 @@ echo " SSL support (module) : ${enable_ssl:=no}" echo " SQL support (module) : ${enable_sql:=no}" echo " PACCT module (EXPERIMENTAL) : ${enable_pacct:=no}" echo " MongoDB destination (module): ${enable_mongodb:=no}" -echo " JSON support (module) : parser=${enable_json_parse:=no}, formatter=${enable_json_format:=no}" +echo " JSON support (module) : ${enable_json:=no}" echo " SMTP support (module) : ${enable_smtp:=no}" diff --git a/modules/Makefile.am b/modules/Makefile.am index eb9548d..8f9d63b 100644 --- a/modules/Makefile.am +++ b/modules/Makefile.am @@ -1 +1 @@ -SUBDIRS = afsocket afsql afstreams affile afprog afuser afmongodb afsmtp csvparser confgen system-source syslogformat pacctformat basicfuncs cryptofuncs dbparser tfjson jsonparser +SUBDIRS = afsocket afsql afstreams affile afprog afuser afmongodb afsmtp csvparser confgen system-source syslogformat pacctformat basicfuncs cryptofuncs dbparser json diff --git a/modules/jsonparser/Makefile.am b/modules/json/Makefile.am similarity index 50% rename from modules/jsonparser/Makefile.am rename to modules/json/Makefile.am index 0925f98..4960ca4 100644 --- a/modules/jsonparser/Makefile.am +++ b/modules/json/Makefile.am @@ -1,22 +1,26 @@ +SUBDIRS = tests + moduledir = @moduledir@ -AM_CPPFLAGS = -I$(top_srcdir)/lib -I../../lib export top_srcdir -if ENABLE_JSON_PARSE -module_LTLIBRARIES := libjsonparser.la -libjsonparser_la_SOURCES = \ - jsonparser.c jsonparser.h \ +if ENABLE_JSON +AM_CPPFLAGS = -I$(top_srcdir)/lib -I../../lib +module_LTLIBRARIES = libjson-plugin.la + +libjson_plugin_la_SOURCES = \ + tfjson.c \ + jsonparser.c \ jsonparser-grammar.y \ jsonparser-parser.c jsonparser-parser.h \ - jsonparser-plugin.c + json-plugin.h -libjsonparser_la_CPPFLAGS = $(AM_CPPFLAGS) -libjsonparser_la_CFLAGS = $(JSON_CFLAGS) -libjsonparser_la_LIBADD = $(MODULE_DEPS_LIBS) -libjsonparser_la_LDFLAGS = $(MODULE_LDFLAGS) $(JSON_LIBS) +libjson_plugin_la_CPPFLAGS = $(AM_CPPFLAGS) +libjson_plugin_la_CFLAGS = $(JSON_CFLAGS) +libjson_plugin_la_LIBADD = $(MODULE_DEPS_LIBS) $(JSON_LIBS) +libjson_plugin_la_LDFLAGS = $(MODULE_LDFLAGS) +endif BUILT_SOURCES = jsonparser-grammar.y jsonparser-grammar.c jsonparser-grammar.h EXTRA_DIST = $(BUILT_SOURCES) jsonparser-grammar.ym -endif include $(top_srcdir)/build/lex-rules.am diff --git a/modules/jsonparser/jsonparser.h b/modules/json/json-plugin.h similarity index 83% rename from modules/jsonparser/jsonparser.h rename to modules/json/json-plugin.h index 15fd27f..e8add05 100644 --- a/modules/jsonparser/jsonparser.h +++ b/modules/json/json-plugin.h @@ -1,6 +1,6 @@ /* - * Copyright (c) 2011 BalaBit IT Ltd, Budapest, Hungary - * Copyright (c) 2011 Gergely Nagy <algernon@balabit.hu> + * Copyright (c) 2011-2012 BalaBit IT Ltd, Budapest, Hungary + * Copyright (c) 2011-2012 Gergely Nagy <algernon@balabit.hu> * * This program is free software; you can redistribute it and/or modify it * under the terms of the GNU General Public License version 2 as published @@ -20,8 +20,8 @@ * COPYING for details. */ -#ifndef JSONPARSER_H_INCLUDED -#define JSONPARSER_H_INCLUDED +#ifndef JSON_PLUGIN_H_INCLUDED +#define JSON_PLUGIN_H_INCLUDED #include "logparser.h" @@ -31,4 +31,6 @@ void log_json_parser_set_prefix(LogParser *p, const gchar *prefix); void log_json_parser_set_marker(LogParser *p, const gchar *marker); LogJSONParser *log_json_parser_new(void); +gboolean tfjson_module_init(GlobalConfig *cfg); + #endif diff --git a/modules/jsonparser/jsonparser-grammar.ym b/modules/json/jsonparser-grammar.ym similarity index 98% rename from modules/jsonparser/jsonparser-grammar.ym rename to modules/json/jsonparser-grammar.ym index cc2830a..e78b8e4 100644 --- a/modules/jsonparser/jsonparser-grammar.ym +++ b/modules/json/jsonparser-grammar.ym @@ -28,7 +28,7 @@ %code { -#include "jsonparser.h" +#include "json-plugin.h" #include "cfg-parser.h" #include "jsonparser-grammar.h" #include "syslog-names.h" diff --git a/modules/jsonparser/jsonparser-parser.c b/modules/json/jsonparser-parser.c similarity index 98% rename from modules/jsonparser/jsonparser-parser.c rename to modules/json/jsonparser-parser.c index 6a88040..e9dacaf 100644 --- a/modules/jsonparser/jsonparser-parser.c +++ b/modules/json/jsonparser-parser.c @@ -20,7 +20,7 @@ * COPYING for details. */ -#include "jsonparser.h" +#include "json-plugin.h" #include "cfg-parser.h" #include "jsonparser-grammar.h" diff --git a/modules/jsonparser/jsonparser-parser.h b/modules/json/jsonparser-parser.h similarity index 100% rename from modules/jsonparser/jsonparser-parser.h rename to modules/json/jsonparser-parser.h diff --git a/modules/jsonparser/jsonparser.c b/modules/json/jsonparser.c similarity index 99% rename from modules/jsonparser/jsonparser.c rename to modules/json/jsonparser.c index 3fb2531..21b605d 100644 --- a/modules/jsonparser/jsonparser.c +++ b/modules/json/jsonparser.c @@ -20,7 +20,7 @@ * COPYING for details. */ -#include "jsonparser.h" +#include "json-plugin.h" #include "logparser.h" #include "scratch-buffers.h" diff --git a/modules/tfjson/tests/Makefile.am b/modules/json/tests/Makefile.am similarity index 76% rename from modules/tfjson/tests/Makefile.am rename to modules/json/tests/Makefile.am index a184b69..2424a73 100644 --- a/modules/tfjson/tests/Makefile.am +++ b/modules/json/tests/Makefile.am @@ -1,5 +1,5 @@ -AM_CFLAGS = -I$(top_srcdir)/lib -I../../../lib -I$(top_srcdir)/libtest -I../../../libtest -I$(top_srcdir)/modules/tfjson -I.. -AM_LDFLAGS = -dlpreopen ../../syslogformat/libsyslogformat.la -dlpreopen ../libtfjson.la +AM_CFLAGS = -I$(top_srcdir)/lib -I../../../lib -I$(top_srcdir)/libtest -I../../../libtest -I$(top_srcdir)/modules/json -I.. +AM_LDFLAGS = -dlpreopen ../../syslogformat/libsyslogformat.la -dlpreopen ../libjson-plugin.la LDADD = $(top_builddir)/lib/libsyslog-ng.la $(top_builddir)/libtest/libsyslog-ng-test.a @TOOL_DEPS_LIBS@ check_PROGRAMS = test_json diff --git a/modules/tfjson/tests/test_json.c b/modules/json/tests/test_json.c similarity index 93% rename from modules/tfjson/tests/test_json.c rename to modules/json/tests/test_json.c index bd18203..582b264 100644 --- a/modules/tfjson/tests/test_json.c +++ b/modules/json/tests/test_json.c @@ -15,7 +15,7 @@ main(int argc G_GNUC_UNUSED, char *argv[] G_GNUC_UNUSED) { app_startup(); init_template_tests(); - plugin_load_module("tfjson", configuration, NULL); + plugin_load_module("json-plugin", configuration, NULL); test_format_json(); diff --git a/modules/tfjson/tfjson.c b/modules/json/tfjson.c similarity index 87% rename from modules/tfjson/tfjson.c rename to modules/json/tfjson.c index 8eb1d42..8716559 100644 --- a/modules/tfjson/tfjson.c +++ b/modules/json/tfjson.c @@ -128,24 +128,31 @@ tf_json_free_state(gpointer s) TEMPLATE_FUNCTION(TFJsonState, tf_json, tf_json_prepare, NULL, tf_json_call, tf_json_free_state, NULL); -static Plugin builtin_tmpl_func_plugins[] = +extern CfgParser jsonparser_parser; + +static Plugin json_plugins[] = { + { + .type = LL_CONTEXT_PARSER, + .name = "json-parser", + .parser = &jsonparser_parser, + }, TEMPLATE_FUNCTION_PLUGIN(tf_json, "format_json"), }; gboolean -tfjson_module_init(GlobalConfig *cfg, CfgArgs *args) +json_module_init(GlobalConfig *cfg, CfgArgs *args) { - plugin_register(cfg, builtin_tmpl_func_plugins, G_N_ELEMENTS(builtin_tmpl_func_plugins)); + plugin_register(cfg, json_plugins, G_N_ELEMENTS(json_plugins)); return TRUE; } const ModuleInfo module_info = { - .canonical_name = "tfjson", + .canonical_name = "json", .version = VERSION, - .description = "The tfjson module provides a JSON formatting template function for syslog-ng.", + .description = "The json module provides JSON parsing & formatting support for syslog-ng.", .core_revision = SOURCE_REVISION, - .plugins = builtin_tmpl_func_plugins, - .plugins_len = G_N_ELEMENTS(builtin_tmpl_func_plugins), + .plugins = json_plugins, + .plugins_len = G_N_ELEMENTS(json_plugins), }; diff --git a/modules/jsonparser/jsonparser-plugin.c b/modules/jsonparser/jsonparser-plugin.c deleted file mode 100644 index 15c4c91..0000000 --- a/modules/jsonparser/jsonparser-plugin.c +++ /dev/null @@ -1,53 +0,0 @@ -/* - * Copyright (c) 2011 BalaBit IT Ltd, Budapest, Hungary - * Copyright (c) 2011 Gergely Nagy <algernon@balabit.hu> - * - * This program is free software; you can redistribute it and/or modify it - * under the terms of the GNU General Public License version 2 as published - * by the Free Software Foundation, or (at your option) any later version. - * - * This library is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * Lesser General Public License for more details. - * - * You should have received a copy of the GNU Lesser General Public - * License along with this library; if not, write to the Free Software - * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA - * - * As an additional exemption you are allowed to compile & link against the - * OpenSSL libraries as published by the OpenSSL project. See the file - * COPYING for details. - */ - -#include "cfg-parser.h" -#include "plugin.h" -#include "jsonparser.h" - -extern CfgParser jsonparser_parser; - -static Plugin jsonparser_plugins[] = -{ - { - .type = LL_CONTEXT_PARSER, - .name = "json-parser", - .parser = &jsonparser_parser, - }, -}; - -gboolean -jsonparser_module_init(GlobalConfig *cfg, CfgArgs *args) -{ - plugin_register(cfg, jsonparser_plugins, G_N_ELEMENTS(jsonparser_plugins)); - return TRUE; -} - -const ModuleInfo module_info = -{ - .canonical_name = "jsonparser", - .version = VERSION, - .description = "The jsonparser module provides JSON parsing support for syslog-ng.", - .core_revision = SOURCE_REVISION, - .plugins = jsonparser_plugins, - .plugins_len = G_N_ELEMENTS(jsonparser_plugins), -}; diff --git a/modules/tfjson/Makefile.am b/modules/tfjson/Makefile.am deleted file mode 100644 index eecbf70..0000000 --- a/modules/tfjson/Makefile.am +++ /dev/null @@ -1,13 +0,0 @@ -SUBDIRS = tests - -moduledir = @moduledir@ -export top_srcdir - -if ENABLE_JSON_FORMAT -AM_CPPFLAGS = -I$(top_srcdir)/lib -I../../lib $(JSON_CFLAGS) -module_LTLIBRARIES = libtfjson.la - -libtfjson_la_SOURCES = tfjson.c -libtfjson_la_LIBADD = $(MODULE_DEPS_LIBS) $(JSON_LIBS) -libtfjson_la_LDFLAGS = $(MODULE_LDFLAGS) -endif -- 1.7.10.4
This new function is meant to be used when one wants to process the result of value-pairs() in a structured manner. It will treat the keys as if they were in dotted-notation, and whenever a new object starts, it will call obj_start_func, whenever an object ends (a new one starts, or we're winding down in the end), it calls obj_end_func. When processing a single key, process_value_func will be called. The obj_start_func and obj_end_func callbacks take six arguments: the current key name, the prefix so far, a pointer to an area where prefix-specific data can be stored, the previous element's name and data on the stack, and the user_data passed to value_pairs_walk(). The process_value_func receives the value too, see the VPWalk* typedefs in value-pairs.h. The prefix-specific data can be NULL, so care must be taken not to dereference it in that case. With this functionality, we will be able to construct structured data out of the flat representation of value-pairs. Signed-off-by: Gergely Nagy <algernon@balabit.hu> --- lib/value-pairs.c | 197 +++++++++++++++++++++++++++++++++++++++++++++++++++++ lib/value-pairs.h | 15 ++++ 2 files changed, 212 insertions(+) diff --git a/lib/value-pairs.c b/lib/value-pairs.c index 91894c8..69563c4 100644 --- a/lib/value-pairs.c +++ b/lib/value-pairs.c @@ -340,6 +340,203 @@ value_pairs_foreach(ValuePairs *vp, VPForeachFunc func, msg, seq_num, user_data); } +typedef struct +{ + GTrashStack stackp; + gchar *key; + gchar *prefix; + gint prefix_len; + + gpointer data; +} vp_walk_stack_t; + +typedef struct +{ + VPWalkCallbackFunc obj_start; + VPWalkCallbackFunc obj_end; + VPWalkValueCallbackFunc process_value; + + gpointer user_data; + vp_walk_stack_t *stack; +} vp_walk_state_t; + +static vp_walk_stack_t * +vp_walker_stack_unwind_until (vp_walk_stack_t **stack, vp_walk_state_t *state, + const gchar *name) +{ + vp_walk_stack_t *t; + + if (!stack) + return NULL; + if (!*stack) + return NULL; + if (strncmp(name, (*stack)->prefix, (*stack)->prefix_len) == 0) + return *stack; + + while ((t = g_trash_stack_pop((GTrashStack **)stack)) != NULL) + { + if (strncmp(name, t->prefix, t->prefix_len) != 0) + { + vp_walk_stack_t *p = g_trash_stack_peek((GTrashStack **)stack); + + if (p) + state->obj_end(t->key, t->prefix, &t->data, + p->prefix, &p->data, + state->user_data); + else + state->obj_end(t->key, t->prefix, &t->data, + NULL, NULL, + state->user_data); + g_free(t->key); + g_free(t->prefix); + g_free(t); + } + else + { + /* This one matched, put it back, PUT IT BACK! */ + g_trash_stack_push((GTrashStack **)stack, t); + break; + } + } + + return *stack; +} + +static void +vp_walker_stack_unwind_all(vp_walk_stack_t **stack, + vp_walk_state_t *state) +{ + vp_walk_stack_t *t; + + while ((t = g_trash_stack_pop((GTrashStack **)stack)) != NULL) + { + vp_walk_stack_t *p = g_trash_stack_peek((GTrashStack **)stack); + + if (p) + state->obj_end(t->key, t->prefix, &t->data, + p->prefix, &p->data, + state->user_data); + else + state->obj_end(t->key, t->prefix, &t->data, + NULL, NULL, + state->user_data); + + g_free(t->key); + g_free(t->prefix); + g_free(t); + } +} + +static vp_walk_stack_t * +vp_walker_stack_push (vp_walk_stack_t **stack, + gchar *key, gchar *prefix) +{ + vp_walk_stack_t *nt = g_new(vp_walk_stack_t, 1); + + nt->key = key; + nt->prefix = prefix; + nt->prefix_len = strlen(nt->prefix); + nt->data = NULL; + + g_trash_stack_push((GTrashStack **)stack, nt); + return nt; +} + +static gchar * +vp_walker_name_split(vp_walk_stack_t **stack, vp_walk_state_t *state, + const gchar *name) +{ + gchar **tokens, *key = NULL; + guint token_cnt, i, start; + + tokens = g_strsplit(name, ".", 0); + token_cnt = g_strv_length(tokens); + + start = g_trash_stack_height((GTrashStack **)stack); + for (i = start; i < token_cnt - 1; i++) + { + gchar *next = tokens[i + 1]; + vp_walk_stack_t *nt, *p = g_trash_stack_peek((GTrashStack **)stack); + + tokens[i + 1] = NULL; + nt = vp_walker_stack_push(stack, g_strdup(tokens[i]), + g_strjoinv(".", tokens)); + tokens[i + 1] = next; + + if (p) + state->obj_start(nt->key, nt->prefix, &nt->data, + p->prefix, &p->data, + state->user_data); + else + state->obj_start(nt->key, nt->prefix, &nt->data, + NULL, NULL, state->user_data); + } + + /* The last token is the key (well, second to last, last being + NULL), so treat that normally. */ + key = g_strdup(tokens[token_cnt - 1]); + g_strfreev(tokens); + + return key; +} + +static gboolean +value_pairs_walker(const gchar *name, const gchar *value, + gpointer user_data) +{ + vp_walk_state_t *state = (vp_walk_state_t *)user_data; + vp_walk_stack_t *st = g_trash_stack_peek((GTrashStack **)&state->stack); + gchar *key; + gboolean result; + + st = vp_walker_stack_unwind_until (&st, state, name); + + key = vp_walker_name_split (&st, state, name); + + if (st) + result = state->process_value(key, st->prefix, value, &st->data, + state->user_data); + else + result = state->process_value(key, NULL, value, NULL, + state->user_data); + + g_free(key); + + /* And the new stack becomes whatever we have in st now. */ + state->stack = st; + + return result; +} + +static gint +vp_walk_cmp(const gchar *s1, const gchar *s2) +{ + return strcmp(s2, s1); +} + +void +value_pairs_walk(ValuePairs *vp, + VPWalkCallbackFunc obj_start_func, + VPWalkValueCallbackFunc process_value_func, + VPWalkCallbackFunc obj_end_func, + LogMessage *msg, gint32 seq_num, + gpointer user_data) +{ + vp_walk_state_t state; + + state.user_data = user_data; + state.obj_start = obj_start_func; + state.obj_end = obj_end_func; + state.process_value = process_value_func; + state.stack = NULL; + + state.obj_start(NULL, NULL, NULL, NULL, NULL, user_data); + value_pairs_foreach_sorted(vp, value_pairs_walker, + (GCompareDataFunc)vp_walk_cmp, msg, seq_num, &state); + vp_walker_stack_unwind_all(&state.stack, &state); + state.obj_end(NULL, NULL, NULL, NULL, NULL, user_data); +} + static void value_pairs_init_set(ValuePairSpec *set) { diff --git a/lib/value-pairs.h b/lib/value-pairs.h index 1de97ca..ae84fd2 100644 --- a/lib/value-pairs.h +++ b/lib/value-pairs.h @@ -31,6 +31,14 @@ typedef struct _ValuePairs ValuePairs; typedef gboolean (*VPForeachFunc)(const gchar *name, const gchar *value, gpointer user_data); +typedef gboolean (*VPWalkValueCallbackFunc)(const gchar *name, const gchar *prefix, + const gchar *value, + gpointer *prefix_data, gpointer user_data); +typedef gboolean (*VPWalkCallbackFunc)(const gchar *name, + const gchar *prefix, gpointer *prefix_data, + const gchar *prev, gpointer *prev_data, + gpointer user_data); + gboolean value_pairs_add_scope(ValuePairs *vp, const gchar *scope); void value_pairs_add_glob_pattern(ValuePairs *vp, const gchar *pattern, gboolean include); void value_pairs_add_pair(ValuePairs *vp, GlobalConfig *cfg, const gchar *key, const gchar *value); @@ -45,6 +53,13 @@ void value_pairs_foreach(ValuePairs *vp, VPForeachFunc func, LogMessage *msg, gint32 seq_num, gpointer user_data); +void value_pairs_walk(ValuePairs *vp, + VPWalkCallbackFunc obj_start_func, + VPWalkValueCallbackFunc process_value_func, + VPWalkCallbackFunc obj_end_func, + LogMessage *msg, gint32 seq_num, + gpointer user_data); + ValuePairs *value_pairs_new(void); void value_pairs_free(ValuePairs *vp); -- 1.7.10.4
Now that we can process dotted-notation properly, make use of that! With this change, $(format-json) becomes smart enough to turn dotted-notation back into proper structures. It does not support arrays yet, however. Also added a test case that uses dotted-notation, and tests that it gets correctly formatted. Signed-off-by: Gergely Nagy <algernon@balabit.hu> --- modules/json/tests/test_json.c | 5 +++- modules/json/tfjson.c | 53 ++++++++++++++++++++++++++++++++++++---- 2 files changed, 52 insertions(+), 6 deletions(-) diff --git a/modules/json/tests/test_json.c b/modules/json/tests/test_json.c index 582b264..f188b7f 100644 --- a/modules/json/tests/test_json.c +++ b/modules/json/tests/test_json.c @@ -7,7 +7,10 @@ test_format_json(void) { assert_template_format("$(format-json MSG=$MSG)", "{\"MSG\":\"árvíztűrőtükörfúrógép\"}"); assert_template_format_with_context("$(format-json MSG=$MSG)", "{\"MSG\":\"árvíztűrőtükörfúrógép\"}{\"MSG\":\"árvíztűrőtükörfúrógép\"}"); - assert_template_format("$(format-json --scope rfc3164)", "{\"DATE\":\"Feb 11 19:58:35\",\"FACILITY\":\"local3\",\"HOST\":\"bzorp\",\"MESSAGE\":\"árvíztűrőtükörfúrógép\",\"PID\":\"23323\",\"PRIORITY\":\"err\",\"PROGRAM\":\"syslog-ng\"}"); + assert_template_format("$(format-json --scope rfc3164)", "{\"PROGRAM\":\"syslog-ng\",\"PRIORITY\":\"err\",\"PID\":\"23323\",\"MESSAGE\":\"árvíztűrőtükörfúrógép\",\"HOST\":\"bzorp\",\"FACILITY\":\"local3\",\"DATE\":\"Feb 11 19:58:35\"}"); + assert_template_format("$(format-json msg.text=$MSG msg.id=42 host=bzorp)", "{\"msg\":{\"text\":\"árvíztűrőtükörfúrógép\",\"id\":\"42\"},\"host\":\"bzorp\"}"); + assert_template_format("$(format-json msg.text.str=$MSG msg.text.len=42 msg.id=42 host=bzorp)", + "{\"msg\":{\"text\":{\"str\":\"árvíztűrőtükörfúrógép\",\"len\":\"42\"},\"id\":\"42\"},\"host\":\"bzorp\"}"); } int diff --git a/modules/json/tfjson.c b/modules/json/tfjson.c index 8716559..c8a2472 100644 --- a/modules/json/tfjson.c +++ b/modules/json/tfjson.c @@ -1,7 +1,7 @@ /* - * Copyright (c) 2002-2011 BalaBit IT Ltd, Budapest, Hungary + * Copyright (c) 2011-2012 BalaBit IT Ltd, Budapest, Hungary * Copyright (c) 2011 Balint Kovacs <blint@balabit.hu> - * Copyright (c) 2011 Gergely Nagy <algernon@balabit.hu> + * Copyright (c) 2011-2012 Gergely Nagy <algernon@balabit.hu> * * This program is free software; you can redistribute it and/or modify it * under the terms of the GNU General Public License version 2 as published @@ -79,11 +79,52 @@ tf_json_object_to_string (struct json_object *jso, } static gboolean -tf_json_foreach (const gchar *name, const gchar *value, gpointer user_data) +tf_json_obj_start(const gchar *name, + const gchar *prefix, gpointer *prefix_data, + const gchar *prev, gpointer *prev_data, + gpointer user_data) { - struct json_object *root = (struct json_object *)user_data; + struct json_object *o; + + if (prefix_data) + { + o = json_object_new_object(); + *prefix_data = o; + o->_to_json_string = tf_json_object_to_string; + } + return FALSE; +} + +static gboolean +tf_json_obj_end(const gchar *name, + const gchar *prefix, gpointer *prefix_data, + const gchar *prev, gpointer *prev_data, + gpointer user_data) +{ + struct json_object *root; + + if (prev_data) + root = (struct json_object *)*prev_data; + else + root = (struct json_object *)user_data; + + if (prefix_data) + json_object_object_add (root, (gchar *) name, (struct json_object *)*prefix_data); + return FALSE; +} + +static gboolean +tf_json_value(const gchar *name, const gchar *prefix, const gchar *value, + gpointer *prefix_data, gpointer user_data) +{ + struct json_object *root; struct json_object *this; + if (prefix_data) + root = (struct json_object *)*prefix_data; + else + root = (struct json_object *)user_data; + this = json_object_new_string ((gchar *) value); json_object_object_add (root, (gchar *) name, this); @@ -98,7 +139,9 @@ tf_json_append(GString *result, ValuePairs *vp, LogMessage *msg) json = json_object_new_object(); json->_to_json_string = tf_json_object_to_string; - value_pairs_foreach(vp, tf_json_foreach, msg, 0, json); + value_pairs_walk(vp, + tf_json_obj_start, tf_json_value, tf_json_obj_end, + msg, 0, json); g_string_append(result, json_object_to_json_string (json)); json_object_put(json); -- 1.7.10.4
With the recent changes to value pairs, we can now turn our internal dotted notation into structures, so do that, and use inserts instead of upserts! This - at the moment - comes with a noticable performance cost, as we'll do extra work on the syslog-ng side (which was previously done on the server side). However, this paves the way for bulk inserts, which will improve performance a lot. (And making the value pairs walker more performant will also help, but that's independent of mongodb.) Signed-off-by: Gergely Nagy <algernon@balabit.hu> --- modules/afmongodb/afmongodb.c | 97 +++++++++++++++++++++++++++++------------ 1 file changed, 68 insertions(+), 29 deletions(-) diff --git a/modules/afmongodb/afmongodb.c b/modules/afmongodb/afmongodb.c index ad06dd4..205ab98 100644 --- a/modules/afmongodb/afmongodb.c +++ b/modules/afmongodb/afmongodb.c @@ -31,6 +31,7 @@ #include "stats.h" #include "nvtable.h" #include "logqueue.h" +#include "value-pairs.h" #include "mongo.h" @@ -86,7 +87,7 @@ typedef struct gchar *ns; GString *current_value; - bson *bson_sel, *bson_upd, *bson_set; + bson *bson; } MongoDBDestDriver; /* @@ -316,10 +317,56 @@ afmongodb_dd_connect(MongoDBDestDriver *self, gboolean reconnect) * Worker thread */ static gboolean -afmongodb_vp_foreach (const gchar *name, const gchar *value, - gpointer user_data) +afmongodb_vp_obj_start(const gchar *name, + const gchar *prefix, gpointer *prefix_data, + const gchar *prev, gpointer *prev_data, + gpointer user_data) { - bson *bson_set = (bson *)user_data; + bson *o; + + if (prefix_data) + { + o = bson_new(); + *prefix_data = o; + } + return FALSE; +} + +static gboolean +afmongodb_vp_obj_end(const gchar *name, + const gchar *prefix, gpointer *prefix_data, + const gchar *prev, gpointer *prev_data, + gpointer user_data) +{ + bson *root; + + if (prev_data) + root = (bson *)*prev_data; + else + root = (bson *)user_data; + + if (prefix_data) + { + bson *d = (bson *)*prefix_data; + + bson_finish (d); + bson_append_document (root, name, d); + bson_free (d); + } + return FALSE; +} + +static gboolean +afmongodb_vp_process_value(const gchar *name, const gchar *prefix, + const gchar *value, + gpointer *prefix_data, gpointer user_data) +{ + bson *o; + + if (prefix_data) + o = (bson *)*prefix_data; + else + o = (bson *)user_data; if (name[0] == '.') { @@ -328,10 +375,10 @@ afmongodb_vp_foreach (const gchar *name, const gchar *value, tx_name[0] = '_'; strncpy(&tx_name[1], name + 1, sizeof(tx_name) - 1); tx_name[sizeof(tx_name) - 1] = 0; - bson_append_string (bson_set, tx_name, value, -1); + bson_append_string (o, tx_name, value, -1); } else - bson_append_string (bson_set, name, value, -1); + bson_append_string (o, name, value, -1); return FALSE; } @@ -355,29 +402,25 @@ afmongodb_worker_insert (MongoDBDestDriver *self) msg_set_context(msg); - bson_reset (self->bson_sel); - bson_reset (self->bson_upd); - bson_reset (self->bson_set); + bson_reset (self->bson); oid = mongo_util_oid_new_with_time (self->last_msg_stamp, self->seq_num); - bson_append_oid (self->bson_sel, "_id", oid); + bson_append_oid (self->bson, "_id", oid); g_free (oid); - bson_finish (self->bson_sel); - value_pairs_foreach (self->vp, afmongodb_vp_foreach, - msg, self->seq_num, self->bson_set); + value_pairs_walk(self->vp, + afmongodb_vp_obj_start, + afmongodb_vp_process_value, + afmongodb_vp_obj_end, + msg, self->seq_num, self->bson); + bson_finish (self->bson); - bson_finish (self->bson_set); - - bson_append_document (self->bson_upd, "$set", self->bson_set); - bson_finish (self->bson_upd); - - if (!mongo_sync_cmd_update (self->conn, self->ns, MONGO_WIRE_FLAG_UPDATE_UPSERT, - self->bson_sel, self->bson_upd)) + if (!mongo_sync_cmd_insert_n(self->conn, self->ns, 1, + (const bson **)&self->bson)) { - msg_error ("Network error while inserting into MongoDB", - evt_tag_int("time_reopen", self->time_reopen), - NULL); + msg_error("Network error while inserting into MongoDB", + evt_tag_int("time_reopen", self->time_reopen), + NULL); success = FALSE; } @@ -416,9 +459,7 @@ afmongodb_worker_thread (gpointer arg) self->current_value = g_string_sized_new(256); - self->bson_sel = bson_new_sized(64); - self->bson_upd = bson_new_sized(512); - self->bson_set = bson_new_sized(512); + self->bson = bson_new_sized(4096); while (!self->writer_thread_terminate) { @@ -462,9 +503,7 @@ afmongodb_worker_thread (gpointer arg) g_free (self->ns); g_string_free (self->current_value, TRUE); - bson_free (self->bson_sel); - bson_free (self->bson_upd); - bson_free (self->bson_set); + bson_free (self->bson); msg_debug ("Worker thread finished", evt_tag_str("driver", self->super.super.id), -- 1.7.10.4
Instead of relying on json-c to format our JSON, do it ourselves, skipping quite a bit of object allocations that we do not need. Signed-off-by: Gergely Nagy <algernon@balabit.hu> --- modules/json/tfjson.c | 152 +++++++++++++++++++++++++++++++++---------------- 1 file changed, 103 insertions(+), 49 deletions(-) diff --git a/modules/json/tfjson.c b/modules/json/tfjson.c index c8a2472..e46cbd4 100644 --- a/modules/json/tfjson.c +++ b/modules/json/tfjson.c @@ -25,10 +25,6 @@ #include "cfg.h" #include "value-pairs.h" -#include <printbuf.h> -#include <json.h> -#include <json_object_private.h> - typedef struct _TFJsonState { TFSimpleFuncState super; @@ -49,33 +45,70 @@ tf_json_prepare(LogTemplateFunction *self, gpointer s, LogTemplate *parent, return TRUE; } -static int -tf_json_object_to_string (struct json_object *jso, - struct printbuf *pb) +typedef struct +{ + gboolean need_comma; + GString *buffer; +} json_state_t; + +static inline void +g_string_append_escaped(GString *dest, const char *str) { - int i = 0; - struct json_object_iter iter; - sprintbuf (pb, "{"); + /* Assumes ASCII! Keep in sync with the switch! */ + static const unsigned char json_exceptions[UCHAR_MAX + 1] = + { + [0x01] = 1, [0x02] = 1, [0x03] = 1, [0x04] = 1, [0x05] = 1, [0x06] = 1, + [0x07] = 1, [0x08] = 1, [0x09] = 1, [0x0a] = 1, [0x0b] = 1, [0x0c] = 1, + [0x0d] = 1, [0x0e] = 1, [0x0f] = 1, [0x10] = 1, [0x11] = 1, [0x12] = 1, + [0x13] = 1, [0x14] = 1, [0x15] = 1, [0x16] = 1, [0x17] = 1, [0x18] = 1, + [0x19] = 1, [0x1a] = 1, [0x1b] = 1, [0x1c] = 1, [0x1d] = 1, [0x1e] = 1, + [0x1f] = 1, ['\\'] = 1, ['"'] = 1 + }; + + const unsigned char *p; - json_object_object_foreachC (jso, iter) + p = (unsigned char *)str; + + while (*p) { - gchar *esc; - - if (i) - sprintbuf (pb, ","); - sprintbuf (pb, "\""); - esc = g_strescape (iter.key, NULL); - sprintbuf (pb, esc); - g_free (esc); - sprintbuf (pb, "\":"); - if (iter.val == NULL) - sprintbuf (pb, "null"); + if (json_exceptions[*p] == 0) + g_string_append_c(dest, *p); else - iter.val->_to_json_string (iter.val, pb); - i++; + { + /* Keep in sync with json_exceptions! */ + switch (*p) + { + case '\b': + g_string_append(dest, "\\b"); + break; + case '\n': + g_string_append(dest, "\\n"); + break; + case '\r': + g_string_append(dest, "\\r"); + break; + case '\t': + g_string_append(dest, "\\t"); + break; + case '\\': + g_string_append(dest, "\\\\"); + break; + case '"': + g_string_append(dest, "\\\""); + break; + default: + { + static const char json_hex_chars[16] = "0123456789abcdef"; + + g_string_append(dest, "\\u00"); + g_string_append_c(dest, json_hex_chars[(*p) >> 4]); + g_string_append_c(dest, json_hex_chars[(*p) & 0xf]); + break; + } + } + } + p++; } - - return sprintbuf(pb, "}"); } static gboolean @@ -84,14 +117,30 @@ tf_json_obj_start(const gchar *name, const gchar *prev, gpointer *prev_data, gpointer user_data) { - struct json_object *o; + json_state_t *state = (json_state_t *)user_data; + gboolean need_comma = FALSE; if (prefix_data) + need_comma = GPOINTER_TO_INT(*prefix_data); + else + need_comma = state->need_comma; + + if (need_comma) + g_string_append_c(state->buffer, ','); + + if (name) { - o = json_object_new_object(); - *prefix_data = o; - o->_to_json_string = tf_json_object_to_string; + g_string_append_c(state->buffer, '"'); + g_string_append_escaped(state->buffer, name); + g_string_append(state->buffer, "\":{"); + state->need_comma = TRUE; } + else + g_string_append_c(state->buffer, '{'); + + if (prefix_data) + *prefix_data=GINT_TO_POINTER(0); + return FALSE; } @@ -101,15 +150,13 @@ tf_json_obj_end(const gchar *name, const gchar *prev, gpointer *prev_data, gpointer user_data) { - struct json_object *root; + json_state_t *state = (json_state_t *)user_data; if (prev_data) - root = (struct json_object *)*prev_data; - else - root = (struct json_object *)user_data; + *prev_data = GINT_TO_POINTER(1); + + g_string_append_c(state->buffer, '}'); - if (prefix_data) - json_object_object_add (root, (gchar *) name, (struct json_object *)*prefix_data); return FALSE; } @@ -117,16 +164,26 @@ static gboolean tf_json_value(const gchar *name, const gchar *prefix, const gchar *value, gpointer *prefix_data, gpointer user_data) { - struct json_object *root; - struct json_object *this; + gboolean need_comma = FALSE; + json_state_t *state = (json_state_t *)user_data; if (prefix_data) - root = (struct json_object *)*prefix_data; + need_comma = GPOINTER_TO_INT(*prefix_data); else - root = (struct json_object *)user_data; + need_comma = state->need_comma; + + if (need_comma) + g_string_append_c(state->buffer, ','); + else if (prefix_data) + *prefix_data = GINT_TO_POINTER(1); - this = json_object_new_string ((gchar *) value); - json_object_object_add (root, (gchar *) name, this); + g_string_append_c(state->buffer, '"'); + g_string_append_escaped(state->buffer, name); + g_string_append(state->buffer, "\":\""); + g_string_append_escaped(state->buffer, value); + g_string_append_c(state->buffer, '"'); + + state->need_comma = TRUE; return FALSE; } @@ -134,17 +191,14 @@ tf_json_value(const gchar *name, const gchar *prefix, const gchar *value, static void tf_json_append(GString *result, ValuePairs *vp, LogMessage *msg) { - struct json_object *json; + json_state_t state; - json = json_object_new_object(); - json->_to_json_string = tf_json_object_to_string; + state.need_comma = FALSE; + state.buffer = result; value_pairs_walk(vp, tf_json_obj_start, tf_json_value, tf_json_obj_end, - msg, 0, json); - - g_string_append(result, json_object_to_json_string (json)); - json_object_put(json); + msg, 0, &state); } static void -- 1.7.10.4
In log_json_parser_process_single(), when parsing an object element, only prepend a prefix if there is one, otherwise g_string_assign() will complain on stderr with an assert, which is not something we'd like. Signed-off-by: Gergely Nagy <algernon@balabit.hu> --- modules/json/jsonparser.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/modules/json/jsonparser.c b/modules/json/jsonparser.c index 21b605d..d3c2b66 100644 --- a/modules/json/jsonparser.c +++ b/modules/json/jsonparser.c @@ -99,7 +99,8 @@ log_json_parser_process_single (struct json_object *jso, json_object_get_string (jso)); break; case json_type_object: - g_string_assign (sb_string (key), prefix); + if (prefix) + g_string_assign (sb_string (key), prefix); g_string_append (sb_string (key), obj_key); g_string_append_c (sb_string (key), '.'); log_json_parser_process_object (jso, sb_string (key)->str, msg); -- 1.7.10.4
Hi, applied all, thanks. I can see some potential performance issues though (lots of memory allocations in walking), that will probably be a bottleneck one day. ----- Original message -----
The patches that will follow this overview soon implement a 'walker' for value-pairs, a way to walk the internal flat structure syslog-ng uses as if it was a tree. This is important for two modules in particular: $(format-json) and the mongodb destination.
The latter already supportes storing structured data, but it is using upserts (with all its downsides) to accomplish that. The former - up until now - was only able to format flat JSON.
With these patches, both change: $(format-json) becomes able to emit structured data, and the mongodb destination starts to use inserts.
The patches also merge the tfjson and jsonparser modules into a single json module, and implements our own JSON formatting for $(format-json). This means that json-glib support is completely removed, as it became unnecessary, and json-c is only used by the parser, but for practical reasons, required to build the plugin. That is, it is no longer possible to build $(format-json) only, but not json-parser, nor the other way around.
The final patch is a bonus, that corrects a small error in jsonparser, and stops it from emitting an assertion warning on stderr in certain cases.
Balazs Scheidler <bazsi77@gmail.com> writes:
applied all, thanks. I can see some potential performance issues though (lots of memory allocations in walking), that will probably be a bottleneck one day.
Yep, they are noticable even now (esp. with format-json; for mongodb, using insert buys us much more performance than the walking takes). I have an idea how to reduce the number of allocations significantly, but I'm not happy with either of my PoC implementations yet. Most likely, I'll transform scratch-buffers into something more generic, but that needs careful thought to make it have a nice and performant API. -- |8]
participants (3)
-
Balazs Scheidler
-
Gergely Nagy
-
Gergely Nagy