- syslog-ng - lists.balabit.hu

tfjson: JSON-generating template function
by Gergely Nagy 05 Feb '11

05 Feb '11

Hi! Looks like the list didn't like me sending the whole patch, so I'll have to omit the patch part this time, and hope that the wall of text below won't hit the size limit. Based on the work of Balint Kovacs <blint(a)balabit.hu>, a $(format_json) template function was implemented: with this function, one can hammer log messages into JSON format. While that is not immediately useful, we have a few things planned that will - with a bit of luck - be able to use this function. The function itself, without arguments, will reformat the log message into JSON. For example, a loggen-generated message will look like this (formatting added by me): { "HOST": "localhost", "HOST_FROM": "localhost", "MESSAGE": "seq: 0000000000, thread: 0000, runid: 1295625842, stamp: 2011-01-21T17:04:02 PADD...", "PROGRAM": "prg00000", "PID": "1234", "SOURCE": "s_tcp", "LEGACY_MSGHDR": "prg00000[1234]: ", ".classifier.class": "unknown" } While this is all cool, there's more! The function accepts parameters: '--select <GLOB>' and '--exclude <GLOB>', where <GLOB> is a shell-like glob pattern. One can have any number of selects or globs. The way this works, is that if a key doesn't match any of the select globs, it will not be included. If it does, but matches any of the exclude globs, it will be dropped. Otherwise it's included. The default select glob is "*", which is why everything is included by default. The globs are matched against the keys: macro names without the dollar sign, patterndb keys, and so on. With this knowledge, we can tweak the template like this: $(format_json --select .classifier.* --select usracct.* --exclude *.*id) Then, given the following message: <86>Jan 21 17:10:21 moria sshd[31930]: pam_unix(sshd:session): session closed for user algernon We'll end up with this JSON: { ".classifier.class": "system", "usracct.username": "algernon", "usracct.type": "logout", "usracct.application": "sshd" } Now, that's not very useful, as we're lacking the HOST and a whole lot of other things. Adding all that as selects would be a bit excessive, therefore one can list macro names (without the dollar sign) along with the select/exclude options, and they'll be included too. Furthermore, one can list key=value pairs, for example, using the log message quite like the one above, and the following template: $(format_json -s .classifier.* -s usracct.* -x *.*id HOST MESSAGE prg='$PROGRAM[$PID]') We'll end up with: { ".classifier.class": "system", "usracct.username": "algernon", "usracct.type": "logout", "usracct.application": "sshd", "MESSAGE": "pam_unix(sshd:session): session closed for user algernon", "prg": "sshd[32216]", "HOST": "localhost" } At the moment, only strings are supported, and there is no way to structure the JSON properly yet. We'll figure out a way to do that too, eventually. There's a few more things in the pipeline for tfjson, so treat this version as a beta. The code is available on the integration/tfjson/base branch of my repository at git://git.madhouse-project.org/syslog-ng/syslog-ng-3.3.git Browsable online at: http://git.madhouse-project.org/syslog-ng/syslog-ng-3.3/tree/modules/tfjson… The bulk of the work was done by Balint, I just happened to accidentally refactor it a tiny bit while reviewing his code. -- |8]

3 4

connecting to mongodb
by Peter Czanik 03 Feb '11

03 Feb '11

Hello, When I try to use mongodb, I receive the following error message: Feb 2 15:33:54 linux-8k4b syslog-ng[9403]: syslog-ng starting up; version='3.3.0alpha0' Feb 2 15:33:54 linux-8k4b syslog-ng[9403]: Error connecting to MongoDB; Feb 2 15:33:54 linux-8k4b syslog-ng[9403]: Error connecting to MongoDB; Feb 2 15:33:54 linux-8k4b syslog-ng[9403]: Network error while inserting into MongoDB; time_reopen='60' The relevant part of my syslog-ng.conf is (taken from an earlier e-mail on the list): destination d_mongodb { mongodb( host("localhost") port(27017) database("syslog-ng") collection("logs") keys("date", "facility", "level", "host", "program", "pid", "message") values("${R_YEAR}-${R_MONTH}-${R_DAY} ${R_HOUR}:${R_MIN}:${R_SEC}", "$FACILITY", "$LEVEL", "$HOST","$PROGRAM", "$PID", "$MSGONLY") ); }; log { source(s_local); destination(d_mongodb); }; Version (latest git) compiled on openSUSE factory: linux-8k4b:/usr/local/etc # /usr/local/sbin/syslog-ng -V syslog-ng 3.3.0alpha0 Installer-Version: 3.3.0alpha0 Revision: Compile-Date: Feb 2 2011 14:27:08 Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-SSL: on Enable-SQL: on Enable-Linux-Caps: off Enable-Pcre: on Enable-Pacct: off linux-8k4b:/usr/local/etc # Using tcpdump I can't even see a connection attempt. Connecton using mongodb tools work fine. linux-8k4b:/usr/local/etc # mongo --version MongoDB shell version: 1.6.6-pre- But as packets don't even reach the mongodb server, it's most likely not a relevant information... Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

2 4

[RFC]: value-pairs()
by Gergely Nagy 02 Feb '11

02 Feb '11

Hi! The idea of a generic framework to set up value pairs came up first during the afmongodb discussion (see Bazsi's ideas at http://article.gmane.org/gmane.comp.syslog-ng/10432) and while working on the tfjson driver, it came up again. I've learnt a lot since the dynamic-variables implementation in afmongodb, and following Bazsi's comments on the list earlier, we managed to talk a little about the value-pairs() feature too. That, and another quick chat with Balint, followed by working on tfjson helped form the proposal below. I'll divide this into two sections: the first part, mostly aimed at people using syslog-ng in production would be a request for comments regarding the syntax of the feature. The second part will be a few notes on the planned implementation. The Syntax ========== value-pairs ( glob-select ("usracct.*") glob-exclude ("*.*id") builtins (no) $HOST $MESSAGE "program_n_pid" = "$PROGRAM[$PID]" ) What we have here, is basically three possible ways to define - or control - value pairs: * Using a function, like glob-select and glob-exclude or builtins (more about these later) * Macro names, with the '$' sign. * "key" = "value" pairs. The way this is supposed to work is that the base set is all the various key=value pairs syslog-ng collected about a particular log message: the built-in macros ($HOST, $MESSAGE, etc), sdata, patterndb - you name it. This is the set we can select or exclude from, the set we can add new pairs to (different values for each value-pairs() instance, of course). With glob-select(), we can mark a subset of the orignal set for inclusion. Subsequent glob-exclude()s can override this, and remove parts of it from the set. Specifying builtins(no) removes every built-in key from the set. Listing known keys, unquoted, '$' prefixed, will add that value to the set, with the key set to the macro name without the dollar sign. For example, $HOST is exactly the same as "HOST" = "$HOST". Finally, "key" = "value" pairs do the obvious thing: add custom pairs to our set. The order of functions does matter, so glob-select("*") glob-exclude("*id") glob-select("usracct.id") will select usracct.id, but no other key ending with "id". This is partly due to me finding this logical, and because if/when we add other functions later on (say, ltrim() or add-prefix() or similar that modify the keys), the order becomes essential. But, back to the purpose of this all: this would be templating on steroids, so to say. Drivers should be able to easily implement support for value-pairs(), which would make system administrators to tweak the way syslog-ng discovered data will appear in the output. Particularly, this functionality is what will make the afmongodb driver shine. This is a much more advanced version of the dynamic-variables() functionality. This will also come in handy for the JSON template function. I leave the rest up to your imagination. Feedback on the syntax would be most appreciated. The Design ========== Value pairs would be stored in a struct like the following: typedef struct { GPatternSpec *pattern; gboolean exclude; } ValuePairPattern; typedef struct { ValuePairPattern **patterns; gboolean builtins; NVRegistry extras_registry; NVTable extras; } ValuePairs; ValuePairs->patterns would be a NULL terminated array (we can allow for a lot of reallocs during init, imo - otherwise we can make it a GPtrArray or something similar to save time on that), where each pattern is either an exclude or a select pattern. The ->builtins boolean would flag whether we should include the builtin variables ($HOST, etc), and would default to TRUE. And ->extras are the extra keys specified explicitly: both in $MACRO and "key"="value" form (the former translated to "MACRO"="$MACRO" form). We'd have the following support API: ValuePairs *value_pairs_new (); void value_pairs_add_pattern (ValuePairs *vp, const gchar *pattern, gboolean exclude); void value_pairs_set_builtins (ValuePairs *vp, gboolean state); void value_pairs_add_extra (ValuePairs *vp, const gchar *key, const gchar *value); void value_pairs_foreach (ValuePairs *vp, VPForeachFunc *func, gpointer user_data); typedef gboolean (*VPForeachFunc) (const gchar *name, const gchar *value, gssize value_len, gpointer user_data); We'd also have a VALUE_PAIRS keyword in the parser (if it is possible, which I think it is), which drivers could use. The basic VALUE_PAIRS stuff would set up a ValuePairs structure, and store it in a global variable, which the driver could then either copy, or just assign it to it's own (since the main parser won't free or touch it anymore anyway). Then, all the drivers have to do, is call value_pairs_foreach(), which would firs iterate over logmsg_registry's keys, taking the specified patterns into account. Then it would iterate over it's own extras, and pass the results to the foreach function one by one, similar to how nv_table_foreach does. That's about it. It's a bit vague, mostly because I did not research if this is possible the way I imagine it yet. Hopefully this all made at least some sense. -- |8]

5 12

fosdem 2011
by Peter Czanik 02 Feb '11

02 Feb '11

Hello, FOSDEM, the largest European free software conference is coming this weekend: http://fosdem.org/2011/ I'll be there wearing Balabit and Syslog-ng T-shirts :-) I'll be mostly at the "distribution miniconf" and join talks about upstream -- downstream relations. If you want to talk abut syslog-ng (or Zorp) or just see somebody from the mailing list in person, I'm happy to meet you there! I'll have my smartbook with me and will try to read my e-mail regularly. Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

Re: [syslog-ng] Systemd support
by Gergely Nagy 02 Feb '11

02 Feb '11

Tom's message probably didn't make it to the list, due to the size of the attachments. But since I suppose more people are interested in systemd support, I'm CCing the list anyway. On Wed, 2011-02-02 at 02:15 +0100, Tom Gundersen wrote: > On Tue, Feb 1, 2011 at 10:19 PM, Balazs Scheidler <bazsi(a)balabit.hu> wrote: > > the suse maintainer has created a patch, and that branch is my preliminary > > integration, but it has changed somewhat since. it was sent in a private > > email, and unfortunately I simply had no chance to work on that. > > > > algernon(a)balabit.hu may have more information. > > It was not clear to me if the patch in your tree is the most > up-to-date one, but it appears to work fine for me. The patches you attached are not enough for proper systemd integration: if you have any unix-dgram() sockets that are not managed by systemd, syslog-ng will break. So will syslog-ng-ctl in certain cases. My tree has a fix for those issues, and to the best of my knowledge, works both on systemd-enabled systems, and non-systemd systems aswell. It's available (for 3.2) at git://git.madhouse-project.org/syslog-ng/syslog-ng-3.2.git On the 'systemd' branch. I'll prepare integration branches for pulling this all into the official tree during the next few days (both for 3.2 and 3.3). > Together, all of this works perfectly on my machine. Let me know if > you want help with testing of other patches (in case what I found was > not up-to-date). Looking forward to seeing systemd support in a > release :-) There's an extra few patches in my tree, on top of Marius Tomaschewski's systemd integration work, that wrap up the systemd support. More testing with those is always welcomed! (Especially since the testing I was able to conduct myself were pretty basic so far) -- |8]

1 0

2.6.38: CAP_SYSLOG
by Sergey Senozhatsky 01 Feb '11

01 Feb '11

Hello, During 2.6.38 development CAP_SYSLOG has been introduced to perform syslog operations, older CAP_SYS_ADMIN is not sufficient anymore. commit 38ef4c2e437d11b5922723504b62824e96761459 commit ce6ada35bdf710d16582cc4869c26722547e6f11 do_syslog now is as follows: int do_syslog(int type, char __user *buf, int len, bool from_file) { [..] if (type == SYSLOG_ACTION_OPEN || !from_file) { if (dmesg_restrict && !capable(CAP_SYSLOG)) goto warn; /* switch to return -EPERM after 2.6.39 */ if ((type != SYSLOG_ACTION_READ_ALL && type != SYSLOG_ACTION_SIZE_BUFFER) && !capable(CAP_SYSLOG)) goto warn; /* switch to return -EPERM after 2.6.39 */ } [..] CAP_SYSLOG introduced to libcap in version 2.20. #define CAP_SYSLOG 34 Sergey

2 3

[PATCH] adding JSON support to 3.2 (if allowable)
by Matthew Hall 01 Feb '11

01 Feb '11

Hello Balint, Gergely, Bazsi, and list, Here I present my backport of tfjson support to 3.2 for consideration of inclusion if this is allowd for that branch. The last send went bad because I am new to git format-patch. Sorry! Regards, Matthew Hall. Matthew Hall (1): tfjson: $(format_json) template function implementation. configure.in | 27 ++++- modules/Makefile.am | 3 +- modules/tfjson/Makefile.am | 11 ++ modules/tfjson/tfjson.c | 305 ++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 344 insertions(+), 2 deletions(-) create mode 100644 modules/tfjson/Makefile.am create mode 100644 modules/tfjson/tfjson.c

1 0

(no subject)
by mhall＠mhcomputing.net 01 Feb '11

01 Feb '11

>From 36ccc97227e1d4b8dcf9cea0ead0662851cb7732 Mon Sep 17 00:00:00 2001 From: Matthew Hall <mhall(a)mhcomputing.net> Date: Mon, 31 Jan 2011 17:43:28 -0800 Subject: [PATCH] tfjson: $(format_json) template function implementation. Implements a $(format_json) template function, largely based on the work of Balint Kovacs <blint(a)balabit.hu>, built on top of the json-c library. The syntax of the template function is as follows: $(format_json [--select <GLOB>] [--exclude <GLOB>] [--skip-builtins] [MACRO_NAMES...] KEY=VALUE) There can be multiple selects, excludes, macro names and key=value pairs. The VALUE part of the key=value pair can contain template macros. Macro names must be listed without the "$" sign, however. If --skip-builtins is specified, all built-in variables will be excluded. As an example, this is a valid argument list (without the line breaks): $(format_json --select .classification.* --select useracct.* --exclude *.*id HOST my_own_key='$SOURCEIP ($HOST)') This will include every key that begins with ".classification." or "useracct.", but will exclude any, that has a dot, and ends with "id". The latter keys will be excluded even if they match any of the select patterns. Furthermore, the $HOST macro will be included (with the key HOST), along with a my_own_key key, whose value will be evaluated as a template, thus, will contain the source IP and the HOST (again, for demo purposes). Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> (cherry picked from commit 9ab6b381433d657b59eb1d19f994f9328fca9547) Conflicts: configure.in modules/Makefile.am removed mongodb references from configure.in and modules/Makefile.am; resolved conflicts there Signed-off-by: Matthew Hall <mhall(a)mhcomputing.net> --- configure.in | 27 ++++- modules/Makefile.am | 3 +- modules/tfjson/Makefile.am | 11 ++ modules/tfjson/tfjson.c | 305 ++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 344 insertions(+), 2 deletions(-) create mode 100644 modules/tfjson/Makefile.am create mode 100644 modules/tfjson/tfjson.c diff --git /tfjson-base/configure.in /syslog-ng-3.2/configure.in new file mode 100644 index 0c6cef2..08d6a6b 100644 --- /tfjson-base/configure.in +++ /syslog-ng-3.2/configure.in @@ -25,6 +25,8 @@ EVTLOG_MIN_VERSION="0.2" OPENSSL_MIN_VERSION="0.9.8" LIBDBI_MIN_VERSION="0.8.0" PCRE_MIN_VERSION="7.3" +IVYKIS_MIN_VERSION="0.18" +JSON_C_MIN_VERSION="0.9" dnl *************************************************************************** dnl Initial setup @@ -144,6 +146,10 @@ AC_ARG_ENABLE(gcov, ,,enable_gcov="no") +AC_ARG_ENABLE(json, + [ --enable-json Enable support for JSON template formatting (default: auto)] + ,,enable_json="auto") + dnl *************************************************************************** dnl Checks for programs. AC_PROG_CC @@ -493,6 +499,23 @@ if test "x$linking_mode" != "xdynamic" -a "x$blb_cv_static_glib" = "xno"; then fi dnl *************************************************************************** +dnl json-glib headers/libraries +dnl *************************************************************************** +if test "x$enable_json" = "xyes" -o "x$enable_json" = "xauto"; then + PKG_CHECK_MODULES(JSON, json >= $JSON_C_MIN_VERSION,,JSON_LIBS="") + if test -z "$JSON_LIBS"; then + if test "x$enable_json" = "xyes"; then + AC_MSG_ERROR(Cannot find json >= $JSON_C_MIN_VERSION.) + else + AC_MSG_WARN(Cannot find json >= $JSON_C_MIN_VERSION.) + fi + enable_json="no" + else + enable_json="yes" + fi +fi + +dnl *************************************************************************** dnl pcre headers/libraries dnl *************************************************************************** @@ -750,6 +773,7 @@ AM_CONDITIONAL(ENABLE_SSL, [test "$enable_ssl" = "yes"]) AM_CONDITIONAL(ENABLE_SQL, [test "$enable_sql" = "yes"]) AM_CONDITIONAL(ENABLE_SUN_STREAMS, [test "$enable_sun_streams" = "yes"]) AM_CONDITIONAL(ENABLE_PACCT, [test "$enable_pacct" = "yes"]) +AM_CONDITIONAL(ENABLE_JSON, [test "$enable_json" = "yes"]) # substitution into manual pages expanded_sysconfdir=[`patheval $sysconfdir | sed -e 's/-/\\\\-/g'`] @@ -803,6 +827,7 @@ AC_OUTPUT(dist.conf modules/pacctformat/Makefile modules/basicfuncs/Makefile modules/convertfuncs/Makefile + modules/tfjson/Makefile scripts/Makefile scripts/update-patterndb doc/Makefile @@ -843,5 +868,5 @@ echo " Linux capability support : ${enable_linux_caps:=no}" echo " PCRE support : ${enable_pcre:=no}" echo " Env wrapper support : ${enable_env_wrapper:=no}" echo " PACCT module (EXPERIMENTAL) : ${enable_pacct:=no}" - +echo " JSON support (module) : ${enable_json:=no}" diff --git /tfjson-base/modules/Makefile.am /syslog-ng-3.2/modules/Makefile.am new file mode 100644 index cfc721e..16e38df 100644 --- /tfjson-base/modules/Makefile.am +++ /syslog-ng-3.2/modules/Makefile.am @@ -1 +1,2 @@ -SUBDIRS = afsocket afsql afstreams affile afprog afuser csvparser confgen syslogformat pacctformat basicfuncs convertfuncs dbparser dummy +SUBDIRS = afsocket afsql afstreams affile afprog afuser csvparser confgen syslogformat pacctformat basicfuncs convertfuncs dbparser dummy \ + tfjson diff --git /tfjson-base/modules/tfjson/Makefile.am /syslog-ng-3.2/modules/tfjson/Makefile.am new file mode 100644 index 0000000..8b30f83 --- /dev/null +++ /syslog-ng-3.2/modules/tfjson/Makefile.am @@ -0,0 +1,11 @@ +moduledir = @moduledir@ +export top_srcdir + +if ENABLE_JSON +AM_CPPFLAGS = -I$(top_srcdir)/lib -I../../lib @JSON_CFLAGS@ +module_LTLIBRARIES = libtfjson.la + +libtfjson_la_SOURCES = tfjson.c +libtfjson_la_LIBADD = ../../lib/libsyslog-ng.la @JSON_LIBS@ +libtfjson_la_LDFLAGS = -avoid-version +endif diff --git /tfjson-base/modules/tfjson/tfjson.c /syslog-ng-3.2/modules/tfjson/tfjson.c new file mode 100644 index 0000000..e57dce3 --- /dev/null +++ /syslog-ng-3.2/modules/tfjson/tfjson.c @@ -0,0 +1,305 @@ +/* + * Copyright (c) 2002-2011 BalaBit IT Ltd, Budapest, Hungary + * Copyright (c) 2011 Balint Kovacs <blint(a)balabit.hu> + * Copyright (c) 2011 Gergely Nagy <algernon(a)balabit.hu> + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License version 2 as published + * by the Free Software Foundation, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA + * + */ + +#include "plugin.h" +#include "templates.h" +#include "filter.h" +#include "filter-expr-parser.h" +#include "cfg.h" + +#include <json.h> + +typedef struct +{ + GPatternSpec **select_patterns; + GPatternSpec **exclude_patterns; + + GHashTable *template_values; + + gboolean skip_builtins; + + /* Temporary variables. */ + GString *res; +} TFJsonState; + +static void +tf_json_free_state (TFJsonState *state) +{ + gint i = 0; + + if (state->select_patterns) + { + while (state->select_patterns[i]) + { + g_pattern_spec_free(state->select_patterns[i]); + i++; + } + g_free (state->select_patterns); + } + if (state->exclude_patterns) + { + i = 0; + while (state->exclude_patterns[i]) + { + g_pattern_spec_free(state->exclude_patterns[i]); + i++; + } + g_free (state->exclude_patterns); + } + + g_hash_table_destroy(state->template_values); + if (state->res) + g_string_free(state->res, TRUE); + g_free(state); +} + +static gboolean +tf_json_prepare(LogTemplateFunction *self, LogTemplate *parent, + gint argc, gchar *argv[], + gpointer *state, GDestroyNotify *state_destroy, + GError **error) +{ + TFJsonState *jstate = g_new0(TFJsonState, 1); + GOptionContext *ctx; + GError *parse_error = NULL; + gchar **selects = NULL; + gchar **excludes = NULL; + + GOptionEntry format_options[] = + { + { "select", 's', 0, G_OPTION_ARG_STRING_ARRAY, &selects, + "Specify which name-value pairs are included in the result. The parameter is a shell-like glob pattern.", + "<glob expression>" }, + { "exclude", 'x', 0, G_OPTION_ARG_STRING_ARRAY, &excludes, + "Specify which name-value pairs to exclude from the result. The parameter is a shell-like glob pattern.", + "<glob expression>" }, + { "skip-builtins", 'D', 0, G_OPTION_ARG_NONE, &jstate->skip_builtins, + "Skip built-in variables, like $HOST and the rest.", NULL }, + { NULL } + }; + + gchar **ar; + gint arc = argc + 1; + gint i; + + ar = g_new(gchar *, arc + 1); + for (i = 0; i < argc; i++) + ar[i + 1] = g_strdup(argv[i]); + ar[0] = g_strdup("format-json"); + ar[argc + 1] = NULL; + + jstate->template_values = + g_hash_table_new_full(g_str_hash, g_str_equal, g_free, + (GDestroyNotify) log_template_unref); + ctx = g_option_context_new("format-json"); + msg_add_option_group(ctx); + g_option_context_add_main_entries(ctx, format_options, NULL); + if (!g_option_context_parse(ctx, &arc, &ar, &parse_error)) + { + /* + msg_error ("Error parsing function arguments", + evt_tag_str("error", parse_error ? + parse_error->message : "Invalid arguments")); + */ + g_set_error(error, LOG_TEMPLATE_ERROR, LOG_TEMPLATE_ERROR_COMPILE, + "Error parsing function arguments"); + g_option_context_free(ctx); + tf_json_free_state(jstate); + g_strfreev(ar); + return FALSE; + } + g_option_context_free(ctx); + + if (selects == NULL) + { + jstate->select_patterns = g_new0(GPatternSpec *, 2); + jstate->select_patterns[0] = g_pattern_spec_new("*"); + } + else + { + gint n = g_strv_length (selects); + jstate->select_patterns = g_new0(GPatternSpec *, n + 1); + + for (i = 0; i < n; i++) + jstate->select_patterns[i] = g_pattern_spec_new(selects[i]); + } + + if (excludes != NULL) + { + gint n = g_strv_length (excludes); + jstate->exclude_patterns = g_new0(GPatternSpec *, n + 1); + + for (i = 0; i < n; i++) + jstate->exclude_patterns[i] = g_pattern_spec_new(excludes[i]); + } + + for (i = 1; i < arc; i++) + { + gchar *key = NULL; + LogTemplate *t = NULL; + + if (g_strstr_len(ar[i], strlen (ar[i]), "=")) + { + gchar **kv = g_strsplit(ar[i], "=", 2); + + key = g_strdup(kv[0]); + t = log_template_new(parent->cfg, NULL, kv[1]); + g_free(kv[0]); + g_free(kv[1]); + g_free(kv); + } + else + { + gchar *m = g_strconcat("$", ar[i], NULL); + + key = g_strdup(ar[i]); + t = log_template_new(parent->cfg, NULL, m); + g_free(m); + } + g_hash_table_insert(jstate->template_values, key, t); + } + + jstate->res = g_string_sized_new(256); + + *state = jstate; + *state_destroy = (GDestroyNotify) tf_json_free_state; + + g_strfreev(ar); + return TRUE; +} + +static gboolean +tf_json_foreach(NVHandle handle, const gchar *name, const gchar *value, + gssize length, gpointer user_data) +{ + gpointer *args = user_data; + json_object *root = args[0]; + TFJsonState *jstate = args[1]; + json_object *this; + gboolean match = FALSE; + gint i = 0; + + if (jstate->skip_builtins && handle < LM_V_MAX) + return FALSE; + + while (jstate->select_patterns[i]) + { + if (g_pattern_match_string(jstate->select_patterns[i], name)) + { + match = TRUE; + break; + } + i++; + } + + i = 0; + while (jstate->exclude_patterns && + jstate->exclude_patterns[i]) + { + if (g_pattern_match_string(jstate->exclude_patterns[i], name)) + { + match = FALSE; + break; + } + i++; + } + + if (match) + { + this = json_object_new_string_len(value, length); + json_object_object_add(root, name, this); + } + + return FALSE; +} + +static void +tf_json_eval (LogTemplateFunction *self, gpointer state, GPtrArray *arg_bufs, + LogMessage **messages, gint num_messages, LogTemplateOptions *opts, + gint tz, gint seq_num) +{ + return; +} + +static json_object * +tf_json_format_message(TFJsonState *jstate, LogMessage *msg) +{ + json_object *root; + gpointer args[] = { NULL, jstate }; + + root = json_object_new_object(); + args[0] = root; + nv_table_foreach(msg->payload, logmsg_registry, tf_json_foreach, + args); + + return root; +} + +static void +tf_json_format_template(gpointer key, gpointer value, gpointer user_data) +{ + json_object *this; + gpointer *args = user_data; + LogMessage *msg = (LogMessage *)args[1]; + TFJsonState *jstate = (TFJsonState *)args[2]; + + g_string_truncate(jstate->res, 0); + log_template_format((LogTemplate *)value, msg, NULL, LTZ_LOCAL, 0, jstate->res); + + this = json_object_new_string(jstate->res->str); + json_object_object_add((json_object *)args[0], (gchar *)key, this); +} + +static void +tf_json_call(LogTemplateFunction *self, gpointer state, GPtrArray *arg_bufs, + LogMessage **messages, gint num_messages, LogTemplateOptions *opts, + gint tz, gint seq_num, GString *result) +{ + gint i; + TFJsonState *jstate = (TFJsonState *)state; + + for (i = 0; i < num_messages; i++) + { + LogMessage *msg = messages[i]; + json_object *json; + gpointer args[] = { NULL, msg, jstate }; + + json = tf_json_format_message(jstate, msg); + args[0] = json; + g_hash_table_foreach(jstate->template_values, tf_json_format_template, + args); + g_string_append(result, json_object_to_json_string (json)); + json_object_put(json); + } +} + +TEMPLATE_FUNCTION(tf_json, tf_json_prepare, tf_json_eval, tf_json_call, NULL); + +static Plugin builtin_tmpl_func_plugins[] = + { + TEMPLATE_FUNCTION_PLUGIN(tf_json, "format_json"), + }; + +gboolean +tfjson_module_init(GlobalConfig *cfg, CfgArgs *args) +{ + plugin_register(cfg, builtin_tmpl_func_plugins, G_N_ELEMENTS(builtin_tmpl_func_plugins)); + return TRUE; +} -- 1.7.0.4

1 0

(no subject)
by mhall＠mhcomputing.net 01 Feb '11

01 Feb '11

From: Matthew Hall <mhall(a)mhcomputing.net> Date: Mon, 31 Jan 2011 18:00:01 -0800 Subject: [PATCH] adding JSON support to 3.2 (if allowable) Hello Balint, Gergely, Bazsi, and list, Here I present my backport of tfjson support to 3.2 for consideration of inclusion if this is allowd for that branch. Regards, Matthew Hall. Matthew Hall (1): tfjson: $(format_json) template function implementation. configure.in | 27 ++++- modules/Makefile.am | 3 +- modules/tfjson/Makefile.am | 11 ++ modules/tfjson/tfjson.c | 305 ++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 344 insertions(+), 2 deletions(-) create mode 100644 modules/tfjson/Makefile.am create mode 100644 modules/tfjson/tfjson.c

1 0

A known syslog-ng issue/bug?
by Xuan Zhong 28 Jan '11

28 Jan '11

Hi, This issue is that if someone looks at a log file generated by syslog-ng with vi, and then “accidentally” writes and saves the file, subsequent writes to the file by syslog-ng don’t appear to work. If syslog-ng closes the file (which happens if it is “idle” for a configurable period, whose default is 1 minute), and then re-opens it when it next needs to write, the writes succeed. Similarly, “service syslog-ng reload” (or restart) will clear the problem. Anyone experienced the similar problem? Thanks Xuan

3 2

Re: [syslog-ng] syslog-ng deadlock if /dev/console locks?
by Patrick H. 26 Jan '11

26 Jan '11

Ah, there might be other differences that you have then. All our machines are almost identical, same OS, only 2 hardware models, same syslog versions, etc, so there wasnt much variance to cause problems. Definitely seems like an HP issue though, and common enough I would have thought something they would have caught and corrected by now :-/ -Patrick Sent: Wed Jan 26 2011 12:55:47 GMT-0500 (Eastern Standard Time) From: Sowell, Brett <brett.sowell(a)amd.com> To: Patrick H. <syslogng(a)feystorm.net> "Krizak, Paul" <Paul.Krizak(a)amd.com>, Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu>, "Petrini, Bryce" <Bryce.Petrini(a)amd.com>, "Hart, Corey" <Corey.Hart(a)amd.com> Subject: Re: [syslog-ng] syslog-ng deadlock if /dev/console locks? > Hi Patrick, > > I had come across your post during research and added the suggested > echo 'h' to our break/recovery test scripts. > > It seemed that, in our case, echoing 'h' to sysrq-trigger would result > in a stable recovery of /dev/console 2/10 times, sometimes temporary > recovery, but more resulted in no effect. > > -Brett > > On 01/26/11 10:17, Patrick H. wrote: >> Nope, just 'echo h', that was it. >> >> -Patrick >> >> Sent: Wed Jan 26 2011 11:16:31 GMT-0500 (Eastern Standard Time) >> From: Paul Krizak <paul.krizak(a)amd.com> >> To: Syslog-ng users' and developers' mailing list >> <syslog-ng(a)lists.balabit.hu> "Patrick H." <syslogng(a)feystorm.net>, >> "Sowell, Brett" <Brett.Sowell(a)amd.com>, "Petrini, Bryce" >> <Bryce.Petrini(a)amd.com>, "Hart, Corey" <Corey.Hart(a)amd.com> >> Subject: Re: [syslog-ng] syslog-ng deadlock if /dev/console locks? >>> Fascinating. So just triggering the kernel to print something to >>> the console (h is "help") caused /dev/console to properly realign >>> and syslog-ng woke back up? You didn't have to restart syslog-ng or >>> reboot the box or anything? >>> >>> >>> Paul Krizak 7171 Southwest Pkwy MS B200.3A >>> MTS Systems Engineer Austin, TX 78735 >>> Advanced Micro Devices Desk: (512) 602-8775 >>> Linux/Unix Systems Engineering Cell: (512) 791-0686 >>> Global IT Infrastructure Fax: (512) 602-0468 >>> >>> On 01/26/11 10:11, Patrick H. wrote: >>>> We ran into this issue when upgrading iLO on all our boxes. When >>>> the iLO >>>> was upgraded, /dev/console went completely unresponsive, and things >>>> started to hang. The solution turned out to be 'echo h > >>>> /proc/sysrq-trigger'. Apparently when the kernel went to write out to >>>> the serial port, it ran into problems and would reinitialize it. After >>>> that everything started working fine. >>>> >>>> -Patrick >>>> >>>> Sent: Wed Jan 26 2011 11:03:37 GMT-0500 (Eastern Standard Time) >>>> From: Sandor Geller <Sandor.Geller(a)morganstanley.com> >>>> To: Syslog-ng users' and developers' mailing list >>>> <syslog-ng(a)lists.balabit.hu> "Sowell, Brett" <Brett.Sowell(a)amd.com>, >>>> "Petrini, Bryce" <Bryce.Petrini(a)amd.com>, "Hart, Corey" >>>> <Corey.Hart(a)amd.com> >>>> Subject: Re: [syslog-ng] syslog-ng deadlock if /dev/console locks? >>>>> Hello, >>>>> >>>>> On Wed, Jan 26, 2011 at 4:12 PM, Paul Krizak<paul.krizak(a)amd.com> >>>>> wrote: >>>>> >>>>>> Hi, we're using syslog-ng 3.1.2 and have run into what appears to >>>>>> be a >>>>>> bug, but I'd like to get the community's opinion before we dig >>>>>> further >>>>>> into it. >>>>>> >>>>>> We have a bunch of HP servers with iLO2 and iLO3 devices, configured >>>>>> with their virtual serial ports on COM1 (ttyS0). We subsequently >>>>>> have >>>>>> the OS (RHEL4, RHEL5) configured to use COM1 as its console (e.g. >>>>>> /dev/console). This is a very standard configuration that allows >>>>>> us to >>>>>> get remote access to the machines without having to purchase the iLO >>>>>> Advanced KVM feature. It also lets us use the Magic SysRq keys >>>>>> to probe >>>>>> dead systems and stuff, so in general it's not something we're >>>>>> keen to >>>>>> change. >>>>>> >>>>>> What we have found, however, is that there are some cases where >>>>>> the iLO >>>>>> will freeze and requires a reboot. When the iLO reboots, >>>>>> however, the >>>>>> kernel's connection to /dev/console (through the virtual serial >>>>>> port) >>>>>> hangs and blocks. Any traffic to /dev/console just sits in the >>>>>> kernel's >>>>>> buffer and is never delivered. Once the buffer is full, the kernel >>>>>> simply blocks on any write to /dev/console. >>>>>> >>>>>> Now this is a Bad Thing in general, and we're working with HP to >>>>>> try and >>>>>> remedy this bug. However, what concerns me is that syslog-ng, when >>>>>> faced with this behavior, also blocks, even for log messages not >>>>>> bound >>>>>> for /dev/console. >>>>>> >>>>> >>>>> syslog-ng uses a single thread (with the exception of database >>>>> destinations) running the event loop so when a read() or a write() >>>>> blocks then it affects the whole log processing >>>>> >>>>> >>>>>> What we have observed is that a system with syslog-ng will keep >>>>>> delivering the occasional console message to /dev/console (ex. >>>>>> *.emerg >>>>>> messages) and meanwhile the file-based log paths keep working. >>>>>> But once >>>>>> /dev/console blocks, the next time a console message is >>>>>> delivered, *all* >>>>>> of syslog-ng blocks waiting for that message to be delivered, and >>>>>> all of >>>>>> the file-based paths block as well. The result is that pretty much >>>>>> everything on the system stops working. For example, you can't >>>>>> log in, >>>>>> even as root, because the login process blocks on the syslog command >>>>>> that writes to /var/log/secure. Anything that uses syslog >>>>>> suddenly blocks. >>>>>> >>>>>> Is this expected behavior? I would think that syslog-ng would be >>>>>> able >>>>>> to continue accepting and delivering messages, even if one of the >>>>>> log >>>>>> paths is stalled on a blocked write. >>>>>> >>>>> >>>>> syslog-ng uses non-blocking I/O for all sources / destinations but >>>>> despite of this the kernel could still block it therefore syslog-ng >>>>> protects reads/writes in logtransport.c with alarm() so it should >>>>> recover when timeout is set and a read/write blocked. For me it looks >>>>> like the timeout is not set in all cases, only file and program >>>>> sources initialise transport->timeout to 10 secs so I'd say this >>>>> isn't >>>>> expected behaviour - it is a bug. >>>>> >>>>> Regards, >>>>> >>>>> Sandor >>>>> ______________________________________________________________________________ >>>>> >>>>> Member info:https://lists.balabit.hu/mailman/listinfo/syslog-ng >>>>> Documentation:http://www.balabit.com/support/documentation/?product=syslog-… >>>>> >>>>> FAQ:http://www.campin.net/syslog-ng/faq.html >>>>> >>>>> >>>

1 0

syslog-ng - Apache Logging
by Steven Shepherd 26 Jan '11

26 Jan '11

I have Apache logging working via a named pipe, however the log looks like: Jan 25 14:44:13 $HOSTNAME $VIRTUAL_HOST: 000.000.000.000 - - [25/Jan/2011:14:44:13 -0600] "GET /foo.html HTTP/1.0" 200 24040 "-" "Wget/1.10.2 (Red Hat modified)" I want to strip the syslog timestamp and $HOSTNAME. However, if I use "template("$MSGONLY\n")", it strips the timestamp, hostname *and* the $virtual_host (obtained from including "%V" in LogFormat and using 'vcommon' on apache server). Any ideas on how to remove the syslog data but leave the vhost data in place? Cheers!

2 2

Problem with getting syslog-ng statistics
by Aleksei Plotnikov 25 Jan '11

25 Jan '11

Hello everyone, I have a problem with getting statistics from syslog-ng 3.0.5 (under Debian 5.0.4) using it's socket file. In our configuration logs received by syslog-ng are being forwarded to python scripts (which in turn send logs to database). Since sometimes these scripts cannot handle the load, we want to monitor syslog-ng statistics for dropped logs. Currently, I run a script from cron which get statistics from syslog-ng socket file like this "echo STATS | socat - /var/lib/syslog-ng/syslog-ng.ctl" and then parse output. Most of time it works great, but during peak times when number of logs get higher ( ~2000 logs/sec instead of ~1500 logs/sec), this cron script cannot get any output from statistics. Also this message appears in /var/log/syslog: syslog-ng[21677]: Error writing control channel; error='Broken pipe (32)' Is it possible to tune syslog-ng statistics somehow, or maybe there is a better way to get statistics? Thanks, Aleksei ---------------------------- Võimalik, et oled TASUTA 1600€ soojamaareisist vaid CV sisestamise kaugusel! Vaata lähemalt SIIT. http://www.cv.ee

2 2

AUTO: Shanif Lakhani is out of the office (returning 02/14/2011)
by Shanif Lakhani 25 Jan '11

25 Jan '11

I am out of the office until 02/14/2011. Please contact Robert Holoweckyj and/or Lori-Ann Walker for any urgent matters. Thanks Note: This is an automated response to your message "syslog-ng Digest, Vol 69, Issue 44" sent on 1/24/11 20:55:19. This is the only notification you will receive while this person is away.

1 0

[review request]: MongoDB destination
by Gergely Nagy 24 Jan '11

24 Jan '11

I prepared three branches, in preparation for a pull request, but there's still some minor work left to do, so this is more like a "review the structure and plan before the pull request, please" request. All three branches are integration branches, and as such, I will rebase them from time to time. All three branches are folded into one commit over their base, signed-off, and should have proper descriptions. * integration/libmongo-client This one adds a git submodule, which pulls in the libmongo-client library from github to modules/afmongodb/libmongo-client. Also adds infrastructure to make autogen.sh, configure and make enter this directory and Do The Right Thing. At the moment, linking with the supplied libmongo-client is the only option, because the library is not yet ready for system-wide install. It is easy to extend this framework to use the system library if it exists - that is one of the missing features which should go into this branch, but isn't there yet. * integration/afmongodb/base This is the basic, bare-bones driver, without support for dynamic values: it depends on the previous one, and doesn't change anything outside of modules/afmongodb/, except lib/stats.[ch], where it adds SCS_MONGO for stat counter support. The current state of the driver is reasonably stable, there are three known problems that I am aware of: + Segfaults whenever the writer thread emits an msg_debug() or similar itself. This is actually a bug in ivykis, for which Bazsi had a fix for, but I can't find the fix in his git tree yet (only the log message, but without the patch). + Locking issues: it's fairly easy to push the driver into a deadlock, where it will just sit and wait, not doing a thing. I've only seen this happen with flow-control enabled, though. This will have to be solved before I'm confident enough to ask for a pull. + The libmongo-client library is built as a shared library, and is installed as such. It should be static-only, and not installed, only linked to afmongodb. This will be implemented mostly in libmongo-client, and an update of the integration/libmongo-client branch will pull that in; then this branch will be updated accordingly. * integration/afmongodb/dyn-vars Based on integration/afmongodb/base, this adds dynamic_values() support. Minor modifications are done to nvtable too, but it's a very small patch, and very straightforward too. However, as discussed earlier, the goal in the long run is to develop a generic framework for this kind of thing, so this here is more like a temporary solution so that people can experiment with the key feature the mongodb provides over the others. Until the generic framework is ready, this, I believe, is an acceptable compromise. -- |8]

2 2

Error starting syslog-ng
by Fabien Bagard 24 Jan '11

24 Jan '11

Hi list ! I have a problem to start syslog-ng. I got a syntax error on line in bold : ###### # sources # all known message sources source s_all { # message generated by Syslog-NG internal(); # standard Linux log source (this is the default place for the syslog() # function to send logs to) unix-stream("/dev/log"); # messages from the kernel file("/proc/kmsg" log_prefix("kernel: ")); }; source s_unix { tcp( port(514) * tls(* key_file("/etc/syslog-ng/certs/logger.parrot.biz.key") cert_file("/etc/syslog-ng/certs/logger.parrot.biz.crt") ca_dir("/etc/syslog-ng/certs/CA/") peer_verify(optional-untrusted) ) ); }; But I can't figure out what ! Can someone point this error to me ? Thanks -- Fabien Bagard IT Department tel + 33 (0)1 48 03 60 40 -------------------------------------------------------------------------------- Parrot SA 174, Quai de Jemmapes | 75010 Paris - France tel + 33 (0)1 48 03 60 60 | fax + 33 (0)1 48 03 70 08 http://www.parrot.com -------------------------------------------------------------------------------- This e-mail message and any attached document(s) are for the sole use of the intended recipient(s)and may contain confidential and legally privileged information. Any unauthorized review, copy, use and/or disclosure is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original.

4 9

Syslog-ng error
by Shawn Cannon 24 Jan '11

24 Jan '11

I am trying to setup a second syslog-ng server that will be part of a load balanced cluster. When the 2nd server tries to connect to my mssql database, it tries to run the CREATE TABLE commands which fails because the table already exists since the first syslog-ng server already created them. How do I tell the 2nd server not to try the CREATE TABLE stuff so it will start working? Shawn Cannon IT Professional http://www.shawncannon.com Contact Me <http://www.linkedin.com/pub/shawn-cannon/5/84/81a> LinkedIn<http://www.linkedin.com/pub/shawn-cannon/5/84/81a> <http://www.facebook.com/rolltidega72> Facebook<http://www.facebook.com/rolltidega72> <http://twitter.com/rolltidega> Twitter <http://twitter.com/rolltidega> My blog: My thoughts on VMworld… a week later…<http://www.shawncannon.com/?p=19>

5 10

syslog-ng 3.1.3-2 - working examples of remote logging
by Steven Shepherd 21 Jan '11

21 Jan '11

OK, so syslog has come a long way and many configuration changes in the years :) Can someone provide working examples of (3.1.X) client/server syslog-ng.conf set up for remote logging purposes such as /var/log/messages, /var/log/kern.log, /var/log/auth.log...etc? Would want on local machine as well as remote syslog server. Will also want to remotely log apache access/error log via syslog-ng as well. Any help/tips/pointers is greatly appreciated! Cheers!

3 2

Open source version: syslog-ng-3.1.3-1.rhel5.i386.rpm
by julian 19 Jan '11

19 Jan '11

what install options are assumed for syslog-ng-3.1.3-1.rhel5.i386.rpm: server, relay, client? It is acting as relay.

3 2

oracle db + syslog-ng integration
by vijay 19 Jan '11

19 Jan '11

hi, Iam Using syslog-ng 3.0.4.when i used below configuration destination d_sql { sql(type(oracle) username("myusername") password("mypassword") database("LOGS") table("msgs_${R_YEAR}${R_MONTH}${R_DAY}") columns("datetime varchar(16)", "host varchar(32)", "program varchar(32)", "pid varchar(8)", "message varchar2") values("$R_DATE", "$HOST", "$PROGRAM", "$PID", "$MSGONLY") indexes("datetime", "host", "program", "pid", "message")); }; iam facing error while starting service syntax error in /usr/local/etc/syslog-ng.conf at line 322. syslog-ng documentation: http://www.balabit.com/support/documentation/?product=syslog-ng mailing list: https://lists.balabit.hu/mailman/listinfo/syslog-ng the line 322 is---------- sql(type(oracle) username("myusername") password("mypassword") vijay.p

2 3

syslog ng open version 3.1.3 does not relay or write to local file
by Bentayeb, Julian A. 18 Jan '11

18 Jan '11

I have syslog-ng-3.1.3-1.rhel5.i386.rpm open version installed and want syslog clients to send to the syslog-ng server; The intention is to have the server send all logs to a final syslog destination and a keep a copy in a file locally. The only thing that is working is I am seeing SYSLOG packest via tcpdump. No messages come to the local file and no messages go to the final destination. I installed the rpm and there was no mode option (server, relay, client) I could chose. Your help is appreciated. Here is the syslog conf I am using: @version:3.0 # # Relay all logs to final destination # options { mark_freq(30); keep_hostname(yes); }; source s_local { unix-stream("/dev/log"); internal(); }; source s_network { syslog(transport(udp)); }; destination d_syslog_udp { syslog("xx.xx.xx.xx" transport("udp") ); }; log { source(s_local); source(s_network); destination(d_syslog_udp); }; # # Keep a copy of logs coming over the network locally in a file for Splunk # source s_network { syslog(transport(udp));}; destination d_file { file("/var/log/messages" ); }; log { source(s_network); destination(d_file); }; Julian Bentayeb Lawrence Livermore National Laboratory Information and Communications Systems 925-424-5702

1 0

Syslogs from Router
by Christopher Petsch 18 Jan '11

18 Jan '11

I am trying to get my router (Netgear FVS338) to send syslogs to my computer but I cant seem to get it configured correctly. Ive googled for the past three days and come up short with little to no help and what little i did find i kept getting syntax errors for. I know I'm doing something wrong but I don't know what. If I could get some help on this or be pointed in a better direction it would be most appricated. Thank you in advance. -- Sincerely, Chris KeyID: 0x348DC808 Keyserver: pool.sks-keyservers.net

5 9

Performance of syslog-ng for Database access (mysql)
by Yu Watanabe 18 Jan '11

18 Jan '11

Hello all. I would like to ask question for the performance of syslog-ng when it is accessing to mysql database. I am using syslog-ng 3.0.9 and thinking of using either one of the method below for the database connection. 1. Using the program driver and fork the "mysql command" to execute the sql. 2. Use the sql driver to access to the table I was wondering which is more efficient for syslog-ng and the server itself (i.e. CPU , memory ) Any advice would be helpful! Thanks, Yu Watanabe

3 6

MongoDB destination driver
by Gergely Nagy 16 Jan '11

16 Jan '11

Greetings! I was contemplating switching one of the websites I maintain from postgresql to one of these document store things, and while wondering about how to evaluate the options best, I stumbled into a short article on the MongoDB website: http://blog.mongodb.org/post/172254834/mongodb-is-fantastic-for-logging, which promptly led me to start pondering about how hard - or easy, as it turns out - would it be to write a MongoDB destination driver for syslog-ng? Only four hours later, I can present you the mongodb destination driver, available from my git repo (note the -b option, it is important): $ git clone -b algernon/dest/mongodb git://git.madhouse-project.org/syslog-ng/syslog-ng-3.2.git Of course, one can browse the sources on the web too: http://git.madhouse-project.org/syslog-ng/syslog-ng-3.2/tree/modules/afmong… It is using the MongoDB C client library (http://www.mongodb.org/display/DOCS/C+Language+Center) - I simply embedded the sources for now, lacking a better option. Once compiled, one can already begin using it with the default options: destination d_mongodb { mongodb(); }; This will try to connect to localhost:27017, and use the logs collection in the syslog-ng database, and will log all the standard fields. Of course, all of those are configurable! To demonstrate all the - currently - available options, the destination definition above is the same as the following: destination d_mongodb { mongodb( host("localhost") port(27017) database("syslog-ng") collection("logs") keys("date", "facility", "level", "host", "program", "pid", "message") values("${R_YEAR}-${R_MONTH}-${R_DAY} ${R_HOUR}:${R_MIN}:${R_SEC}", "$FACILITY", "$LEVEL", "$HOST", "$PROGRAM", "$PID", "$MSGONLY") ); }; A few things, like authentication and some template options are not configurable yet, partly because I didn't figure out what they're good for, or how they work. But I will get there at some point, especially if there's interest in said features. All in all, I'm very happy that I could cook up a fairly simple destination driver within a few hours, having no prior experience writing one. The syslog-ng code is amazing, by the way, it was a breeze to navigate through and find the stuff I needed to make this driver work. Mind you, this is very new code, and I haven't tested it extensively, but I do have some great plans involving syslog-ng and mongodb >;) Hope you like the code, and perhaps find it useful! -- |8]

6 38

syslog-ng 3.2.2 has been released
by devel＠balabit.hu 16 Jan '11

16 Jan '11

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.2.2 SUMMARY : new stable release DATE : Jan 16, 2011 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (3.2.2) has been released. For latest fixes in the 3.2.x feature branch you are recommended to upgrade to this version. CHANGES: 3.2.2 Sat, 15 Jan 2011 13:50:35 +0100 Bugfixes: * Fixed a possible segmentation fault when the port number is changed for a TCP source, the configuration is reloaded and there were open connections for the old port, which send messages after the SIGHUP. This behaviour has been broken since 3.0.1. * Fixed a possible security issue on FreeBSD and on platforms where mode_t is an unsigned 16 bit value. On these platforms syslog-ng may be using 0xFFFF as the permission bits causing log files to be world readable/writable/executable/setuid. * Fixed leaking the contents of internal() messages (such as MARK or the statistics message). * Fixed current time tracking when calculating the time in the correllation engine. * When the patterndb file got reloaded the correllation state was dropped. This behaviour was fixed. * Really ignore invalid persist-state files, which caused syslog-ng startup to fail previously. * Added the missing support for blocks inside log {} statements. * Fixed a configuration init error when the same db-parser() instance is referenced from multiple log paths. * Fixed handling the port() options for SQL destinations. Platform specific: * Added cygwin support to the system() source. Build related: * syslog-ng modules are now linked with "-module -no-undefined" parameters, pdbtool and unit tests are using -dlpreopen when explicitly linking against such modules. * The core patterndb functionality got split off to a separate library installed to $libdir to make it easier to be used by pdbtool. * Fixed support for an explicit --exec-prefix configure parameter. Earlier if exec_prefix was different from prefix, the installation layout produced unworkable binaries. * If no OpenSSL libraries are available, pdbtool patternize still can work, although in this case proper UUID generation is not possible. * If syslog-ng is compiled against an old glib (earlier than 2.13), it'll not use an API that is present in newer ones. Please note however that there might be other similar compatibility issues with old Glib versions. * Updated cygwin packaging files. * Don't use -wno-pointer-sign in dbparser if gcc doesn't support it. This will emit a lot of warnings, but still make the code possible to compile with older gcc versions. Credits: syslog-ng is developed as a community project. All changes and improvements requires effort, and this effort is really appreciated. Writing code, testing changes or simply providing use-cases and information on one's setup will make syslog-ng better. Here are the people, listed in no specific order who made this release possible. * Balázs Németh (BalaBit) * Sándor Gellér (Morgan Stanley) * Péter Czanik (BalaBit) * Owen Mann (Interactive Data) * Zhengxiang Pan (Alcatel Lucent) * Corinna Vinschen (RedHat) * Eric Berggren (Apple) * Gergely Nagy (BalaBit) * Matthew Hall * LEBRETON Philippe * Craig Bell * Steven Chamberlain * Balázs Scheidler (BalaBit) Thanks for their efforts, it is appreciated. DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

1 0