- syslog-ng - lists.balabit.hu

consecutive pattern parsers, and some other pattern matching questions
by Valentijn Sessink 01 Mar '11

01 Mar '11

Hello list, I'm trying to setup a pattern DB for Postfix, and I'm running into the a couple of problems. 1) One of the log messages I would like to catch is: connect from smtp.example.com[xxxx:xxxx:300:40c1::23] Now I'd like to catch both hostname and IP-address, and maybe I'd like to feed the IP address into some sort of program later. So I thought I'd better put these in individual variables. However, the matching rule <pattern>connect from @ESTRING:postfix.remotehost:[@@IPvANY:postfix.remoteip@]</pattern> doesn't work, because of the double "@@" - which is handled as an escaped "@", instead of two consecutive pattern parsers. So my first question is: how can I have two consecutive pattern parsers in a pattern? 2) Other messages say things like: 64A7F3001E7: from=<something(a)example.com> ..... The syslog-ng OSE admin guide tells me to use @QSTRING:<>@ to match the mail address; but this shows an error. Is @QSTRING:<&gt@ the correct way to proceed? (Or is this impossible with the current implementation?) 3) My third question boils down to: is it possible to correlate *one* single message into *two* separate trails? Would that just work by adding the same pattern to two different contexts? The question comes from Postfix using a bunch of small, interconnected programs, so a log trail of a single mail message will change characteristics during it's journey. For example, when a message comes in, smtpd will log: connect from smtp.example.com[xxxx:xxxx:300:40c1::23] ... it could then hand over the message with: 599903001E7: client=smtp.example.com[xxxx:xxxx:300:40c1::23] ... and then other programs continue the log trail useing this queue-ID 599903001E7 as a marker. So here are basically two events intertwined: a connection (that just says "connect..." and "... disconnect" and has a context-scope of "program); and a longer event that starts with the "connect..." and ends with delivery - or even forwarding - of the mail message - and probably has a "host" context-scope. 4) and finally: is there a good way to immediately end a certain context-scope? For example, after "disconnect from smtp.example.com[xxxx:xxxx:300:40c1::23]", the smtp phase is over, so there's no use keeping this context in memory anymore. Would adding something like "context-timeout=0" to the "disconnect" pattern work? Best regards, Valentijn

3 8

[PATCH] tfjson: $(format_json) template function implementation.
by Matthew Hall 01 Mar '11

01 Mar '11

Implements a $(format_json) template function, largely based on the work of Balint Kovacs <blint(a)balabit.hu>, built on top of the json-c library. The syntax of the template function is as follows: $(format_json [--select <GLOB>] [--exclude <GLOB>] [--skip-builtins] [MACRO_NAMES...] KEY=VALUE) There can be multiple selects, excludes, macro names and key=value pairs. The VALUE part of the key=value pair can contain template macros. Macro names must be listed without the "$" sign, however. If --skip-builtins is specified, all built-in variables will be excluded. As an example, this is a valid argument list (without the line breaks): $(format_json --select .classification.* --select useracct.* --exclude *.*id HOST my_own_key='$SOURCEIP ($HOST)') This will include every key that begins with ".classification." or "useracct.", but will exclude any, that has a dot, and ends with "id". The latter keys will be excluded even if they match any of the select patterns. Furthermore, the $HOST macro will be included (with the key HOST), along with a my_own_key key, whose value will be evaluated as a template, thus, will contain the source IP and the HOST (again, for demo purposes). Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> (cherry picked from commit 9ab6b381433d657b59eb1d19f994f9328fca9547) Conflicts: configure.in modules/Makefile.am removed mongodb references from configure.in and modules/Makefile.am; resolved conflicts there Signed-off-by: Matthew Hall <mhall(a)mhcomputing.net> --- configure.in | 27 ++++- modules/Makefile.am | 3 +- modules/tfjson/Makefile.am | 11 ++ modules/tfjson/tfjson.c | 305 ++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 344 insertions(+), 2 deletions(-) create mode 100644 modules/tfjson/Makefile.am create mode 100644 modules/tfjson/tfjson.c diff --git /tfjson-base/configure.in /syslog-ng-3.2/configure.in new file mode 100644 index 0c6cef2..08d6a6b 100644 --- /tfjson-base/configure.in +++ /syslog-ng-3.2/configure.in @@ -25,6 +25,8 @@ EVTLOG_MIN_VERSION="0.2" OPENSSL_MIN_VERSION="0.9.8" LIBDBI_MIN_VERSION="0.8.0" PCRE_MIN_VERSION="7.3" +IVYKIS_MIN_VERSION="0.18" +JSON_C_MIN_VERSION="0.9" dnl *************************************************************************** dnl Initial setup @@ -144,6 +146,10 @@ AC_ARG_ENABLE(gcov, ,,enable_gcov="no") +AC_ARG_ENABLE(json, + [ --enable-json Enable support for JSON template formatting (default: auto)] + ,,enable_json="auto") + dnl *************************************************************************** dnl Checks for programs. AC_PROG_CC @@ -493,6 +499,23 @@ if test "x$linking_mode" != "xdynamic" -a "x$blb_cv_static_glib" = "xno"; then fi dnl *************************************************************************** +dnl json-glib headers/libraries +dnl *************************************************************************** +if test "x$enable_json" = "xyes" -o "x$enable_json" = "xauto"; then + PKG_CHECK_MODULES(JSON, json >= $JSON_C_MIN_VERSION,,JSON_LIBS="") + if test -z "$JSON_LIBS"; then + if test "x$enable_json" = "xyes"; then + AC_MSG_ERROR(Cannot find json >= $JSON_C_MIN_VERSION.) + else + AC_MSG_WARN(Cannot find json >= $JSON_C_MIN_VERSION.) + fi + enable_json="no" + else + enable_json="yes" + fi +fi + +dnl *************************************************************************** dnl pcre headers/libraries dnl *************************************************************************** @@ -750,6 +773,7 @@ AM_CONDITIONAL(ENABLE_SSL, [test "$enable_ssl" = "yes"]) AM_CONDITIONAL(ENABLE_SQL, [test "$enable_sql" = "yes"]) AM_CONDITIONAL(ENABLE_SUN_STREAMS, [test "$enable_sun_streams" = "yes"]) AM_CONDITIONAL(ENABLE_PACCT, [test "$enable_pacct" = "yes"]) +AM_CONDITIONAL(ENABLE_JSON, [test "$enable_json" = "yes"]) # substitution into manual pages expanded_sysconfdir=[`patheval $sysconfdir | sed -e 's/-/\\\\-/g'`] @@ -803,6 +827,7 @@ AC_OUTPUT(dist.conf modules/pacctformat/Makefile modules/basicfuncs/Makefile modules/convertfuncs/Makefile + modules/tfjson/Makefile scripts/Makefile scripts/update-patterndb doc/Makefile @@ -843,5 +868,5 @@ echo " Linux capability support : ${enable_linux_caps:=no}" echo " PCRE support : ${enable_pcre:=no}" echo " Env wrapper support : ${enable_env_wrapper:=no}" echo " PACCT module (EXPERIMENTAL) : ${enable_pacct:=no}" - +echo " JSON support (module) : ${enable_json:=no}" diff --git /tfjson-base/modules/Makefile.am /syslog-ng-3.2/modules/Makefile.am new file mode 100644 index cfc721e..16e38df 100644 --- /tfjson-base/modules/Makefile.am +++ /syslog-ng-3.2/modules/Makefile.am @@ -1 +1,2 @@ -SUBDIRS = afsocket afsql afstreams affile afprog afuser csvparser confgen syslogformat pacctformat basicfuncs convertfuncs dbparser dummy +SUBDIRS = afsocket afsql afstreams affile afprog afuser csvparser confgen syslogformat pacctformat basicfuncs convertfuncs dbparser dummy \ + tfjson diff --git /tfjson-base/modules/tfjson/Makefile.am /syslog-ng-3.2/modules/tfjson/Makefile.am new file mode 100644 index 0000000..8b30f83 --- /dev/null +++ /syslog-ng-3.2/modules/tfjson/Makefile.am @@ -0,0 +1,11 @@ +moduledir = @moduledir@ +export top_srcdir + +if ENABLE_JSON +AM_CPPFLAGS = -I$(top_srcdir)/lib -I../../lib @JSON_CFLAGS@ +module_LTLIBRARIES = libtfjson.la + +libtfjson_la_SOURCES = tfjson.c +libtfjson_la_LIBADD = ../../lib/libsyslog-ng.la @JSON_LIBS@ +libtfjson_la_LDFLAGS = -avoid-version +endif diff --git /tfjson-base/modules/tfjson/tfjson.c /syslog-ng-3.2/modules/tfjson/tfjson.c new file mode 100644 index 0000000..e57dce3 --- /dev/null +++ /syslog-ng-3.2/modules/tfjson/tfjson.c @@ -0,0 +1,305 @@ +/* + * Copyright (c) 2002-2011 BalaBit IT Ltd, Budapest, Hungary + * Copyright (c) 2011 Balint Kovacs <blint(a)balabit.hu> + * Copyright (c) 2011 Gergely Nagy <algernon(a)balabit.hu> + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License version 2 as published + * by the Free Software Foundation, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA + * + */ + +#include "plugin.h" +#include "templates.h" +#include "filter.h" +#include "filter-expr-parser.h" +#include "cfg.h" + +#include <json.h> + +typedef struct +{ + GPatternSpec **select_patterns; + GPatternSpec **exclude_patterns; + + GHashTable *template_values; + + gboolean skip_builtins; + + /* Temporary variables. */ + GString *res; +} TFJsonState; + +static void +tf_json_free_state (TFJsonState *state) +{ + gint i = 0; + + if (state->select_patterns) + { + while (state->select_patterns[i]) + { + g_pattern_spec_free(state->select_patterns[i]); + i++; + } + g_free (state->select_patterns); + } + if (state->exclude_patterns) + { + i = 0; + while (state->exclude_patterns[i]) + { + g_pattern_spec_free(state->exclude_patterns[i]); + i++; + } + g_free (state->exclude_patterns); + } + + g_hash_table_destroy(state->template_values); + if (state->res) + g_string_free(state->res, TRUE); + g_free(state); +} + +static gboolean +tf_json_prepare(LogTemplateFunction *self, LogTemplate *parent, + gint argc, gchar *argv[], + gpointer *state, GDestroyNotify *state_destroy, + GError **error) +{ + TFJsonState *jstate = g_new0(TFJsonState, 1); + GOptionContext *ctx; + GError *parse_error = NULL; + gchar **selects = NULL; + gchar **excludes = NULL; + + GOptionEntry format_options[] = + { + { "select", 's', 0, G_OPTION_ARG_STRING_ARRAY, &selects, + "Specify which name-value pairs are included in the result. The parameter is a shell-like glob pattern.", + "<glob expression>" }, + { "exclude", 'x', 0, G_OPTION_ARG_STRING_ARRAY, &excludes, + "Specify which name-value pairs to exclude from the result. The parameter is a shell-like glob pattern.", + "<glob expression>" }, + { "skip-builtins", 'D', 0, G_OPTION_ARG_NONE, &jstate->skip_builtins, + "Skip built-in variables, like $HOST and the rest.", NULL }, + { NULL } + }; + + gchar **ar; + gint arc = argc + 1; + gint i; + + ar = g_new(gchar *, arc + 1); + for (i = 0; i < argc; i++) + ar[i + 1] = g_strdup(argv[i]); + ar[0] = g_strdup("format-json"); + ar[argc + 1] = NULL; + + jstate->template_values = + g_hash_table_new_full(g_str_hash, g_str_equal, g_free, + (GDestroyNotify) log_template_unref); + ctx = g_option_context_new("format-json"); + msg_add_option_group(ctx); + g_option_context_add_main_entries(ctx, format_options, NULL); + if (!g_option_context_parse(ctx, &arc, &ar, &parse_error)) + { + /* + msg_error ("Error parsing function arguments", + evt_tag_str("error", parse_error ? + parse_error->message : "Invalid arguments")); + */ + g_set_error(error, LOG_TEMPLATE_ERROR, LOG_TEMPLATE_ERROR_COMPILE, + "Error parsing function arguments"); + g_option_context_free(ctx); + tf_json_free_state(jstate); + g_strfreev(ar); + return FALSE; + } + g_option_context_free(ctx); + + if (selects == NULL) + { + jstate->select_patterns = g_new0(GPatternSpec *, 2); + jstate->select_patterns[0] = g_pattern_spec_new("*"); + } + else + { + gint n = g_strv_length (selects); + jstate->select_patterns = g_new0(GPatternSpec *, n + 1); + + for (i = 0; i < n; i++) + jstate->select_patterns[i] = g_pattern_spec_new(selects[i]); + } + + if (excludes != NULL) + { + gint n = g_strv_length (excludes); + jstate->exclude_patterns = g_new0(GPatternSpec *, n + 1); + + for (i = 0; i < n; i++) + jstate->exclude_patterns[i] = g_pattern_spec_new(excludes[i]); + } + + for (i = 1; i < arc; i++) + { + gchar *key = NULL; + LogTemplate *t = NULL; + + if (g_strstr_len(ar[i], strlen (ar[i]), "=")) + { + gchar **kv = g_strsplit(ar[i], "=", 2); + + key = g_strdup(kv[0]); + t = log_template_new(parent->cfg, NULL, kv[1]); + g_free(kv[0]); + g_free(kv[1]); + g_free(kv); + } + else + { + gchar *m = g_strconcat("$", ar[i], NULL); + + key = g_strdup(ar[i]); + t = log_template_new(parent->cfg, NULL, m); + g_free(m); + } + g_hash_table_insert(jstate->template_values, key, t); + } + + jstate->res = g_string_sized_new(256); + + *state = jstate; + *state_destroy = (GDestroyNotify) tf_json_free_state; + + g_strfreev(ar); + return TRUE; +} + +static gboolean +tf_json_foreach(NVHandle handle, const gchar *name, const gchar *value, + gssize length, gpointer user_data) +{ + gpointer *args = user_data; + json_object *root = args[0]; + TFJsonState *jstate = args[1]; + json_object *this; + gboolean match = FALSE; + gint i = 0; + + if (jstate->skip_builtins && handle < LM_V_MAX) + return FALSE; + + while (jstate->select_patterns[i]) + { + if (g_pattern_match_string(jstate->select_patterns[i], name)) + { + match = TRUE; + break; + } + i++; + } + + i = 0; + while (jstate->exclude_patterns && + jstate->exclude_patterns[i]) + { + if (g_pattern_match_string(jstate->exclude_patterns[i], name)) + { + match = FALSE; + break; + } + i++; + } + + if (match) + { + this = json_object_new_string_len(value, length); + json_object_object_add(root, name, this); + } + + return FALSE; +} + +static void +tf_json_eval (LogTemplateFunction *self, gpointer state, GPtrArray *arg_bufs, + LogMessage **messages, gint num_messages, LogTemplateOptions *opts, + gint tz, gint seq_num) +{ + return; +} + +static json_object * +tf_json_format_message(TFJsonState *jstate, LogMessage *msg) +{ + json_object *root; + gpointer args[] = { NULL, jstate }; + + root = json_object_new_object(); + args[0] = root; + nv_table_foreach(msg->payload, logmsg_registry, tf_json_foreach, + args); + + return root; +} + +static void +tf_json_format_template(gpointer key, gpointer value, gpointer user_data) +{ + json_object *this; + gpointer *args = user_data; + LogMessage *msg = (LogMessage *)args[1]; + TFJsonState *jstate = (TFJsonState *)args[2]; + + g_string_truncate(jstate->res, 0); + log_template_format((LogTemplate *)value, msg, NULL, LTZ_LOCAL, 0, jstate->res); + + this = json_object_new_string(jstate->res->str); + json_object_object_add((json_object *)args[0], (gchar *)key, this); +} + +static void +tf_json_call(LogTemplateFunction *self, gpointer state, GPtrArray *arg_bufs, + LogMessage **messages, gint num_messages, LogTemplateOptions *opts, + gint tz, gint seq_num, GString *result) +{ + gint i; + TFJsonState *jstate = (TFJsonState *)state; + + for (i = 0; i < num_messages; i++) + { + LogMessage *msg = messages[i]; + json_object *json; + gpointer args[] = { NULL, msg, jstate }; + + json = tf_json_format_message(jstate, msg); + args[0] = json; + g_hash_table_foreach(jstate->template_values, tf_json_format_template, + args); + g_string_append(result, json_object_to_json_string (json)); + json_object_put(json); + } +} + +TEMPLATE_FUNCTION(tf_json, tf_json_prepare, tf_json_eval, tf_json_call, NULL); + +static Plugin builtin_tmpl_func_plugins[] = + { + TEMPLATE_FUNCTION_PLUGIN(tf_json, "format_json"), + }; + +gboolean +tfjson_module_init(GlobalConfig *cfg, CfgArgs *args) +{ + plugin_register(cfg, builtin_tmpl_func_plugins, G_N_ELEMENTS(builtin_tmpl_func_plugins)); + return TRUE; +} -- 1.7.0.4

5 8

syslog-ng crashed
by Subrahmanyam Manuguri 01 Mar '11

01 Mar '11

Hi, I am using the following version of syslog-ng server on a PowerPC based system running Montavista Linux. syslog-ng --version syslog-ng 3.0.3 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.0#master#08c9bf9d98e4e021756adc12c42605840140ba8b Compile-Date: Jul 23 2009 14:23:17 Enable-Threads: off Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-Sun-Door: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-SSL: on Enable-SQL: off Enable-Linux-Caps: on Enable-Pcre: on Of late, I am running into the syslog-ng crashes during the boot up of the system. At the boot up time the number of messages are significantly more. I was able to collect the stack trace. (gdb) bt #0 0x00000000 in ?? () #1 0x10024d0c in log_pipe_queue () #2 0x10024c1c in log_multiplexer_queue () #3 0x100304a8 in log_pipe_queue () #4 0x10030440 in log_source_group_queue () #5 0x1000f218 in log_pipe_queue () #6 0x1000f1b4 in log_pipe_forward_msg () #7 0x1000f218 in log_pipe_queue () #8 0x1000f1b4 in log_pipe_forward_msg () #9 0x10045030 in log_pipe_queue () #10 0x10044f08 in log_source_queue () #11 0x100318d0 in log_pipe_queue () #12 0x1003185c in log_reader_handle_line () #13 0x10031bf4 in log_reader_fetch_log () #14 0x100315b8 in log_reader_fd_dispatch () #15 0x0ff132e4 in g_main_context_dispatch () from /usr/lib/libglib-2.0.so.0 #16 0x0ff16c9c in ?? () from /usr/lib/libglib-2.0.so.0 #17 0x0ff17248 in g_main_context_iteration () from /usr/lib/libglib-2.0.so.0 #18 0x10006200 in main_loop_run () #19 0x100069c4 in main () (gdb) The configuration i am running with is options { time_sleep(10); log_fetch_limit(250); log_fifo_size(2000); keep_timestamp(no); dns_cache(no); dns_cache_hosts(/etc/hosts); use_dns(persist_only); flush_timeout(200); perm(0644); stats_freq(3600); stats_level(0); ts_format(iso); time_reopen(10); create_dirs(yes); dir_perm(755); }; Can somebody please help me in tweaking the right parameters to avoid the problem (if that solves the problem)? If this is a known problem, is there a patch available? Appreciate your help very much. Thanks, Subra

2 4

follow_freq for unix-stream/unix-dgram sources
by Aleksei Plotnikov 01 Mar '11

01 Mar '11

Hello, In "syslog-ng 3.0 Administrator Guide" one of options for unix-stream and unix-datagram sources is "follow_freq", which is default 1 according to manual. But when I explicitly set it in source config to "unix-stream("/dev/log" follow_freq(1))" or any other positive value, syslog-ng start to consume all memory and than crashes after some time. Also log is full of such messages: "syslog-ng[11189]: Error invoking seek on followed file; error='Illegal seek (29)'" and when I do strace on syslog-ng process, it shows: lseek(12, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek) lseek(13, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek) lseek(14, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek) and 12,13,14 is file descriptors for /dev/log. Looks like syslog-ng is trying to lseek on socket, what is not possible. It happened when I tried different syslog-ng versions 3.0.5 and 3.1.2 on debian lenny 5.0.6 and ubuntu 10.10. So, is follow_freq applicable in such case and maybe it should be noted in manual? Also is it possible to make /dev/log not-blocking to sender, when syslog-ng cannot process messages from it. In our case, when follow_freq was set to 1, syslog-ng was not able to read from /dev/log/ - other processes which were sending logs to /dev/log are stucked too. It happens with both unix-stream and unix-dgram sockets. Or is there any good way to avoid such behaviour? Thanks.

3 2

ANSI NULLS??
by Shawn Cannon 01 Mar '11

01 Mar '11

I have a question about logging to a Microsoft SQL database. Is there a way to have the data log to the database with ANSI NULLS set to TRUE? We need that capability and I am not sure where to turn that on. Thanks!

2 1

Invitation to connect on LinkedIn
by Roc Zhou via LinkedIn 28 Feb '11

28 Feb '11

LinkedIn ------------Roc Zhou requested to add you as a connection on LinkedIn: ------------------------------------------ Cosmin, I'd like to add you to my professional network on LinkedIn. - Roc Accept invitation from Roc Zhou http://www.linkedin.com/e/-uz7o7o-gkp2i0wr-3r/2QmX6bZJs-PJn0df2BKk2xSyRmW36… View invitation from Roc Zhou http://www.linkedin.com/e/-uz7o7o-gkp2i0wr-3r/2QmX6bZJs-PJn0df2BKk2xSyRmW36… ------------------------------------------ DID YOU KNOW you can conduct a more credible and powerful reference check using LinkedIn? Enter the company name and years of employment or the prospective employee to find their colleagues that are also in your network. This provides you with a more balanced set of feedback to evaluate that new hire. http://www.linkedin.com/e/-uz7o7o-gkp2i0wr-3r/rsr/inv-27/ -- (c) 2011, LinkedIn Corporation

1 0

Why the 'a' versions came out first ?
by Mr. James W. Laferriere 25 Feb '11

25 Feb '11

Hello Bazsi (& anyone else in the know) , While I am not a PE customer and more of a listener on these maillists I noticed this release with a kind of "Now why did they release the 'a' versions before the non-updated ?" . This just did not sound right So I thought I'd ask . Feb 24 devel(a)balabit.hu (4867) [syslog-ng-announce] syslog-ng Premium Edition 3.0.7a has been released Feb 24 devel(a)balabit.hu (4917) [syslog-ng-announce] syslog-ng Premium Edition 4.0.1a has been released Feb 25 devel(a)balabit.hu (6657) [syslog-ng-announce] syslog-ng Premium Edition 3.0.7 has been released Feb 25 devel(a)balabit.hu (8134) [syslog-ng-announce] syslog-ng Premium Edition 4.0.1 has been released Feb 25 devel(a)balabit.hu (4919) [syslog-ng-announce] syslog-ng Premium Edition 3.2.1b has been released Tia , JimL -- +------------------------------------------------------------------+ | James W. Laferriere | System Techniques | Give me VMS | | Network&System Engineer | 3237 Holden Road | Give me Linux | | babydr(a)baby-dragons.com | Fairbanks, AK. 99709 | only on AXP | +------------------------------------------------------------------+

2 1

Firewalling with syslog-ng, a working prototype
by Valentijn Sessink 24 Feb '11

24 Feb '11

Hi list, For a week or so, I'm gathering the building blocks for a sort of low-tech intrusion detection/prevention system. My "itch": having a system that acts "real time" on the log messages that various daemons produce; having it low profile; easy to get it to act (i.e. no scripts that call scripts that call other scripts). For example, if sshd says "invalid user", I'd like the firewall to act on this, with as little steps in between as possible. Luckily, syslog-ng is able to find patterns all by itself, so I'm able to "skip the middle man", i.e. I can use syslog-ng directly on the firewalling rules. And what is better: I'm not even using the program() call! I'm currently running such a system in pre-production and I'm delighted. It's really easy to build. It works like a charm. Here's how: 0) Install syslog-ng. 1) To get started, get the sshd pattern from the Balabit git web frontend: http://git.balabit.hu/?p=bazsi/syslog-ng-patterndb.git;a=blob;f=access/sshd… and put it in /etc/syslog-ng/patterndb.d/, run pdbtool merge -D . -p /var/lib/syslog-ng/patterndb.xml 2) Setup your syslog-ng.conf file. Add, in the appropriate sections: # a destination; we have a dual destination here, a file to see the # blocked hosts, and an iptables-destination in /proc to block them. destination d_syslogblock { file("/proc/net/xt_recent/syslogblock" template("+${usracct.device}\n")); file("/var/log/syslogblock"); }; # a parser for the pattern-DB we made in step 1 parser pattern_db { db_parser( file("/var/lib/syslog-ng/patterndb.xml")); }; # a filter to filter the parser results filter f_syslogblock { tags("secevt") and match("REJECT" value("secevt.verdict")); }; # and finally, the log itself: log { source(s_src); parser(pattern_db); filter(f_syslogblock); destination(d_syslogblock); }; 3) Now set up your firewalling. A minimalistic firewall would look like: ip6tables -N block ip6tables -A INPUT -i lo -j ACCEPT ip6tables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT ip6tables -A INPUT -m state --state INVALID -j DROP ip6tables -A INPUT -m recent --rcheck --name block \ --seconds 3600 -j DROP # see note (1) below ip6tables -A INPUT -m recent --rcheck --name syslogblock \ --seconds 900 --hitcount 15 -j block ip6tables -A block -m recent --rcheck --name block --set 4) Repeat step 3, but with "iptables" instead of "ip6tables". (I used ip6tables on purpose, because it's time you get ready for IPv6 if you aren't). Ready. Not too hard, isn't it? It works as follows. syslog-ng will put the IP-address of any wrong password or username into the /proc/net/xt_recent/syslogblock file, with a "+" in front of the address. This is the iptables way of telling the "recent" module it should add a certain address. Now the IPtables rules tell, that if someone is seen in this "syslogblock" list more that 15 times within 15 minutes, then (s)he will be added to the "block" list. And if someone is in the "block" list, (s)he will be blocked for an hour. Note (1): if you replace the "--rcheck" here with an "--update" statement, the block will last even longer: "--rcheck" says: were you blocked less than an hour ago? Then we can't let you in. While "--update" says: were you blocked less than an hour ago, but you're here *again*? We'll block you for *another* hour, starting now. So the latter means that you actually need to be quiet for 60 minutes to be able to log in again. There is only one caveat in this whole setup: if you (or syslog-ng) inadvertedly send garbage to the /proc/net/xt_recent/syslogblock file, then this file will be closed. Syslog-ng will reopen it after the "reopen" timeout, so your message will probably come through; but should something change in the working of syslog-ng, like should syslog-ng try to *re-send* such a broken message, then this will, effectively, block the rest of the blocking messages. (I don't think this will happen, because "fprintf" is not a reliable transport, so to say, but you'll never know). I'll be implementing this solution for a couple of other servers in the next weeks or so, and I'll report if there is more to say about it. Oh, BTW: I think that these two lines: ip6tables -A INPUT -m recent --rcheck --name syslogblock --seconds 900 --hitcount 15 -j block ip6tables -A block -m recent --rcheck --name block --set can be concatenated so that you don't need a separate "block" chain anymore, as follows: ip6tables -A INPUT -m recent --rcheck --name syslogblock --seconds 900 --hitcount 15 -m recent --rcheck --name block --set But I'm not sure, I'll have to check. Best regards, Valentijn -- http://www.openoffice.nl/ Open Office - Linux Office Solutions Valentijn Sessink v.sessink#openoffice.nl +31(0)20-4214059

5 14

Syntax check / reload / duplicate messages issues in 3.2.2 (and 3.1.1)
by Jakub Jankowski 23 Feb '11

23 Feb '11

Hi, I've came across a weird bug in syslog-ng OSE 3.1.1, which is also present in 3.2.2: when I add a non-existant destination to syslog-ng config: [root@test-syslog-ng ~]# echo 'log { source(s_sys); filter(f_default); destination(d_nonexistent); };' >> /etc/syslog-ng/syslog-ng.conf it passes syntax checking ($SYSLOGNG_OPTIONS is basically --no-caps -p /pid/file/path - I need no-caps because this is virtual environment): [root@test-syslog-ng ~]# . /etc/sysconfig/syslog-ng && syslog-ng -s $SYSLOGNG_OPTIONS [root@test-syslog-ng ~]# echo $? 0 but after /etc/init.d/syslog-ng reload (HUP) this happens: Feb 18 14:17:40 test-syslog-ng syslog-ng[30198]: Error in configuration, unresolved destination reference; destination='d_nonexistent' Feb 18 14:17:40 test-syslog-ng syslog-ng[30198]: Error in configuration, unresolved destination reference; destination='d_nonexistent' Feb 18 14:17:40 test-syslog-ng syslog-ng[30198]: Error initializing new configuration, reverting to old config; Feb 18 14:17:40 test-syslog-ng syslog-ng[30198]: Error initializing new configuration, reverting to old config; Feb 18 14:17:40 test-syslog-ng syslog-ng[30198]: Configuration reload request received, reloading configuration; Feb 18 14:17:40 test-syslog-ng syslog-ng[30198]: Configuration reload request received, reloading configuration; Afterwards, everything is logged twice(!) [root@test-syslog-ng ~]# R=$RANDOM; echo $R; logger -t test $R 578 [root@test-syslog-ng ~]# grep "test: $R" /var/log/messages Feb 18 14:38:28 test-syslog-ng test: 578 Feb 18 14:38:28 test-syslog-ng test: 578 [root@test-syslog-ng ~]# Another reload, and afterwards every message is logged 3 times: [root@test-syslog-ng ~]# service syslog-ng reload Reloading syslog-ng: [ OK ] [root@test-syslog-ng ~]# R=$RANDOM; echo $R; logger -t test $R 13143 [root@test-syslog-ng ~]# grep "test: $R" /var/log/messages Feb 18 14:39:57 test-syslog-ng test: 13143 Feb 18 14:39:57 test-syslog-ng test: 13143 Feb 18 14:39:57 test-syslog-ng test: 13143 [root@test-syslog-ng ~]# Can anyone confirm this behaviour? It's all fine if I stop syslog-ng and start it again (well, try to start): [root@test-syslog-ng ~]# service syslog-ng start Starting syslog-ng: Error in configuration, unresolved destination reference; destination='d_nonexistent' [FAILED] I see two bugs here: 1) syntax checking (-s) is, well, just syntax checking - but doesn't guarantee that config is valid. 2) after reloading configuration with unresolved destination N times, every message is logged 1+N times. My environment: CentOS5, syslog-ng (without any patches) built from my own .spec, which I made by adapting EPEL srpm (2.x) to 3.x. [root@test-syslog-ng ~]# syslog-ng -V syslog-ng 3.2.2 Installer-Version: 3.2.2 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.2#master#1d3f396485eb47b1ff6aa18ac4f1c4cd51c0ea4c Compile-Date: Jan 26 2011 11:44:50 Enable-Threads: on Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-SSL: on Enable-SQL: on Enable-Linux-Caps: on Enable-Pcre: off Enable-Pacct: off [root@test-syslog-ng ~]# Config file is pretty straightforward: [root@test-syslog-ng ~]# cat /etc/syslog-ng/syslog-ng.conf @version: 3.2 options { flush_lines (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (yes); keep_hostname (yes); }; source s_sys { file ("/proc/kmsg" program_override("kernel") flags(no-multi-line)); unix-stream ("/dev/log" flags(no-multi-line)); internal(); # udp(ip(0.0.0.0) port(514) flags(no-multi-line)); }; destination d_cons { file("/dev/console" group("root")); }; destination d_mesg { file("/var/log/messages" group("root")); }; destination d_auth { file("/var/log/secure" group("root")); }; destination d_mail { file("/var/log/maillog" group("root") flush_lines(10)); }; destination d_spol { file("/var/log/spooler" group("root")); }; destination d_boot { file("/var/log/boot.log" group("root")); }; destination d_cron { file("/var/log/cron" group("root")); }; destination d_kern { file("/var/log/kern" group("root")); }; destination d_mlal { usertty("*"); }; # Facilities/levels filter f_kernel { facility(kern); }; filter f_default { level(info..emerg) and not (facility(mail) or facility(authpriv) or facility(cron)); }; filter f_auth { facility(authpriv); }; filter f_mail { facility(mail); }; filter f_emergency { level(emerg); }; filter f_news { facility(uucp) or (facility(news) and level(crit..emerg)); }; filter f_boot { facility(local7); }; filter f_cron { facility(cron); }; log { source(s_sys); filter(f_kernel); destination(d_kern); }; log { source(s_sys); filter(f_default); destination(d_mesg); }; log { source(s_sys); filter(f_auth); destination(d_auth); }; log { source(s_sys); filter(f_mail); destination(d_mail); }; log { source(s_sys); filter(f_emergency); destination(d_mlal); }; log { source(s_sys); filter(f_news); destination(d_spol); }; log { source(s_sys); filter(f_boot); destination(d_boot); }; log { source(s_sys); filter(f_cron); destination(d_cron); }; [root@test-syslog-ng ~]# Regards, -- Jakub Jankowski|shasta@toxcorp.com|http://toxcorp.com/ GPG: FCBF F03D 9ADB B768 8B92 BB52 0341 9037 A875 942D

2 1

PatternDB: macros extraction from URLs
by Ioan Indreias 23 Feb '11

23 Feb '11

Hello all, We are using OSE 3.2.1 version and till now we have managed to configure most of the patterns we need. However we have reached to a point where we need some hints from the users of this list. The problem is the following: how to extract macros when the order of them are not known (like in a URL). For example we would like to extract 'user' and 'action' from an URL like the one mentioned below: APP[9988]: WEB[0011]: http://abc.example.com/query.php?user=test1&action=login&host=prod1&device=… HTTP 1.1 Unfortunately the "user" and "action" could be placed anywhere in the URL (as the URL is not created by aour application) thus we have to create something like this: <pattern>http://abc.example.com/query.php@ESTRING::u@ser=@ESTRING:user:&@action=…</pattern> <pattern>http://abc.example.com/query.php@ESTRING::u@ser=@ESTRING:user:&@ESTRING…</pattern> <pattern>http://abc.example.com/query.php@ESTRING::a@ction=@ESTRING:action:&@use…</pattern> <pattern>http://abc.example.com/query.php@ESTRING::a@ction=@ESTRING:action:&@EST…</pattern> and so on.... Not to mention if we need to extract the 'device' macro as well - the number of patterns grow significantly. Have somebody some hints on how to optimize the extraction of macros when them are not in an known order? Thank you in advance, Ioan

6 7

Newer libtool and autoconf versions?
by Corinna Vinschen 23 Feb '11

23 Feb '11

Hi, Right now, the current 3.2.1 sources can't be build on Cygwin. Apart from two minor problems which are fixed in git (bugzilla items 101 and 102) and a refresh of the files in contrib/cygwin-packaging (bugzilla entry 103), there's a somewhat more troubeling problem. The source tarballs contain libtool/autoconf-generated files which have been created with older versions of these tools, namely libtool 2.2.6b and autoconf 2.65. In order to get a working libtool *.la dependency mechanism on Cygwin, these files have to be created with pretty new versions of libtool/autoconf, right now at least libtool 2.4 and autoconf 2.67 (automake is not affected). Otherwise, with the old versions, building the shared libs fails due to a libtool bug. While I can rectify this situation locally by running autoreconf, it would be nice if building syslog-ng from a tarball would work out of the box on Cygwin, just as on other systems. So, here's my question: Is it possible to generate subsequent source tarballs using these newer versions of libtool and autoconf? Thanks in advance, Corinna -- Corinna Vinschen Cygwin Project Co-Leader Red Hat

5 32

Help required for Syslog-ng compilation on AIX 5.2.
by Rakesh Rajasekharan 23 Feb '11

23 Feb '11

Hi, I need some urgent help or advice to address a make error that I am getting during my compilation of syslog-ng on AIX machine using the --enable-static-linking option I am using gcc3.3.2 versoin. Below was my configure options that I used ./configure --enable-mixed-linking --prefix=/opt/soe/local/syslog-ng-3.0.7 configure goes fine but the make breaks with the below error. I first doubted flex for this but I do have that installed on my machine. mv -f .deps/main.Tpo .deps/main.Po gcc -std=gnu99 -g -O2 -Wall -D_LARGE_FILES=1 -L/usr/local/lib -o syslog-ng main.o libsyslog-ng.a -static -lrt -lnsl -lfl -L/usr/local/lib -lglib-2.0 -lintl -liconv -L/opt/soe/local/syslog-ng-3.0.7/eventlog/lib -levtlog -ldl globals.o ld: 0711-340 SEVERE ERROR Symbol yyout in object libsyslog-ng.a[cfg-lex.o] A csect with multiple labels cannot be moved into the TOC. ld: 0711-340 SEVERE ERROR Symbol yyin in object libsyslog-ng.a[cfg-lex.o] A csect with multiple labels cannot be moved into the TOC. ld: 0711-340 SEVERE ERROR Symbol yyout in object libsyslog-ng.a[cfg-lex.o] A csect with multiple labels cannot be moved into the TOC. ld: 0711-340 SEVERE ERROR Symbol yyin in object libsyslog-ng.a[cfg-lex.o] A csect with multiple labels cannot be moved into the TOC. ld: 0711-340 SEVERE ERROR Symbol yyout in object libsyslog-ng.a[cfg-lex.o] A csect with multiple labels cannot be moved into the TOC. ld: 0711-340 SEVERE ERROR Symbol yyout in object libsyslog-ng.a[cfg-lex.o] A csect with multiple labels cannot be moved into the TOC. ld: 0711-340 SEVERE ERROR Symbol yylineno in object libsyslog-ng.a[cfg-lex.o] A csect with multiple labels cannot be moved into the TOC. ld: 0711-340 SEVERE ERROR Symbol yylineno in object libsyslog-ng.a[cfg-lex.o] A csect with multiple labels cannot be moved into the TOC. ld: 0711-340 SEVERE ERROR Symbol yylineno in object libsyslog-ng.a[cfg-lex.o] A csect with multiple labels cannot be moved into the TOC. ld: 0711-340 SEVERE ERROR Symbol yylineno in object libsyslog-ng.a[cfg-lex.o] A csect with multiple labels cannot be moved into the TOC. ld: 0711-340 SEVERE ERROR Symbol yylineno in object libsyslog-ng.a[cfg-lex.o] A csect with multiple labels cannot be moved into the TOC. ld: 0711-340 SEVERE ERROR Symbol yylineno in object libsyslog-ng.a[cfg-lex.o] A csect with multiple labels cannot be moved into the TOC. ld: 0711-340 SEVERE ERROR Symbol yylineno in object libsyslog-ng.a[cfg-lex.o] A csect with multiple labels cannot be moved into the TOC. ld: 0711-340 SEVERE ERROR Symbol yylineno in object libsyslog-ng.a[cfg-lex.o] A csect with multiple labels cannot be moved into the TOC. ld: 0711-340 SEVERE ERROR Symbol yylineno in object libsyslog-ng.a[cfg-lex.o] A csect with multiple labels cannot be moved into the TOC. ld: 0711-224 WARNING: Duplicate symbol: macros ld: 0711-345 Use the -bloadmap or -bnoquiet option to obtain more information. Please provide your inputs. Thanks & Reagrds, Rakesh

1 0

[Bug 113] New: hang on disk access seem to cause network logging to stop
by bugzilla＠bugzilla.balabit.com 23 Feb '11

23 Feb '11

https://bugzilla.balabit.com/show_bug.cgi?id=113 Summary: hang on disk access seem to cause network logging to stop Product: syslog-ng Version: 3.0.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: arekm(a)maven.pl Type of the Report: bug Estimated Hours: 0.0 syslog-ng deadlocks in some disk operation (faulty hardware; controler somehow locks up etc) then network logging to another host is also stopping. This is very unfortunate because network logging could give provide me a hints on why disks deadlocks etc. Also if we have few hard disks mounted in /mnt/test1, /mnt/test2, we log things into /mnt/test1/logfile, /mnt/test2/logfile and test test1 disk deadlocks then syslog-ng will also stop logging to test2/logfile. Basically one deadlocked device locks everything else. Major pain here. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3

syslog_ng 3.0.7 make fails on RHEL.
by Rakesh Rajasekharan 23 Feb '11

23 Feb '11

Hi, I am finding no success trying to compile syslog-ng 3.0.7 on the RHEL machine using the --enable-static-linking mode. I get stuck in the make stage. I am using GNU gcc 3.4.5 version .Below is the error that appears in make. gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I.. -I/usr/local/include/glib-2.0 -I/usr/local/lib/glib-2.0/include -I/opt/soe/local/syslog-ng-3.0.7/eventlog/include/eventlog -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -g -O2 -Wall -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c mv -f .deps/main.Tpo .deps/main.Po gcc -std=gnu99 -g -O2 -Wall -o syslog-ng main.o libsyslog-ng.a -static -lrt -lnsl -lfl -L/usr/local/lib -lglib-2.0 -liconv -L/opt/soe/local/syslog-ng-3.0.7/eventlog/lib -levtlog -ldl -lwrap -lcap globals.o /usr/local/lib/gcc/i686-pc-linux-gnu/3.4.5/../../../../i686-pc-linux-gnu/bin/ld: cannot find -lcap collect2: ld returned 1 exit status make[2]: *** [syslog-ng] Error 1 make[2]: Leaving directory `/compile/syslog-ng-3.0.7/src' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/compile/syslog-ng-3.0.7' make: *** [all] Error 2 [root@CSCSVSLGG189v syslog-ng-3.0.7]# which ld /usr/local/bin/ld [root@CSCSVSLGG189v syslog-ng-3.0.7]# ld -V GNU ld version 2.17 Supported emulations: elf_i386 i386linux Can you please advice on this. Thanks & Regards, Rakesh

3 4

using correlation to filter out some messages ?
by Guillaume Rousse 22 Feb '11

22 Feb '11

Hello. I've had a look at message correlation in the 3.2 manual, but it's still a bit obscure for me to understand if it could allow me to achieve filtering of pairs of log messages, as described here: https://lists.balabit.hu/pipermail/syslog-ng/2010-January/013857.html -- BOFH excuse #360: Your parity check is overdrawn and you're out of cache.

5 19

abandon libdbi and write a new sql driver?
by Patrick H. 22 Feb '11

22 Feb '11

Reading through the mailing list lately I've seen a fair amount of problems reported with libdbi. Given that it also doesnt support prepared statements which is a very important feature, I'm wondering if libdbi should just be dropped entirely. Libdbi seems to be of use to applications that want just very basic sql support, and not for anything involving high performance or reliability. I know its no small undertaking to write a new DBI for all the common databases out there, but if syslog-ng is supposed to be high-performing, libdbi does not fit in with that. I dont know if it'd be best to start a new project to replace libdbi but accomplish the same thing of providing a common API for all databases, or to just write separate modules for syslog-ng 3.2+ for each database out there. Writing modules would certainly be easier as they could be written independently by people who best know each database API. This is all just my opinion, but I for one am all for it. In my production environment, we do logging to a database, and libdbi just didnt have the flexibility or performance we needed, so I ended up writing a separate program which connects to the database to perform the inserts. Would be nice to avoid having to do that. I know I've also heard many recommendations of piping out to a perl script to do the inserts. A perl script should never do its job faster than C code. -Patrick

6 9

syslog-ng 3.2.2 OSE takes 100% CPU after source files get truncated.
by Mailing Lists 22 Feb '11

22 Feb '11

Hi, Not sure if syslog-ng has a bug tracker, but I couldn't find one anyway, so posting this here. Here is the problem: I have source file that is a log file that is been truncated periodically (by log rotate). When the truncation happens syslog-ng process starts taking 100% CPU, and not doing any logging. This behavior remains until source file receives a write. To reproduce: 1. configure syslog-ng with following lines: source s_logfiles { file("/var/log/applogs/mysql/mysqld.log" flags(no-parse) program_override("mysqld.log")); file("/var/log/applogs/mysql/mysql-slow.log" flags(no-parse) program_override("mysql-slow.log")); }; destination d_mysql_logall { file("/var/log/mysqlall.log"); }; log { source(s_logfiles); destination(d_mysql_logall);}; 2. start syslog-ng 3. output some text into source files: for i in /var/log/applogs/mysql/mysqld.log /var/log/applogs/mysql/mysql-slow.log ; do echo "some text" >> $i; done 4. truncate source files: for i in /var/log/applogs/mysql/mysqld.log /var/log/applogs/mysql/mysql-slow.log ; do > $i; done 5. observe syslog-ng takes 100% CPU, doing nothing. 6. output some text into source files again: for i in /var/log/applogs/mysql/mysqld.log /var/log/applogs/mysql/mysql-slow.log ; do echo "some text, again" >> $i; done 7. observe syslog-ng returned to normal operation. comment: some times you'll need going through steps 3 - 5 a few times to trigger the bug. My config: syslog-ng 3.2.2 Installer-Version: 3.2.2 Revision: ssh+git://bazsi@git.balabit //var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.2#master#1d3f396485eb47b1ff6aa18ac4f1c4cd51c0ea4c Compile-Date: Jan 20 2011 11:34:35 Enable-Threads: off Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-SSL: on Enable-SQL: off Enable-Linux-Caps: off Enable-Pcre: on Enable-Pacct: off Regards.

3 18

[Bug 109] New: building on Cygwin fails in modules/dbparser
by bugzilla＠bugzilla.balabit.com 22 Feb '11

22 Feb '11

https://bugzilla.balabit.com/show_bug.cgi?id=109 Summary: building on Cygwin fails in modules/dbparser Product: syslog-ng Version: 3.2.x Platform: PC OS/Version: Other Status: NEW Severity: blocker Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: vinschen(a)redhat.com Type of the Report: regression Estimated Hours: 0.0 Created an attachment (id=35) --> (https://bugzilla.balabit.com/attachment.cgi?id=35) Patch to fix build problem in modules/dbparser I just tested the new implementation from git, and gcc fails to build libsyslog-ng-patterndb.dll.a since a lot of function calls are missing: libtool: link: gcc -std=gnu99 -shared .libs/radix.o .libs/patterndb.o .libs/t imerwheel.o .libs/patternize.o -lssl -lcrypto -lz -lrt /usr/lib/libgmodule-2.0.dll.a -L/usr/lib /usr/lib/libglib-2.0.dll.a /usr/lib/libintl.dll.a /usr/lib/libiconv.dll.a /usr/lib/libevtlog.dll.a -lresolv /usr/lib/libpcre.dll.a -O2 -Wl,--export-all-symbols -o .libs/cygsyslog-ng-patterndb-0.dll -Wl,--enable-auto-image-base -Xlinker --out-implib -Xlinker .libs/libsyslog-ng-patterndb.dll.a Creating library file: .libs/libsyslog-ng-patterndb.dll.a .libs/radix.o: In function `r_new_pnode': modules/dbparser/radix.c:374: undefined reference to `_log_msg_get_value_handl e' modules/dbparser/radix.c:618: undefined reference to `_msg_limit_internal_mess age' modules/dbparser/radix.c:618: undefined reference to `_log_msg_get_value_name' modules/dbparser/radix.c:618: undefined reference to `_msg_event_create' modules/dbparser/radix.c:629: undefined reference to `_msg_limit_internal_mess age' modules/dbparser/radix.c:567: undefined reference to `_msg_event_create' modules/dbparser/radix.c:674: undefined reference to `_msg_limit_internal_mess age' modules/dbparser/radix.c:674: undefined reference to `_msg_event_create' etc. The reason is that the dependency to libsyslog is missing. Would you mind to apply the attached patch? It fixes the build problem and the resulting pdbtool appears to work fine. Thanks, Corinna -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

filter/parse based on certificate attributes?
by Christian Tramnitz 22 Feb '11

22 Feb '11

With syslog 3.0 and ssl support using mutual certificate authentication is it possible to parse/filter/tag based on attributes in the certificate? I'd like to have a common listener and then relay or append messages based on client-cert attribute values... (Sorry if this message is arriving more than once, I seem to have problems with posting through Gmane.) Regards, Christian

4 4

dbtype in SQL logging
by Hendrik Visage 22 Feb '11

22 Feb '11

Hi there, found the following entry in the Administrator's guide(s): <quote> columns Type: string list Default: "date", "facility", "level", "host", "program", "pid", "message" *Description:* Name of the columns storing the data in *fieldname [dbtype]*format. The *[dbtype]* parameter is optional, and specifies the type of the field. By default, syslog-ng creates *text* columns. Note that not every database engine can index text fields. </quote> So now before I go search the source code for answers, what types of "[dbtype]" is supported/handled by syslog-ng 3.2.2? More precisely, I'm running on HP-UX 11.31 and preference is using PostgreSQL Thank you HEndrik

2 1

syslog-ng 3.2.2
by Rakesh Rajasekharan 20 Feb '11

20 Feb '11

Hi , I was trying to compile the recently releases 3.2.2 version. I notoiced that the --enable-static-linking option is not present in the configure script. Can someone please confirm. Thanks, Rakesh

3 2

Default name value pair
by John Kristoff 20 Feb '11

20 Feb '11

Perhaps the functionality is there and I don't realize it, and this is not critical if it's not, but it would come in handy to be able to do something like the following: destination d_sql { sql( ... values( "${FOO:-default_value}", ...) In other words, being able to specify a default value for the name-value pair when one doesn't exist much like you can do in UNIX shell scripts. Similarly, but in my case otherwise, using patterndb, it would be handy to be able to set a name-value pair per <pattern>. I believe right now you can only do do based on the upper level <patterns> hierarchy. To help clarify the reasoning, I have a database table with a field that I'd like to fill with at least a default value, but that field might not exist in the log message in some cases. For instance, BIND 9.x query logs can log the "view", but unless you configure views for all the possible classes, which is generally not done, queries for other classes will not have the view identifier. Compare these common examples: <test_message>queries: info: client 127.0.0.1#1024: view lopback: query: 1.0.0.127.in-addr.arpa IN PTR +</test_message> <test_message>queries: info: client 192.0.2.1#1024: query: version.bind CH TXT + (192.0.2.1)</test_message> Right now the view field gets logged as a null, which is OK, but I'd rather it be inserted with a default view of 'default' in this example. John

2 1

duplicate config elements in sql destinations
by John Kristoff 20 Feb '11

20 Feb '11

I have a syslog-ng.conf that contains multiple destination sections that all begin like this: destination d_sql_[foo] { sql( type(pgsql) host("127.0.0.1") database("bar") ... "[foo]" differs between each section as well as the table name, columns and values. However, syslog-ng complains with the following: Internal error, duplicate configuration elements refer to the same persistent config; name='afsql_dd(pgsql,127.0.0.1,,bar)' Things appear to work, but apparently syslog-ng is confused by the duplicate database destination, even if the tables are different. Is this merely cosmetic or does this have wider implications? Perhaps they each end up sharing the same fifo buffer when they shouldn't for instance? John

2 1

[PATCH][patternize] support for custom delimiters
by Balint Kovacs 20 Feb '11

20 Feb '11

Hi, with some time at my hands over the weekend I was finally able finish support for multiple delimiters in 'pdbtool patternize'. Rationale: The current implementation of patternize only splits up log messages at spaces, which is inaccurate - while working with patterns, I found that most of the volatile content in messages is enclosed in or followed by some kind of separator character(s). So the best way of handling this would be to make the user be able to define the delimiters and provide a sane default. Implementation: The delimiter list is extracted from each message separately after tokenization (this way it delimiters do not affect frequent word collection) and then added to the end of the cluster key in the SLCT phase to ensure unicity. When printing the pattern, the delimiter list is popped from the cluster key and as it is in the same order as the extracted words, we print delimiter with the same index as the word's within the for loop. The user input needs to be sanitized, we have to make sure that the space char is always in the delimiter list and that there are no duplicates. I have also changed the the parser marker char from * to 0x1A ("substitute" char) to prevent parser "false positives" in case a words starts with *. I also added the class and provider properties to each rule to make the generated xml xsd-compliant. The default delimiter list is :&~?![]=,;()'" And the results: Running time and memory consumption have almost doubled(!), as far as I could test, this comes from the fact, that we have way many more tokens than with splitting at spaces, so with the current structure we can't really do much about that. BUT accuracy has increased quite a lot, I think there is a visible difference. For the log message: core.session(3): (svc/intra_HTTP:1486416/http): Server connection established; server_fd='46', server_address='AF_INET(1.2.3.4:80)', server_zone='Zone(internet, 0.0.0.0/0)', server_local='AF_INET(1.2.3.4:52007)', server_protocol='TCP' With the tokenizing at spaces: core.session(3): @ESTRING:: @Server connection established; @ESTRING:: @@ESTRING:: @server_zone='Zone(internet, 0.0.0.0/0)', @ESTRING:: @server_protocol='TCP' And with the new default delimiters: core.session(3): (svc/intra_HTTP:@ESTRING::)@: Server connection established; server_fd='@ESTRING::'@, server_address='AF_INET(@ESTRING:::@80)', server_zone='Zone(internet, 0.0.0.0/0)', server_local='AF_INET(1.2.3.4:@ESTRING::)@', server_protocol='TCP' (please note that the log sample used for generating these patterns was grepped on intra_HTTP, that's why the zorp service name and server port 80 are found as a frequent words and not ESTRING'd) Regards, Balint

2 1

[development] open questions
by Balazs Scheidler 20 Feb '11

20 Feb '11

Hi, I had some distraction in the last two weeks, thus I was unable to follow the syslog-ng mailing list/emails as closely I would have wanted. I've finished reading all the unread emails in my syslog-ng folder, and I wanted to summarize the status quo: Bugfixes/maintenance (Bazsi): * I've merged 3.2 into 3.3 to get the latest changes, bugfixes. libsyslog-ng-patterndb.so (Corinna, Marius Tomaschewski, Peter Czanik): * causes problems on cygwin & SUSE, so probably an alternative solution is needed. * my current idea is either to use Corinna's idea and put pdbtool in /usr/lib/syslog-ng (which I personally don't really like, as /usr/lib is not the place for binaries), or that libsyslog-ng-patterndb.so should be moved /usr/lib _after_ compilation by the distribution build scripts. * Marius, Corinna what's your take on this issue? $(format-json) template function (Balint, Algernon, Matthew) * waiting for the generic value-pairs() implementation * it could probably be added to 3.2 as well, if value-pairs doesn't prove to be risky in itself (or else they'd have to be put in the tfjson module). mongodb dynamic values support (Algernon) * waiting for the value-pairs() implementation (Algernon) * this is 3.3 stuff systemd support (Marius Tomaschewski, Tom Gundersen, Algernon) * should be more-or-less ready, Algernon promised to rebase it against both 3.2 and 3.3 , so that I can integrate into both. LogQueue scalability (Bazsi): * I was working on making LogQueue more scalable -- Bazsi

2 3