- syslog-ng - lists.balabit.hu

[PATCH 1/2] [dbparser] inherit nvpairs from parent in correlation action
by balint.kovacs＠balabit.com 16 Nov '12

16 Nov '12

From: Balint Kovacs <blint(a)blint.hu> This patch implements an inherit-name-value-pairs attribute for message type actions is patterndb correlation. In case this is set to TRUE, the triggering log message is cloned and all name-value pairs are automagically copied to the emitted message. The values set in the values XML subtree overwrite the original values. In case it is set to FALSE or unspecified, a new message is generated and only the values set manually are added to it. <actions> <action> <message inherit-name-value-pairs='TRUE'> <values> <value name="PROGRAM">my-program-name</value> <value name="MSG">Hello new message!</value> <value name="contextid">${session-id}@2</value> </values> </message> </action> </actions> Signed-off-by: Balint Kovacs <blint(a)blint.hu> --- modules/dbparser/patterndb-int.h | 1 + modules/dbparser/patterndb.c | 32 +++++++++++++++++++++++++++++--- 2 files changed, 30 insertions(+), 3 deletions(-) diff --git a/modules/dbparser/patterndb-int.h b/modules/dbparser/patterndb-int.h index 0434847..154b686 100644 --- a/modules/dbparser/patterndb-int.h +++ b/modules/dbparser/patterndb-int.h @@ -126,6 +126,7 @@ typedef struct _PDBAction guint8 content_type; guint16 rate; guint32 id:8, rate_quantum:24; + gboolean inherit_nvpairs; union { PDBMessage message; diff --git a/modules/dbparser/patterndb.c b/modules/dbparser/patterndb.c index 8720ea0..168a24c 100644 --- a/modules/dbparser/patterndb.c +++ b/modules/dbparser/patterndb.c @@ -388,6 +388,17 @@ pdb_action_set_trigger(PDBAction *self, const gchar *trigger, GError **error) g_set_error(error, 0, 1, "Unknown trigger type: %s", trigger); } +void +pdb_action_set_inheritance(PDBAction *self, const gchar *inherit_nvpairs, GError **error) +{ + if (strcmp(inherit_nvpairs, "TRUE") == 0) + self->inherit_nvpairs = TRUE; + else if (strcmp(inherit_nvpairs, "FALSE") == 0) + self->inherit_nvpairs = FALSE; + else + g_set_error(error, 0, 1, "Unknown inheritance type: %s", inherit_nvpairs); +} + PDBAction * pdb_action_new(gint id) { @@ -398,6 +409,7 @@ pdb_action_new(gint id) self->trigger = RAT_MATCH; self->content_type = RAC_NONE; self->id = id; + self->inherit_nvpairs = FALSE; return self; } @@ -552,9 +564,18 @@ pdb_rule_run_actions(PDBRule *self, gint trigger, PatternDB *db, PDBContext *con case RAC_NONE: break; case RAC_MESSAGE: - genmsg = log_msg_new_empty(); - genmsg->flags |= LF_LOCAL; - genmsg->timestamps[LM_TS_STAMP] = msg->timestamps[LM_TS_STAMP]; + if (action->inherit_nvpairs) + { + LogPathOptions path_options = LOG_PATH_OPTIONS_INIT; + path_options.ack_needed = FALSE; + genmsg = log_msg_clone_cow(msg, &path_options); + } + else + { + genmsg = log_msg_new_empty(); + genmsg->flags |= LF_LOCAL; + genmsg->timestamps[LM_TS_STAMP] = msg->timestamps[LM_TS_STAMP]; + } if (context) { switch (context->key.scope) @@ -942,6 +963,11 @@ pdb_loader_start_element(GMarkupParseContext *context, const gchar *element_name } else if (strcmp(element_name, "message") == 0) { + for (i = 0; attribute_names[i]; i++) + { + if (strcmp(attribute_names[i], "inherit-name-value-pairs") == 0) + pdb_action_set_inheritance(state->current_action, attribute_values[i], error); + } if (!state->in_action) { *error = g_error_new(1, 0, "Unexpected <message> element, it must be inside an action"); -- 1.7.9.5

5 9

[PATCH (3.4)] afmongodb: Add support for datetime fields
by Gergely Nagy 15 Nov '12

15 Nov '12

For TTL collections primarly (but useful for other scenarios aswell), introduce a new option: datetime(FIELD_NAME). This allows one to mark a single field anywhere in the structure as a datetime field, and the driver will attempt to convert it to a proper datetime, by assuming it is a unix timestamp. If the conversion fails, or the field is not found, it will not be inserted. Usage: mongodb(datetime("some.date") value-pairs( key("MESSAGE") pair("some.date", "$UNIXTIME"))) This will result in something like: { "_id" : ObjectId("509b8ffc1a1006213a000001"), "some" : { "date" : ISODate("2012-11-08T10:56:59Z") }, "MESSAGE" : "foobar" } Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/afmongodb/afmongodb-grammar.ym | 2 ++ modules/afmongodb/afmongodb-parser.c | 1 + modules/afmongodb/afmongodb.c | 61 +++++++++++++++++++++++++++++--- modules/afmongodb/afmongodb.h | 1 + 4 files changed, 61 insertions(+), 4 deletions(-) diff --git a/modules/afmongodb/afmongodb-grammar.ym b/modules/afmongodb/afmongodb-grammar.ym index f108f26..40eccc0 100644 --- a/modules/afmongodb/afmongodb-grammar.ym +++ b/modules/afmongodb/afmongodb-grammar.ym @@ -53,6 +53,7 @@ extern ValuePairsTransformSet *last_vp_transset; %token KW_SERVERS %token KW_SAFE_MODE %token KW_PATH +%token KW_DATETIME %% @@ -97,6 +98,7 @@ afmongodb_option | KW_USERNAME '(' string ')' { afmongodb_dd_set_user(last_driver, $3); free($3); } | KW_PASSWORD '(' string ')' { afmongodb_dd_set_password(last_driver, $3); free($3); } | KW_SAFE_MODE '(' yesno ')' { afmongodb_dd_set_safe_mode(last_driver, $3); } + | KW_DATETIME '(' string ')' { afmongodb_dd_set_datetime(last_driver, $3); free($3); } | value_pair_option { afmongodb_dd_set_value_pairs(last_driver, $1); } | dest_driver_option ; diff --git a/modules/afmongodb/afmongodb-parser.c b/modules/afmongodb/afmongodb-parser.c index 8aa55c1..1c51154 100644 --- a/modules/afmongodb/afmongodb-parser.c +++ b/modules/afmongodb/afmongodb-parser.c @@ -39,6 +39,7 @@ static CfgLexerKeyword afmongodb_keywords[] = { { "host", KW_HOST }, { "port", KW_PORT }, { "path", KW_PATH }, + { "datetime", KW_DATETIME }, { NULL } }; diff --git a/modules/afmongodb/afmongodb.c b/modules/afmongodb/afmongodb.c index 205ab98..bb267a5 100644 --- a/modules/afmongodb/afmongodb.c +++ b/modules/afmongodb/afmongodb.c @@ -22,6 +22,7 @@ */ #include <time.h> +#include <stdlib.h> #include "afmongodb.h" #include "afmongodb-parser.h" @@ -32,6 +33,7 @@ #include "nvtable.h" #include "logqueue.h" #include "value-pairs.h" +#include "scratch-buffers.h" #include "mongo.h" @@ -55,6 +57,7 @@ typedef struct gint port; gboolean safe_mode; + gchar *datetime_field; gchar *user; gchar *password; @@ -207,6 +210,15 @@ afmongodb_dd_set_safe_mode(LogDriver *d, gboolean state) self->safe_mode = state; } +void +afmongodb_dd_set_datetime(LogDriver *d, const gchar *field) +{ + MongoDBDestDriver *self = (MongoDBDestDriver *)d; + + g_free(self->datetime_field); + self->datetime_field = g_strdup(field); +} + /* * Utilities */ @@ -343,7 +355,7 @@ afmongodb_vp_obj_end(const gchar *name, if (prev_data) root = (bson *)*prev_data; else - root = (bson *)user_data; + root = (bson *)(((gpointer *)user_data)[0]); if (prefix_data) { @@ -357,6 +369,30 @@ afmongodb_vp_obj_end(const gchar *name, } static gboolean +afmongodb_field_is_datetime(const gchar *prefix, const gchar *name, + const gchar *datetime_field) +{ + size_t plen = 0; + + if (!datetime_field) + return FALSE; + + if (prefix) + plen = strlen(prefix); + + if (strncmp(datetime_field, prefix, plen) != 0) + return FALSE; + + if (datetime_field[plen] != '.') + return FALSE; + + if (strcmp(&datetime_field[plen + 1], name) != 0) + return FALSE; + + return TRUE; +} + +static gboolean afmongodb_vp_process_value(const gchar *name, const gchar *prefix, const gchar *value, gpointer *prefix_data, gpointer user_data) @@ -366,7 +402,7 @@ afmongodb_vp_process_value(const gchar *name, const gchar *prefix, if (prefix_data) o = (bson *)*prefix_data; else - o = (bson *)user_data; + o = (bson *)(((gpointer *)user_data)[0]); if (name[0] == '.') { @@ -378,7 +414,20 @@ afmongodb_vp_process_value(const gchar *name, const gchar *prefix, bson_append_string (o, tx_name, value, -1); } else - bson_append_string (o, name, value, -1); + { + MongoDBDestDriver *self = (MongoDBDestDriver *)(((gpointer *)user_data)[1]); + + if (afmongodb_field_is_datetime(prefix, name, self->datetime_field)) + { + gchar *endptr; + gint64 datetime = (gint64)strtoul(value, &endptr, 10) * 1000; + + if (endptr[0] == '\0') + bson_append_utc_datetime(o, name, datetime); + } + else + bson_append_string (o, name, value, -1); + } return FALSE; } @@ -390,6 +439,7 @@ afmongodb_worker_insert (MongoDBDestDriver *self) guint8 *oid; LogMessage *msg; LogPathOptions path_options = LOG_PATH_OPTIONS_INIT; + gpointer user_data[] = { NULL, self }; afmongodb_dd_connect(self, TRUE); @@ -408,11 +458,13 @@ afmongodb_worker_insert (MongoDBDestDriver *self) bson_append_oid (self->bson, "_id", oid); g_free (oid); + user_data[0] = self->bson; + value_pairs_walk(self->vp, afmongodb_vp_obj_start, afmongodb_vp_process_value, afmongodb_vp_obj_end, - msg, self->seq_num, self->bson); + msg, self->seq_num, user_data); bson_finish (self->bson); if (!mongo_sync_cmd_insert_n(self->conn, self->ns, 1, @@ -650,6 +702,7 @@ afmongodb_dd_free(LogPipe *d) string_list_free(self->servers); if (self->vp) value_pairs_free(self->vp); + g_free(self->datetime_field); log_dest_driver_free(d); } diff --git a/modules/afmongodb/afmongodb.h b/modules/afmongodb/afmongodb.h index 6a61616..76f8b83 100644 --- a/modules/afmongodb/afmongodb.h +++ b/modules/afmongodb/afmongodb.h @@ -39,6 +39,7 @@ void afmongodb_dd_set_password(LogDriver *d, const gchar *password); void afmongodb_dd_set_value_pairs(LogDriver *d, ValuePairs *vp); void afmongodb_dd_set_safe_mode(LogDriver *d, gboolean state); void afmongodb_dd_set_path(LogDriver *d, const gchar *path); +void afmongodb_dd_set_datetime(LogDriver *d, const gchar *field); gboolean afmongodb_dd_check_address(LogDriver *d, gboolean local); -- 1.7.10.4

2 4

Replace hostname with a hostname extracted from the message
by Clayton Dukes 14 Nov '12

14 Nov '12

Hi Folks, How can I extract a hostname from the message and replace the source ip with that name/ip address? Sample message: Nov 13 18:11:00 myhostname mymessage: Original Host=192.168.6.3:LOW_THRESHOLD_EVENT - 0 AUTHORIZED sessions So, I need a rewrite rule(?) to take 192.168.6.3 and replace $HOST with it: Nov 13 18:11:00 92.168.6.3 mymessage:LOW_THRESHOLD_EVENT - 0 AUTHORIZED sessions ______________________________________________________________ Clayton Dukes ______________________________________________________________

2 6

[RFC]: value-pairs key rewriting outside of key()
by Gergely Nagy 14 Nov '12

14 Nov '12

Hi! One thing I realized just now, is that there was a slight design mistake made when doing the value-pairs() key rewriting functionality: it was baked into the key() statement, so that as of this writing, one can only rewrite keys that were explicitly selected. While this sounds good on paper, it has one major drawback: If I want to rewrite keys that were selected by the scope, I have to explicitly select those keys myself. This makes a lot of things far more complicated than they need be, and this is why there's a hack in the mongodb destination that replaces leading dots with an underscore. Instead of doing that, I'd like to move the mongodb destination to use the key rewriting functionality, and while there, also teach $(format-json) to do something similar. Fortunately, it's only the config parser that needs an update, as the code below it was written in a way that allows me to do what I want. So, I'd like to add another way to configure key rewriting: value-pairs(scope(nv-pairs) rekey(".*", shift(1), add-prefix("_"))) This is very similar to the already existing way, except it's not inside a key(), but in the main value-pairs() body, thus, it will not add extra keys to the set, but will apply the transformations to keys already in it. For format-json, it would be fairly similar: $(format-json --scope nv-pairs --rekey .* --shift 1 --add-prefix "_") This functionality will then make it possible to rewrite keys that were implicitly added to the set by the scope, which in turn would make me much happier. As an FYI, I'll push this change to my merge-queue/3.4 branch tomorrow, unless anyone objects to it, or has a better suggestion by then. -- |8]

1 0

[Bug 205] New: Docs for macro MSGHDR incorrect
by bugzilla＠bugzilla.balabit.com 10 Nov '12

10 Nov '12

https://bugzilla.balabit.com/show_bug.cgi?id=205 Summary: Docs for macro MSGHDR incorrect Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 The docs state; MSGHDR Description: The name and the pid of the program that sent the log message in PROGRAM: PID format. Includes a trailing whitespace. Note that the macro returns an empty value if both the program and pid fields of the message are empty. When this macro is used the format is PROGRAM[PID]: including a trailing space. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 204] New: dbparser docs not clear for @NUMBER@
by bugzilla＠bugzilla.balabit.com 10 Nov '12

10 Nov '12

https://bugzilla.balabit.com/show_bug.cgi?id=204 Summary: dbparser docs not clear for @NUMBER@ Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: packaging AssignedTo: folti(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 the @NUMBER@ parser should state that a leading minus sign is handled for non-hex numbers and that commas are not permitted. Alternatively, making @NUMBER@ accept commas would be nice, or allowing parameters for additional characters, although comma would be the only one I could think of. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5

Debugging tools for syslog-ng
by Anton Koldaev 10 Nov '12

10 Nov '12

Hi, I wanted to know if syslog-ng developers has some tools like mysqltuner or just a shell scripts to check syslog-ng configuration and get some recommendations on tuning? There is a lot of information about all the buffers/window sizes, etc. and it's very complicated to set the proper values if I have a bunch of different sources/destinations. For example if I'm using flow-control+multiple destinations it can stop reading the source at any time and I have no idea when and why it's happening and which value should I tune. -- Best regards, Koldaev Anton

3 12

rsyslog client produces "Error processing log message"
by Andreas Heinlein 08 Nov '12

08 Nov '12

Hello, we have a centralised log server running syslog-ng 3.1 OSE on Debian 6.0. On the client side, we were using syslog-ng but now I'd like to use rsyslog instead (for several reasons). Transport should be TLS-encrypted TCP. I have set up a connection between the two, but apparently syslog-ng fails to parse the log messages sent by rsyslog. Every log line goes like this: Nov 6 11:15:31 admin2-desktop syslog-ng[1578]: Error processing log message: <13>Nov 6 11:15:31 admin2-desktop ah: Test4 Does anyone have an idea what to configure with either rsyslog or syslog-ng so the two understand each other? Relevant server side config: source s_all { syslog(ip(172.16.x.x) port(6514) max_connections(50) tls( key_file("/etc/ssl/private/xxx.pem") cert_file("/etc/ssl/private/xxx.pem") ca_dir("/etc/ssl/certs") peer_verify(optional-untrusted) ) ); }; Relevant client side config: $DefaultNetstreamDriverCAFile /etc/ssl/certs/xxx.pem $DefaultNetstreamDriver gtls # use gtls netstream driver $ActionSendStreamDriverMode 1 # require TLS for the connection $ActionSendStreamDriverAuthMode anon # server is NOT authenticated *.* @@(o)mail:6514;RSYSLOG_TraditionalForwardFormat # (o) seems to be neccessary or syslog-ng will not even accept the connection; RSYSLOG_TraditionalForwardFormat apparently doesn't matter Thanks, Andreas

3 7

syslog-ng Insider - November 2012
by Peter Czanik 08 Nov '12

08 Nov '12

Dear syslog-ng users, This is the 18th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com <mailto:documentation@balabit.com> FEATURED NEWS syslog-ng 3.3.7 is released --------------------------- Version 3.3.7 was released last week, right before the “All Saints Day” holidays. It has many fixes for FreeBSD, so it should compile with the bundled ivykis on FreeBSD 8 and 9. Patches to compile it on FreeBSD 7 will also be included in http://www.freshports.org/sysutils/syslog-ng/ Packages for all supported or development versions of openSUSE are also available: https://build.opensuse.org/project/show?project=home%3Aczanik%3Asyslog-ng33 Packages for Debian and Ubuntu are available at http://asylum.madhouse-project.org/projects/debian/ Announcement for 3.3.7: https://lists.balabit.hu/pipermail/syslog-ng/2012-October/019602.html syslog-ng @ conferences ----------------------- syslog-ng was presented at two conferences in October. Fedora users and developers gathered in Paris: https://czanik.blogs.balabit.com/2012/10/fudcon-fedora-and-syslog-ng/ while openSUSE, SLES and Gentoo developers had their conference in Prague at the technical university: https://czanik.blogs.balabit.com/2012/10/osc-opensuse-and-syslog-ng/ syslog-ng 3.4 news ------------------ Many new features were added during the previous month. AMQP destination, syslog parser, a couple of new parsers for patterndb contributed by Evan Rempel and a few more smaller features and fixes. For a complete list check https://github.com/balabit/syslog-ng-3.4/commits/master NEW RELEASES: * syslog-ng OSE 3.3.7: https://lists.balabit.hu/pipermail/syslog-ng/2012-October/019602.html ARCHIVE http://insider.blogs.balabit.com/ -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

Errors building syslog-ng-3.3.7 under FreeBSD 8.3
by C. L. Martinez 08 Nov '12

08 Nov '12

Hi all, I am trying to build syslog-ng 3.3.7 in a FreeBSD 8.3 i386 host, but make returns me these errors: syslog-ng Open Source Edition 3.3.7 configured Compiler options: compiler : gcc -std=gnu99 compiler options : -g -O2 -Wall -pthread -D_REENTRANT -I/usr/local/include/glib-2.0 -I/usr/local/include/eventlog -I/usr/local/include -I/usr/local/include -I$(top_srcdir)/lib/ivykis/lib/include -I$(top_builddir)/lib/ivykis/lib/include -I$(top_srcdir)/lib/ivykis/modules/include -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 linker flags : prefix : /data/soft/syslog-ng linking mode : dynamic __thread keyword : yes Submodules: ivykis : internal libmongo-client : internal Features: Debug symbols : no GCC profiling : no Memtrace : no IPV6 support : yes spoof-source support : yes tcp-wrapper support : yes Linux capability support : no PCRE support : yes Env wrapper support : no systemd support : no (unit dir: none) Modules: Module search path : /data/soft/syslog-ng/lib/syslog-ng Default module list : affile,afprog,afsocket,afuser,basicfuncs,csvparser,dbparser,syslogformat Sun STREAMS support (module): no SSL support (module) : yes SQL support (module) : no PACCT module (EXPERIMENTAL) : no MongoDB destination (module): yes JSON support (module) : no (using no) root@fbsd02:/tmp/u/syslog-ng-3.3.7# make make all-recursive Making all in lib "Makefile", line 1101: Need an operator make: fatal errors encountered -- cannot continue *** Error code 1 Stop in /tmp/u/syslog-ng-3.3.7. *** Error code 1 Stop in /tmp/u/syslog-ng-3.3.7 My package list: bison-2.5.1,1 A parser generator from FSF, (mostly) compatible with Yacc compat6x-i386-6.4.604000.200810_3 A convenience package to install the compat6x libraries eventlog-0.2.12 Replacement of the simple syslog() API providing structure flex-2.5.37 Fast lexical analyzer generator gamin-0.1.10_4 A file and directory monitoring system gettext-0.18.1.1 GNU gettext package gio-fam-backend-2.28.8_1 FAM backend for GLib\'s GIO library glib-2.28.8_4 Some useful routines of C programming (current stable versi gmake-3.82_1 GNU version of 'make' utility libee-0.3.2 An event expression library inspired by CEE libestr-0.1.2 A library for some string essentials libiconv-1.14 A character set conversion library libnet11-1.1.2.1_4,1 A C library for creating IP packets libsysinfo-0.0.2 GNU libc's sysinfo port for FreeBSD libtool-2.4.2 Generic shared library support script m4-1.4.16_1,1 GNU m4 openssl-1.0.1_4 SSL and crypto library pcre-8.31 Perl Compatible Regular Expressions library perl-5.14.2_2 Practical Extraction and Report Language pkgconf-0.8.8 pkg-config compatible utility which does not depend on glib python27-2.7.3_3 An interpreted object-oriented programming language

2 3