November 2012 - syslog-ng - lists.balabit.hu

merging stuff from merge-queue/3.4
by Balazs Scheidler 22 Nov '12

22 Nov '12

Hi, I have started merging the branch $subject, however I've found some patches I couldn't merge. system-source: Use follow-freq(1) on FreeBSD prior to 9.1 Are you sure time based polling would work here? I mean time based polling also checks if the size of the file has changed, as it was meant to be used only with regular files. I'm afraid that the inability to use kqueue on /dev/klog would prevent using kqueue completely, as the syslog-ng file source code requires either regular files or pollable devices. dbparser: support inheriting properties in a corellation action I merged this one, although I have some doubts. It uses log_msg_clone_cow(), which does have some tricky semantics, as it assumes that the cloned message will not change, once cloned. I was thinking about removing clone_cow() completely because of this tricky stuff, and whenever we need a clone we're about to change the LogMessage anyway, and delaying the actual change means that we have to do two malloc() calls, instead of one. Can you test whether changing the message that triggered the action with a rewrite rule, keeps the cloned one intact? This would need a parallel path to the db-parser(). dbparser: Make the context length available to action conditions I didn't pick this one. I was thinking about using a template function to return the number of entries in the current context. This patch changes the log message while it could keep it read-only, and even if I wanted to do this macro-like, I'd have created a real macro. Both template formatting and template functions receive the context as arguments anyway, so the required information is readily available. -- Bazsi

5 8

[PATCH 0/3] Type hinting
by Gergely Nagy 22 Nov '12

22 Nov '12

The patches that will follow this overview implement type-hinting support into LogTemplate and value-pairs(), then extend the mongodb destination and format-json to use these hints. In the end, we'll do things like: mongodb(value-pairs(scope("selected-macros", "nv-pairs") pair("DATE", datetime("$UNIXTIME")) pair("PID", int("$PID")) pair(".bool", boolean("true")))); And this will use the appropriate types when inserting into mongodb. The $(format-json) template function works similarly: template("$(format-json --scope selected-macros,nv-pairs DATE=datetime($UNIXTIME) PID=int($PID) .bool=boolean(TRUE))\n"); I tried to make the code as performant as possible, there are no extra allocations performed, and in case there is no casting to perform, it's just an extra integer comparsion. It can be tweaked further, if so need be, but for now, I'm happy with the results. Nevertheless, there are a couple of things I'm not entirely satisfied with (namely, abusing the ScratchBuffers to store the type hint on the first byte; or the way the value-pairs CLI interface parses type hints), and I will fix them in the coming days. While using this isn't the most convenient thing ever, and setting default types is not possible from within drivers (technically, it is, but it's awkward), it's a step in the right direction. Once nvpairs gains proper type support, that'll make things a whole lot sweeter! This - and any future updates - are available from my feature/3.4/templates/cast-typehints branch.

2 9

RFC: syslog-parser
by Balazs Scheidler 22 Nov '12

22 Nov '12

Hi, I've figured it is not nice from me that I keep pushing stuff to "master" without getting proper review from the list. So here's a attempt to change that. I've just pushed an experimental patch to the "syslog-parser" branch on github that allows the syslog parsing functionality to be used as a syslog-ng parser: https://github.com/balabit/syslog-ng-3.4/tree/syslog-parser Here's a short description on why this is useful (quoting the commit message): This patch creates a new parser, to explicitly parse messages as syslog messages. This can be used to selectively parse these messages based on some filters. Imagine this use-case: log { source { udp(port(2000) flags(no-parse)); }; parser { log { junction { log { filter { netmask("127.0.0.1/32"); }; parser { syslog-parser(); }; flags(final); }; log { filter { netmask("127.0.0.2/32"); }; parser { csv-parser(columns("C1", "C2", "C3")); }; flags(final); }; }; }; }; destination { file("/home/bazsi/logs/qqq" template("$(format-json --key *)\n")); }; }; Messages from 127.0.0.1 will get parsed as syslog messages, while 127.0.0.2 as a csv-parser() style messages. I'd welcome any kind of feedback, code or functionality wise. I still have some stuff to fix in this, but after initial feedback I'm going to merge it to master. Thanks in advance. -- Bazsi

6 8

Filter netmask not working as expected
by Agus 20 Nov '12

20 Nov '12

Hi guys, Just new, and created the following conf for testing purposes. The problem is that i get the logs in both destinations despite the filter. @version: 3.4 @include "scl.conf" options { keep_hostname(yes); normalize_hostnames(yes); threaded(yes); ts_format(iso); # Adds TZ #use_fqdn(yes); use_dns(no); }; source s_local { system(); internal(); }; source s_network { udp(); }; destination d_local { file("/var/log/messages"); }; destination d_my_mac { file("/var/log/mymac"); }; filter f_my_mac { netmask(10.24.18.2/255.255.255.255); }; log { source(s_network); filter(f_my_mac); destination(d_my_mac); flags(final); }; log { source(s_local); # uncomment this line to open port 514 to receive messages source(s_network); #destination(d_central_udp); destination(d_local); }; as netmask i also tried cidr /24 and same thing. The problem is that i get the logs in both destinations. I only want to have them in my_mac Thanks!

2 2

[PATCH 1/2] [dbparser] inherit nvpairs from parent in correlation action
by balint.kovacs＠balabit.com 16 Nov '12

16 Nov '12

From: Balint Kovacs <blint(a)blint.hu> This patch implements an inherit-name-value-pairs attribute for message type actions is patterndb correlation. In case this is set to TRUE, the triggering log message is cloned and all name-value pairs are automagically copied to the emitted message. The values set in the values XML subtree overwrite the original values. In case it is set to FALSE or unspecified, a new message is generated and only the values set manually are added to it. <actions> <action> <message inherit-name-value-pairs='TRUE'> <values> <value name="PROGRAM">my-program-name</value> <value name="MSG">Hello new message!</value> <value name="contextid">${session-id}@2</value> </values> </message> </action> </actions> Signed-off-by: Balint Kovacs <blint(a)blint.hu> --- modules/dbparser/patterndb-int.h | 1 + modules/dbparser/patterndb.c | 32 +++++++++++++++++++++++++++++--- 2 files changed, 30 insertions(+), 3 deletions(-) diff --git a/modules/dbparser/patterndb-int.h b/modules/dbparser/patterndb-int.h index 0434847..154b686 100644 --- a/modules/dbparser/patterndb-int.h +++ b/modules/dbparser/patterndb-int.h @@ -126,6 +126,7 @@ typedef struct _PDBAction guint8 content_type; guint16 rate; guint32 id:8, rate_quantum:24; + gboolean inherit_nvpairs; union { PDBMessage message; diff --git a/modules/dbparser/patterndb.c b/modules/dbparser/patterndb.c index 8720ea0..168a24c 100644 --- a/modules/dbparser/patterndb.c +++ b/modules/dbparser/patterndb.c @@ -388,6 +388,17 @@ pdb_action_set_trigger(PDBAction *self, const gchar *trigger, GError **error) g_set_error(error, 0, 1, "Unknown trigger type: %s", trigger); } +void +pdb_action_set_inheritance(PDBAction *self, const gchar *inherit_nvpairs, GError **error) +{ + if (strcmp(inherit_nvpairs, "TRUE") == 0) + self->inherit_nvpairs = TRUE; + else if (strcmp(inherit_nvpairs, "FALSE") == 0) + self->inherit_nvpairs = FALSE; + else + g_set_error(error, 0, 1, "Unknown inheritance type: %s", inherit_nvpairs); +} + PDBAction * pdb_action_new(gint id) { @@ -398,6 +409,7 @@ pdb_action_new(gint id) self->trigger = RAT_MATCH; self->content_type = RAC_NONE; self->id = id; + self->inherit_nvpairs = FALSE; return self; } @@ -552,9 +564,18 @@ pdb_rule_run_actions(PDBRule *self, gint trigger, PatternDB *db, PDBContext *con case RAC_NONE: break; case RAC_MESSAGE: - genmsg = log_msg_new_empty(); - genmsg->flags |= LF_LOCAL; - genmsg->timestamps[LM_TS_STAMP] = msg->timestamps[LM_TS_STAMP]; + if (action->inherit_nvpairs) + { + LogPathOptions path_options = LOG_PATH_OPTIONS_INIT; + path_options.ack_needed = FALSE; + genmsg = log_msg_clone_cow(msg, &path_options); + } + else + { + genmsg = log_msg_new_empty(); + genmsg->flags |= LF_LOCAL; + genmsg->timestamps[LM_TS_STAMP] = msg->timestamps[LM_TS_STAMP]; + } if (context) { switch (context->key.scope) @@ -942,6 +963,11 @@ pdb_loader_start_element(GMarkupParseContext *context, const gchar *element_name } else if (strcmp(element_name, "message") == 0) { + for (i = 0; attribute_names[i]; i++) + { + if (strcmp(attribute_names[i], "inherit-name-value-pairs") == 0) + pdb_action_set_inheritance(state->current_action, attribute_values[i], error); + } if (!state->in_action) { *error = g_error_new(1, 0, "Unexpected <message> element, it must be inside an action"); -- 1.7.9.5

5 9

[PATCH (3.4)] afmongodb: Add support for datetime fields
by Gergely Nagy 15 Nov '12

15 Nov '12

For TTL collections primarly (but useful for other scenarios aswell), introduce a new option: datetime(FIELD_NAME). This allows one to mark a single field anywhere in the structure as a datetime field, and the driver will attempt to convert it to a proper datetime, by assuming it is a unix timestamp. If the conversion fails, or the field is not found, it will not be inserted. Usage: mongodb(datetime("some.date") value-pairs( key("MESSAGE") pair("some.date", "$UNIXTIME"))) This will result in something like: { "_id" : ObjectId("509b8ffc1a1006213a000001"), "some" : { "date" : ISODate("2012-11-08T10:56:59Z") }, "MESSAGE" : "foobar" } Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/afmongodb/afmongodb-grammar.ym | 2 ++ modules/afmongodb/afmongodb-parser.c | 1 + modules/afmongodb/afmongodb.c | 61 +++++++++++++++++++++++++++++--- modules/afmongodb/afmongodb.h | 1 + 4 files changed, 61 insertions(+), 4 deletions(-) diff --git a/modules/afmongodb/afmongodb-grammar.ym b/modules/afmongodb/afmongodb-grammar.ym index f108f26..40eccc0 100644 --- a/modules/afmongodb/afmongodb-grammar.ym +++ b/modules/afmongodb/afmongodb-grammar.ym @@ -53,6 +53,7 @@ extern ValuePairsTransformSet *last_vp_transset; %token KW_SERVERS %token KW_SAFE_MODE %token KW_PATH +%token KW_DATETIME %% @@ -97,6 +98,7 @@ afmongodb_option | KW_USERNAME '(' string ')' { afmongodb_dd_set_user(last_driver, $3); free($3); } | KW_PASSWORD '(' string ')' { afmongodb_dd_set_password(last_driver, $3); free($3); } | KW_SAFE_MODE '(' yesno ')' { afmongodb_dd_set_safe_mode(last_driver, $3); } + | KW_DATETIME '(' string ')' { afmongodb_dd_set_datetime(last_driver, $3); free($3); } | value_pair_option { afmongodb_dd_set_value_pairs(last_driver, $1); } | dest_driver_option ; diff --git a/modules/afmongodb/afmongodb-parser.c b/modules/afmongodb/afmongodb-parser.c index 8aa55c1..1c51154 100644 --- a/modules/afmongodb/afmongodb-parser.c +++ b/modules/afmongodb/afmongodb-parser.c @@ -39,6 +39,7 @@ static CfgLexerKeyword afmongodb_keywords[] = { { "host", KW_HOST }, { "port", KW_PORT }, { "path", KW_PATH }, + { "datetime", KW_DATETIME }, { NULL } }; diff --git a/modules/afmongodb/afmongodb.c b/modules/afmongodb/afmongodb.c index 205ab98..bb267a5 100644 --- a/modules/afmongodb/afmongodb.c +++ b/modules/afmongodb/afmongodb.c @@ -22,6 +22,7 @@ */ #include <time.h> +#include <stdlib.h> #include "afmongodb.h" #include "afmongodb-parser.h" @@ -32,6 +33,7 @@ #include "nvtable.h" #include "logqueue.h" #include "value-pairs.h" +#include "scratch-buffers.h" #include "mongo.h" @@ -55,6 +57,7 @@ typedef struct gint port; gboolean safe_mode; + gchar *datetime_field; gchar *user; gchar *password; @@ -207,6 +210,15 @@ afmongodb_dd_set_safe_mode(LogDriver *d, gboolean state) self->safe_mode = state; } +void +afmongodb_dd_set_datetime(LogDriver *d, const gchar *field) +{ + MongoDBDestDriver *self = (MongoDBDestDriver *)d; + + g_free(self->datetime_field); + self->datetime_field = g_strdup(field); +} + /* * Utilities */ @@ -343,7 +355,7 @@ afmongodb_vp_obj_end(const gchar *name, if (prev_data) root = (bson *)*prev_data; else - root = (bson *)user_data; + root = (bson *)(((gpointer *)user_data)[0]); if (prefix_data) { @@ -357,6 +369,30 @@ afmongodb_vp_obj_end(const gchar *name, } static gboolean +afmongodb_field_is_datetime(const gchar *prefix, const gchar *name, + const gchar *datetime_field) +{ + size_t plen = 0; + + if (!datetime_field) + return FALSE; + + if (prefix) + plen = strlen(prefix); + + if (strncmp(datetime_field, prefix, plen) != 0) + return FALSE; + + if (datetime_field[plen] != '.') + return FALSE; + + if (strcmp(&datetime_field[plen + 1], name) != 0) + return FALSE; + + return TRUE; +} + +static gboolean afmongodb_vp_process_value(const gchar *name, const gchar *prefix, const gchar *value, gpointer *prefix_data, gpointer user_data) @@ -366,7 +402,7 @@ afmongodb_vp_process_value(const gchar *name, const gchar *prefix, if (prefix_data) o = (bson *)*prefix_data; else - o = (bson *)user_data; + o = (bson *)(((gpointer *)user_data)[0]); if (name[0] == '.') { @@ -378,7 +414,20 @@ afmongodb_vp_process_value(const gchar *name, const gchar *prefix, bson_append_string (o, tx_name, value, -1); } else - bson_append_string (o, name, value, -1); + { + MongoDBDestDriver *self = (MongoDBDestDriver *)(((gpointer *)user_data)[1]); + + if (afmongodb_field_is_datetime(prefix, name, self->datetime_field)) + { + gchar *endptr; + gint64 datetime = (gint64)strtoul(value, &endptr, 10) * 1000; + + if (endptr[0] == '\0') + bson_append_utc_datetime(o, name, datetime); + } + else + bson_append_string (o, name, value, -1); + } return FALSE; } @@ -390,6 +439,7 @@ afmongodb_worker_insert (MongoDBDestDriver *self) guint8 *oid; LogMessage *msg; LogPathOptions path_options = LOG_PATH_OPTIONS_INIT; + gpointer user_data[] = { NULL, self }; afmongodb_dd_connect(self, TRUE); @@ -408,11 +458,13 @@ afmongodb_worker_insert (MongoDBDestDriver *self) bson_append_oid (self->bson, "_id", oid); g_free (oid); + user_data[0] = self->bson; + value_pairs_walk(self->vp, afmongodb_vp_obj_start, afmongodb_vp_process_value, afmongodb_vp_obj_end, - msg, self->seq_num, self->bson); + msg, self->seq_num, user_data); bson_finish (self->bson); if (!mongo_sync_cmd_insert_n(self->conn, self->ns, 1, @@ -650,6 +702,7 @@ afmongodb_dd_free(LogPipe *d) string_list_free(self->servers); if (self->vp) value_pairs_free(self->vp); + g_free(self->datetime_field); log_dest_driver_free(d); } diff --git a/modules/afmongodb/afmongodb.h b/modules/afmongodb/afmongodb.h index 6a61616..76f8b83 100644 --- a/modules/afmongodb/afmongodb.h +++ b/modules/afmongodb/afmongodb.h @@ -39,6 +39,7 @@ void afmongodb_dd_set_password(LogDriver *d, const gchar *password); void afmongodb_dd_set_value_pairs(LogDriver *d, ValuePairs *vp); void afmongodb_dd_set_safe_mode(LogDriver *d, gboolean state); void afmongodb_dd_set_path(LogDriver *d, const gchar *path); +void afmongodb_dd_set_datetime(LogDriver *d, const gchar *field); gboolean afmongodb_dd_check_address(LogDriver *d, gboolean local); -- 1.7.10.4

2 4

Replace hostname with a hostname extracted from the message
by Clayton Dukes 14 Nov '12

14 Nov '12

Hi Folks, How can I extract a hostname from the message and replace the source ip with that name/ip address? Sample message: Nov 13 18:11:00 myhostname mymessage: Original Host=192.168.6.3:LOW_THRESHOLD_EVENT - 0 AUTHORIZED sessions So, I need a rewrite rule(?) to take 192.168.6.3 and replace $HOST with it: Nov 13 18:11:00 92.168.6.3 mymessage:LOW_THRESHOLD_EVENT - 0 AUTHORIZED sessions ______________________________________________________________ Clayton Dukes ______________________________________________________________

2 6

[RFC]: value-pairs key rewriting outside of key()
by Gergely Nagy 14 Nov '12

14 Nov '12

Hi! One thing I realized just now, is that there was a slight design mistake made when doing the value-pairs() key rewriting functionality: it was baked into the key() statement, so that as of this writing, one can only rewrite keys that were explicitly selected. While this sounds good on paper, it has one major drawback: If I want to rewrite keys that were selected by the scope, I have to explicitly select those keys myself. This makes a lot of things far more complicated than they need be, and this is why there's a hack in the mongodb destination that replaces leading dots with an underscore. Instead of doing that, I'd like to move the mongodb destination to use the key rewriting functionality, and while there, also teach $(format-json) to do something similar. Fortunately, it's only the config parser that needs an update, as the code below it was written in a way that allows me to do what I want. So, I'd like to add another way to configure key rewriting: value-pairs(scope(nv-pairs) rekey(".*", shift(1), add-prefix("_"))) This is very similar to the already existing way, except it's not inside a key(), but in the main value-pairs() body, thus, it will not add extra keys to the set, but will apply the transformations to keys already in it. For format-json, it would be fairly similar: $(format-json --scope nv-pairs --rekey .* --shift 1 --add-prefix "_") This functionality will then make it possible to rewrite keys that were implicitly added to the set by the scope, which in turn would make me much happier. As an FYI, I'll push this change to my merge-queue/3.4 branch tomorrow, unless anyone objects to it, or has a better suggestion by then. -- |8]

1 0

[Bug 205] New: Docs for macro MSGHDR incorrect
by bugzilla＠bugzilla.balabit.com 10 Nov '12

10 Nov '12

https://bugzilla.balabit.com/show_bug.cgi?id=205 Summary: Docs for macro MSGHDR incorrect Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 The docs state; MSGHDR Description: The name and the pid of the program that sent the log message in PROGRAM: PID format. Includes a trailing whitespace. Note that the macro returns an empty value if both the program and pid fields of the message are empty. When this macro is used the format is PROGRAM[PID]: including a trailing space. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 204] New: dbparser docs not clear for @NUMBER@
by bugzilla＠bugzilla.balabit.com 10 Nov '12

10 Nov '12

https://bugzilla.balabit.com/show_bug.cgi?id=204 Summary: dbparser docs not clear for @NUMBER@ Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: packaging AssignedTo: folti(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 the @NUMBER@ parser should state that a leading minus sign is handled for non-hex numbers and that commas are not permitted. Alternatively, making @NUMBER@ accept commas would be nice, or allowing parameters for additional characters, although comma would be the only one I could think of. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5