August 2012 - syslog-ng - lists.balabit.hu

[PATCH (3.4)] afmongodb: Support connecting to unix sockets
by Gergely Nagy 18 Aug '12

18 Aug '12

The next version of libmongo-client (0.1.6) will support connecting via unix sockets, and the mongodb destination driver should follow. To achieve this, the internal copy was updated to a version that already supports this, and the minimum required version was also bumped to 0.1.6. With that in place, the path() option is introduced, which, when set, will tell the driver to connect via the specified path to a unix socket. It does this by setting port to MONGO_CONN_LOCAL, that's how a path is recognised by libmongo-client. However, setting path() conflicts with host(), port() and servers(): the first cannot be used when any of the other three was previously set, nor the other way around. The configuration grammar was updated to notice this case, and bail out with an error if such a scenario is detected. Based on the work of Conrad Hoffmann <ch(a)bitfehler.net>. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- configure.in | 2 +- modules/afmongodb/afmongodb-grammar.ym | 26 ++++++- modules/afmongodb/afmongodb-parser.c | 1 + modules/afmongodb/afmongodb.c | 118 +++++++++++++++++++++++--------- modules/afmongodb/afmongodb.h | 3 + modules/afmongodb/libmongo-client | 2 +- 6 files changed, 115 insertions(+), 37 deletions(-) diff --git a/configure.in b/configure.in index a0410ed..294066a 100644 --- a/configure.in +++ b/configure.in @@ -28,7 +28,7 @@ IVYKIS_MIN_VERSION="0.30.1" JSON_C_MIN_VERSION="0.9" JSON_GLIB_MIN_VERSION="0.12" PCRE_MIN_VERSION="6.1" -LMC_MIN_VERSION="0.1.0" +LMC_MIN_VERSION="0.1.6" dnl *************************************************************************** dnl Initial setup diff --git a/modules/afmongodb/afmongodb-grammar.ym b/modules/afmongodb/afmongodb-grammar.ym index 71dd46d..f108f26 100644 --- a/modules/afmongodb/afmongodb-grammar.ym +++ b/modules/afmongodb/afmongodb-grammar.ym @@ -52,6 +52,7 @@ extern ValuePairsTransformSet *last_vp_transset; %token KW_COLLECTION %token KW_SERVERS %token KW_SAFE_MODE +%token KW_PATH %% @@ -69,14 +70,33 @@ afmongodb_options ; afmongodb_option - : KW_SERVERS '(' string_list ')' { afmongodb_dd_set_servers(last_driver, $3); } + : KW_SERVERS '(' string_list ')' + { + CHECK_ERROR(afmongodb_dd_check_address(last_driver, FALSE), @3, "Can't mix path() & servers()"); + afmongodb_dd_set_servers(last_driver, $3); + } + | KW_HOST '(' string ')' + { + CHECK_ERROR(afmongodb_dd_check_address(last_driver, FALSE), @3, "Can't mix path() & host()"); + afmongodb_dd_set_host(last_driver, $3); + free($3); + } + | KW_PORT '(' LL_NUMBER ')' + { + CHECK_ERROR(afmongodb_dd_check_address(last_driver, FALSE), @3, "Can't mix path() & port()"); + afmongodb_dd_set_port(last_driver, $3); + } + | KW_PATH '(' string ')' + { + CHECK_ERROR(afmongodb_dd_check_address(last_driver, TRUE), @3, "Can't mix path() with host() or server()"); + afmongodb_dd_set_path(last_driver, $3); + free($3); + } | KW_DATABASE '(' string ')' { afmongodb_dd_set_database(last_driver, $3); free($3); } | KW_COLLECTION '(' string ')' { afmongodb_dd_set_collection(last_driver, $3); free($3); } | KW_USERNAME '(' string ')' { afmongodb_dd_set_user(last_driver, $3); free($3); } | KW_PASSWORD '(' string ')' { afmongodb_dd_set_password(last_driver, $3); free($3); } | KW_SAFE_MODE '(' yesno ')' { afmongodb_dd_set_safe_mode(last_driver, $3); } - | KW_HOST '(' string ')' { afmongodb_dd_set_host(last_driver, $3); free($3); } - | KW_PORT '(' LL_NUMBER ')' { afmongodb_dd_set_port(last_driver, $3); } | value_pair_option { afmongodb_dd_set_value_pairs(last_driver, $1); } | dest_driver_option ; diff --git a/modules/afmongodb/afmongodb-parser.c b/modules/afmongodb/afmongodb-parser.c index 9d72878..8aa55c1 100644 --- a/modules/afmongodb/afmongodb-parser.c +++ b/modules/afmongodb/afmongodb-parser.c @@ -38,6 +38,7 @@ static CfgLexerKeyword afmongodb_keywords[] = { { "safe_mode", KW_SAFE_MODE }, { "host", KW_HOST }, { "port", KW_PORT }, + { "path", KW_PATH }, { NULL } }; diff --git a/modules/afmongodb/afmongodb.c b/modules/afmongodb/afmongodb.c index 5af34c9..aeed3d3 100644 --- a/modules/afmongodb/afmongodb.c +++ b/modules/afmongodb/afmongodb.c @@ -50,7 +50,7 @@ typedef struct gchar *coll; GList *servers; - gchar *host; + gchar *address; gint port; gboolean safe_mode; @@ -116,8 +116,8 @@ afmongodb_dd_set_host(LogDriver *d, const gchar *host) { MongoDBDestDriver *self = (MongoDBDestDriver *)d; - g_free(self->host); - self->host = g_strdup(host); + g_free(self->address); + self->address = g_strdup(host); } void @@ -138,6 +138,39 @@ afmongodb_dd_set_servers(LogDriver *d, GList *servers) } void +afmongodb_dd_set_path(LogDriver *d, const gchar *path) +{ + MongoDBDestDriver *self = (MongoDBDestDriver *)d; + + g_free(self->address); + self->address = g_strdup(path); + self->port = MONGO_CONN_LOCAL; +} + +gboolean +afmongodb_dd_check_address(LogDriver *d, gboolean local) +{ + MongoDBDestDriver *self = (MongoDBDestDriver *)d; + + if (local) + { + if ((self->port != 0 || + self->port != MONGO_CONN_LOCAL) && + self->address != NULL) + return FALSE; + if (self->servers) + return FALSE; + } + else + { + if (self->port == MONGO_CONN_LOCAL && + self->address != NULL) + return FALSE; + } + return TRUE; +} + +void afmongodb_dd_set_database(LogDriver *d, const gchar *database) { MongoDBDestDriver *self = (MongoDBDestDriver *)d; @@ -182,8 +215,12 @@ afmongodb_dd_format_stats_instance(MongoDBDestDriver *self) { static gchar persist_name[1024]; - g_snprintf(persist_name, sizeof(persist_name), - "mongodb,%s,%u,%s,%s", self->host, self->port, self->db, self->coll); + if (self->port == MONGO_CONN_LOCAL) + g_snprintf(persist_name, sizeof(persist_name), + "mongodb,%s,%s,%s", self->address, self->db, self->coll); + else + g_snprintf(persist_name, sizeof(persist_name), + "mongodb,%s,%u,%s,%s", self->address, self->port, self->db, self->coll); return persist_name; } @@ -192,8 +229,12 @@ afmongodb_dd_format_persist_name(MongoDBDestDriver *self) { static gchar persist_name[1024]; - g_snprintf(persist_name, sizeof(persist_name), - "afmongodb(%s,%u,%s,%s)", self->host, self->port, self->db, self->coll); + if (self->port == MONGO_CONN_LOCAL) + g_snprintf(persist_name, sizeof(persist_name), + "afmongodb(%s,%s,%s)", self->address, self->db, self->coll); + else + g_snprintf(persist_name, sizeof(persist_name), + "afmongodb(%s,%u,%s,%s)", self->address, self->port, self->db, self->coll); return persist_name; } @@ -221,7 +262,7 @@ afmongodb_dd_connect(MongoDBDestDriver *self, gboolean reconnect) if (reconnect && self->conn) return TRUE; - self->conn = mongo_sync_connect(self->host, self->port, FALSE); + self->conn = mongo_sync_connect(self->address, self->port, FALSE); if (!self->conn) { msg_error ("Error connecting to MongoDB", NULL); @@ -467,31 +508,45 @@ afmongodb_dd_init(LogPipe *s) value_pairs_add_scope(self->vp, "nv-pairs"); } - if (self->host) - { - gchar *srv = g_strdup_printf ("%s:%d", self->host, - (self->port) ? self->port : 27017); - self->servers = g_list_prepend (self->servers, srv); - g_free (self->host); - } - - self->host = NULL; - self->port = 27017; - if (!mongo_util_parse_addr(g_list_nth_data(self->servers, 0), &self->host, - &self->port)) + if (self->port != MONGO_CONN_LOCAL) { - msg_error("Cannot parse the primary host", - evt_tag_str("primary", g_list_nth_data(self->servers, 0)), - NULL); - return FALSE; + if (self->address) + { + gchar *srv = g_strdup_printf ("%s:%d", self->address, + (self->port) ? self->port : 27017); + self->servers = g_list_prepend (self->servers, srv); + g_free (self->address); + } + + if (!self->servers) + afmongodb_dd_set_servers((LogDriver *)self, g_list_append (NULL, g_strdup ("127.0.0.1:27017"))); + + self->address = NULL; + self->port = 27017; + if (!mongo_util_parse_addr(g_list_nth_data(self->servers, 0), + &self->address, + &self->port)) + { + msg_error("Cannot parse the primary host", + evt_tag_str("primary", g_list_nth_data(self->servers, 0)), + NULL); + return FALSE; + } } - msg_verbose("Initializing MongoDB destination", - evt_tag_str("host", self->host), - evt_tag_int("port", self->port), - evt_tag_str("database", self->db), - evt_tag_str("collection", self->coll), - NULL); + if (self->port == MONGO_CONN_LOCAL) + msg_verbose("Initializing MongoDB destination", + evt_tag_str("address", self->address), + evt_tag_str("database", self->db), + evt_tag_str("collection", self->coll), + NULL); + else + msg_verbose("Initializing MongoDB destination", + evt_tag_str("address", self->address), + evt_tag_int("port", self->port), + evt_tag_str("database", self->db), + evt_tag_str("collection", self->coll), + NULL); self->queue = log_dest_driver_acquire_queue(&self->super, afmongodb_dd_format_persist_name(self)); @@ -548,7 +603,7 @@ afmongodb_dd_free(LogPipe *d) g_free(self->coll); g_free(self->user); g_free(self->password); - g_free(self->host); + g_free(self->address); string_list_free(self->servers); if (self->vp) value_pairs_free(self->vp); @@ -609,7 +664,6 @@ afmongodb_dd_new(void) self->super.super.super.queue = afmongodb_dd_queue; self->super.super.super.free_fn = afmongodb_dd_free; - afmongodb_dd_set_servers((LogDriver *)self, g_list_append (NULL, g_strdup ("127.0.0.1:27017"))); afmongodb_dd_set_database((LogDriver *)self, "syslog"); afmongodb_dd_set_collection((LogDriver *)self, "messages"); afmongodb_dd_set_safe_mode((LogDriver *)self, FALSE); diff --git a/modules/afmongodb/afmongodb.h b/modules/afmongodb/afmongodb.h index e1f4b57..6a61616 100644 --- a/modules/afmongodb/afmongodb.h +++ b/modules/afmongodb/afmongodb.h @@ -38,5 +38,8 @@ void afmongodb_dd_set_user(LogDriver *d, const gchar *user); void afmongodb_dd_set_password(LogDriver *d, const gchar *password); void afmongodb_dd_set_value_pairs(LogDriver *d, ValuePairs *vp); void afmongodb_dd_set_safe_mode(LogDriver *d, gboolean state); +void afmongodb_dd_set_path(LogDriver *d, const gchar *path); + +gboolean afmongodb_dd_check_address(LogDriver *d, gboolean local); #endif diff --git a/modules/afmongodb/libmongo-client b/modules/afmongodb/libmongo-client index 58f3814..7896ce7 160000 --- a/modules/afmongodb/libmongo-client +++ b/modules/afmongodb/libmongo-client @@ -1 +1 @@ -Subproject commit 58f3814cad94bcd78216c7ac971c8435d17a9242 +Subproject commit 7896ce7dbc239e4069787fee336875a9b9a6f6f5 -- 1.7.10.4

2 1

[PATCH (3.4)] afsocket: Implement tcp_keep_alive_time()
by Gergely Nagy 18 Aug '12

18 Aug '12

With the new tcp_keep_alive_time() option one can control on a per-source basis the time after which a TCP connection will start sending TCP keepalive probes (assuming so-keepalive(yes)). While it is accepted for all stream sources, it only makes sense for TCP-based streams. By default, it's value is 0, which means using the kernel default (2 hours). Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/afsocket/afsocket-grammar.ym | 2 ++ modules/afsocket/afsocket-parser.c | 1 + modules/afsocket/afsocket.c | 11 +++++++++++ modules/afsocket/afsocket.h | 2 ++ 4 files changed, 16 insertions(+), 0 deletions(-) diff --git a/modules/afsocket/afsocket-grammar.ym b/modules/afsocket/afsocket-grammar.ym index 7970ca9..48b0526 100644 --- a/modules/afsocket/afsocket-grammar.ym +++ b/modules/afsocket/afsocket-grammar.ym @@ -83,6 +83,7 @@ TLSContext *last_tls_context; %token KW_SO_SNDBUF %token KW_SO_RCVBUF %token KW_SO_KEEPALIVE +%token KW_TCP_KEEP_ALIVE_TIME %token KW_SPOOF_SOURCE %token KW_KEEP_ALIVE @@ -252,6 +253,7 @@ source_afinet_tcp_option source_afsocket_stream_params : KW_KEEP_ALIVE '(' yesno ')' { afsocket_sd_set_keep_alive(last_driver, $3); } + | KW_TCP_KEEP_ALIVE_TIME '(' LL_NUMBER ')' { afsocket_sd_set_keep_alive_time(last_driver, $3); } | KW_MAX_CONNECTIONS '(' LL_NUMBER ')' { afsocket_sd_set_max_connections(last_driver, $3); } ; diff --git a/modules/afsocket/afsocket-parser.c b/modules/afsocket/afsocket-parser.c index 44671ca..bd8f5d6 100644 --- a/modules/afsocket/afsocket-parser.c +++ b/modules/afsocket/afsocket-parser.c @@ -64,6 +64,7 @@ static CfgLexerKeyword afsocket_keywords[] = { { "so_sndbuf", KW_SO_SNDBUF }, { "so_keepalive", KW_SO_KEEPALIVE }, { "tcp_keep_alive", KW_SO_KEEPALIVE, 0, KWS_OBSOLETE, "so_keepalive" }, + { "tcp_keep_alive_time",KW_TCP_KEEP_ALIVE_TIME }, { "spoof_source", KW_SPOOF_SOURCE }, { "transport", KW_TRANSPORT }, { "max_connections", KW_MAX_CONNECTIONS }, diff --git a/modules/afsocket/afsocket.c b/modules/afsocket/afsocket.c index a7cc7bc..4a71b11 100644 --- a/modules/afsocket/afsocket.c +++ b/modules/afsocket/afsocket.c @@ -43,6 +43,7 @@ #include <arpa/inet.h> #include <sys/stat.h> #include <unistd.h> +#include <netinet/tcp.h> #if ENABLE_TCP_WRAPPER #include <tcpd.h> @@ -79,6 +80,8 @@ afsocket_setup_socket(gint fd, SocketOptions *sock_options, AFSocketDirection di setsockopt(fd, SOL_SOCKET, SO_BROADCAST, &sock_options->broadcast, sizeof(sock_options->broadcast)); } setsockopt(fd, SOL_SOCKET, SO_KEEPALIVE, &sock_options->keepalive, sizeof(sock_options->keepalive)); + if (sock_options->keepalive_time > 0) + setsockopt(fd, SOL_TCP, TCP_KEEPIDLE, &sock_options->keepalive_time, sizeof(sock_options->keepalive_time)); return TRUE; } @@ -386,6 +389,14 @@ afsocket_sd_set_keep_alive(LogDriver *s, gint enable) } void +afsocket_sd_set_keep_alive_time(LogDriver *s, gint timeout) +{ + AFSocketSourceDriver *self = (AFSocketSourceDriver *) s; + + self->keepalive_timeout = timeout; +} + +void afsocket_sd_set_max_connections(LogDriver *s, gint max_connections) { AFSocketSourceDriver *self = (AFSocketSourceDriver *) s; diff --git a/modules/afsocket/afsocket.h b/modules/afsocket/afsocket.h index 392e249..fd11d28 100644 --- a/modules/afsocket/afsocket.h +++ b/modules/afsocket/afsocket.h @@ -58,6 +58,7 @@ typedef struct _SocketOptions gint rcvbuf; gint broadcast; gint keepalive; + gint keepalive_time; } SocketOptions; gboolean afsocket_setup_socket(gint fd, SocketOptions *sock_options, AFSocketDirection dir); @@ -78,6 +79,7 @@ struct _AFSocketSourceDriver gint max_connections; gint num_connections; gint listen_backlog; + gint keepalive_timeout; GList *connections; SocketOptions *sock_options_ptr; -- 1.7.9

2 1

[PATCH] db-parser(): implement a MAC address parser
by Balint Kovacs 18 Aug '12

18 Aug '12

While talking about netfilter logs recently, we found, that a MAC address parser may come in handy when parsing these messages, so here it comes. I hope others might find it useful as well. Balint

2 2

[PATCH (3.4)] jsonparser: Clone the marker() setting when cloning the parser
by Gergely Nagy 18 Aug '12

18 Aug '12

Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/jsonparser/jsonparser.c | 1 + 1 file changed, 1 insertion(+) diff --git a/modules/jsonparser/jsonparser.c b/modules/jsonparser/jsonparser.c index 7331ad9..891b9f0 100644 --- a/modules/jsonparser/jsonparser.c +++ b/modules/jsonparser/jsonparser.c @@ -201,6 +201,7 @@ log_json_parser_clone (LogPipe *s) cloned = (LogJSONParser *) log_json_parser_new (); log_json_parser_set_prefix ((LogParser *)cloned, self->prefix); + log_json_parser_set_marker ((LogParser *)cloned, self->marker); return &cloned->super.super; } -- 1.7.10

2 2

[PATCH (3.4)] afsmtp: Fix the module linkage
by Gergely Nagy 18 Aug '12

18 Aug '12

Use $(MODULE_DEPS_LIBS) and $(MODULE_LDFLAGS) instead of trying to grab all the libs. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/afsmtp/Makefile.am | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/modules/afsmtp/Makefile.am b/modules/afsmtp/Makefile.am index bb5b94d..34c0e98 100644 --- a/modules/afsmtp/Makefile.am +++ b/modules/afsmtp/Makefile.am @@ -6,10 +6,10 @@ export top_srcdir if ENABLE_SMTP module_LTLIBRARIES = libafsmtp.la -libafsmtp_la_CFLAGS = $(LIBESMTP_CFLAGS) $(IVYKIS_CFLAGS) $(EVENTLOG_CFLAGS) $(SYSLOG_NG_CFLAGS) +libafsmtp_la_CFLAGS = $(LIBESMTP_CFLAGS) libafsmtp_la_SOURCES = afsmtp-grammar.y afsmtp.c afsmtp.h afsmtp-parser.c afsmtp-parser.h -libafsmtp_la_LIBADD = $(LIBESMTP_LIBS) $(IVYKIS_LIBS) $(EVENTLOG_LIBS) $(SYSLOG_NG_LIBS) -libafsmtp_la_LDFLAGS = -avoid-version +libafsmtp_la_LIBADD = $(LIBESMTP_LIBS) $(MODULE_DEPS_LIBS) +libafsmtp_la_LDFLAGS = $(MODULE_LDFLAGS) endif -- 1.7.10.4

2 1

[Bug 62] PROGRAM macro problem
by bugzilla＠bugzilla.balabit.com 18 Aug '12

18 Aug '12

https://bugzilla.balabit.com/show_bug.cgi?id=62 Gergely Nagy <algernon(a)balabit.hu> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |algernon(a)balabit.hu Resolution| |FIXED Status|NEW |RESOLVED --- Comment #4 from Gergely Nagy <algernon(a)balabit.hu> 2012-08-18 13:26:40 --- syslog-ng 3.4 has a $(sanitize) template function, which will replace "/" with "_", so this issue can be considered resolved, I believe. The same function can be configured to replace parens and pretty much anything else too, if so need be. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 0

[PATCH 0/1] [RESEND] libmongo-client: Add support for Unix sockets
by Conrad Hoffmann 16 Aug '12

16 Aug '12

Hey once more. Here is a first revision of my earlier patch. Changes to last version: - Better error handling/input checking (thanks to Balazs for the feedback) - Free everything that was allocated before bailing out after errors - Some style fixes, hope it's ok now Further feedback welcome! Conrad Hoffmann (1): Add support for Unix sockets. examples/mongo-dump.c | 22 +++++++++++++++++----- src/mongo-client.c | 39 +++++++++++++++++++++++++++++++++++++++ src/mongo-client.h | 12 ++++++++++++ src/mongo-sync.c | 23 +++++++++++++++++++++++ src/mongo-sync.h | 15 +++++++++++++++ 5 files changed, 106 insertions(+), 5 deletions(-) -- 1.7.11.4

2 7

Compiling 3.4
by Conrad Hoffmann 14 Aug '12

14 Aug '12

Greetings! I am having some difficulties compiling current master (at the time of this writing) of syslog-ng 3.4. The first error is this: filter-expr-parser.c:30:5: error: conflicting types for 'filter_expr_parse' In file included from filter-expr-parser.c:27:0: filter-expr-grammar.h:404:5: note: previous declaration of 'filter_expr_parse' was here make[4]: *** [filter-expr-parser.lo] Error Now, at first I thought, since filter-expr-grammar.h is generated, it might have something to do with my bison version or the likes. But then I realized that the offending declaration in filter-expr-parser.c:30: int filter_expr_parse(CfgLexer *lexer, FilterExprNode **node); directly contradicts its use int the same file below (line 74): .parse = (gint (*)(CfgLexer *, gpointer *, gpointer)) filter_expr_parse, filter-expr-parser.c is not generated, so I don't quite get how this compiles at all for anyone? Am I missing something? The actual usage (line 74) matches the generated declaration, so I can just comment out the offending one in line 30. But I suspect this it not how it's supposed to work... Same thing happens for parser-expr-parser and rewrite-expr-parser... Thanks for any hints, Conrad

3 5

phpsyslogng-2.8.
by Jose Aguero M. 14 Aug '12

14 Aug '12

Hi all: I have installed ubuntu server with syslog-ng 2.8, but I need to know how to install a client on linux centos. I need help thanks

2 1

[PATCH] [0/1] libmongo-client: add support for Unix sockets
by Conrad Hoffmann 14 Aug '12

14 Aug '12

Greetings! This is patch agains the master branch of syslog-ng-3.4. It's more of a proof-of-concept thing, but in a quick session I put support for Unix sockets into libmongo-client. It may not be stunningly beautiful, but I actually used it (along with a second patch, which I will send shortly) to have syslog-ng write to mongodb via a Unix socket. I realize libmongo-client is also on GitHub, but the submodule pointed to a different repo, so I decided to just dump this here. :) If you have any interest at all, I could also clean this up some more. Maybe split across two commits, too. Let me know. Conrad Hoffmann (1): Add support for Unix sockets. examples/mongo-dump.c | 22 +++++++++++++++++----- src/mongo-client.c | 40 ++++++++++++++++++++++++++++++++++++++++ src/mongo-client.h | 12 ++++++++++++ src/mongo-sync.c | 23 +++++++++++++++++++++++ src/mongo-sync.h | 15 +++++++++++++++ 5 files changed, 107 insertions(+), 5 deletions(-) -- 1.7.11.4

3 3