- syslog-ng - lists.balabit.hu

syslog-ng with MySQL
by James McDonald 21 Aug '12

21 Aug '12

Trying to incorporate a MySQL db (on a separate server) with my two syslog-ng servers, so that the logs can be searchable. Does anyone have any experience with this? Any tips or documentation would be helpful. I didn't find a whole lot on the Balabit website when I searched for it.

4 10

special characters in filenames
by Daniel Neubacher 21 Aug '12

21 Aug '12

Hello there, I've got a pretty annoying problem with syslog-ng 3.3.5 and log filenames. I think my syslog-ng.cfg is a little unconventional because I've configured for each log file a file statement and a uniq program name to keep the filenames from the client on the server. The cfg I've attached bellow is only a small part of it because I can't show you everything. In the original syslog-ng.cfg I've configured about 120 individual log files. Now to the problem. Each day I'm getting log file names like this on my log server: /log/applogs/2012/08/17/perl-1.hk.fra1.xing.com/t.file(d_applogs#0,/log/applogs/2012/08/17/mx2-2.xing.com/syslog-ng)=0', /log/applogs/2012/08/15/syslog-2.log.fra2.xing.com/ropped='dst.file(d_syslog#0,/log/syslog /log/syslog/2012/08/17/ fai-1.ops.fra1 fai? This is pretty annoying because my script which is compressing the logs is failing on these files. Does anybody know this problem? Server cfg: @version: 3.3 options { threaded(yes); owner("root"); group("root"); perm(0660); dir_owner("root"); dir_group("root"); dir_perm(0770); create_dirs(yes); stats_freq(600); stats_level(2); chain_hostnames(no); check_hostname(yes); keep_hostname(no); dns_cache(yes); dns_cache_size(16384); dns_cache_expire(3600); dns_cache_expire_failed(60); use_fqdn(yes); log_msg_size(128000); log_fifo_size(1000000); }; template t_plain { template("$MSG\n"); template_escape(no); }; filter f_syslog { program(^auth.log) or program(^cron.log) or program(^daemon.log) or program(^kern.log) or program(^lpr.log) or program(^mail.log) or program(^syslog.log) or program(^user.log) or program(^uucp.log) or program(^mail.info) or program(^mail.warn) or program(^mail.err) or program(^news.crit) or program(^news.err) or program(^news.notice) or program(^debug.log) or program(^error.log) or program(^messages.log) or program(^ppp.log); }; filter f_applogs { not filter(f_syslog) and program(".*.log"); }; source s_src { unix-dgram("/dev/log" max-connections(500)); internal(); file("/proc/kmsg" program_override("kernel")); }; source s_net { udp( log_fetch_limit(400) so_rcvbuf(51200000) keep_timestamp(yes) port(514) ); tcp( max-connections(1000) so_rcvbuf(51200000) so_keepalive(yes) keep_timestamp(yes) port(514) ); syslog(); }; destination d_messages { file("/var/log/messages"); }; destination d_syslog { file("/log/syslog/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST/$PROGRAM" template(t_plain)); }; destination d_syslog_onefile { file("/log/syslog_onefile/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST"); }; destination d_applogs { file("/log/applogs/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST/$PROGRAM" template(t_plain)); }; destination d_perlhk {tcp("perl-1.hk.fra1.xing.com" port(514));}; log { source(s_src); destination(d_messages); }; log { source(s_net); filter(f_syslog); destination(d_syslog); }; log { source(s_net); filter(f_syslog); destination(d_syslog_onefile); }; log { source(s_net); filter(f_applogs); destination(d_applogs); }; log { source(s_net); filter(f_applogs); destination(d_perlhk); }; Client: @version: 3.3 options { threaded(yes); use_dns(yes); use_fqdn(yes); dns_cache(yes); dns_cache_size(16384); dns_cache_expire(3600); dns_cache_expire_failed(10); log_msg_size(128000); log_fifo_size(100000); normalize_hostnames(yes); check_hostname(yes); bad_hostname("^gconfd$"); create_dirs(yes); owner("root"); group("root"); perm(0640); stats_freq(3600); time_reopen(30); }; # Applogs source s_perl_applogs { file(/www/applogs/admin.log follow_freq(1) flags(no-parse) program_override(admin.log)); file(/www/applogs/fcgid.log follow_freq(1) flags(no-parse) program_override(fcgid.log)); }; ##Ruby source s_ruby_applogs { file("/virtual/cra/shared/log/production.log" follow_freq(1) flags(no-parse) program_override(production.log)); }; source s_syslog { file("/var/log/auth.log" follow_freq(1) flags(no-parse) program_override("auth.log")); file("/var/log/cron.log" follow_freq(1) flags(no-parse) program_override("cron.log")); file("/var/log/daemon.log" follow_freq(1) flags(no-parse) program_override("daemon.log")); file("/var/log/kern.log" follow_freq(1) flags(no-parse) program_override("kern.log")); file("/var/log/lpr.log" follow_freq(1) flags(no-parse) program_override("lpr.log")); file("/var/log/mail.log" follow_freq(1) flags(no-parse) program_override("mail.log")); file("/var/log/syslog" follow_freq(1) flags(no-parse) program_override("syslog.log")); file("/var/log/user.log" follow_freq(1) flags(no-parse) program_override("user.log")); file("/var/log/uucp.log" follow_freq(1) flags(no-parse) program_override("uucp.log")); file("/var/log/mail/mail.info" follow_freq(1) flags(no-parse) program_override("mail.info")); file("/var/log/mail/mail.warn" follow_freq(1) flags(no-parse) program_override("mail.warn")); file("/var/log/mail/mail.err" follow_freq(1) flags(no-parse) program_override("mail.err")); file("/var/log/news/news.crit" follow_freq(1) flags(no-parse) program_override("news.crit")); file("/var/log/news/news.err" follow_freq(1) flags(no-parse) program_override("news.err")); file("/var/log/news/news.notice" follow_freq(1) flags(no-parse) program_override("news.notice")); file("/var/log/debug" follow_freq(1) flags(no-parse) program_override("debug.log")); file("/var/log/error" follow_freq(1) flags(no-parse) program_override("error.log")); file("/var/log/messages" follow_freq(1) flags(no-parse) program_override("messages.log")); file("/var/log/ppp.log" follow_freq(1) flags(no-parse) program_override("ppp.log")); }; destination syslog-1.log.fra1 { udp("syslog-1.log.fra1.xing.com" port(514)); }; destination syslog-2.log.fra1 { tcp("syslog-2.log.fra1.xing.com" port(514)); }; destination syslog-1.log.fra2 { tcp("syslog-1.log.fra2.xing.com" port(514)); }; destination syslog-2.log.fra2 { tcp("syslog-2.log.fra2.xing.com" port(514)); }; log { source(s_all); destination(syslog-1.log.fra1); }; log { source(s_syslog); source(s_perl_applogs); source(s_ruby_applogs); destination(syslog-2.log.fra1); destination(syslog-1.log.fra2); destination(syslog-2.log.fra2); }; -- Daniel Neubacher, Network Administrator daniel.neubacher(a)xing.com<mailto:daniel.neubacher@xing.com> XING AG Gaensemarkt 43, 20354 Hamburg, Germany Tel. +49 40 419131-28, Fax +49 40 419131-11 Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 98807 Exec. Board (Vorstand): Dr. Stefan Groß-Selbeck (Vorsitzender), Dr. Thomas Vollmoeller, Ingo Chu, Dr. Helmut Becker, Jens Pape Chairman of the Supervisory Board (Aufsichtsratsvorsitzender): Dr. Neil Sunderland This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly forbidden and may be unlawful.

3 3

[PATCH (3.4)] confgen: Implement a built-in system() generator
by Gergely Nagy 19 Aug '12

19 Aug '12

The system SCL plugin relies on a script (generate-system-source.sh) to generate the block system() expands into. While this is flexible and easy to modifiy, it has the disadvantage of running an external script at startup, which is undesirable in many situations, including when using a strict AppArmor profile. However, a generated system() is useful in a lot of situations, ranging from creating portable, system-agnostic configuration files to supporting sysvinit and systemd with the same config. For these reasons, this patch implements a built-in system() generator, that does not rely on the generate-system-source.sh script at all, but does everything itself, from C. It should be used similarly to the existing generator: @module confgen context(source) name(system) system(yes) This will result in the very same system() expansion as the script's as of this patch. The downside is that if one of the generator gets extended, the other must be updated too, to keep them in sync. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/confgen/confgen-plugin.c | 174 +++++++++++++++++++++++++++++++++++++- scl/system/plugin.conf | 5 ++ 2 files changed, 175 insertions(+), 4 deletions(-) diff --git a/modules/confgen/confgen-plugin.c b/modules/confgen/confgen-plugin.c index ffd24b2..90428b9 100644 --- a/modules/confgen/confgen-plugin.c +++ b/modules/confgen/confgen-plugin.c @@ -28,8 +28,13 @@ #include "messages.h" #include "plugin.h" +#include <sys/utsname.h> +#include <sys/types.h> +#include <sys/stat.h> +#include <stdlib.h> #include <string.h> #include <errno.h> +#include <unistd.h> gboolean confgen_generate(CfgLexer *lexer, gint type, const gchar *name, CfgArgs *args, gpointer user_data) @@ -87,10 +92,157 @@ confgen_generate(CfgLexer *lexer, gint type, const gchar *name, CfgArgs *args, g return TRUE; } +static void +confgen_sysblock_add_unix_dgram(GString *sysblock, const gchar *path, + const gchar *perms) +{ + g_string_append_printf(sysblock, "unix-dgram(\"%s\"", path); + if (perms) + g_string_append_printf(sysblock, " perms(%s)", perms); + g_string_append(sysblock, ");\n"); +} + +static void +confgen_sysblock_add_file(GString *sysblock, const gchar *path, + gint follow_freq, const gchar *prg_override, + const gchar *flags) +{ + g_string_append_printf(sysblock, "file(\"%s\"", path); + if (follow_freq >= 0) + g_string_append_printf(sysblock, " follow-freq(%d)", follow_freq); + if (prg_override) + g_string_append_printf(sysblock, " program-override(\"%s\")", prg_override); + if (flags) + g_string_append_printf(sysblock, " flags(%s)", flags); + g_string_append(sysblock, ");\n"); +} + +static void +confgen_sysblock_add_module(GString *sysblock, const gchar *mod) +{ + g_string_append_printf(sysblock, "@module %s\n", mod); +} + +static void +confgen_sysblock_add_sun_streams(GString *sysblock, const gchar *path, + const gchar *door) +{ + g_string_append_printf(sysblock, "sun-streams(\"%s\"", path); + if (door) + g_string_append_printf(sysblock, " door(\"%s\")", door); + g_string_append(sysblock, ");\n"); +} + +static void +confgen_sysblock_add_pipe(GString *sysblock, const gchar *path, gint pad_size) +{ + g_string_append_printf(sysblock, "pipe(\"%s\"", path); + if (pad_size >= 0) + g_string_append_printf(sysblock, " pad_size(%d)", pad_size); + g_string_append(sysblock, ");\n"); +} + +gboolean +confgen_generate_system(CfgLexer *lexer, gint type, const gchar *name, + CfgArgs *args, gpointer user_data) +{ + gchar buf[256]; + GString *sysblock; + struct utsname u; + + g_snprintf(buf, sizeof(buf), "%s confgen %s", cfg_lexer_lookup_context_name_by_type(type), name); + if (!cfg_args_validate(args, NULL, buf)) + { + msg_error("confgen: confgen invocations do not process arguments, but your argument list is not empty", + evt_tag_str("context", cfg_lexer_lookup_context_name_by_type(type)), + evt_tag_str("block", name), + NULL); + return FALSE; + } + + sysblock = g_string_sized_new(1024); + + if (uname(&u) != 0) + { + msg_error("confgen: Cannot get information about the running kernel", + evt_tag_errno("error", errno), + NULL); + return FALSE; + } + + if (strcmp(u.sysname, "Linux") == 0) + { + char *log = "/dev/log"; + + if (getenv("LISTEN_FDS") != NULL) + { + struct stat sbuf; + + if (stat("/run/systemd/journal/syslog", &sbuf) == 0) + { + if (S_ISSOCK(sbuf.st_mode)) + log = "/run/systemd/journal/syslog"; + } + } + + confgen_sysblock_add_unix_dgram(sysblock, log, NULL); + confgen_sysblock_add_file(sysblock, "/proc/kmsg", -1, "kernel", "kernel"); + } + else if (strcmp(u.sysname, "SunOS") == 0) + { + confgen_sysblock_add_module(sysblock, "afstreams"); + + if (strcmp(u.release, "5.8") == 0) + confgen_sysblock_add_sun_streams(sysblock, "/dev/log", NULL); + else if (strcmp(u.release, "5.9") == 0) + confgen_sysblock_add_sun_streams(sysblock, "/dev/log", "/etc/.syslog_door"); + else + confgen_sysblock_add_sun_streams(sysblock, "/dev/log", "/var/run/syslog_door"); + } + else if (strcmp(u.sysname, "FreeBSD") == 0) + { + confgen_sysblock_add_unix_dgram(sysblock, "/var/run/log", NULL); + confgen_sysblock_add_unix_dgram(sysblock, "/var/run/logpriv", "0600"); + confgen_sysblock_add_file(sysblock, "/dev/klog", 0, "kernel", "no-parse"); + } + else if (strcmp(u.sysname, "GNU/kFreeBSD") == 0) + { + confgen_sysblock_add_unix_dgram(sysblock, "/var/run/log", NULL); + confgen_sysblock_add_file(sysblock, "/dev/klog", 0, "kernel", NULL); + } + else if (strcmp(u.sysname, "HP-UX") == 0) + { + confgen_sysblock_add_pipe(sysblock, "/dev/pipe", 2048); + } + else if (strcmp(u.sysname, "AIX") == 0 || + strcmp(u.sysname, "OSF1") == 0 || + strncmp(u.sysname, "CYGWIN", 6) == 0) + { + confgen_sysblock_add_unix_dgram(sysblock, "/dev/log", NULL); + } + else + { + msg_error("system(): Error detecting platform, unable to define the system() source. " + "Please send your system information to the developers!", + evt_tag_str("sysname", u.sysname), + evt_tag_str("release", u.release), + NULL); + return FALSE; + } + + if (!cfg_lexer_include_buffer(lexer, buf, sysblock->str, sysblock->len)) + { + g_string_free(sysblock, TRUE); + return FALSE; + } + + return TRUE; +} + gboolean confgen_module_init(GlobalConfig *cfg, CfgArgs *args) { - const gchar *name, *context, *exec; + const gchar *name, *context, *exec, *sys; name = cfg_args_get(args, "name"); if (!name) @@ -107,13 +259,27 @@ confgen_module_init(GlobalConfig *cfg, CfgArgs *args) return FALSE; } exec = cfg_args_get(args, "exec"); - if (!exec) + sys = cfg_args_get(args, "system"); + + if (!exec && !sys) + { + msg_error("confgen: exec or system argument expected", + NULL); + return FALSE; + } + + if (exec && sys) { - msg_error("confgen: exec argument expected", + msg_error("confgen: exec and system arguments are mutually excluse", NULL); return FALSE; } - cfg_lexer_register_block_generator(cfg->lexer, cfg_lexer_lookup_context_type_by_name(context), name, confgen_generate, g_strdup(exec), g_free); + + if (exec) + cfg_lexer_register_block_generator(cfg->lexer, cfg_lexer_lookup_context_type_by_name(context), name, confgen_generate, g_strdup(exec), g_free); + if (sys) + cfg_lexer_register_block_generator(cfg->lexer, cfg_lexer_lookup_context_type_by_name(context), + name, confgen_generate_system, NULL, NULL); return TRUE; } diff --git a/scl/system/plugin.conf b/scl/system/plugin.conf index 5ba652e..d150fee 100644 --- a/scl/system/plugin.conf +++ b/scl/system/plugin.conf @@ -21,3 +21,8 @@ ############################################################################# @module confgen context(source) name(system) exec("`scl-root`/system/generate-system-source.sh") + +# To use a built-in discovery system, that does not rely on a script, +# comment out the line above, and uncomment the next one: + +#@module confgen context(source) name(system) system(yes) -- 1.7.10

2 2

[PATCH (3.4)] afmongodb: Support connecting to unix sockets
by Gergely Nagy 18 Aug '12

18 Aug '12

The next version of libmongo-client (0.1.6) will support connecting via unix sockets, and the mongodb destination driver should follow. To achieve this, the internal copy was updated to a version that already supports this, and the minimum required version was also bumped to 0.1.6. With that in place, the path() option is introduced, which, when set, will tell the driver to connect via the specified path to a unix socket. It does this by setting port to MONGO_CONN_LOCAL, that's how a path is recognised by libmongo-client. However, setting path() conflicts with host(), port() and servers(): the first cannot be used when any of the other three was previously set, nor the other way around. The configuration grammar was updated to notice this case, and bail out with an error if such a scenario is detected. Based on the work of Conrad Hoffmann <ch(a)bitfehler.net>. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- configure.in | 2 +- modules/afmongodb/afmongodb-grammar.ym | 26 ++++++- modules/afmongodb/afmongodb-parser.c | 1 + modules/afmongodb/afmongodb.c | 118 +++++++++++++++++++++++--------- modules/afmongodb/afmongodb.h | 3 + modules/afmongodb/libmongo-client | 2 +- 6 files changed, 115 insertions(+), 37 deletions(-) diff --git a/configure.in b/configure.in index a0410ed..294066a 100644 --- a/configure.in +++ b/configure.in @@ -28,7 +28,7 @@ IVYKIS_MIN_VERSION="0.30.1" JSON_C_MIN_VERSION="0.9" JSON_GLIB_MIN_VERSION="0.12" PCRE_MIN_VERSION="6.1" -LMC_MIN_VERSION="0.1.0" +LMC_MIN_VERSION="0.1.6" dnl *************************************************************************** dnl Initial setup diff --git a/modules/afmongodb/afmongodb-grammar.ym b/modules/afmongodb/afmongodb-grammar.ym index 71dd46d..f108f26 100644 --- a/modules/afmongodb/afmongodb-grammar.ym +++ b/modules/afmongodb/afmongodb-grammar.ym @@ -52,6 +52,7 @@ extern ValuePairsTransformSet *last_vp_transset; %token KW_COLLECTION %token KW_SERVERS %token KW_SAFE_MODE +%token KW_PATH %% @@ -69,14 +70,33 @@ afmongodb_options ; afmongodb_option - : KW_SERVERS '(' string_list ')' { afmongodb_dd_set_servers(last_driver, $3); } + : KW_SERVERS '(' string_list ')' + { + CHECK_ERROR(afmongodb_dd_check_address(last_driver, FALSE), @3, "Can't mix path() & servers()"); + afmongodb_dd_set_servers(last_driver, $3); + } + | KW_HOST '(' string ')' + { + CHECK_ERROR(afmongodb_dd_check_address(last_driver, FALSE), @3, "Can't mix path() & host()"); + afmongodb_dd_set_host(last_driver, $3); + free($3); + } + | KW_PORT '(' LL_NUMBER ')' + { + CHECK_ERROR(afmongodb_dd_check_address(last_driver, FALSE), @3, "Can't mix path() & port()"); + afmongodb_dd_set_port(last_driver, $3); + } + | KW_PATH '(' string ')' + { + CHECK_ERROR(afmongodb_dd_check_address(last_driver, TRUE), @3, "Can't mix path() with host() or server()"); + afmongodb_dd_set_path(last_driver, $3); + free($3); + } | KW_DATABASE '(' string ')' { afmongodb_dd_set_database(last_driver, $3); free($3); } | KW_COLLECTION '(' string ')' { afmongodb_dd_set_collection(last_driver, $3); free($3); } | KW_USERNAME '(' string ')' { afmongodb_dd_set_user(last_driver, $3); free($3); } | KW_PASSWORD '(' string ')' { afmongodb_dd_set_password(last_driver, $3); free($3); } | KW_SAFE_MODE '(' yesno ')' { afmongodb_dd_set_safe_mode(last_driver, $3); } - | KW_HOST '(' string ')' { afmongodb_dd_set_host(last_driver, $3); free($3); } - | KW_PORT '(' LL_NUMBER ')' { afmongodb_dd_set_port(last_driver, $3); } | value_pair_option { afmongodb_dd_set_value_pairs(last_driver, $1); } | dest_driver_option ; diff --git a/modules/afmongodb/afmongodb-parser.c b/modules/afmongodb/afmongodb-parser.c index 9d72878..8aa55c1 100644 --- a/modules/afmongodb/afmongodb-parser.c +++ b/modules/afmongodb/afmongodb-parser.c @@ -38,6 +38,7 @@ static CfgLexerKeyword afmongodb_keywords[] = { { "safe_mode", KW_SAFE_MODE }, { "host", KW_HOST }, { "port", KW_PORT }, + { "path", KW_PATH }, { NULL } }; diff --git a/modules/afmongodb/afmongodb.c b/modules/afmongodb/afmongodb.c index 5af34c9..aeed3d3 100644 --- a/modules/afmongodb/afmongodb.c +++ b/modules/afmongodb/afmongodb.c @@ -50,7 +50,7 @@ typedef struct gchar *coll; GList *servers; - gchar *host; + gchar *address; gint port; gboolean safe_mode; @@ -116,8 +116,8 @@ afmongodb_dd_set_host(LogDriver *d, const gchar *host) { MongoDBDestDriver *self = (MongoDBDestDriver *)d; - g_free(self->host); - self->host = g_strdup(host); + g_free(self->address); + self->address = g_strdup(host); } void @@ -138,6 +138,39 @@ afmongodb_dd_set_servers(LogDriver *d, GList *servers) } void +afmongodb_dd_set_path(LogDriver *d, const gchar *path) +{ + MongoDBDestDriver *self = (MongoDBDestDriver *)d; + + g_free(self->address); + self->address = g_strdup(path); + self->port = MONGO_CONN_LOCAL; +} + +gboolean +afmongodb_dd_check_address(LogDriver *d, gboolean local) +{ + MongoDBDestDriver *self = (MongoDBDestDriver *)d; + + if (local) + { + if ((self->port != 0 || + self->port != MONGO_CONN_LOCAL) && + self->address != NULL) + return FALSE; + if (self->servers) + return FALSE; + } + else + { + if (self->port == MONGO_CONN_LOCAL && + self->address != NULL) + return FALSE; + } + return TRUE; +} + +void afmongodb_dd_set_database(LogDriver *d, const gchar *database) { MongoDBDestDriver *self = (MongoDBDestDriver *)d; @@ -182,8 +215,12 @@ afmongodb_dd_format_stats_instance(MongoDBDestDriver *self) { static gchar persist_name[1024]; - g_snprintf(persist_name, sizeof(persist_name), - "mongodb,%s,%u,%s,%s", self->host, self->port, self->db, self->coll); + if (self->port == MONGO_CONN_LOCAL) + g_snprintf(persist_name, sizeof(persist_name), + "mongodb,%s,%s,%s", self->address, self->db, self->coll); + else + g_snprintf(persist_name, sizeof(persist_name), + "mongodb,%s,%u,%s,%s", self->address, self->port, self->db, self->coll); return persist_name; } @@ -192,8 +229,12 @@ afmongodb_dd_format_persist_name(MongoDBDestDriver *self) { static gchar persist_name[1024]; - g_snprintf(persist_name, sizeof(persist_name), - "afmongodb(%s,%u,%s,%s)", self->host, self->port, self->db, self->coll); + if (self->port == MONGO_CONN_LOCAL) + g_snprintf(persist_name, sizeof(persist_name), + "afmongodb(%s,%s,%s)", self->address, self->db, self->coll); + else + g_snprintf(persist_name, sizeof(persist_name), + "afmongodb(%s,%u,%s,%s)", self->address, self->port, self->db, self->coll); return persist_name; } @@ -221,7 +262,7 @@ afmongodb_dd_connect(MongoDBDestDriver *self, gboolean reconnect) if (reconnect && self->conn) return TRUE; - self->conn = mongo_sync_connect(self->host, self->port, FALSE); + self->conn = mongo_sync_connect(self->address, self->port, FALSE); if (!self->conn) { msg_error ("Error connecting to MongoDB", NULL); @@ -467,31 +508,45 @@ afmongodb_dd_init(LogPipe *s) value_pairs_add_scope(self->vp, "nv-pairs"); } - if (self->host) - { - gchar *srv = g_strdup_printf ("%s:%d", self->host, - (self->port) ? self->port : 27017); - self->servers = g_list_prepend (self->servers, srv); - g_free (self->host); - } - - self->host = NULL; - self->port = 27017; - if (!mongo_util_parse_addr(g_list_nth_data(self->servers, 0), &self->host, - &self->port)) + if (self->port != MONGO_CONN_LOCAL) { - msg_error("Cannot parse the primary host", - evt_tag_str("primary", g_list_nth_data(self->servers, 0)), - NULL); - return FALSE; + if (self->address) + { + gchar *srv = g_strdup_printf ("%s:%d", self->address, + (self->port) ? self->port : 27017); + self->servers = g_list_prepend (self->servers, srv); + g_free (self->address); + } + + if (!self->servers) + afmongodb_dd_set_servers((LogDriver *)self, g_list_append (NULL, g_strdup ("127.0.0.1:27017"))); + + self->address = NULL; + self->port = 27017; + if (!mongo_util_parse_addr(g_list_nth_data(self->servers, 0), + &self->address, + &self->port)) + { + msg_error("Cannot parse the primary host", + evt_tag_str("primary", g_list_nth_data(self->servers, 0)), + NULL); + return FALSE; + } } - msg_verbose("Initializing MongoDB destination", - evt_tag_str("host", self->host), - evt_tag_int("port", self->port), - evt_tag_str("database", self->db), - evt_tag_str("collection", self->coll), - NULL); + if (self->port == MONGO_CONN_LOCAL) + msg_verbose("Initializing MongoDB destination", + evt_tag_str("address", self->address), + evt_tag_str("database", self->db), + evt_tag_str("collection", self->coll), + NULL); + else + msg_verbose("Initializing MongoDB destination", + evt_tag_str("address", self->address), + evt_tag_int("port", self->port), + evt_tag_str("database", self->db), + evt_tag_str("collection", self->coll), + NULL); self->queue = log_dest_driver_acquire_queue(&self->super, afmongodb_dd_format_persist_name(self)); @@ -548,7 +603,7 @@ afmongodb_dd_free(LogPipe *d) g_free(self->coll); g_free(self->user); g_free(self->password); - g_free(self->host); + g_free(self->address); string_list_free(self->servers); if (self->vp) value_pairs_free(self->vp); @@ -609,7 +664,6 @@ afmongodb_dd_new(void) self->super.super.super.queue = afmongodb_dd_queue; self->super.super.super.free_fn = afmongodb_dd_free; - afmongodb_dd_set_servers((LogDriver *)self, g_list_append (NULL, g_strdup ("127.0.0.1:27017"))); afmongodb_dd_set_database((LogDriver *)self, "syslog"); afmongodb_dd_set_collection((LogDriver *)self, "messages"); afmongodb_dd_set_safe_mode((LogDriver *)self, FALSE); diff --git a/modules/afmongodb/afmongodb.h b/modules/afmongodb/afmongodb.h index e1f4b57..6a61616 100644 --- a/modules/afmongodb/afmongodb.h +++ b/modules/afmongodb/afmongodb.h @@ -38,5 +38,8 @@ void afmongodb_dd_set_user(LogDriver *d, const gchar *user); void afmongodb_dd_set_password(LogDriver *d, const gchar *password); void afmongodb_dd_set_value_pairs(LogDriver *d, ValuePairs *vp); void afmongodb_dd_set_safe_mode(LogDriver *d, gboolean state); +void afmongodb_dd_set_path(LogDriver *d, const gchar *path); + +gboolean afmongodb_dd_check_address(LogDriver *d, gboolean local); #endif diff --git a/modules/afmongodb/libmongo-client b/modules/afmongodb/libmongo-client index 58f3814..7896ce7 160000 --- a/modules/afmongodb/libmongo-client +++ b/modules/afmongodb/libmongo-client @@ -1 +1 @@ -Subproject commit 58f3814cad94bcd78216c7ac971c8435d17a9242 +Subproject commit 7896ce7dbc239e4069787fee336875a9b9a6f6f5 -- 1.7.10.4

2 1

[PATCH (3.4)] afsocket: Implement tcp_keep_alive_time()
by Gergely Nagy 18 Aug '12

18 Aug '12

With the new tcp_keep_alive_time() option one can control on a per-source basis the time after which a TCP connection will start sending TCP keepalive probes (assuming so-keepalive(yes)). While it is accepted for all stream sources, it only makes sense for TCP-based streams. By default, it's value is 0, which means using the kernel default (2 hours). Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/afsocket/afsocket-grammar.ym | 2 ++ modules/afsocket/afsocket-parser.c | 1 + modules/afsocket/afsocket.c | 11 +++++++++++ modules/afsocket/afsocket.h | 2 ++ 4 files changed, 16 insertions(+), 0 deletions(-) diff --git a/modules/afsocket/afsocket-grammar.ym b/modules/afsocket/afsocket-grammar.ym index 7970ca9..48b0526 100644 --- a/modules/afsocket/afsocket-grammar.ym +++ b/modules/afsocket/afsocket-grammar.ym @@ -83,6 +83,7 @@ TLSContext *last_tls_context; %token KW_SO_SNDBUF %token KW_SO_RCVBUF %token KW_SO_KEEPALIVE +%token KW_TCP_KEEP_ALIVE_TIME %token KW_SPOOF_SOURCE %token KW_KEEP_ALIVE @@ -252,6 +253,7 @@ source_afinet_tcp_option source_afsocket_stream_params : KW_KEEP_ALIVE '(' yesno ')' { afsocket_sd_set_keep_alive(last_driver, $3); } + | KW_TCP_KEEP_ALIVE_TIME '(' LL_NUMBER ')' { afsocket_sd_set_keep_alive_time(last_driver, $3); } | KW_MAX_CONNECTIONS '(' LL_NUMBER ')' { afsocket_sd_set_max_connections(last_driver, $3); } ; diff --git a/modules/afsocket/afsocket-parser.c b/modules/afsocket/afsocket-parser.c index 44671ca..bd8f5d6 100644 --- a/modules/afsocket/afsocket-parser.c +++ b/modules/afsocket/afsocket-parser.c @@ -64,6 +64,7 @@ static CfgLexerKeyword afsocket_keywords[] = { { "so_sndbuf", KW_SO_SNDBUF }, { "so_keepalive", KW_SO_KEEPALIVE }, { "tcp_keep_alive", KW_SO_KEEPALIVE, 0, KWS_OBSOLETE, "so_keepalive" }, + { "tcp_keep_alive_time",KW_TCP_KEEP_ALIVE_TIME }, { "spoof_source", KW_SPOOF_SOURCE }, { "transport", KW_TRANSPORT }, { "max_connections", KW_MAX_CONNECTIONS }, diff --git a/modules/afsocket/afsocket.c b/modules/afsocket/afsocket.c index a7cc7bc..4a71b11 100644 --- a/modules/afsocket/afsocket.c +++ b/modules/afsocket/afsocket.c @@ -43,6 +43,7 @@ #include <arpa/inet.h> #include <sys/stat.h> #include <unistd.h> +#include <netinet/tcp.h> #if ENABLE_TCP_WRAPPER #include <tcpd.h> @@ -79,6 +80,8 @@ afsocket_setup_socket(gint fd, SocketOptions *sock_options, AFSocketDirection di setsockopt(fd, SOL_SOCKET, SO_BROADCAST, &sock_options->broadcast, sizeof(sock_options->broadcast)); } setsockopt(fd, SOL_SOCKET, SO_KEEPALIVE, &sock_options->keepalive, sizeof(sock_options->keepalive)); + if (sock_options->keepalive_time > 0) + setsockopt(fd, SOL_TCP, TCP_KEEPIDLE, &sock_options->keepalive_time, sizeof(sock_options->keepalive_time)); return TRUE; } @@ -386,6 +389,14 @@ afsocket_sd_set_keep_alive(LogDriver *s, gint enable) } void +afsocket_sd_set_keep_alive_time(LogDriver *s, gint timeout) +{ + AFSocketSourceDriver *self = (AFSocketSourceDriver *) s; + + self->keepalive_timeout = timeout; +} + +void afsocket_sd_set_max_connections(LogDriver *s, gint max_connections) { AFSocketSourceDriver *self = (AFSocketSourceDriver *) s; diff --git a/modules/afsocket/afsocket.h b/modules/afsocket/afsocket.h index 392e249..fd11d28 100644 --- a/modules/afsocket/afsocket.h +++ b/modules/afsocket/afsocket.h @@ -58,6 +58,7 @@ typedef struct _SocketOptions gint rcvbuf; gint broadcast; gint keepalive; + gint keepalive_time; } SocketOptions; gboolean afsocket_setup_socket(gint fd, SocketOptions *sock_options, AFSocketDirection dir); @@ -78,6 +79,7 @@ struct _AFSocketSourceDriver gint max_connections; gint num_connections; gint listen_backlog; + gint keepalive_timeout; GList *connections; SocketOptions *sock_options_ptr; -- 1.7.9

2 1

[PATCH] db-parser(): implement a MAC address parser
by Balint Kovacs 18 Aug '12

18 Aug '12

While talking about netfilter logs recently, we found, that a MAC address parser may come in handy when parsing these messages, so here it comes. I hope others might find it useful as well. Balint

2 2

[PATCH (3.4)] jsonparser: Clone the marker() setting when cloning the parser
by Gergely Nagy 18 Aug '12

18 Aug '12

Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/jsonparser/jsonparser.c | 1 + 1 file changed, 1 insertion(+) diff --git a/modules/jsonparser/jsonparser.c b/modules/jsonparser/jsonparser.c index 7331ad9..891b9f0 100644 --- a/modules/jsonparser/jsonparser.c +++ b/modules/jsonparser/jsonparser.c @@ -201,6 +201,7 @@ log_json_parser_clone (LogPipe *s) cloned = (LogJSONParser *) log_json_parser_new (); log_json_parser_set_prefix ((LogParser *)cloned, self->prefix); + log_json_parser_set_marker ((LogParser *)cloned, self->marker); return &cloned->super.super; } -- 1.7.10

2 2

[PATCH (3.4)] afsmtp: Fix the module linkage
by Gergely Nagy 18 Aug '12

18 Aug '12

Use $(MODULE_DEPS_LIBS) and $(MODULE_LDFLAGS) instead of trying to grab all the libs. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/afsmtp/Makefile.am | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/modules/afsmtp/Makefile.am b/modules/afsmtp/Makefile.am index bb5b94d..34c0e98 100644 --- a/modules/afsmtp/Makefile.am +++ b/modules/afsmtp/Makefile.am @@ -6,10 +6,10 @@ export top_srcdir if ENABLE_SMTP module_LTLIBRARIES = libafsmtp.la -libafsmtp_la_CFLAGS = $(LIBESMTP_CFLAGS) $(IVYKIS_CFLAGS) $(EVENTLOG_CFLAGS) $(SYSLOG_NG_CFLAGS) +libafsmtp_la_CFLAGS = $(LIBESMTP_CFLAGS) libafsmtp_la_SOURCES = afsmtp-grammar.y afsmtp.c afsmtp.h afsmtp-parser.c afsmtp-parser.h -libafsmtp_la_LIBADD = $(LIBESMTP_LIBS) $(IVYKIS_LIBS) $(EVENTLOG_LIBS) $(SYSLOG_NG_LIBS) -libafsmtp_la_LDFLAGS = -avoid-version +libafsmtp_la_LIBADD = $(LIBESMTP_LIBS) $(MODULE_DEPS_LIBS) +libafsmtp_la_LDFLAGS = $(MODULE_LDFLAGS) endif -- 1.7.10.4

2 1

[Bug 62] PROGRAM macro problem
by bugzilla＠bugzilla.balabit.com 18 Aug '12

18 Aug '12

https://bugzilla.balabit.com/show_bug.cgi?id=62 Gergely Nagy <algernon(a)balabit.hu> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |algernon(a)balabit.hu Resolution| |FIXED Status|NEW |RESOLVED --- Comment #4 from Gergely Nagy <algernon(a)balabit.hu> 2012-08-18 13:26:40 --- syslog-ng 3.4 has a $(sanitize) template function, which will replace "/" with "_", so this issue can be considered resolved, I believe. The same function can be configured to replace parens and pretty much anything else too, if so need be. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 0

[PATCH 0/1] [RESEND] libmongo-client: Add support for Unix sockets
by Conrad Hoffmann 16 Aug '12

16 Aug '12

Hey once more. Here is a first revision of my earlier patch. Changes to last version: - Better error handling/input checking (thanks to Balazs for the feedback) - Free everything that was allocated before bailing out after errors - Some style fixes, hope it's ok now Further feedback welcome! Conrad Hoffmann (1): Add support for Unix sockets. examples/mongo-dump.c | 22 +++++++++++++++++----- src/mongo-client.c | 39 +++++++++++++++++++++++++++++++++++++++ src/mongo-client.h | 12 ++++++++++++ src/mongo-sync.c | 23 +++++++++++++++++++++++ src/mongo-sync.h | 15 +++++++++++++++ 5 files changed, 106 insertions(+), 5 deletions(-) -- 1.7.11.4

2 7