- syslog-ng - lists.balabit.hu

Announcing Debian packages for syslog-ng 3.3
by Gergely Nagy 23 Aug '11

23 Aug '11

Hi! As I've hinted about it in an earlier mail, I've been working on packaging syslog-ng 3.3beta2 (mostly to scratch an own itch: I want 3.3 packaged, and split into smaller parts :P), and as of this writing, packages are slowly flowing in to the newly created repository: # For Debian unstable & testing: deb http://packages.madhouse-project.org/debian unstable syslog-ng deb-src http://packages.madhouse-project.org/debian unstable syslog-ng # For Debian stable (squeeze): # (this also contains libmongo-client and libevtlog backported to # squeeze) deb http://packages.madhouse-project.org/debian stable syslog-ng deb-src http://packages.madhouse-project.org/debian stable syslog-ng For now, only amd64 binaries are built, if there's a need, I can build i386 and powerpc packages aswell. These Debian packages are NOT the stock debian syslog-ng package updated for 3.3: the packaging was pretty much redone from scratch (and they're still a work in progress). Nevertheless, I believe they're in a usable shape, and could be useful for people on Debian wishing to play with syslog-ng 3.3. Anyway, the repositories above contain the following packages: * syslog-ng This was turned into a meta-package, that doesn't really have any contents, but depends on the rest of the syslog-ng packages. It is provided both as a clear upgrade path from the stock Debian packages, and as a way to install every plugin syslog-ng has. * syslog-ng-core This is what used to be the syslog-ng package in stock Debian: the standard feature set, minus a few plugins, which were split out. See below! * syslog-ng-module-sql Contains the SQL destination module. Split out into a separate package, so that the core does not have to depend on libdbi. * syslog-ng-module-mongodb The MongoDB destination driver. * syslog-ng-module-json The $(format-json) (aka tfjson) plugin. * syslog-ng-dev Development files (headers, tools, etc) to build third-party syslog-ng modules. This part is non-functional at the moment, it will be fixed at some point in the not too distant future, but since this part is mainly useful for me alone, I didn't care that much. If there's any issues with the repository, or with the packages in them, please let me know, so I can fix them! I will keep updating the packages, to fix any bugs I - or anyone else - finds. -- |8]

1 1

libafsocket.so
by Peter Czanik 23 Aug '11

23 Aug '11

Hello, I'm working on the 3.3 beta2 package for openSUSE and I wonder, if there is any reason to keep libafsocket-notls.so in the package. -rwxr-xr-x 1 root root 117965 Aug 19 11:18 libafsocket-notls.so lrwxrwxrwx 1 root root 18 Aug 23 10:14 libafsocket.so -> libafsocket-tls.so -rwxr-xr-x 1 root root 117965 Aug 19 11:18 libafsocket-tls.so Only afsocket is referenced from modules.conf, which is a link to libafsocket-tls.so If I remove the link and the notls version and rename the tls to libafsocket.so, then I could get rid of a couple of post build check warnings (which are soon to be turned into errors), and also re-claim 100+k from my HDD. Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

2 2

Problem with program_override in upgrade from 3.0.8 to 3.2.4
by Javi Polo 22 Aug '11

22 Aug '11

Hello there I've been using syslog-ng for a long time, no problems so far, till today ... :p I'm using Open Source Edition, upgrading from 3.0.8 to 3.2.4, installing from the .run file to /opt Today I wanted to update our syslog-ng's to the latest version and found that for some reason, when I override a program via program_override, the PROGRAM macro is empty when I send it to another loghost. program_override seems to be working, as locally writen files show so I did upgrade both syslog-ng, the client and the logserver When I switched back to the old version I found everything began working again Here's the conflicting config in the client: #CLIENT CONFIG source s_apache_access { file("/var/log/apache2/access.log" program_override ("apache_access")); }; destination d_logserver01 { tcp("logserver01"); }; destination d_tmp { file("/var/log/tmp.log" template("$HOST $PROGRAM $MESSAGE\n")); }; log { source(s_apache_error); destination(d_logserver01); destination(d_tmp); flags(final); }; And here's on the server: #LOGSERVER CONFIG filter f_tmp { host(web0[1-2]); }; filter f_apache_error { program(apache_error); }; destination d_apache_error { file("/array/logs/${S_YEAR}/${S_MONTH}/${S_DAY}/${HOST}/apache2/error.log" template("$ISODATE $HOST ${PROGRAM}: $MESSAGE\n") ); }; destination d_tmp { file("/array/logs/${S_YEAR}/${S_MONTH}/${S_DAY}/${HOST}/tmp.log" template("$HOST $PROGRAM $MSG\n")); }; log { source(s_tcp); filter(f_apache_error); destination(d_apache_error); flags(final); }; log { source(s_tcp); filter(f_tmp); destination(d_tmp); flags(final); }; After the update those are the results of tmp.log On client: web01 apache_error Aug 02 20:53:06 2011] [error] [client 192.168.0.5] client denied by server configuration: /var/www/asdadsasdas On server: web01 Aug 02 20:53:06 2011] [error] [client 192.168.50.65] client denied by server configuration: /var/www/asdadsasdas As you can see, the program_name has dissapeared I suspect is a bug, but I'm not sure as there might be some change in syslog-ng behaviour that I did miss :? Anybody could lend me a hand? thanks in advance :) -- Javi Polo Administrador de Sistemas Tel 93 734 97 70 Fax 93 734 97 71 jpolo(a)wtransnet.com

2 4

Quick question about libsyslog-ng.so.0
by Gergely Nagy 21 Aug '11

21 Aug '11

Hi! As part of packaging syslog-ng, and splitting it into smaller packages, I face the problem of having to create a devel package. Now, I could go the easy way (as I've done with the current set of Debian packages) and create a syslog-ng-core package, which would contain libsyslog-ng.so.0, the plugins and the rest, and a syslog-ng-dev package with the headers and other tools. This could work, but the downside is that on an automated buildd, one does not neccessarily want to replace the running syslogd (even if that's already syslog-ng, which it is in my case) just to be able to build a package. So I was thinking about splitting the libsyslog-ng.so library out into its own package. And herein lies the problem! The SONAME of the library is currently not set, thus, libtool defaults to libsyslog-ng.so.0. The problem with this, is that the SONAME will remain the same, even if the library breaks its ABI underneath. Which will be a problem for any third party module linked against this library: a new version of syslog-ng comes out, all of them will break at runtime, in various interesting ways, perhaps even after days of working, when the module runs over an incompatible change. This can be worked around on the packaging level, by ensuring that a new version of the libsyslog-ng package will conflict with any older versions, but... that's a pain to maintain, and very fragile. So instead, I'd like to propose a different way: use the -release flag of Libtool. This will make the library's SONAME contain syslog-ng's version aswell, thus, every upgrade will use a different soname (like libsyslog-ng-3.3.0beta2.so), and modules will break at startup time, which is much better, as we don't trip over unexpected changes. It is also tremendously easier to package a semi-private library this way. (Though, this has a few downsides as well from a packaging point of view, but considerably less.) So here I am, seeking input on the issue! By the way, a third option also exists: maintaining ABI and API compatibility, and only bumping the SONAME where something did really change incompatibly. This would be the best option, but this also takes a lot more work to maintain properly (I'm happy to help with this, though, I'm already doing this for libmongo-client; but on the other hand, libsyslog-ng really is something syslog-ng specific, and not a general purpose library, so it might not worth the trouble to go this far). -- |8]

2 2

A couple of notes regarding packagability
by Gergely Nagy 20 Aug '11

20 Aug '11

Hi! I started to play around with creating various packages of 3.3.0beta2, and noticed a couple of annoying bits and pieces (none of this is serious, I can workaround most if not all of them, but the packaging would be a lot simpler if I wouldn't have to): * I sent a patch back in May, which made it possible to use a system-wide libmongo-client installation, but it does not appear to be merged yet. It is attached to this email, diffed against current git head. * autogen.sh is not executable in a fresh checkout, and this repeatedly bites me over and over again. * autogen.sh will always try to call git submodule init when autogenerating the submodules. I'd like a way to skip this step, and just use whatever is in the subdirectories, assuming there IS something. The reason I want this, is because I'm building from a git checkout, not a make dist'd tarball, and my build chroots do not have network access (nor do I want to give them any). * It would also be nice if autogen.sh used the --copy flag for libtoolize and automake. * When the system-wide libmongo-client support patch is applied, and syslog-ng is built with --with-libmongo-client=system, make distclean will fail, as it tries to clean in modules/afmongodb/libmongo-client/, while that hasn't been configured in. I do not know an easy solution around this, except maybe overriding DIST_SUBDIRS with SUBDIRS for the distclean-recursive target in modules/afmongodb/ - but that sounds like a crude hack, and I haven't tested if it actually works. This is all, so far. I'll have another packaging sprint sometime tomorrow, and see if I find anything else. I'll try to find time to submit patches for the above, but I'm not sure if I'll be able to this weekend. It is very likely, that I'll submit a patch for system() to recognise GNU/kFreeBSD and create the appropriate entries. PS: syslog-ng 3.3.0beta2 packages for Debian squeeze & sid should be available by the end of the coming week, for i386, amd64 and kfreebsd-amd64 at least. I just have to iron out a few issues and add some finishing touches to my build farm. -- |8]

2 2

syslog-ng-crypto.so is versioned again
by Peter Czanik 20 Aug '11

20 Aug '11

Hello, This was once fixed in http://git.balabit.hu/?p=bazsi/syslog-ng-3.3.git;a=commitdiff;h=dbb63b6f2d8… but now I see it again. Verified both on FreeBSD 8.1 and openSUSE factory, versioned libraries are installed under lib/syslog/ Caught, as usual, by the openSUSE post build checks... Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

2 1

[PATCH] affile: Implement size based filename rotation.
by Gergely Nagy 20 Aug '11

20 Aug '11

Based on a patch made by Sergei Zhirikov for syslog-ng 3.1, this patch implements a new size-limit() option for file based destinations. The syntax is as follows: destination d_messages { file("/var/log/messages" size-limit(104857600)); }; destination d_syslog { file("/var/log/syslog" size-limit(10485760 "/var/log/syslog.${SEQNUM}")); }; That is, size-limit() takes a file size limit (in bytes), and optionally a filename template to rotate to. If no filename template is specified, it will use the template used for the file itself, and append "-${UNIXTIME}.${SEQNUM}". The file is rotated after it reaches the specified size limit, therefore, it will usually be a message bigger than requested. And while rotating, if the filename to rotate to already exists, it will be overwritten. It can also be configured globally (only the limit, not the template) using the file-size-limit() global option. Signed-off-by: Sergei Zhirikov <sfzhi(a)yahoo.com> Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- lib/cfg-grammar.y | 2 + lib/cfg-parser.c | 1 + lib/cfg.c | 1 + lib/cfg.h | 1 + lib/logproto.c | 15 +++- lib/logproto.h | 14 ++++- modules/affile/affile-grammar.ym | 3 + modules/affile/affile-parser.c | 1 + modules/affile/affile.c | 137 ++++++++++++++++++++++++++++++++++---- modules/affile/affile.h | 3 + 10 files changed, 160 insertions(+), 18 deletions(-) diff --git a/lib/cfg-grammar.y b/lib/cfg-grammar.y index 4748de4..ab4fd83 100644 --- a/lib/cfg-grammar.y +++ b/lib/cfg-grammar.y @@ -213,6 +213,7 @@ %token KW_OWNER 10250 %token KW_GROUP 10251 %token KW_PERM 10252 +%token KW_FILE_SIZE_LIMIT 10253 %token KW_DIR_OWNER 10260 %token KW_DIR_GROUP 10261 @@ -653,6 +654,7 @@ options_item | KW_RECV_TIME_ZONE '(' string ')' { configuration->recv_time_zone = g_strdup($3); free($3); } | KW_SEND_TIME_ZONE '(' string ')' { configuration->template_options.time_zone[LTZ_SEND] = g_strdup($3); free($3); } | KW_LOCAL_TIME_ZONE '(' string ')' { configuration->template_options.time_zone[LTZ_LOCAL] = g_strdup($3); free($3); } + | KW_FILE_SIZE_LIMIT '(' LL_NUMBER ')' { configuration->file_size_limit = $3; } ; /* START_RULES */ diff --git a/lib/cfg-parser.c b/lib/cfg-parser.c index 589206a..2763a4c 100644 --- a/lib/cfg-parser.c +++ b/lib/cfg-parser.c @@ -101,6 +101,7 @@ static CfgLexerKeyword main_keywords[] = { { "default_priority", KW_DEFAULT_LEVEL, 0x0300 }, { "default_facility", KW_DEFAULT_FACILITY, 0x0300 }, { "threaded", KW_THREADED, 0x0303 }, + { "file_size_limit", KW_FILE_SIZE_LIMIT, 0x0303 }, { "value", KW_VALUE, 0x0300 }, diff --git a/lib/cfg.c b/lib/cfg.c index 006f416..896eed3 100644 --- a/lib/cfg.c +++ b/lib/cfg.c @@ -339,6 +339,7 @@ cfg_new(gint version) self->file_uid = 0; self->file_gid = 0; self->file_perm = 0600; + self->file_size_limit = 0; self->dir_uid = 0; self->dir_gid = 0; self->dir_perm = 0700; diff --git a/lib/cfg.h b/lib/cfg.h index 3083a51..2b18316 100644 --- a/lib/cfg.h +++ b/lib/cfg.h @@ -88,6 +88,7 @@ struct _GlobalConfig gint file_uid; gint file_gid; gint file_perm; + off_t file_size_limit; gint dir_uid; gint dir_gid; diff --git a/lib/logproto.c b/lib/logproto.c index 92476ac..70286d5 100644 --- a/lib/logproto.c +++ b/lib/logproto.c @@ -241,6 +241,8 @@ typedef struct _LogProtoFileWriter gint buf_count; gint fd; gint sum_len; + LogProtoFileWriterCallback callback; + void *context; struct iovec buffer[0]; } LogProtoFileWriter; @@ -257,6 +259,7 @@ log_proto_file_writer_flush(LogProto *s) { LogProtoFileWriter *self = (LogProtoFileWriter *)s; gint rc, i, i0, sum, ofs; + off_t pos; /* we might be called from log_writer_deinit() without having a buffer at all */ @@ -265,8 +268,11 @@ log_proto_file_writer_flush(LogProto *s) /* lseek() is used instead of O_APPEND, as on NFS O_APPEND performs * poorly, as reported on the mailing list 2008/05/29 */ + pos = lseek(self->fd, 0, SEEK_END); + + if (self->callback) + self->callback(self->fd, pos, self->context); - lseek(self->fd, 0, SEEK_END); rc = writev(self->fd, self->buffer, self->buf_count); if (rc < 0) @@ -414,7 +420,8 @@ log_proto_file_writer_prepare(LogProto *s, gint *fd, GIOCondition *cond) } LogProto * -log_proto_file_writer_new(LogTransport *transport, gint flush_lines) +log_proto_file_writer_new_with_callback(LogTransport *transport, gint flush_lines, + LogProtoFileWriterCallback callback, void *context) { if (flush_lines == 0) /* the flush-lines option has not been specified, use a default value */ @@ -428,6 +435,8 @@ log_proto_file_writer_new(LogTransport *transport, gint flush_lines) self->fd = transport->fd; self->buf_size = flush_lines; + self->callback = callback; + self->context = context; self->super.prepare = log_proto_file_writer_prepare; self->super.post = log_proto_file_writer_post; self->super.flush = log_proto_file_writer_flush; @@ -436,8 +445,6 @@ log_proto_file_writer_new(LogTransport *transport, gint flush_lines) return &self->super; } - - typedef struct _LogProtoBufferedServerState { /* NOTE: that if you add/remove structure members you have to update diff --git a/lib/logproto.h b/lib/logproto.h index 443de16..a7cf200 100644 --- a/lib/logproto.h +++ b/lib/logproto.h @@ -36,6 +36,7 @@ typedef struct _LogProtoFileReader LogProtoFileReader; typedef enum { LPS_SUCCESS, + LPS_RETRY, LPS_ERROR, LPS_EOF, } LogProtoStatus; @@ -152,7 +153,18 @@ LogProto *log_proto_dgram_server_new(LogTransport *transport, gint max_msg_size, */ LogProto *log_proto_text_server_new(LogTransport *transport, gint max_msg_size, guint flags); -LogProto *log_proto_file_writer_new(LogTransport *transport, gint flush_lines); +/* LogProtoFileWriter + */ +typedef void (*LogProtoFileWriterCallback)(gint fd, off_t size, void *context); +LogProto *log_proto_file_writer_new_with_callback(LogTransport *transport, gint flush_lines, + LogProtoFileWriterCallback callback, + void *context); + +static inline LogProto * +log_proto_file_writer_new(LogTransport *transport, gint flush_lines) +{ + return log_proto_file_writer_new_with_callback(transport, flush_lines, NULL, NULL); +} /* * LogProtoTextClient diff --git a/modules/affile/affile-grammar.ym b/modules/affile/affile-grammar.ym index 5915e5f..34c8e9b 100644 --- a/modules/affile/affile-grammar.ym +++ b/modules/affile/affile-grammar.ym @@ -61,6 +61,7 @@ extern LogDriver *last_driver; %token KW_FSYNC %token KW_FOLLOW_FREQ %token KW_OVERWRITE_IF_OLDER +%token KW_SIZE_LIMIT %type <ptr> source_affile %type <ptr> source_affile_params @@ -164,6 +165,8 @@ dest_affile_option | KW_OVERWRITE_IF_OLDER '(' LL_NUMBER ')' { affile_dd_set_overwrite_if_older(last_driver, $3); } | KW_FSYNC '(' yesno ')' { affile_dd_set_fsync(last_driver, $3); } | KW_LOCAL_TIME_ZONE '(' string ')' { affile_dd_set_local_time_zone(last_driver, $3); free($3); } + | KW_SIZE_LIMIT '(' LL_NUMBER ')' { affile_dd_set_file_size_limit(last_driver, $3, NULL); } + | KW_SIZE_LIMIT '(' LL_NUMBER string ')'{ affile_dd_set_file_size_limit(last_driver, $3, $4); free($4); } ; dest_afpipe_params diff --git a/modules/affile/affile-parser.c b/modules/affile/affile-parser.c index 1e0bb5f..cf25e42 100644 --- a/modules/affile/affile-parser.c +++ b/modules/affile/affile-parser.c @@ -38,6 +38,7 @@ static CfgLexerKeyword affile_keywords[] = { { "remove_if_older", KW_OVERWRITE_IF_OLDER, 0, KWS_OBSOLETE, "overwrite_if_older" }, { "overwrite_if_older", KW_OVERWRITE_IF_OLDER }, { "follow_freq", KW_FOLLOW_FREQ, }, + { "size_limit", KW_SIZE_LIMIT }, { NULL } }; diff --git a/modules/affile/affile.c b/modules/affile/affile.c index 168cc1b..1e1403a 100644 --- a/modules/affile/affile.c +++ b/modules/affile/affile.c @@ -472,6 +472,12 @@ struct _AFFileDestWriter time_t last_open_stamp; time_t time_reopen; struct iv_timer reap_timer; + struct + { + LogMessage *current_msg; + GString *filename; + gint32 seq; + } cb_context; gboolean reopen_pending, queue_pending; }; @@ -514,9 +520,77 @@ affile_dw_reap(gpointer s) } static gboolean +affile_dw_open_file(AFFileDestWriter *self, int *fd) +{ + int flags; + + if (self->owner->flags & AFFILE_PIPE) + flags = O_RDWR | O_NOCTTY | O_NONBLOCK | O_LARGEFILE; + else + flags = O_WRONLY | O_CREAT | O_NOCTTY | O_NONBLOCK | O_LARGEFILE; + + return affile_open_file(self->filename, flags, + self->owner->file_uid, self->owner->file_gid, self->owner->file_perm, + self->owner->dir_uid, self->owner->dir_gid, self->owner->dir_perm, + !!(self->owner->flags & AFFILE_CREATE_DIRS), FALSE, !!(self->owner->flags & AFFILE_PIPE), fd); +} + +static void +affile_dw_writer_callback(gint fd, off_t size, void *context) +{ + AFFileDestWriter *self = (AFFileDestWriter *) context; + int reopen_fd; + + if (self->owner->size_limit <= 0) + return; + + if ((size > 0) && (size >= self->owner->size_limit)) + { + g_string_truncate(self->cb_context.filename, 0); + log_template_format(self->owner->size_fn_format, self->cb_context.current_msg, + NULL, LTZ_LOCAL, self->cb_context.seq, NULL, + self->cb_context.filename); + + if (rename(self->filename, self->cb_context.filename->str) == 0) + { + if (affile_dw_open_file(self, &reopen_fd)) + { + if (dup2(reopen_fd, fd) < 0) + { + msg_error("Error switching to new log file", + evt_tag_str("old_filename", self->filename), + evt_tag_str("new_filename", self->cb_context.filename->str), + evt_tag_errno(EVT_TAG_OSERROR, errno), + NULL); + } + else + self->cb_context.seq++; + close(reopen_fd); + } + else + { + msg_error("Error opening file for writing", + evt_tag_str("old_filename", self->filename), + evt_tag_str("new_filename", self->cb_context.filename->str), + evt_tag_errno(EVT_TAG_OSERROR, errno), + NULL); + } + } + else + { + msg_error("Error renaming overgrown file", + evt_tag_str("old_filename", self->filename), + evt_tag_str("new_filanem", self->cb_context.filename->str), + evt_tag_errno(EVT_TAG_OSERROR, errno), + NULL); + } + } +} + +static gboolean affile_dw_reopen(AFFileDestWriter *self) { - int fd, flags; + int fd; struct stat st; self->last_open_stamp = self->last_msg_stamp; @@ -531,16 +605,7 @@ affile_dw_reopen(AFFileDestWriter *self) unlink(self->filename); } - if (self->owner->flags & AFFILE_PIPE) - flags = O_RDWR | O_NOCTTY | O_NONBLOCK | O_LARGEFILE; - else - flags = O_WRONLY | O_CREAT | O_NOCTTY | O_NONBLOCK | O_LARGEFILE; - - - if (affile_open_file(self->filename, flags, - self->owner->file_uid, self->owner->file_gid, self->owner->file_perm, - self->owner->dir_uid, self->owner->dir_gid, self->owner->dir_perm, - !!(self->owner->flags & AFFILE_CREATE_DIRS), FALSE, !!(self->owner->flags & AFFILE_PIPE), &fd)) + if (affile_dw_open_file(self, &fd)) { guint write_flags; @@ -550,7 +615,9 @@ affile_dw_reopen(AFFileDestWriter *self) log_writer_reopen(self->writer, self->owner->flags & AFFILE_PIPE ? log_proto_text_client_new(log_transport_plain_new(fd, write_flags)) - : log_proto_file_writer_new(log_transport_plain_new(fd, write_flags), self->owner->writer_options.flush_lines)); + : log_proto_file_writer_new_with_callback(log_transport_plain_new(fd, write_flags), + self->owner->writer_options.flush_lines, + affile_dw_writer_callback, self)); main_loop_call((void * (*)(void *)) affile_dw_arm_reaper, self, TRUE); } @@ -635,6 +702,8 @@ affile_dw_queue(LogPipe *s, LogMessage *lm, const LogPathOptions *path_options, if (self->last_open_stamp == 0) self->last_open_stamp = self->last_msg_stamp; + self->cb_context.current_msg = lm; + if (!log_writer_opened((LogWriter *) self->writer) && !self->reopen_pending && (self->last_open_stamp < self->last_msg_stamp - self->time_reopen)) @@ -667,7 +736,8 @@ static void affile_dw_free(LogPipe *s) { AFFileDestWriter *self = (AFFileDestWriter *) s; - + + g_string_free(self->cb_context.filename, TRUE); log_pipe_unref(self->writer); self->writer = NULL; g_free(self->filename); @@ -697,6 +767,8 @@ affile_dw_new(AFFileDestDriver *owner, const gchar *filename) /* we have to take care about freeing filename later. This avoids a move of the filename. */ self->filename = g_strdup(filename); + self->cb_context.filename = g_string_sized_new(1024); + self->cb_context.seq = 1; g_static_mutex_init(&self->lock); return self; } @@ -737,6 +809,33 @@ affile_dd_set_file_perm(LogDriver *s, gint file_perm) self->file_perm = file_perm; } +void +affile_dd_set_file_size_limit(LogDriver *s, off_t file_size_limit, const gchar *template) +{ + AFFileDestDriver *self = (AFFileDestDriver *) s; + + self->size_limit = file_size_limit; + + if (!self->size_fn_format) + self->size_fn_format = log_template_new(configuration, NULL); + + if (template) + log_template_compile(self->size_fn_format, template, NULL); + else + { + if (self->filename_template) + { + gchar *fn = g_strdup_printf ("%s-${UNIXTIME}.${SEQNUM}", self->filename_template->template); + log_template_compile(self->size_fn_format, fn, NULL); + g_free (fn); + } + else + { + msg_error("Error setting size-rotated filename template", NULL); + } + } +} + void affile_dd_set_dir_uid(LogDriver *s, const gchar *dir_uid) { @@ -889,6 +988,16 @@ affile_dd_init(LogPipe *s) self->dir_perm = cfg->dir_perm; if (self->time_reap == -1) self->time_reap = cfg->time_reap; + if (self->size_limit == -1) + { + gchar *fn = g_strdup_printf("%s-${UNIXTIME}.${SEQNUM}", + self->filename_template->template); + + self->size_limit = cfg->file_size_limit; + self->size_fn_format = log_template_new(cfg, NULL); + log_template_compile(self->size_fn_format, fn, NULL); + g_free (fn); + } log_writer_options_init(&self->writer_options, cfg, 0); log_template_options_init(&self->template_fname_options, cfg); @@ -1147,6 +1256,7 @@ affile_dd_free(LogPipe *s) g_assert(self->single_writer == NULL && self->writer_hash == NULL); log_template_options_destroy(&self->template_fname_options); + log_template_unref(self->size_fn_format); log_template_unref(self->filename_template); log_writer_options_destroy(&self->writer_options); log_dest_driver_free(s); @@ -1175,6 +1285,7 @@ affile_dd_new(gchar *filename, guint32 flags) self->flags |= AFFILE_NO_EXPAND; } self->time_reap = -1; + self->size_limit = -1; log_template_options_defaults(&self->template_fname_options); g_static_mutex_init(&self->lock); return &self->super.super; diff --git a/modules/affile/affile.h b/modules/affile/affile.h index f418e0d..e961fed 100644 --- a/modules/affile/affile.h +++ b/modules/affile/affile.h @@ -75,6 +75,8 @@ typedef struct _AFFileDestDriver gint overwrite_if_older; gboolean use_time_recvd; gint time_reap; + off_t size_limit; + LogTemplate *size_fn_format; } AFFileDestDriver; LogDriver *affile_dd_new(gchar *filename, guint32 flags); @@ -91,5 +93,6 @@ void affile_dd_set_create_dirs(LogDriver *s, gboolean create_dirs); void affile_dd_set_fsync(LogDriver *s, gboolean enable); void affile_dd_set_overwrite_if_older(LogDriver *s, gint overwrite_if_older); void affile_dd_set_local_time_zone(LogDriver *s, const gchar *local_time_zone); +void affile_dd_set_file_size_limit(LogDriver *s, off_t file_size_limit, const gchar *template); #endif -- 1.7.0.4

3 3

snmptrapd to syslog-ng 3.1
by Smart, Dan 20 Aug '11

20 Aug '11

My interest is in network device syslog and traps. I'm trying to receive traps, and then process them in Simple Event Correlator (SEC). I've got SEC working fine with standard remote syslog. After reading everything I could find, I found a discussion from 2008 about losing the source hostname when sending the trap to syslog. I'm trying the source program method, and eliminating multi-line traps. As I understand that syslog-ng is looking for Standard Out from the program, I specified -f in snmptrapd to stop forking, and -Lo to send output to standard output. I'm getting nothing in my d_debug file. Any suggestions? There is also a web page with a filter and rewrite recipe for traps. Not sure why I need this if I am sending the trap directly to SEC. See https://lists.balabit.hu/pipermail/syslog-ng/2008-November/012200.html And http://bazsi.blogs.balabit.com/2008/11/syslog-ng-3-0-and-snmp-traps/ -=Dan=- ========= syslog-ng.conf ================= # options { long_hostnames(off); use_dns(yes); use_fqdn(no); keep_hostname(yes); owner("root"); group("adm"); perm(0640); stats_freq(0); bad_hostname("^gconfd$"); ts_format(iso); flush_lines(100); log_fetch_limit(100); log_fifo_size(2048); dir_perm(0755); }; source s_program { program("/usr/sbin/snmptrapd -a -f -Lo --disableAuthorization=yes", flags(no-multi-line)); }; destination d_debug { file("/var/log/syslog-ng-debug" owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes)); }; destination d_sec { program("/usr/local/bin/sec -input=\"-\" -conf=/usr/local/etc/sec.conf" flags(no-multi-line) ); }; log { source(s_program); destination(d_sec); destination(d_debug); flags(flow-control); };

2 6

Store syslog occurrence frequency instead of adding all of them to the DB
by Marcos Tang 19 Aug '11

19 Aug '11

Hi, I am using Syslog-NG which stores the Syslog to a remote MySQL server for central logging. Now, I am running into a situation a lot of records inside the MySQL database is "almost" exactly the same, just except the timestamp is different. For example, I get "file system is full" every minute for a host and this message repeats itself periodically. In order to better manage the MySQL database, do you have any suggestions how I can reduce the number of overall records inside the DB, but add the occurrence frequency instead? For example, syslog message "file system is full" occurs every minutes on the system. There will be 24 x 60 "file system is full" records on my MySQL DB after 1 day. If no one fix it, I will get 10 x 24 x 60 "file system is full" records for 10 hosts having the same problem for 1 day. Can those records being "processed" some how and when I search the MySQL DB, I only see ONE record list the following only? Total occurrenceMessage content ===================== 14,400File system is full If I can do that, it will definitely reduce a log of space of my DB and speed up the query. It seems this is a database problem as well. Please let me know your idea at anytime. Regards, Marcos

5 7

openSUSE 3.3beta2 packages, nice article, poll
by Peter Czanik 19 Aug '11

19 Aug '11

Hello, I just sent out the syslog-ng newsletter a few days ago, but there are already fresh news. But first of all: If you got 1 minute, please fill our syslog-ng performance survey at https://www.surveymonkey.com/s/6ZQDVH6 There was a new release of syslog-ng OSE 3.3 last weekend, it's now at beta2. FreeBSD ports was the first to add support for it, and now here are the openSUSE packages for testing: http://download.opensuse.org/repositories/home:/czanik:/syslog-ng33/ or if more interested in package sources, then check https://build.opensuse.org/project/show?project=home%3Aczanik%3Asyslog-ng33 where these packages were built. Systemd support is being reworked in openSUSE, so please stick to sysvinit (which is the default) when testing these packages. There is one known problem affecting functionality: mongodb support is broken. It's already fixed in git, so it will be part of the next release, or if there is a request for it, I can add the patch to the beta2 package. I just found a nice introductory article about syslog-ng at Linux Journal, which describes the basics and also some more interesting topics and provides good explanations. You can read it at http://www.linuxjournal.com/content/creating-centralized-syslog-server It was written on Debian, but most information should apply to any Linux distribution. Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

2 1

mongodb stats problem
by Peter Czanik 17 Aug '11

17 Aug '11

Hello, I just ran into this problem while checking Google Alerts about syslog-ng: http://www.quora.com/Error-starting-syslog-ng-with-mongo-module Is it addressed with the latest patch in git, or it's something new? Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

2 1

syslog-ng Insider - August 2011
by Peter Czanik 17 Aug '11

17 Aug '11

Dear syslog-ng users, This is the 5th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com <mailto:documentation@balabit.com> FEATURED NEWS Please test 3.3 beta2! -------------------- Version 3.3 is a major step in syslog-ng's development, as it is now fully multi-threaded, and is also the first release with considerable amount of community developed features. A new beta was released this weekend! It received many changes and fixes since the beta1 release. According to internal testing, it works nice and stable, but we already received some external problem reports. So your feedback is very valuable, especially if you could test it in real world situations with logs and configurations we could never imagine ourselves. As threading is a major new feature, which is not enabled by default, please try it by adding “threaded(yes)” to your options in syslog-ng.conf Please download sources from http://www.balabit.com/downloads/files?path=/syslog-ng/sources/3.3.0beta2 ! For a list of changes and fixes check the announcment at http://lists.balabit.hu/pipermail/syslog-ng-announce/2011-August/000117.html If you use FreeBSD, ports is already updated to beta2: http://www.freshports.org/sysutils/syslog-ng3-devel/ Documentation is also available: http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guid… Windows in focus ---------------- The syslog-ng application is often used in a Windows environment. There is now an ongoing blog series about syslog clients for Windows. BalaBit also released some patterns for Windows. The patterns are available from http://czanik.blogs.balabit.com/2011/07/patterns-for-windows-server-2008/ Blogs about syslog clients: http://czanik.blogs.balabit.com/2011/07/eventlog-to-syslog/ http://czanik.blogs.balabit.com/2011/07/snare/ http://czanik.blogs.balabit.com/2011/07/some-more-applications-to-forward-w… To be continued... Performance ----------- Both syslog-ng OSE and PE are now multi threaded, which brings performance to extreme. On the test machine HDD and multiple gigabit Ethernet lines were the limiting factor, not syslog-ng. The test were done using PE, but OSE performance should be similar: http://pzolee.blogs.balabit.com/2011/07/do-you-want-to-process-800-000-mess… BalaBit call the features and technologies aimed compliance at a high performance HSRL (High Spead Reliable Logging). POLL Please take a minute to answer three syslog-ng related questions at https://www.surveymonkey.com/s/6ZQDVH6 OTHER SHORT NEWS * A few more words on CVE-2011-1951: http://bazsi.blogs.balabit.com/2011/07/on-cve-2011-1951-bug-or-security-iss… * Logs as a movie: http://algernon.blogs.balabit.com/2011/06/twelve-days-of-log/ * ELSA updated: http://ossectools.blogspot.com/2011/07/elsa-vmware-appliance-available.html NEW RELEASES * syslog-ng OSE 3.3 beta2 was released: http://lists.balabit.hu/pipermail/syslog-ng-announce/2011-August/000117.html ARCHIVE http://insider.blogs.balabit.com/ -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

internal() messages looping back on PostgreSQL restart
by Aaron 17 Aug '11

17 Aug '11

I have syslog-ng logging to a remote PostgreSQL database and to a local log file as well. If at any point the Postgres database restarts and syslog-ng happens to attempt to log to the database for the brief period that Postgres is unavailable, my local messages are flooded with internal() looping back messages similar to this: syslog-ng[31587]: syslog-ng internal() messages are looping back, preventing loop by suppressing further messages; recurse_count='3' syslog-ng[31587]: syslog-ng internal() messages are looping back, preventing loop by suppressing further messages; recurse_count='3' syslog-ng[31587]: syslog-ng internal() messages are looping back, preventing loop by suppressing further messages; recurse_count='2' My guess is that syslog-ng panics when it is unable to connect to the PostgreSQL database. The problem is that syslog-ng never re-establishes the connection, and writes several thousands of the above messages to the local log file each second, which can become a little problematic resource-wise. In order to make syslog-ng behave again, I need to manually restart the service, otherwise it would continue to flood my local messages and never reconnect to the database. I thought it might be a problem with my configuration, and I tried using the time_reopen() option but that seemed to have no effect. Is this perhaps a bug? Syslog-ng version: 3.2.4

2 1

[PATCH] afmongodb: Add stats locking.
by Gergely Nagy 17 Aug '11

17 Aug '11

Lock / unlock the stats when registering the counters. Reported-by: Costa Farber <costaf(a)wix.com> Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/afmongodb/afmongodb.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/modules/afmongodb/afmongodb.c b/modules/afmongodb/afmongodb.c index f82f10e..9e155ed 100644 --- a/modules/afmongodb/afmongodb.c +++ b/modules/afmongodb/afmongodb.c @@ -439,12 +439,14 @@ afmongodb_dd_init(LogPipe *s) self->queue = log_dest_driver_acquire_queue(&self->super, afmongodb_dd_format_persist_name(self)); + stats_lock(); stats_register_counter(0, SCS_MONGODB | SCS_DESTINATION, self->super.super.id, afmongodb_dd_format_stats_instance(self), SC_TYPE_STORED, &self->stored_messages); stats_register_counter(0, SCS_MONGODB | SCS_DESTINATION, self->super.super.id, afmongodb_dd_format_stats_instance(self), SC_TYPE_DROPPED, &self->dropped_messages); + stats_unlock(); log_queue_set_counters(self->queue, self->stored_messages, self->dropped_messages); afmongodb_dd_start_thread(self); -- 1.7.0.4

2 1

Frontend viewing logfiles
by Volker Lieder 17 Aug '11

17 Aug '11

Hi there, we have installed syslog-ng-pe with file encryption. Is there a webfrontend where we can search through the encrypted logfiles? e.g i have to set the cert-key-file in a config and then i can search the logfiles via webfrontend. Any ideas? Regards, Volker Lieder -- Mit freundlichen Grüßen/Best regards Volker Lieder System Administrator mail volker(a)its-lieder.de ITS-Lieder Kleinlindenerstrasse 3 35398 Giessen / Germany http://www.its-lieder.de UST-ID: DE260836372 ========================================================== Jegliche Stellungnahmen und Meinungen dieser E-Mail sind alleine die des Autors und nicht notwendigerweise die der Firma. Falls erforderlich, können Sie eine gesonderte schriftliche Bestätigung anfordern. Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the company. If verification is required please request a hard-copy version.

3 4

syslog-ng 3.3.0beta2 has been released
by devel＠balabit.hu 17 Aug '11

17 Aug '11

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.3.0beta2 SUMMARY : new stable release DATE : Aug 15, 2011 ------------------------------------------------------------------------------ DESCRIPTION: A new version of syslog-ng Open Source Edition (3.3.0beta2) has been released. CHANGES: 3.3.0beta2 Fri, 12 Aug 2011 09:00:17 +0200 Highlights ========== This release has started using the officially assigned port numbers for RFC5424 based log transports. syslog-ng will properly use the old defaults if the configuration @version indicates an old config. This release contains all identified bug fixes published by the BalaBit syslog-ng team in their syslog-ng PE 4.1 release, additional features will be integrated into syslog-ng 3.4 as 3.3 is already feature-frozen. See more details on the list of bugs fixed in the "bugfixes" section below. This release got a "beta" tag instead of a "release-candidate" because of the level of public testing it has received. However all currently known bugs are believed to be fixed and this should be much better than beta1 was. Bugfixes: ========= * Fixed a memory leak in macro based destination files. * Fixed the "dropped" and "stored" counters for the SQL destination, which were swapped previously. * Fixed file sources to properly start the source file from the beginning when restoring the persistent state failed. * Fixed BSD timestamp parsing, that caused an extra syslog header to be added to messages which had a single digit day-of-the-month (e.g. the first ten days for every month). * Fixed a possible crasher at exit, caused by the destinations being flushed and the associated sources already cleaned up. * Fixed a destination specific memory leak, which caused some hundred bytes to be leaked when the configuration was reloaded. * Set O_CLOEXEC for child program pipes, in order to avoid inheriting those to other children. * Allow RFC5424 structured data blocks to have no parameters (e.g. "[foo]", which only has an SDID and no SDPARAM). * Fixed the value of the $PRI macro which expanded to an empty string if the priority was 0 (= kern.emerg) * Fixed a race conditions in template compilation and expansion, that could cause segmentation faults and other nastiness. * Fixed a race condition in macro based file destinations, which caused crashes when multiple sources wrote into the same file, when the filename expanded to the same name from the multiple connections. (e.g. using $PROGRAM in the filename and receiving messages from different hosts running the same programs). * Fixed a possible segmentation fault in certificate validation, when the peer certificate contains an X.509 certificate purpose extension. * Try to load modules with the .a extension on AIX. * Fixed a possible crash in the socket source with optional(yes). * Made stats counters and their registration thread safe, which either caused crashes if stats_level() > 1 or miscalculated values due to parallel threads accessing the same counters. * Fixed a possible crash for unknown or mistyped source flags. * Fixed an fd and memory leak for opened destination connections. * Fixed possible crashes in SIGHUP handling in case syslog-ng is busy processing messages. * Fixed a failed assertion in persistent-state handling, triggered when multiple file sources are busy processing messages and recording their current file position. * Fixed SCL block expansion, when passing the empty value for a parameter. * The final flush that happens right before syslog-ng exits honoured the throttle() value. Since this is an attempt to squeeze out everything we have to the destination, it's best to ignore throttle in this case. * Fixed a possible use-after-free and other crash causing problems in file macro based destinations. * Fixed a massive memory leak of all processed messages, which were triggered by a change past v3.3.0beta1, but was found in a git snapshot. * Fixed keep-alive(yes) behaviour for destination drivers. * Messages generated by the db-parser() will be marked as local messages. * Implemented locking callbacks for OpenSSL, without which enabling SSL and threading mode almost guaranteed a crash in case multiple connections were running SSL. * Fixed an issue with the SQL destination, which can cause stalls in message processing. Build changes: ============== * Added --with-module-path configure option that specifies the default search path for modules. The help for the older --with-module-dir option was clarified, it indicates where syslog-ng Makefiles install modules. * The configure script automatically adds -pthread to the CC command line. Hopefully this is indeed supported on all platforms (worked on the ones I've tried), and avoids having to play dirty tricks with macros like _REENTRANT and _THREAD_SAFE. * Link unit test programs against -livykis in case we're using a system-installed ivykis, rather than our private copy. * Fixed compile issues on Solaris. * Updated to a newer libmongo-client (v0.1.3) * Updated to a newer ivykis version (v0.19 with syslog-ng specific patches) Performance improvements: ========================= * Use the GSlice allocator for objects allocated in the fast path. * Avoid another use of sprintf in the LogWriter fastpath. * Use per-thread caches for time related functions and the DNS cache. * Use the string representation of IP addresses in the DNS cache, as inet_ntop() performs much worse than the cache. Other changes: ============== * Functional tests: properly detect the presence of sqlite to do the sql() testing. * The minimum permissible value for the log-iw-size was changed to 100 from 10, as the per-connection window calculation has changed. Earlier all connections shared the same window, starting with 3.3, the window is distributed evenly to all connections, to avoid starving (e.g. one connection starves the rest by using up the window space) and to avoid thread synchronization in the message processing fast path. * Ported per-facility, per-severity, per-program and per-host counters from the PE version. * Other systemd unit files. Credits: ======== syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: * Dave Reisner (ArchLinux) * Nix (esperi.org.uk) * Hendrik Visage * Viacheslav Biriukov * Andras Mitzki (BalaBit) * Attila Nagy (BalaBit) * Gyorgy Fischhof (BalaBit) * Viktor Juhasz (BalaBit) * Balazs Scheidler (BalaBit) * Peter Czanik (BalaBit) * Gergely Nagy (BalaBit) * Peter Gyorko (BalaBit) * Tamas Pal (BalaBit) * Zoltan Pallagi (BalaBit) * Robert Fekete (BalaBit) DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

3 2

Symantec SCSP & Syslog-ng
by Hendrik Visage 16 Aug '11

16 Aug '11

Hi there, Anybody else using this combination? (Perhaps even on HP/UX??) Hendrik

1 0

[Bug 130] New: Changes to resolv.conf do not apply to DNS resolution
by bugzilla＠bugzilla.balabit.com 15 Aug '11

15 Aug '11

https://bugzilla.balabit.com/show_bug.cgi?id=130 Summary: Changes to resolv.conf do not apply to DNS resolution Product: syslog-ng Version: 3.0.x Platform: All OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: Kevin.Guthrie(a)itron.com Type of the Report: bug Estimated Hours: 0.0 This was seen using 3.0.1 on an ARM processor running Linux 2.6.32. It is linked against glibc 2.11. The unit is an embedded device to run independently. It contains a modem to handle communication with head end systems. The configuration was set up to to use a fully qualified domain name to a UDP port. The unit was booted up originally with an empty /etc/resolv.conf file. The syslog-ng daemon was started up prior to the modem establishing a connection. The resolv.conf file is not populated at the time that the syslog-ng daemon is initialized. Errors in the local log show the DNS failure, as expected. However, after the modem connection is established and the resolv.conf file is updated, syslog-ng continues to fail DNS resolution. Attempts to use SIGHUP against the process do not correct the issue. Only killing and restarting the daemon corrects the issue. That should not be necessary. Based off information found at http://sources.redhat.com/bugzilla/show_bug.cgi?id=3675, syslog-ng needs to call the res_init function in order to allow DNS resolution to succeed. This could be limited to the SIGHUP handler or prior to any attempt to perform DNS resolution. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

mongodb problem
by Peter Czanik 14 Aug '11

14 Aug '11

Hello, My 'make dist' problem is fixed ( http://git.balabit.hu/?p=bazsi/syslog-ng-3.3.git;a=commit;h=d01c8bdd1957d4d… ). Running 'make' works, but once I run 'make inst' I run into the following problem: gmake[4]: Entering directory `/usr/ports/sysutils/syslog-ng3-devel/work/syslog-ng-3.3.0beta1/modules/afmongodb' Makefile:798: warning: overriding recipe for target `check' Makefile:648: warning: ignoring old recipe for target `check' gmake[4]: Nothing to be done for `install-exec-am'. test -z "/usr/local/lib/syslog-ng" || ../.././install-sh -c -d "/usr/local/lib/syslog-ng" /bin/sh ../../libtool --mode=install /usr/bin/install -c -o root -g wheel libafmongodb.la '/usr/local/lib/syslog-ng' libtool: install: warning: relinking `libafmongodb.la' libtool: install: (cd /usr/ports/sysutils/syslog-ng3-devel/work/syslog-ng-3.3.0beta1/modules/afmongodb; /bin/sh /usr/ports/sysutils/syslog-ng3-devel/work/syslog-ng-3.3.0beta1/libtool --tag CC --mode=relink cc -std=gnu99 -I./libmongo-client/src -I/usr/local/include/libnet115 -I/usr/local/include -O2 -pipe -fno-strict-aliasing -Wall -pthread -avoid-version -module -no-undefined -rpath=/usr/local/lib -o libafmongodb.la -rpath /usr/local/lib/syslog-ng libafmongodb_la-afmongodb-grammar.lo libafmongodb_la-afmongodb.lo libafmongodb_la-afmongodb-parser.lo ../../lib/libsyslog-ng.la -L./libmongo-client/src -lmongo-client ) libtool: relink: cc -std=gnu99 -shared -fPIC -DPIC .libs/libafmongodb_la-afmongodb-grammar.o .libs/libafmongodb_la-afmongodb.o .libs/libafmongodb_la-afmongodb-parser.o -Wl,-rpath -Wl,/usr/local/lib -L/usr/ports/sysutils/syslog-ng3-devel/work/syslog-ng-3.3.0beta1/lib/ivykis/lib/.libs -L/usr/local/lib -lsyslog-ng -L../lib/ivykis/lib -L../lib/ivykis/modules -lrt -lgmodule-2.0 -lgthread-2.0 -levtlog /usr/ports/sysutils/syslog-ng3-devel/work/syslog-ng-3.3.0beta1/lib/ivykis/lib/.libs/libivykis.a -lpthread -L./libmongo-client/src -lmongo-client -lglib-2.0 -lintl -liconv -lpcre -lcrypto -lssl -O2 -pthread -pthread -Wl,-soname -Wl,libafmongodb.so -o .libs/libafmongodb.so /usr/bin/ld: cannot find -lmongo-client libtool: install: error: relink `libafmongodb.la' with the above command before installing it gmake[4]: *** [install-moduleLTLIBRARIES] Error 1 gmake[4]: Leaving directory `/usr/ports/sysutils/syslog-ng3-devel/work/syslog-ng-3.3.0beta1/modules/afmongodb' gmake[3]: *** [install-am] Error 2 gmake[3]: Leaving directory `/usr/ports/sysutils/syslog-ng3-devel/work/syslog-ng-3.3.0beta1/modules/afmongodb' gmake[2]: *** [install] Error 2 gmake[2]: Leaving directory `/usr/ports/sysutils/syslog-ng3-devel/work/syslog-ng-3.3.0beta1/modules/afmongodb' gmake[1]: *** [install-recursive] Error 1 gmake[1]: Leaving directory `/usr/ports/sysutils/syslog-ng3-devel/work/syslog-ng-3.3.0beta1/modules' gmake: *** [install-recursive] Error 1 *** Error code 2 It is from the latest git sources on FreeBSD 8.1. Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

3 7

syslog-ng 3.1.2-1~lucid1 not writing to daemon.log, syslog, messages, etc.
by Mike Gracy 14 Aug '11

14 Aug '11

Using stock config, but added a few lines to tail some files and push them to a splunk server. It won't log the system log messages, but it pushes the additional logs out that I added (s_nginx_*). I commented out my config and looking at the debug output, it looks like it's not matching anything in the base/default config. # First, set some global options. options { long_hostnames(off); flush_lines(0); use_dns(no); use_fqdn(no); owner("root"); group("adm"); perm(0640); stats_freq(0); bad_hostname("^gconfd$"); }; ######################## # Sources ######################## # This is the default behavior of sysklogd package # Logs may come from unix stream, but not from another machine. # source s_src { unix-dgram("/dev/log"); internal(); file("/proc/kmsg" program_override("kernel")); }; # If you wish to get logs from remote machine you should uncomment # this and comment the above source line. # #source s_net { tcp(ip(127.0.0.1) port(1000) authentication(required) encrypt(allow)); }; # Additional sources to monitor destination d_remote { udp("10.170.230.60" , port(4957)); }; source s_nginx_20 { file ("/iw-deploy/log/access-client-internal.log" follow_freq(1)); }; source s_nginx_21 { file ("/iw-deploy/log/access-client.log" follow_freq(1)); }; source s_nginx_22 { file ("/iw-deploy/log/access-server.log" follow_freq(1)); }; source s_nginx_23 { file ("/iw-deploy/log/celeryd.log" follow_freq(1)); }; source s_nginx_24 { file ("/iw-deploy/log/error.log" follow_freq(1)); }; source s_nginx_25 { file ("/iw-deploy/log/installer.log" follow_freq(1)); }; source s_nginx_26 { file ("/iw-deploy/log/island_gen.log" follow_freq(1)); }; source s_nginx_27 { file ("/iw-deploy/log/persistence.log" follow_freq(1)); }; source s_nginx_28 { file ("/iw-deploy/log/sec.log" follow_freq(1)); }; source s_nginx_29 { file ("/iw-deploy/log/worship1.log" follow_freq(1)); }; source s_nginx_30 { file ("/iw-deploy/log/worship2.log" follow_freq(1)); }; source s_nginx_31 { file ("/iw-deploy/log/worship_ff1.log" follow_freq(1)); }; source s_nginx_32 { file ("/iw-deploy/log/worship_router.log" follow_freq(1)); }; source s_nginx_33 { file ("/iw-deploy/log/persistence.err" follow_freq(1) ); }; source s_nginx_34 { file ("/iw-deploy/log/client-errors.log" follow_freq(1) ); }; log { source(s_nginx_20); destination(d_remote); }; log { source(s_nginx_21); destination(d_remote); }; log { source(s_nginx_22); destination(d_remote); }; log { source(s_nginx_23); destination(d_remote); }; log { source(s_nginx_24); destination(d_remote); }; log { source(s_nginx_25); destination(d_remote); }; log { source(s_nginx_26); destination(d_remote); }; log { source(s_nginx_27); destination(d_remote); }; log { source(s_nginx_28); destination(d_remote); }; log { source(s_nginx_29); destination(d_remote); }; log { source(s_nginx_30); destination(d_remote); }; log { source(s_nginx_31); destination(d_remote); }; log { source(s_nginx_32); destination(d_remote); }; log { source(s_nginx_33); destination(d_remote); }; log { source(s_nginx_34); destination(d_remote); }; ###### # destinations # some standard log files destination df_auth { file("/var/log/auth.log"); }; destination df_syslog { file("/var/log/syslog"); }; destination df_cron { file("/var/log/cron.log"); }; destination df_daemon { file("/var/log/daemon.log"); }; destination df_kern { file("/var/log/kern.log"); }; destination df_lpr { file("/var/log/lpr.log"); }; destination df_mail { file("/var/log/mail.log"); }; destination df_user { file("/var/log/user.log"); }; destination df_uucp { file("/var/log/uucp.log"); }; ###### # destinations # some standard log files destination df_auth { file("/var/log/auth.log"); }; destination df_syslog { file("/var/log/syslog"); }; destination df_cron { file("/var/log/cron.log"); }; destination df_daemon { file("/var/log/daemon.log"); }; destination df_kern { file("/var/log/kern.log"); }; destination df_lpr { file("/var/log/lpr.log"); }; destination df_mail { file("/var/log/mail.log"); }; destination df_user { file("/var/log/user.log"); }; destination df_uucp { file("/var/log/uucp.log"); }; # these files are meant for the mail system log files # and provide re-usable destinations for {mail,cron,...}.info, # {mail,cron,...}.notice, etc. destination df_facility_dot_info { file("/var/log/$FACILITY.info"); }; destination df_facility_dot_notice { file("/var/log/$FACILITY.notice"); }; destination df_facility_dot_warn { file("/var/log/$FACILITY.warn"); }; destination df_facility_dot_err { file("/var/log/$FACILITY.err"); }; destination df_facility_dot_crit { file("/var/log/$FACILITY.crit"); }; # these files are meant for the news system, and are kept separated # because they should be owned by "news" instead of "root" destination df_news_dot_notice { file("/var/log/news/news.notice" owner("news")); }; destination df_news_dot_err { file("/var/log/news/news.err" owner("news")); }; destination df_news_dot_crit { file("/var/log/news/news.crit" owner("news")); }; # some more classical and useful files found in standard syslog configurations destination df_debug { file("/var/log/debug"); }; destination df_messages { file("/var/log/messages"); }; # pipes # a console to view log messages under X destination dp_xconsole { pipe("/dev/xconsole"); }; # consoles # this will send messages to everyone logged in destination du_all { usertty("*"); }; ###### # filters # all messages from the auth and authpriv facilities filter f_auth { facility(auth, authpriv); }; # all messages except from the auth and authpriv facilities filter f_syslog { not facility(auth, authpriv); }; # respectively: messages from the cron, daemon, kern, lpr, mail, news, user, # and uucp facilities filter f_cron { facility(cron); }; filter f_daemon { facility(daemon); }; filter f_kern { facility(kern); }; filter f_lpr { facility(lpr); }; filter f_mail { facility(mail); }; filter f_news { facility(news); }; filter f_user { facility(user); }; filter f_uucp { facility(uucp); }; # some filters to select messages of priority greater or equal to info, warn, # and err # (equivalents of syslogd's *.info, *.warn, and *.err) filter f_at_least_info { level(info..emerg); }; filter f_at_least_notice { level(notice..emerg); }; filter f_at_least_warn { level(warn..emerg); }; filter f_at_least_err { level(err..emerg); }; filter f_at_least_crit { level(crit..emerg); }; # all messages of priority debug not coming from the auth, authpriv, news, and # mail facilities filter f_debug { level(debug) and not facility(auth, authpriv, news, mail); }; # all messages of info, notice, or warn priority not coming form the auth, # authpriv, cron, daemon, mail, and news facilities filter f_messages { level(info,notice,warn) and not facility(auth,authpriv,cron,daemon,mail,news); }; # messages with priority emerg filter f_emerg { level(emerg); }; # complex filter for messages usually sent to the xconsole filter f_xconsole { facility(daemon,mail) or level(debug,info,notice,warn) or (facility(news) and level(crit,err,notice)); }; ###### # logs # order matters if you use "flags(final);" to mark the end of processing in a # "log" statement # these rules provide the same behavior as the commented original syslogd rules # auth,authpriv.* /var/log/auth.log log { source(s_src); filter(f_auth); destination(df_auth); }; # *.*;auth,authpriv.none -/var/log/syslog log { source(s_src); filter(f_syslog); destination(df_syslog); }; # this is commented out in the default syslog.conf # cron.* /var/log/cron.log #log { # source(s_src); # filter(f_cron); # destination(df_cron); #}; # daemon.* -/var/log/daemon.log log { source(s_src); filter(f_daemon); destination(df_daemon); }; # kern.* -/var/log/kern.log log { source(s_src); filter(f_kern); destination(df_kern); }; # lpr.* -/var/log/lpr.log log { source(s_src); filter(f_lpr); destination(df_lpr); }; # mail.* -/var/log/mail.log log { source(s_src); filter(f_mail); destination(df_mail); }; # user.* -/var/log/user.log log { source(s_src); filter(f_user); destination(df_user); }; # uucp.* /var/log/uucp.log log { source(s_src); filter(f_uucp); destination(df_uucp); }; # mail.info -/var/log/mail.info log { source(s_src); filter(f_mail); filter(f_at_least_info); destination(df_facility_dot_info); }; # mail.warn -/var/log/mail.warn log { source(s_src); filter(f_mail); filter(f_at_least_warn); destination(df_facility_dot_warn); }; # mail.err /var/log/mail.err log { source(s_src); filter(f_mail); filter(f_at_least_err); destination(df_facility_dot_err); }; # news.crit /var/log/news/news.crit log { source(s_src); filter(f_news); filter(f_at_least_crit); destination(df_news_dot_crit); }; # news.err /var/log/news/news.err log { source(s_src); filter(f_news); filter(f_at_least_err); destination(df_news_dot_err); }; # news.notice /var/log/news/news.notice log { source(s_src); filter(f_news); filter(f_at_least_notice); destination(df_news_dot_notice); }; # *.=debug;\ # auth,authpriv.none;\ # news.none;mail.none -/var/log/debug log { source(s_src); filter(f_debug); destination(df_debug); }; # *.=info;*.=notice;*.=warn;\ # auth,authpriv.none;\ # cron,daemon.none;\ # mail,news.none -/var/log/messages log { source(s_src); filter(f_messages); destination(df_messages); }; # *.emerg * log { source(s_src); filter(f_emerg); destination(du_all); }; # daemon.*;mail.*;\ # news.crit;news.err;news.notice;\ # *.=debug;*.=info;\ # *.=notice;*.=warn |/dev/xconsole log { source(s_src); filter(f_xconsole); destination(dp_xconsole); };

2 1

Struggling with regexp, store-matches and csv-parser macros
by Remy Zandwijk 14 Aug '11

14 Aug '11

Hi list, so, I am new to syslog-ng and I am struggling with the features regexp/store-matches and csv-parser macros. I searched the web and mailinglist archive for these two subjects, but that didn't answer my questions. The syslog-ng 3.1.3 I'm using is running on Debian Squeeze. filter f_mytest { facility(local0) and level(debug) and match("^www" value("MSGHDR") flags("store-matches")); }; template t_mytest { template("${MSG}--${1}\n"); } destination d_mytest { file("/var/tmp/mytest.log"); }; log { source(s_src); filter(f_mytest); template(t_mytest); destination(f_mytest); }; When I send a message to syslog-ng with: logger -p local0.debug -t www testmessage nothing appears in the logfile. However, when flags("store-matches") is omitted, the message appears like expected. Why does this filter not work? I am playing around with the csv-parser function as well. The (user-defined) macros can be used in file() functions, but I found out that they don't work in owner() and group() functions. Is that expected? Thanks, Remy

2 2

How to manage the Syslog-NG messages coming from different sites? Each sites has their Syslog-NG clients?
by Marcos Tang 12 Aug '11

12 Aug '11

Hi, I have a problem to manage the Syslog-NG messages sending from different Syslog-NG clients at different remote sites. For example, I have 10 sites (site1, site2, site3 to site 10) running Syslog-NG clients. Each site has one Syslog-NG server and all the Syslog messages will be forwarded to the centralized Syslog-NG server and insert into the MySQL database at the headquarter finally. Now, I can see all the Syslog messages at the headquarter MySQL database; but it is hard for me to manage them. For example, if I know the hostname of a particular host, I can query the MySQL database to search for the Syslog related to that host. However, if I want to know the Syslog messages coming from a particular site such as "site1", how can I do that? There is no "site1" information inside the Syslog messages. I am thinking if I can modify the Syslog-NG configuration file such that I can add some information such as "site1" and make it as part of the Syslog messages, I can query the MySQL database for "site1" pattern. But I am not sure if I can do that or not. If you have any suggestions/ideas, please let me know that. Regards, Marcos

6 6

[PATCH] afsql: Fix sql stuck problema
by Juhasz Viktor 11 Aug '11

11 Aug '11

[sql] send a signal to the database thread only in case when the queue is syncronised, because in other cases the database thread wake up unnecessary many times in case of threaded mode And the sql stucked if the explicit commit is turned on and flush_lines and log_iw_size and fetch_limit and log_fifo_size have the same value. The reason of the stuck was that the queue become full but the database thread can't pop any messages form the queue. So it can be occurred that the reader finished the io_job and suspended and the database thread wait for the signal with 0 read message. Signed-off-by: Viktor Juhasz <jviktor(a)balabit.hu> diff --git a/lib/logqueue.c b/lib/logqueue.c index 29c39e4..43ed69e 100644 --- a/lib/logqueue.c +++ b/lib/logqueue.c @@ -55,6 +55,17 @@ log_queue_reset_parallel_push(LogQueue *self) g_static_mutex_unlock(&self->lock); } +void +log_queue_set_parallel_push(LogQueue *self, gint notify_limit, LogQueuePushNotifyFunc parallel_push_notify, gpointer user_data, GDestroyNotify user_data_destroy) +{ + g_static_mutex_lock(&self->lock); + self->parallel_push_notify = parallel_push_notify; + self->parallel_push_data = user_data; + self->parallel_push_notify_limit = notify_limit; + self->parallel_push_data_destroy = user_data_destroy; + g_static_mutex_unlock(&self->lock); +} + /* * * @batch_items: the number of items processed in a batch (e.g. the number of items the consumer is preferred to process at a single invocation) diff --git a/lib/logqueue.h b/lib/logqueue.h index 1d86f3f..ef04c99 100644 --- a/lib/logqueue.h +++ b/lib/logqueue.h @@ -139,6 +139,7 @@ log_queue_set_throttle(LogQueue *self, gint throttle) void log_queue_push_notify(LogQueue *self); void log_queue_reset_parallel_push(LogQueue *self); +void log_queue_set_parallel_push(LogQueue *self, gint notify_limit, LogQueuePushNotifyFunc parallel_push_notify, gpointer user_data, GDestroyNotify user_data_destroy); gboolean log_queue_check_items(LogQueue *self, gint batch_items, gboolean *partial_batch, gint *timeout, LogQueuePushNotifyFunc parallel_push_notify, gpointer user_data, GDestroyNotify user_data_destroy); void log_queue_set_counters(LogQueue *self, StatsCounterItem *stored_messages, StatsCounterItem *dropped_messages); void log_queue_init_instance(LogQueue *self, const gchar *persist_name); diff --git a/modules/afsql/afsql.c b/modules/afsql/afsql.c index 039270d..bbdd1d2 100644 --- a/modules/afsql/afsql.c +++ b/modules/afsql/afsql.c @@ -116,6 +116,7 @@ typedef struct _AFSqlDestDriver dbi_conn dbi_ctx; GHashTable *validated_tables; guint32 failed_message_counter; + } AFSqlDestDriver; static gboolean dbi_initialized = FALSE; @@ -685,11 +686,12 @@ afsql_dd_insert_db(AFSqlDestDriver *self) success = log_queue_pop_head(self->queue, &msg, &path_options, (self->flags & AFSQL_DDF_EXPLICIT_COMMITS), FALSE); g_mutex_unlock(self->db_thread_mutex); if (!success) - return TRUE; + { + return TRUE; + } } msg_set_context(msg); - table = g_string_sized_new(32); value = g_string_sized_new(256); query_string = g_string_sized_new(512); @@ -767,7 +769,6 @@ afsql_dd_insert_db(AFSqlDestDriver *self) if (success && self->flush_lines_queued != -1) { self->flush_lines_queued++; - if (self->flush_lines && self->flush_lines_queued == self->flush_lines && !afsql_dd_commit_txn(self, TRUE)) return FALSE; } @@ -1155,6 +1156,16 @@ afsql_dd_deinit(LogPipe *s) } static void +afsql_dd_queue_notify(gpointer user_data) +{ + AFSqlDestDriver *self = (AFSqlDestDriver *) user_data; + g_mutex_lock(self->db_thread_mutex); + g_cond_signal(self->db_thread_wakeup_cond); + log_queue_reset_parallel_push(self->queue); + g_mutex_unlock(self->db_thread_mutex); +} + +static void afsql_dd_queue(LogPipe *s, LogMessage *msg, const LogPathOptions *path_options, gpointer user_data) { AFSqlDestDriver *self = (AFSqlDestDriver *) s; @@ -1165,7 +1176,9 @@ afsql_dd_queue(LogPipe *s, LogMessage *msg, const LogPathOptions *path_options, log_queue_push_tail(self->queue, msg, path_options); if (queue_was_empty && !self->db_thread_suspended) - g_cond_signal(self->db_thread_wakeup_cond); + { + log_queue_set_parallel_push(self->queue, 1, afsql_dd_queue_notify, self, NULL); + } g_mutex_unlock(self->db_thread_mutex); }

2 2

syslog-ng 3.3 git snapshot
by Peter Czanik 10 Aug '11

10 Aug '11

Hello, There were many fixes added since the last 3.3 beta1 release. These are currently available only from git. I just learned, that accessing git is troublesome for those behind strict firewalls. So I prepared a tgz from the latest git snapshot with 'make dist'. It is available at http://people.balabit.hu/czanik/syslog-ng-3.3.0beta1.tar.gz (don't be confused by the 'beta1' suffix, it's from this mornings git checkout) Known problem: mongodb support is broken, when using the output of 'make dist'. Please use the --disable-mongodb configure option to disable mongodb for now. I'll try to periodically update this snapshot until an 'official' release is available. Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

3 2

syslog-ng event capability
by curious curious 08 Aug '11

08 Aug '11

Hi, What are the capabilities , limits of syslog-ng daemon? When does it fail to send the logs to the destination? How many events/logs per second can it handle without dropping anything? Regards

2 1