May 2012 - syslog-ng - lists.balabit.hu

[PATCH] afsocket: Fix unix-dgram initialisation
by Gergely Nagy 04 May '12

04 May '12

When initializing an unix-dgram socket, set self->fd to the acquired socket FD, so that fds inherited from systemd actually get used and polled on. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/afsocket/afsocket.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/modules/afsocket/afsocket.c b/modules/afsocket/afsocket.c index 5b192f4..c1bee5c 100644 --- a/modules/afsocket/afsocket.c +++ b/modules/afsocket/afsocket.c @@ -679,7 +679,7 @@ afsocket_sd_init(LogPipe *s) if (sock == -1 && !afsocket_open_socket(self->bind_addr, !!(self->flags & AFSOCKET_STREAM), &sock)) return self->super.super.optional; } - self->fd = -1; + self->fd = sock; if (!self->setup_socket(self, sock)) { -- 1.7.9.1

2 3

[PATCH] afsmtp: Fix afsmtp linking when configured with --with-libesmtp
by Gergely Nagy 04 May '12

04 May '12

In case configure received a --with-libesmtp=/path option, we forgot to add -lesmtp to LIBESMTP_LIBS, which resulted in afsmtp getting compiled, but not linked to libesmtp, which is obviously not desirable. This patch fixes that, and adds the -lesmtp. Reported-by: Peter Czanik <czanik(a)balabit.hu> Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- configure.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/configure.in b/configure.in index 2d56bbd..03cc76c 100644 --- a/configure.in +++ b/configure.in @@ -815,7 +815,7 @@ if test "x$enable_smtp" != "xno" && test "x$with_libesmtp" != "no"; then CPPFLAGS="$CPPFLAGS -I$with_libesmtp/include" LDFLAGS="$LDFLAGS -L$with_libesmtp/lib" AC_CHECK_HEADER(libesmtp.h, [LIBESMTP_CFLAGS="-I$with_libesmtp/include" - LIBESMTP_LIBS="-L$with_libesmtp/lib"], [libesmtp=no]) + LIBESMTP_LIBS="-L$with_libesmtp/lib -lesmtp"], [libesmtp=no]) CPPFLAGS="$CPPFLAGS_SAVE" LDFLAGS="$LDFLAGS_SAVE" else -- 1.7.10

1 0

[PATCH] lib/gprocess.c: Ignore SIGHUP inside the supervisor.
by Gergely Nagy 04 May '12

04 May '12

Reloading syslog-ng's config is often done via a killall -HUP syslog-ng command, which also sends a signal to the supervisor. However, the supervisor did not handle said signal, thus it exited as is the default for it. To work around this issue, lets ignore SIGHUP inside the supervisor. Reported-by: Patrick Hemmer Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- lib/gprocess.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/lib/gprocess.c b/lib/gprocess.c index aaefbab..d68521d 100644 --- a/lib/gprocess.c +++ b/lib/gprocess.c @@ -1055,10 +1055,15 @@ g_process_perform_supervise(void) pid_t pid; gboolean first = TRUE, exited = FALSE; gchar proc_title[PROC_TITLE_SPACE]; + struct sigaction sa; g_snprintf(proc_title, PROC_TITLE_SPACE, "supervising %s", process_opts.name); g_process_setproctitle(proc_title); + memset(&sa, 0, sizeof(sa)); + sa.sa_handler = SIG_IGN; + sigaction(SIGHUP, &sa, NULL); + while (1) { if (pipe(init_result_pipe) != 0) -- 1.7.10

1 0

syslog-ng Insider - May 2012
by Peter Czanik 03 May '12

03 May '12

Dear syslog-ng users, This is the 13th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com <mailto:documentation@balabit.com> FEATURED NEWS Have you tried to add custom information to log messages, fix mis-formatted logs or anonymize logs? --------------------------------- The next long-time-supported release of SSB version 3 LTS is about to be released. This release includes a switch to 64-bit architecture, a huge performance improvement in the indexing/searching feature for a large number of message and search patterns and a couple of new features, too. The following post plans to introduce those new features to you. The updated User’s Manual will contain a detailed description of them — this post is written more to serve as teaser and to highlight some of the use cases we had in mind when we’d planned the features, and, of course, to ask for your feedback about them. http://gyp.blogs.balabit.com/2012/03/new-features-in-ssb-3-lts-rewriting-pa… Admin guide: http://www.balabit.com/sites/default/files/documents/ssb-3.0-guides/ssb-v3.… syslog-ng participates in GSoC ------------------------------ This year syslog-ng participates in GSoC under the umbrella of openSUSE. We have one student accepted, who will work on syslog-ng’s mongodb destination. The story of BalaBit's GSoC participation: http://czanik.blogs.balabit.com/2012/04/syslog-ng-on-gsoc-thanks-opensuse/ What the student will do: http://algernon.blogs.balabit.com/2012/05/in-the-land-of-open-source-the-fr… MongoDB howto ------------- One of the major reasons to update to 3.3 other than threading is MongoDB. It allows great flexibility when using patterns and storing parsed data from logs. The syslog-ng documentation covers most information necessary to use MongoDB, but this HowTo compiles all these into a single document and extends it with features from the upcoming syslog-ng 3.4 version. http://asylum.madhouse-project.org/blog/2012/04/26/mongodb-howto/ OTHER SHORT NEWS * There is now an open source LogStore reader work in progress: https://github.com/algernon/balabit.logstore * syslog-ng 3.4 alpha packages are now available in the FreeBSD ports and also for openSUSE: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… * syslog-ng and ELSA were presented at LOADays: http://czanik.blogs.balabit.com/2012/04/syslog-ng-at-linux-open-administrat… NEW RELEASES: * syslog-ng OSE 3.3.5: https://lists.balabit.hu/pipermail/syslog-ng-announce/2012-May/000142.html * syslog-ng OSE 3.4 alpha2: https://lists.balabit.hu/pipermail/syslog-ng-announce/2012-May/000143.html * libumberlog 0.2.1: http://algernon.github.com/libumberlog/ ARCHIVE http://insider.blogs.balabit.com/ -- Peter Czanik (CzP)<czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

smtp on FreeBSD
by Peter Czanik 02 May '12

02 May '12

Hello, I tried to compile syslog-ng 3.4 alpha git on FreeBSD 8.1 with smtp support, but it does not start: fb81# /usr/local/etc/rc.d/syslog-ng start Starting syslog_ng. Error opening plugin module; module='afsmtp', error='/usr/local/lib/syslog-ng/libafsmtp.so: Undefined symbol "_smtp_message_str_cb"' smtp support can only be disabled by removing libesmtp from the system... There is also a segfault now for each incoming log (nothing appears in log files), so I'm checking if it's related to any of the enabled modules. At first glance it seems to be something FreeBSD specific, I have all modules enabled on openSUSE, and logging seems to work (at least the basics, I checked). Bye, -- Peter Czanik (CzP)<czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

4 18

3.4 packages
by Peter Czanik 02 May '12

02 May '12

Hello, I created packages from current 3.4 git for openSUSE and helped to update the FreeBSD syslog-ng-devel port. They had some light testing and seem to work. Please refer to http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… for download information! The openSUSE packages have smtp enabled (in a separate sub package, untested), on FreeBSD the module can't be loaded, so marked as broken. Please let me know, if you run into any problems! Bye, -- Peter Czanik (CzP)<czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

2 2

[RFC]: LogThreadedDestDriver - making threaded destinations better
by Gergely Nagy 01 May '12

01 May '12

Hi! There have been a number of fixes going into afsql or afmongodb (or both) recently, fixing bugs that affected both, for the simple reason that when I made afmongodb, I copied parts of the architecture from afsql. Now there is the SMTP destination aswell, suffering from the same issues, and a Redis destination is in the works aswell. Therefore, it is about time to rethink how these drivers that all look very similar under the hood could be rebased onto a common foundation, something that abstracts away all the common stuff, such as handling the worker thread, and the queue method - so all the affected drivers need to do, is implement their own tasks, and skip all the copy-paste-ish boilerplate they have now. To this end, we had a short brainstorming with Bazsi, and I'll try to summarize what I remember (with a few later ideas added). The goal is to create a new class, based on LogDestDriver, for all those destination drivers that use a worker thread to do their blocking operations. This new class, which for the time being shall be named LogThreadedDestDriver shall implement the following things: * It must handle the setup, starting and stopping of the worker thread, with a module-supplied worker function, and that function shall be assisted by class functions to do the dirty work. * The queue handling must be safe and free of races (which pretty much means we have to find a way to wrap log_queue_pop_head into a synchronous call) * The class must be prepared to work not only with single messages, but in bulk aswell (think SQL's transactions, or the upcoming MongoDB bulk insert) * The class must implement its own queue method, downstream modules should not need to care about that. * We must be able to wake up the worker thread, even if there is nothing to handle: this is needed for a clean shutdown. And that's pretty much it. The class would have callbacks for the following things: * A pair of callbacks to call during the driver's _init() and _deinit() functions. These can either be implemented as callbacks, and the driver will then use it's own init/deinit and set it in its parent, and call the module-set callbacks from those. Or, the class can provide _init/_deinit methods, and leave it up to the module to implement its own methods, and call the class methods itself. The former makes drivers that do not require extra init/deinit simpler, the latter has less callbacks and looks neater. When implementing, I'll probably go with the latter. * An 'do-the-work' callback, which is called whenever work needs to be performed, and an error callback to fire whenever the do-the-work failed. I'm not entirely sure how best to handle the transaction case here, but that's something that needs to be done around here. One possibility is to allow two kinds of do-the-work callbacks: one for the simple case where no transactions are done (afsmtp and current afmongodb) and one that supports transactions/bulk inserts (afsql and future afmongodb and redis). Off the top of my head, this should be enough to move the boring parts of the current threaded destination drivers into a common class. Once the transaction case is figured out, all of our drivers can be reimplemented on top of this class, reducing copied and slightly altered code by a HUGE margin, and finally making it so that we have a single place to fix possible bugs in, instead of 3 slightly different copies of the same thing. As a side effect, time_reopen and stored/dropped counter handling would get moved from individual drivers into this base class, along with flush_timeout and flush_lines (if those still exist in 3.4) and a couple of other things. I plan to have a Proof of Concept patch ready by friday night, we'll see how it goes! (And shortly after, a redis destination will follow) -- |8]

2 2

Multi-core use in threaded mode
by Martin Holste 01 May '12

01 May '12

I'm seeing that on a very busy syslog-ng 3.3.4 box, CPU never rises above 103% and log loss occurs. Does threaded mode not allow separate threads to utilize multiple cores? I need to scale above a single CPU.

6 8

[PATCH (3.4) 0/3] afsql code cleanups
by Gergely Nagy 01 May '12

01 May '12

The patches following this mail are my attempts at cleaning up the afsql driver here and there, mostly afsql_dd_insert_db(), in preparation for moving towards LogThreadedDestDriver (which will be posted separately a bit later). The migration is possible without these patches, but these are worth applying even without the LogThreadedDestDriver patches, as these present no real architectural change, just a little refactoring. I think the code could be cleaned up further, but for the time being, I only did what I thought necessary to prepare for migration to LogThreadedDestDriver.

2 5

[PATCH (3.4)] afsmtp: Add SCS_SMTP to lib/stats.h
by Gergely Nagy 01 May '12

01 May '12

Add SCS_SMTP to lib/stats.h, so we get proper smtp stats too. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- lib/stats.h | 1 + modules/afsmtp/afsmtp.c | 4 ---- 2 files changed, 1 insertion(+), 4 deletions(-) diff --git a/lib/stats.h b/lib/stats.h index 99936da..9174408 100644 --- a/lib/stats.h +++ b/lib/stats.h @@ -70,6 +70,7 @@ enum SCS_SEVERITY = 24, SCS_FACILITY = 25, SCS_SENDER = 26, + SCS_SMTP = 27, SCS_MAX, SCS_SOURCE_MASK = 0xff }; diff --git a/modules/afsmtp/afsmtp.c b/modules/afsmtp/afsmtp.c index bd8adfe..841100d 100644 --- a/modules/afsmtp/afsmtp.c +++ b/modules/afsmtp/afsmtp.c @@ -33,10 +33,6 @@ #include <libesmtp.h> -#ifndef SCS_SMTP -#define SCS_SMTP 0 -#endif - typedef struct { gchar *name; -- 1.7.10

2 1