April 2012 - syslog-ng - lists.balabit.hu

[PATCH] afsocket: Fix unix-dgram initialisation
by Gergely Nagy 04 May '12

04 May '12

When initializing an unix-dgram socket, set self->fd to the acquired socket FD, so that fds inherited from systemd actually get used and polled on. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/afsocket/afsocket.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/modules/afsocket/afsocket.c b/modules/afsocket/afsocket.c index 5b192f4..c1bee5c 100644 --- a/modules/afsocket/afsocket.c +++ b/modules/afsocket/afsocket.c @@ -679,7 +679,7 @@ afsocket_sd_init(LogPipe *s) if (sock == -1 && !afsocket_open_socket(self->bind_addr, !!(self->flags & AFSOCKET_STREAM), &sock)) return self->super.super.optional; } - self->fd = -1; + self->fd = sock; if (!self->setup_socket(self, sock)) { -- 1.7.9.1

2 3

smtp on FreeBSD
by Peter Czanik 02 May '12

02 May '12

Hello, I tried to compile syslog-ng 3.4 alpha git on FreeBSD 8.1 with smtp support, but it does not start: fb81# /usr/local/etc/rc.d/syslog-ng start Starting syslog_ng. Error opening plugin module; module='afsmtp', error='/usr/local/lib/syslog-ng/libafsmtp.so: Undefined symbol "_smtp_message_str_cb"' smtp support can only be disabled by removing libesmtp from the system... There is also a segfault now for each incoming log (nothing appears in log files), so I'm checking if it's related to any of the enabled modules. At first glance it seems to be something FreeBSD specific, I have all modules enabled on openSUSE, and logging seems to work (at least the basics, I checked). Bye, -- Peter Czanik (CzP)<czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

4 18

3.4 packages
by Peter Czanik 02 May '12

02 May '12

Hello, I created packages from current 3.4 git for openSUSE and helped to update the FreeBSD syslog-ng-devel port. They had some light testing and seem to work. Please refer to http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… for download information! The openSUSE packages have smtp enabled (in a separate sub package, untested), on FreeBSD the module can't be loaded, so marked as broken. Please let me know, if you run into any problems! Bye, -- Peter Czanik (CzP)<czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

2 2

[RFC]: LogThreadedDestDriver - making threaded destinations better
by Gergely Nagy 01 May '12

01 May '12

Hi! There have been a number of fixes going into afsql or afmongodb (or both) recently, fixing bugs that affected both, for the simple reason that when I made afmongodb, I copied parts of the architecture from afsql. Now there is the SMTP destination aswell, suffering from the same issues, and a Redis destination is in the works aswell. Therefore, it is about time to rethink how these drivers that all look very similar under the hood could be rebased onto a common foundation, something that abstracts away all the common stuff, such as handling the worker thread, and the queue method - so all the affected drivers need to do, is implement their own tasks, and skip all the copy-paste-ish boilerplate they have now. To this end, we had a short brainstorming with Bazsi, and I'll try to summarize what I remember (with a few later ideas added). The goal is to create a new class, based on LogDestDriver, for all those destination drivers that use a worker thread to do their blocking operations. This new class, which for the time being shall be named LogThreadedDestDriver shall implement the following things: * It must handle the setup, starting and stopping of the worker thread, with a module-supplied worker function, and that function shall be assisted by class functions to do the dirty work. * The queue handling must be safe and free of races (which pretty much means we have to find a way to wrap log_queue_pop_head into a synchronous call) * The class must be prepared to work not only with single messages, but in bulk aswell (think SQL's transactions, or the upcoming MongoDB bulk insert) * The class must implement its own queue method, downstream modules should not need to care about that. * We must be able to wake up the worker thread, even if there is nothing to handle: this is needed for a clean shutdown. And that's pretty much it. The class would have callbacks for the following things: * A pair of callbacks to call during the driver's _init() and _deinit() functions. These can either be implemented as callbacks, and the driver will then use it's own init/deinit and set it in its parent, and call the module-set callbacks from those. Or, the class can provide _init/_deinit methods, and leave it up to the module to implement its own methods, and call the class methods itself. The former makes drivers that do not require extra init/deinit simpler, the latter has less callbacks and looks neater. When implementing, I'll probably go with the latter. * An 'do-the-work' callback, which is called whenever work needs to be performed, and an error callback to fire whenever the do-the-work failed. I'm not entirely sure how best to handle the transaction case here, but that's something that needs to be done around here. One possibility is to allow two kinds of do-the-work callbacks: one for the simple case where no transactions are done (afsmtp and current afmongodb) and one that supports transactions/bulk inserts (afsql and future afmongodb and redis). Off the top of my head, this should be enough to move the boring parts of the current threaded destination drivers into a common class. Once the transaction case is figured out, all of our drivers can be reimplemented on top of this class, reducing copied and slightly altered code by a HUGE margin, and finally making it so that we have a single place to fix possible bugs in, instead of 3 slightly different copies of the same thing. As a side effect, time_reopen and stored/dropped counter handling would get moved from individual drivers into this base class, along with flush_timeout and flush_lines (if those still exist in 3.4) and a couple of other things. I plan to have a Proof of Concept patch ready by friday night, we'll see how it goes! (And shortly after, a redis destination will follow) -- |8]

2 2

Multi-core use in threaded mode
by Martin Holste 01 May '12

01 May '12

I'm seeing that on a very busy syslog-ng 3.3.4 box, CPU never rises above 103% and log loss occurs. Does threaded mode not allow separate threads to utilize multiple cores? I need to scale above a single CPU.

6 8

[PATCH (3.4) 0/3] afsql code cleanups
by Gergely Nagy 01 May '12

01 May '12

The patches following this mail are my attempts at cleaning up the afsql driver here and there, mostly afsql_dd_insert_db(), in preparation for moving towards LogThreadedDestDriver (which will be posted separately a bit later). The migration is possible without these patches, but these are worth applying even without the LogThreadedDestDriver patches, as these present no real architectural change, just a little refactoring. I think the code could be cleaned up further, but for the time being, I only did what I thought necessary to prepare for migration to LogThreadedDestDriver.

2 5

[PATCH (3.4)] afsmtp: Add SCS_SMTP to lib/stats.h
by Gergely Nagy 01 May '12

01 May '12

Add SCS_SMTP to lib/stats.h, so we get proper smtp stats too. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- lib/stats.h | 1 + modules/afsmtp/afsmtp.c | 4 ---- 2 files changed, 1 insertion(+), 4 deletions(-) diff --git a/lib/stats.h b/lib/stats.h index 99936da..9174408 100644 --- a/lib/stats.h +++ b/lib/stats.h @@ -70,6 +70,7 @@ enum SCS_SEVERITY = 24, SCS_FACILITY = 25, SCS_SENDER = 26, + SCS_SMTP = 27, SCS_MAX, SCS_SOURCE_MASK = 0xff }; diff --git a/modules/afsmtp/afsmtp.c b/modules/afsmtp/afsmtp.c index bd8adfe..841100d 100644 --- a/modules/afsmtp/afsmtp.c +++ b/modules/afsmtp/afsmtp.c @@ -33,10 +33,6 @@ #include <libesmtp.h> -#ifndef SCS_SMTP -#define SCS_SMTP 0 -#endif - typedef struct { gchar *name; -- 1.7.10

2 1

syslog-ng 3.3.5 and openssl
by Alexander Komyagin 01 May '12

01 May '12

Hi! I noticed that when I'm compiling syslog-ng 3.3.5 with openssl enabled (at least in my environment), libsyslog-ng-crypto.so somehow is not linked with libssl (checked with ldd), thus resulting with unresolved symbols error on syslog-ng startup for afsocket module. Attached patch fixes the issue. --- syslog-ng-3.3.5.orig/lib/Makefile.am 2012-03-31 22:23:43.000000000 +0400 +++ syslog-ng-3.3.5/lib/Makefile.am 2012-04-26 17:32:58.000000000 +0400 @@ -12,7 +12,7 @@ libsyslog_ng_la_LDFLAGS = -no-undefined -release @VERSION@ module_LTLIBRARIES = libsyslog-ng-crypto.la -libsyslog_ng_crypto_la_LIBADD = @CORE_DEPS_LIBS@ libsyslog-ng.la +libsyslog_ng_crypto_la_LIBADD = @CORE_DEPS_LIBS@ @OPENSSL_LIBS@ libsyslog-ng.la libsyslog_ng_crypto_la_LDFLAGS = -no-undefined -avoid-version # this is intentionally formatted so conflicts are less likely to arise. one name in every line. -- Best wishes, Alexander Komyagin

2 1

[PATCH] lib/gprocess.c: Ignore SIGHUP inside the supervisor.
by Gergely Nagy 01 May '12

01 May '12

Reloading syslog-ng's config is often done via a killall -HUP syslog-ng command, which also sends a signal to the supervisor. However, the supervisor did not handle said signal, thus it exited as is the default for it. To work around this issue, lets ignore SIGHUP inside the supervisor. Reported-by: Patrick Hemmer Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- lib/gprocess.c | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-) diff --git a/lib/gprocess.c b/lib/gprocess.c index aaefbab..9246d88 100644 --- a/lib/gprocess.c +++ b/lib/gprocess.c @@ -1110,7 +1110,9 @@ g_process_perform_supervise(void) g_process_message("Initialization failed but the daemon did not exit, even when forced to, trying to recover; pid='%d'", pid); continue; } - + + sigignore(SIGHUP); + if (process_opts.check_fn && (process_opts.check_period >= 0)) { gint i = 1; -- 1.7.9.1

2 1

3.4 on opensuse factory
by Peter Czanik 01 May '12

01 May '12

Hello, When trying to run syslog-ng 3.4 on openSUSE factory, the supervisor process starts, but it's child segfaults continuously. I tried to revert the big nvtable patch, which solved my problem previously, but it did not help. Here are two backtraces, practically the same: With the nvtable patch: (gdb) bt #0 0x00007f3e447516c6 in __memcpy_ssse3_back () from /lib64/libc.so.6 #1 0x00007f3e44f1a3ff in nv_table_add_value () from /lib64/libsyslog-ng-3.4.0alpha1.so #2 0x00007f3e44f0858f in log_msg_set_value () from /lib64/libsyslog-ng-3.4.0alpha1.so #3 0x00007f3e44f015db in log_src_driver_queue_method () from /lib64/libsyslog-ng-3.4.0alpha1.so #4 0x00007f3e44f12bfe in log_source_queue () from /lib64/libsyslog-ng-3.4.0alpha1.so #5 0x00007f3e44efa008 in afinter_source_post () from /lib64/libsyslog-ng-3.4.0alpha1.so #6 0x00007f3e44f338d4 in iv_event_run_pending_events () from /lib64/libsyslog-ng-3.4.0alpha1.so #7 0x00007f3e44f33c49 in iv_event_raw_got_event () from /lib64/libsyslog-ng-3.4.0alpha1.so #8 0x00007f3e44f3534a in iv_main () from /lib64/libsyslog-ng-3.4.0alpha1.so #9 0x00007f3e44f18532 in main_loop_run () from /lib64/libsyslog-ng-3.4.0alpha1.so #10 0x00000000004016e1 in main () With the nvtable patch (https://github.com/bazsi/syslog-ng-3.4/commit/1c3a37764d98d5a37f2cd3c79f7f4…) reverted: (gdb) bt #0 0x00007f6fdb8b86c6 in __memcpy_ssse3_back () from /lib64/libc.so.6 #1 0x00007f6fdc0818bf in nv_table_add_value () from /lib64/libsyslog-ng-3.4.0alpha1.so #2 0x00007f6fdc06f620 in log_msg_set_value () from /lib64/libsyslog-ng-3.4.0alpha1.so #3 0x00007f6fdc0685cb in log_src_driver_queue_method () from /lib64/libsyslog-ng-3.4.0alpha1.so #4 0x00007f6fdc079dbe in log_source_queue () from /lib64/libsyslog-ng-3.4.0alpha1.so #5 0x00007f6fdc060ff8 in afinter_source_post () from /lib64/libsyslog-ng-3.4.0alpha1.so #6 0x00007f6fdc09afc4 in iv_event_run_pending_events () from /lib64/libsyslog-ng-3.4.0alpha1.so #7 0x00007f6fdc09b339 in iv_event_raw_got_event () from /lib64/libsyslog-ng-3.4.0alpha1.so #8 0x00007f6fdc09ca3a in iv_main () from /lib64/libsyslog-ng-3.4.0alpha1.so #9 0x00007f6fdc07f942 in main_loop_run () from /lib64/libsyslog-ng-3.4.0alpha1.so #10 0x00000000004016e1 in main () Is it something in syslog-ng, or I should rather report it at openSUSE? Bye, -- Peter Czanik (CzP)<czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

3 3