- syslog-ng - lists.balabit.hu

[ANN]: balabit.logstore - Open Source LogStore reader API
by Gergely Nagy 04 May '12

04 May '12

--------------------------------------------------------------- PACKAGE : com.balabit/logstore VERSION : 0.0.1 SUMMARY : First alpha release DATE : 2012 April 24 HOMEPAGE : http://algernon.github.com/balabit.logstore --------------------------------------------------------------- DESCRIPTION: The balabit.logstore project is a library written in Clojure, that tries to provide a convenient API to read syslog-ng PE LogStore files. The reason behind the implementation is to have an independent, open source reader for the LogStore file format, so that one is not tied to syslog-ng to read one's logs stored in this format. An open implementation makes it possible to read these logs on systems where syslog-ng is not installed, or where the lgstool program is not available. LogStore itself is a storage format for log messages, supporting encryption, compression, secure timestamps, serialized messages (think name-value pairs as opposed to just a plain, formatted string) and a lot more. This format is being opened, and this library is meant to provide an implenetation for a LogStore reader, that will eventually support reading all kinds of LogStore files. EXAMPLE: (ns example (:require [balabit.logstore :as logstore])) (defn lgstool-cat ([] (lgstool-cat "resources/logstores/short.compressed.store")) ([filename] (logstore/with-file filename (let [indexed-records (zipmap (range (count (logstore/records))) (logstore/records)) chunk-records (keys (filter #(= :chunk (:type (val %))) indexed-records)) print-msgs (fn [index] (dorun (map println (:messages (logstore/nth-record index)))))] (dorun (map print-msgs chunk-records)))))) (lgstool-cat "/path/to/logstore/messages.store") More examples can be found bundled with the sources, and as part of the documentation: http://algernon.github.com/balabit.logstore/#balabit.logstore.examples FEATURES: The library - being in very early alpha stage - can already read some LogStore files, and transparently decompress them, collect meta-data about both the logstore, and the information stored within. It comes with an extensive test suite and a lot of examples. FUTURE DIRECTION: We are working on making the library able to deserialize name-value pairs, to make it support decrypting logstores, and paralell to that, we're working on a Java API, so that it can be used from a more familiar language, and still provide a convenient, easy to use API. In the future, we also plan to publish ready to use JAR files that one can easily use from any Java or Clojure project. DOWNLOADS: The source is available from the git repository at github: git://github.com/algernon/balabit.logstore.git Documentation and more information about the library is available on its homepage at http://algernon.github.com/balabit.logstore/. People interested in the development of the library can also look at the developer docs at: http://algernon.github.com/balabit.logstore/developer-api.html -- |8]

2 2

[PATCH] afsocket: Support a smooth upgrade path to recent systemd
by Gergely Nagy 04 May '12

04 May '12

When the journal was introduced in systemd, it started to pass /run/systemd/journal/syslog to syslogds (it was passing down /dev/log before). Since we verify that the FD to open matches the expected filename, we didn't pick up the fd due to the filename mismatch. This can be fixed on the system() source level, where it can decide which to generate into the config. However, for setups that do not use system() for one reason or the other, this does not work. For these systems that do not use the system() source, the best solution is to try and replace /dev/log with /run/systemd/journal/syslog at run-time. This patch does just that, and ties it to config versions 3.4 or earlier. This means that post-3.4, the automatic replacement will stop working, and users will get the generic failed to acquire socket error message instead. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/afsocket/afunix.c | 56 ++++++++++++++++++++++++++++++++++----------- 1 file changed, 43 insertions(+), 13 deletions(-) diff --git a/modules/afsocket/afunix.c b/modules/afsocket/afunix.c index 8145f1a..a5644ac 100644 --- a/modules/afsocket/afunix.c +++ b/modules/afsocket/afunix.c @@ -72,7 +72,8 @@ afunix_sd_set_perm(LogDriver *s, gint perm) #if ENABLE_SYSTEMD static gboolean -afunix_sd_acquire_socket(AFSocketSourceDriver *s, gint *result_fd) +afunix_sd_acquire_named_socket(AFSocketSourceDriver *s, gint *result_fd, + const gchar *filename) { AFUnixSourceDriver *self = (AFUnixSourceDriver *) s; gint fd, fds; @@ -101,12 +102,12 @@ afunix_sd_acquire_socket(AFSocketSourceDriver *s, gint *result_fd) for (fd = SD_LISTEN_FDS_START; fd < SD_LISTEN_FDS_START + fds; fd++) { /* check if any type is available */ - if (sd_is_socket_unix(fd, 0, -1, self->filename, 0)) + if (sd_is_socket_unix(fd, 0, -1, filename, 0)) { int type = (self->super.flags & AFSOCKET_STREAM) ? SOCK_STREAM : SOCK_DGRAM; /* check if it matches our idea of the socket type */ - if (sd_is_socket_unix(fd, type, -1, self->filename, 0)) + if (sd_is_socket_unix(fd, type, -1, filename, 0)) { *result_fd = fd; break; @@ -114,7 +115,7 @@ afunix_sd_acquire_socket(AFSocketSourceDriver *s, gint *result_fd) else { msg_error("The systemd supplied UNIX domain socket is of a different type, check the configured driver and the matching systemd unit file", - evt_tag_str("filename", self->filename), + evt_tag_str("filename", filename), evt_tag_int("systemd-sock-fd", fd), evt_tag_str("expecting", type == SOCK_STREAM ? "unix-stream()" : "unix-dgram()"), NULL); @@ -130,31 +131,60 @@ afunix_sd_acquire_socket(AFSocketSourceDriver *s, gint *result_fd) */ msg_debug("Ignoring systemd supplied fd as it is not a UNIX domain socket", - evt_tag_str("filename", self->filename), + evt_tag_str("filename", filename), evt_tag_int("systemd-sock-fd", fd), NULL); } } } - else - return TRUE; if (*result_fd != -1) { g_fd_set_nonblock(*result_fd, TRUE); g_fd_set_cloexec(*result_fd, TRUE); msg_verbose("Acquired systemd socket", - evt_tag_str("filename", self->filename), + evt_tag_str("filename", filename), evt_tag_int("systemd-sock-fd", *result_fd), NULL); + return TRUE; } - else + return TRUE; +} + +static gboolean +afunix_sd_acquire_socket(AFSocketSourceDriver *s, gint *result_fd) +{ + AFUnixSourceDriver *self = (AFUnixSourceDriver *) s; + gboolean fd_ok; + GlobalConfig *cfg = log_pipe_get_config(&s->super.super.super); + + fd_ok = afunix_sd_acquire_named_socket(s, result_fd, self->filename); + + if (fd_ok && (*result_fd == -1) && (strcmp(self->filename, "/dev/log") == 0)) { - msg_debug("Failed to acquire systemd socket, trying to open ourselves", - evt_tag_str("filename", self->filename), - NULL); + fd_ok = afunix_sd_acquire_named_socket(s, result_fd, "/run/systemd/journal/syslog"); + + if (fd_ok && *result_fd > -1) + { + if (cfg->version <= 0x0304) + { + msg_warning("WARNING: systemd detected while using /dev/log; migrating automatically to /run/systemd/journal/syslog. Please update your configuration to use the system() source.", + evt_tag_str("id", self->super.super.super.id), + NULL); + + g_free(self->filename); + self->filename = g_strdup("/run/systemd/journal/syslog"); + return TRUE; + } + } } - return TRUE; + + if (!fd_ok) + msg_debug("Failed to acquire systemd socket, trying to open ourselves", + evt_tag_str("filename", self->filename), + NULL); + + return fd_ok; } #else -- 1.7.10

1 0

[PATCH 0/2] systemd enhancements
by Gergely Nagy 04 May '12

04 May '12

The two patches that follow improve syslog-ng's support for systemd-enabled platforms. The first patch simply teaches the system() source to prefer /run/systemd/journal/syslog over /dev/log if running under systemd. That alone makes configs that use system() work under both sysvinit and systemd without any change at all to the config. However, for configs that use /dev/log hard-coded, the upgrade path isn't that smooth. In that case, we still check whether running under systemd, and if so, but we couldn't accept any of the offered fds, we check again with /run/systemd/journal/syslog as the filename, and replace the hard-coded one if the other filename worked. The second patch implements the above - I'm not entirely sold on the idea, and haven't explored the possible side-effects yet, but I believe it's reasonably safe for the vast majority of cases. For now, the second patch is mostly meant as a proposal, and will need a review before it can be considered for a merge.

2 5

[PATCH] afsocket: Fix unix-dgram initialisation
by Gergely Nagy 04 May '12

04 May '12

When initializing an unix-dgram socket, set self->fd to the acquired socket FD, so that fds inherited from systemd actually get used and polled on. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/afsocket/afsocket.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/modules/afsocket/afsocket.c b/modules/afsocket/afsocket.c index 5b192f4..c1bee5c 100644 --- a/modules/afsocket/afsocket.c +++ b/modules/afsocket/afsocket.c @@ -679,7 +679,7 @@ afsocket_sd_init(LogPipe *s) if (sock == -1 && !afsocket_open_socket(self->bind_addr, !!(self->flags & AFSOCKET_STREAM), &sock)) return self->super.super.optional; } - self->fd = -1; + self->fd = sock; if (!self->setup_socket(self, sock)) { -- 1.7.9.1

2 3

[PATCH] afsmtp: Fix afsmtp linking when configured with --with-libesmtp
by Gergely Nagy 04 May '12

04 May '12

In case configure received a --with-libesmtp=/path option, we forgot to add -lesmtp to LIBESMTP_LIBS, which resulted in afsmtp getting compiled, but not linked to libesmtp, which is obviously not desirable. This patch fixes that, and adds the -lesmtp. Reported-by: Peter Czanik <czanik(a)balabit.hu> Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- configure.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/configure.in b/configure.in index 2d56bbd..03cc76c 100644 --- a/configure.in +++ b/configure.in @@ -815,7 +815,7 @@ if test "x$enable_smtp" != "xno" && test "x$with_libesmtp" != "no"; then CPPFLAGS="$CPPFLAGS -I$with_libesmtp/include" LDFLAGS="$LDFLAGS -L$with_libesmtp/lib" AC_CHECK_HEADER(libesmtp.h, [LIBESMTP_CFLAGS="-I$with_libesmtp/include" - LIBESMTP_LIBS="-L$with_libesmtp/lib"], [libesmtp=no]) + LIBESMTP_LIBS="-L$with_libesmtp/lib -lesmtp"], [libesmtp=no]) CPPFLAGS="$CPPFLAGS_SAVE" LDFLAGS="$LDFLAGS_SAVE" else -- 1.7.10

1 0

[PATCH] lib/gprocess.c: Ignore SIGHUP inside the supervisor.
by Gergely Nagy 04 May '12

04 May '12

Reloading syslog-ng's config is often done via a killall -HUP syslog-ng command, which also sends a signal to the supervisor. However, the supervisor did not handle said signal, thus it exited as is the default for it. To work around this issue, lets ignore SIGHUP inside the supervisor. Reported-by: Patrick Hemmer Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- lib/gprocess.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/lib/gprocess.c b/lib/gprocess.c index aaefbab..d68521d 100644 --- a/lib/gprocess.c +++ b/lib/gprocess.c @@ -1055,10 +1055,15 @@ g_process_perform_supervise(void) pid_t pid; gboolean first = TRUE, exited = FALSE; gchar proc_title[PROC_TITLE_SPACE]; + struct sigaction sa; g_snprintf(proc_title, PROC_TITLE_SPACE, "supervising %s", process_opts.name); g_process_setproctitle(proc_title); + memset(&sa, 0, sizeof(sa)); + sa.sa_handler = SIG_IGN; + sigaction(SIGHUP, &sa, NULL); + while (1) { if (pipe(init_result_pipe) != 0) -- 1.7.10

1 0

syslog-ng Insider - May 2012
by Peter Czanik 03 May '12

03 May '12

Dear syslog-ng users, This is the 13th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com <mailto:documentation@balabit.com> FEATURED NEWS Have you tried to add custom information to log messages, fix mis-formatted logs or anonymize logs? --------------------------------- The next long-time-supported release of SSB version 3 LTS is about to be released. This release includes a switch to 64-bit architecture, a huge performance improvement in the indexing/searching feature for a large number of message and search patterns and a couple of new features, too. The following post plans to introduce those new features to you. The updated User’s Manual will contain a detailed description of them — this post is written more to serve as teaser and to highlight some of the use cases we had in mind when we’d planned the features, and, of course, to ask for your feedback about them. http://gyp.blogs.balabit.com/2012/03/new-features-in-ssb-3-lts-rewriting-pa… Admin guide: http://www.balabit.com/sites/default/files/documents/ssb-3.0-guides/ssb-v3.… syslog-ng participates in GSoC ------------------------------ This year syslog-ng participates in GSoC under the umbrella of openSUSE. We have one student accepted, who will work on syslog-ng’s mongodb destination. The story of BalaBit's GSoC participation: http://czanik.blogs.balabit.com/2012/04/syslog-ng-on-gsoc-thanks-opensuse/ What the student will do: http://algernon.blogs.balabit.com/2012/05/in-the-land-of-open-source-the-fr… MongoDB howto ------------- One of the major reasons to update to 3.3 other than threading is MongoDB. It allows great flexibility when using patterns and storing parsed data from logs. The syslog-ng documentation covers most information necessary to use MongoDB, but this HowTo compiles all these into a single document and extends it with features from the upcoming syslog-ng 3.4 version. http://asylum.madhouse-project.org/blog/2012/04/26/mongodb-howto/ OTHER SHORT NEWS * There is now an open source LogStore reader work in progress: https://github.com/algernon/balabit.logstore * syslog-ng 3.4 alpha packages are now available in the FreeBSD ports and also for openSUSE: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… * syslog-ng and ELSA were presented at LOADays: http://czanik.blogs.balabit.com/2012/04/syslog-ng-at-linux-open-administrat… NEW RELEASES: * syslog-ng OSE 3.3.5: https://lists.balabit.hu/pipermail/syslog-ng-announce/2012-May/000142.html * syslog-ng OSE 3.4 alpha2: https://lists.balabit.hu/pipermail/syslog-ng-announce/2012-May/000143.html * libumberlog 0.2.1: http://algernon.github.com/libumberlog/ ARCHIVE http://insider.blogs.balabit.com/ -- Peter Czanik (CzP)<czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

smtp on FreeBSD
by Peter Czanik 02 May '12

02 May '12

Hello, I tried to compile syslog-ng 3.4 alpha git on FreeBSD 8.1 with smtp support, but it does not start: fb81# /usr/local/etc/rc.d/syslog-ng start Starting syslog_ng. Error opening plugin module; module='afsmtp', error='/usr/local/lib/syslog-ng/libafsmtp.so: Undefined symbol "_smtp_message_str_cb"' smtp support can only be disabled by removing libesmtp from the system... There is also a segfault now for each incoming log (nothing appears in log files), so I'm checking if it's related to any of the enabled modules. At first glance it seems to be something FreeBSD specific, I have all modules enabled on openSUSE, and logging seems to work (at least the basics, I checked). Bye, -- Peter Czanik (CzP)<czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

4 18

3.4 packages
by Peter Czanik 02 May '12

02 May '12

Hello, I created packages from current 3.4 git for openSUSE and helped to update the FreeBSD syslog-ng-devel port. They had some light testing and seem to work. Please refer to http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… for download information! The openSUSE packages have smtp enabled (in a separate sub package, untested), on FreeBSD the module can't be loaded, so marked as broken. Please let me know, if you run into any problems! Bye, -- Peter Czanik (CzP)<czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

2 2

[RFC]: LogThreadedDestDriver - making threaded destinations better
by Gergely Nagy 01 May '12

01 May '12

Hi! There have been a number of fixes going into afsql or afmongodb (or both) recently, fixing bugs that affected both, for the simple reason that when I made afmongodb, I copied parts of the architecture from afsql. Now there is the SMTP destination aswell, suffering from the same issues, and a Redis destination is in the works aswell. Therefore, it is about time to rethink how these drivers that all look very similar under the hood could be rebased onto a common foundation, something that abstracts away all the common stuff, such as handling the worker thread, and the queue method - so all the affected drivers need to do, is implement their own tasks, and skip all the copy-paste-ish boilerplate they have now. To this end, we had a short brainstorming with Bazsi, and I'll try to summarize what I remember (with a few later ideas added). The goal is to create a new class, based on LogDestDriver, for all those destination drivers that use a worker thread to do their blocking operations. This new class, which for the time being shall be named LogThreadedDestDriver shall implement the following things: * It must handle the setup, starting and stopping of the worker thread, with a module-supplied worker function, and that function shall be assisted by class functions to do the dirty work. * The queue handling must be safe and free of races (which pretty much means we have to find a way to wrap log_queue_pop_head into a synchronous call) * The class must be prepared to work not only with single messages, but in bulk aswell (think SQL's transactions, or the upcoming MongoDB bulk insert) * The class must implement its own queue method, downstream modules should not need to care about that. * We must be able to wake up the worker thread, even if there is nothing to handle: this is needed for a clean shutdown. And that's pretty much it. The class would have callbacks for the following things: * A pair of callbacks to call during the driver's _init() and _deinit() functions. These can either be implemented as callbacks, and the driver will then use it's own init/deinit and set it in its parent, and call the module-set callbacks from those. Or, the class can provide _init/_deinit methods, and leave it up to the module to implement its own methods, and call the class methods itself. The former makes drivers that do not require extra init/deinit simpler, the latter has less callbacks and looks neater. When implementing, I'll probably go with the latter. * An 'do-the-work' callback, which is called whenever work needs to be performed, and an error callback to fire whenever the do-the-work failed. I'm not entirely sure how best to handle the transaction case here, but that's something that needs to be done around here. One possibility is to allow two kinds of do-the-work callbacks: one for the simple case where no transactions are done (afsmtp and current afmongodb) and one that supports transactions/bulk inserts (afsql and future afmongodb and redis). Off the top of my head, this should be enough to move the boring parts of the current threaded destination drivers into a common class. Once the transaction case is figured out, all of our drivers can be reimplemented on top of this class, reducing copied and slightly altered code by a HUGE margin, and finally making it so that we have a single place to fix possible bugs in, instead of 3 slightly different copies of the same thing. As a side effect, time_reopen and stored/dropped counter handling would get moved from individual drivers into this base class, along with flush_timeout and flush_lines (if those still exist in 3.4) and a couple of other things. I plan to have a Proof of Concept patch ready by friday night, we'll see how it goes! (And shortly after, a redis destination will follow) -- |8]

2 2