- syslog-ng - lists.balabit.hu

[PATCH] loggen: Fix a possible division by zero.
by Gergely Nagy 03 May '11

03 May '11

In some cases where we could not send a single message, loggen tried to divide by zero shortly before printing the statistics at the end. This was trivially reproducible by using -R with a file that contained invalid syslog lines. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- tests/loggen/loggen.c | 7 ++++++- 1 files changed, 6 insertions(+), 1 deletions(-) diff --git a/tests/loggen/loggen.c b/tests/loggen/loggen.c index bf5d111..c5399a7 100644 --- a/tests/loggen/loggen.c +++ b/tests/loggen/loggen.c @@ -467,7 +467,12 @@ gen_messages(send_data_t send_func, void *send_func_ud, int thread_id, FILE *rea printf("%d;%lu.%06lu;%.2lf;%lu\n", thread_id, (long) diff_tv.tv_sec, (long) diff_tv.tv_usec, (((double) (count - last_count) * USEC_PER_SEC) / diff_usec), count); if (readfrom) - raw_message_length = sum_linelen/count; + { + if (count) + raw_message_length = sum_linelen/count; + else + raw_message_length = 0; + } free(testsdata); return count; } -- 1.7.2.5

2 1

[new shared lib] libsyslog-ng-crypto.so
by Balazs Scheidler 03 May '11

03 May '11

Hi, I just wanted to give a heads-up for people doing development on syslog-ng code, that I've introduced libsyslog-ng-crypto.so in the core, which is to contain core code that depends on libssl. You probably know, that one of the driving factors for modularizing syslog-ng was to make the core of syslog-ng independent of OpenSSL, since that is not available during early boot on a number of systems. Anyhow, there's a lot of code, that should be shared between plugins, but would depend on OpenSSL. To resolve this problem, I've now created libsyslog-ng-crypto.so, installed into $prefix/lib/syslog-ng (e.g. a private lib, which lives in the same place as the plugins), which is now responsible for: - seeding the random number generator - TLS transport code (tlstransport.c, earlier in the afsocket module) - TLS context code (tlscontext.c, -"-) I guess some others may follow. Plugins that would use openssl, should _always_ link against libsyslog-ng-crypto.so, as the random number generator is seeded when this library is loaded. (of course not all platforms need explicit PRNG seeding, but some do). Now as I'm reviewing the patch again, it is quite dirty, contains some unrelated changes, so don't take it as an example, but now a lot of others have accumulated on top of it, so I don't want to rebase and change the history. This is the patch I'm talking about: Author: Balazs Scheidler <bazsi(a)balabit.hu> 2011-04-30 20:17:44 Committer: Balazs Scheidler <bazsi(a)balabit.hu> 2011-04-30 20:17:44 Parent: adff96aa401c8ef398a617c14daf7dc45f85cab7 (logreader: fixed parsing flags containing dashes or underscores) Child: 2b65655028da7cc216aa747d0bb206fe28b93955 (logwriter: initialize the queue_filled event earlier) Branches: master, remotes/origin/master Follows: v3.3.0alpha2 Precedes: libsyslog-ng-crypto.so: This patch introduces a new shared object where openssl dependent core goes The TLS code is sufficiently independent from the afsocket plugin, however currently they go in the afsocket module directory. This patch introduces a new shared object called libsyslog-ng-crypto.so, which contains TLS code, random number initialization and basically everything that would introduce an openssl dependency. Signed-off-by: Balazs Scheidler <bazsi(a)balabit.hu> -- Bazsi

1 0

rewrite problem
by Micah Anderson 03 May '11

03 May '11

Hi, I've been trying to use the rewrite functionality using 3.1.3, but I am having some problems: 1. when having a rewrite setup, I can denial of service syslog-ng :( if I do this: rewrite r_ip { subst("([[:digit:]]{1,3}\.){3}[[:digit:]]{1,3}", "\[REDACTED\]", value("MESSAGE"), type("pcre"), flags("global")); }; filter f_mail { facility(mail) and not facility(local1) and not program("^postfix/"); }; destination df_mail { file("/var/log/mail.log"); }; log { source(s_src); filter(f_mail); rewrite(r_ip); destination(df_mail); flags(final); }; and then I do: $ logger -p mail.info hi syslog-ng starts going crazy, it eats 100% of the CPU and the memory. If you are lucky, the OOM killer kicks in and stops it before your system goes down. I originally thought that this was due to a bad regexp I had, but the above regexp works fine for normal mail syslog messages. However, in the process of trying to make a better rewrite rule, I found that I've created ones where normal mail messages do this too. 2. \. doesn't match literal periods Using the above rewrite, one would expect that the regexp would match three times a number containing 1-3 digits, each time followed by a literal period. Then a number containing 1-3 digits. A basic, and not very complete IPv4 regexp, that would match something like: 192.168.25.1. In fact, it does match that IPv4 address, but it is also matching other things that do not have periods in them. For example: rewrite r_ip { subst("([[:digit:]]{1,3}\.){3}[[:digit:]]{1,3}", "\[REDACTED\]", value("MESSAGE"), type("pcre"), flags("global")); }; $ logger -i -p mail.info 'Apr 21 20:38:58 algae postfix/smtp[14817]: EB4A2168: to=<482092-subscribe(a)bugs.debian.org>, relay=192.168.164.34[192.168.164.34]:587, delay=2.9, delays=0.19/0.14/2/0.58, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 6722214C119)' I would expect that the two IP addressesw in the above to be matched and replaced, but this is what happens: Apr 22 12:45:11 algae logger[6234]: Apr [REDACTED] algae postfix/smtp[14817]: EB4A2168: to=<482092-subscribe(a)bugs.debian.org>, relay=[REDACTED][[REDACTED]]:587, delay=2.9, delays=[REDACTED]/2/0.58, dsn=2.0.0, status=sent ([REDACTED] Ok: queued as [REDACTED]) Its matching the queue ID 'EB4A2168' and its also matching '6722214C119', neither of those have periods in them, and EB4A2168 is not 0-9, which is what :digit: should be matching. I tried to replace [[:digit:]] with \d, but that caused issue #1 to happen. micah --

3 7

differing linking options
by Balazs Scheidler 03 May '11

03 May '11

Hi, @Corinna: I'm ccing you because of a cygwin question at the end of this email. Thanks for answering it if you have time. While cherry-picking changes from your tree at git://git.balabit.hu/syslog-ng-team/syslog-ng-core-4.1.git I've noticed that you are changing linking command lines for modules, which would break mixed linking mode used by Linux distributions in their native packages. Since it's bad to differ on things like this, I've figured it might make sense to come to a solution that's good for both of us. This is the patch that adds the new libraries: commit 23ca7a66c5a612ee7135c2543d707c7e484571a1 Author: Tamas Pal <folti(a)balabit.hu> Date: Tue Mar 29 11:42:00 2011 +0200 [autoconfuse]: make system specific libafsocket symlink with the proper shlib extension. Added proper LIBS dependencies, to prevent linking error in case libtool fails to parse .la files properly And this is the hunk that is (for instance) problematic: -libafsocket_notls_la_LIBADD = ../../lib/libsyslog-ng.la $(LIBNET_LIBS) $(LIBWRAP_LIBS) +libafsocket_notls_la_LIBADD = ../../lib/libsyslog-ng.la $(LIBNET_LIBS) $(LIBWRAP_LIBS) $(GLIB_LIBS) $(EVTLOG_LIBS) $(IVYKIS_LIBS) The technical issue at hand is that glib, evtlog and ivykis can all be static libraries in mixed mode, which would result in their inclusion into each module shared object, causing trouble at runtime. (not to mention that the static libs are probably not compiled with -fPIC so they cannot be linked into .so files). I do have a problem with AIX linking with 3.2 now, which is probably the root cause that you've added these libraries into the Makefiles in the first place. AIX linker is quite restrictive on how symbols are reexported from the main executable, but I do remember I could successfully create a linking mode that worked on AIX and wasn't broken on Linux in mixed mode. Clearly adding the libraries to that command line unconditionally is not such a combination. Mixed mode doesn't need to be supported on AIX, (I wouldn't dare to try to support that with non-GNU linkers). I think the breakage is rooted by the addition of -no-undefined to the libtool command line, which causes the AIX linker to ignore symbols exported by the main executable. -no-undefined was added to catch errors on Cygwin, which doesn't allow to link DLLs with undefined symbols in them. The questions are: 1) Since last time we've checked that OSE 3.2 built on cygwin, which doesn't include those libs, I think the cygwin linker would accept symbols exported by the dependencies of libsyslog-ng.so. Is that right Corinna? 2) AIX seems to have worked without -no-undefined (I've just retried that with 3.2.3, and confirmed that works). 3) @syslog-ng-team: Are there any other reasons you've included the libs explicitly on those command lines? So I'm proposing to make -no-undefined conditional, and only include it on non-AIX versions, and skip the listing of libs provided by libsyslog-ng.so from LIBADD. What do you think? -- Bazsi

2 1

Re: [syslog-ng] [PATCH] Use CAP_SYSLOG instead of CAP_SYS_ADMIN, if available.
by Gergely Nagy 03 May '11

03 May '11

Serge Hallyn <serge.hallyn(a)canonical.com> writes: > Quoting Balazs Scheidler (bazsi(a)balabit.hu) >> Hi, >> >> Thanks for the great stuff, it is really appreciated. >> >> I've some notes, please let me know if you or Serge would like to do >> that, or I should. > > Thanks, by all means please go ahead, your suggestions certainly sound > cleaner, and I'd probably end up producing something that ended up > requiring at least one more review cycle :) In that case, I'll have a round of fun with it (in a day or two, if all goes well). -- |8]

1 0

syslog-ng 3.2.3 build failure
by Sandor Geller 02 May '11

02 May '11

Hello, The below change breaks linking syslog-ng-ctl on OSes where the resolver library provides inet_aton so the -lresolv linker flag is needed. So far I ran into the issue only on Solaris 8 which isn't a modern OS I know... --- syslog-ng-3.2.2/syslog-ng/Makefile.in 2011-01-15 17:15:10.000000000 +0100 +++ syslog-ng-3.2.3/syslog-ng/Makefile.in 2011-05-01 21:47:51.000000000 +0200 @@ -60,7 +60,7 @@ syslog_ng_DEPENDENCIES = am_syslog_ng_ctl_OBJECTS = syslog-ng-ctl.$(OBJEXT) syslog_ng_ctl_OBJECTS = $(am_syslog_ng_ctl_OBJECTS) -syslog_ng_ctl_DEPENDENCIES = ../lib/libsyslog-ng.la +syslog_ng_ctl_DEPENDENCIES = ../lib/gsocket.lo ../lib/gsockaddr.lo am_syslog_ng_wrapper_OBJECTS = wrapper.$(OBJEXT) syslog_ng_wrapper_OBJECTS = $(am_syslog_ng_wrapper_OBJECTS) syslog_ng_wrapper_LDADD = $(LDADD) The fix seems to be trivial and hopefully doesn't have side-effects on any OS: --- syslog-ng-3.2.3/configure.in 2011-05-01 21:46:51.000000000 +0200 +++ syslog-ng-3.2.3-hacked/configure.in 2011-05-02 14:06:27.000000000 +0200 @@ -845,6 +845,7 @@ AC_SUBST(SOURCE_REVISION) AC_SUBST(PACKAGE_FILES_SUFFIX) AC_SUBST(PACKAGE_SUFFIX) +AC_SUBST(RESOLV_LIBS) AC_OUTPUT(dist.conf diff -urN syslog-ng-3.2.3/syslog-ng/Makefile.am syslog-ng-3.2.3-hacked/syslog-ng/Makefile.am --- syslog-ng-3.2.3/syslog-ng/Makefile.am 2011-02-23 11:15:06.000000000 +0100 +++ syslog-ng-3.2.3-hacked/syslog-ng/Makefile.am 2011-05-02 13:53:44.000000000 +0200 @@ -19,7 +19,7 @@ syslog_ng_wrapper_SOURCES = wrapper.c syslog_ng_ctl_SOURCES = syslog-ng-ctl.c -syslog_ng_ctl_LDADD = ../lib/gsocket.lo ../lib/gsockaddr.lo @BASE_LIBS@ @GLIB_LIBS@ +syslog_ng_ctl_LDADD = ../lib/gsocket.lo ../lib/gsockaddr.lo @BASE_LIBS@ @GLIB_LIBS@ @RESOLV_LIBS@ if ENABLE_ENV_WRAPPER Regards, Sandor

2 2

PCRE subst example from documentation produces error
by Daniel Maher 02 May '11

02 May '11

Hello, While following the example given here : http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.2-gui… I copied and pasted the string, then attempted to restart the daemon; unfortunately, this was the result : --- Error parsing rewrite expression, syntax error, unexpected LL_IDENTIFIER, expecting ')' in /etc/syslog-ng/syslog-ng.conf at line 354, column 27: {subst("a*", "?", field("message") type("pcre") flags("utf8" "global")); }; ^^^^^ syslog-ng documentation: http://www.balabit.com/support/documentation/?product=syslog-ng mailing list: https://lists.balabit.hu/mailman/listinfo/syslog-ng Cannot load /etc/syslog-ng/syslog-ng.conf --- Have I managed to paste incorrectly, or is the documentation incorrect, or is it something else entirely ? # syslog -V syslog-ng 3.2.2 Installer-Version: 3.2.2 Revision: Compile-Date: Apr 20 2011 16:19:28 Enable-Threads: off Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: off Enable-SSL: on Enable-SQL: off Enable-Linux-Caps: off Enable-Pcre: on Enable-Pacct: off Thank you. -- Daniel Maher « can't talk, too busy calculating computrons. »

2 1

[PATCH 0/3]: value-pairs() & related patches
by Gergely Nagy 01 May '11

01 May '11

Hi! For a multitude of reasons, ranging from the feature freeze looming over my head to hope that discrete patches are easier to review and merge than my awfully chaotic git tree, I'll be posting a couple of patches in a short while, as soon as I figure out what magic arguments I need to give to git send-email to make it send the batch the way I imagined. value-pairs() ============= The first one will be the value-pairs() implementation (also available on my integration/value-pairs branch), which is the base of the two others that will follow. While the value-pairs() implementation could still use some work, especially in the performance department, I believe it's good enough for initial use, and the benefits of having it (so that the mongodb driver can make use of it) far outweight the downsides. The implementation itself is reasonably light, it hardly touches anything outside it's own two files: the grammar and the makefile only, to hook itself in. If not used by a driver, then it does not change functionality at all. Therefore, I believe this could go in before the feature freeze. afmongodb value-pairs() port ============================ The next patch will be the port of the already existing mongodb destination driver to use value-pairs(). Also available on my integration/afmongodb-vp branch. This means that the driver will finally be able to do what it's good at: store arbitrary information in the database: whether that information comes via SDATA or patterndb, it doesn't matter, it will be stored in a structured manner. In my opinion, this is the major selling point of the mongodb destination driver at the moment (I have some other grand plans with it in the future, but that's more invasive, less important, and can easily wait 'till 3.4 ;). Without value-pairs(), afmongodb is pretty much useless. tfson() ======= For extra fun & profit, Balint Kovacs' format_json has been ported to use value-pairs(), and this will be the third patch in the series, and is also available on my integration/tfjson branch. This one's the least important, though it's dead simple, therefore I'd be very happy if it made past the feature freeze aswell. -- |8]

3 15

syslog-ng 3.2.3 has been released
by devel＠balabit.hu 01 May '11

01 May '11

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.2.3 SUMMARY : new stable release DATE : May 1, 2011 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (3.2.3) has been released. For latest fixes in the 3.2.x feature branch you are recommended to upgrade to this version. CHANGES: 3.2.3 Sun, 01 May 2011 19:05:32 +0200 Highlights: This is a maintenance release for the 3.2 branch, which contains several important functionality fixes in the db-parser() correllation code, an important security fix for FreeBSD & HP-UX (CVE-2011-0343) and build fixes for cygwin and mixed mode linking. Security fixes: * Fixed a possible security issue on Debian/kFreeBSD and on platforms where mode_t is an unsigned 16 bit value (FreeBSD, HP-UX). On these platforms syslog-ng may be using 0xFFFF as the permission bits. (CVE-2011-0343) Bugfixes: * Fixes an y2k38 problem that causes syslog-ng to use 100% CPU time in case mark messages are enabled and the UNIX timestamps overflows a signed 32 bit counter, which happens in 19th, January 2038 * Fixed file() destination to work on device nodes (e.g. files in /dev). Without this change, syslog-ng started using 100% CPU time if given devices as destinations that couldn't always consume data. * The code to restore the last file position for source files will not accept file-position past the file size, and will restart the file from the beginning instead. * Don't attempt to remember the current file position for source files that are read with follow-freq(0), e.g. /dev/klog and /proc/kmsg. These are special files which do not have the notion of file position, so no need to remember them. Regular files should always be read with follow-freq() set to nonzero, which is the default. * Fixed linking unit tests and other tools in mixed linking mode. * Fixed compilation on cygwin, especially lot of efforts went into linker compatibility. * Fixed building on platforms where PCRE is not in the standard include path. * Accept catch-all flag on log statements as well as catchall, as this was incorrectly documented in the past. db-parser() & pdbtool bugfixes: * Fixed @XX style message reference parsing used in correllation rules. * Fixed a segfault in the $(grep) template function when processing the parameters failed. * Fixed segfault in "pdbtool match --debug-pattern" in case the pattern doesn't match. * Fixed "pdbtool test" as previously all patterndb.xml files were reported to be invalid, even valid ones. * Fixed correllation timer related issue that caused some timers not to expire in rare cases. db-parser() & pdbtool changes: * Added support for enclosing template function arguments in parenthesis, in which case the quotes within the parentheses are not removed. For example: $(grep ('$FACILITY' == 'syslog')) This makes writing $(grep) and $(if) arguments much easier. * dbparser() the @NUMBER@ and @FLOAT@ parsers are able to parse negative numbers. * Added debug messages to dbparser() correllation so that it becomes easier to diagnose db-parser() problems. * Added -d (for --debug) and -v (for --verbose) options to pdbtool in order to make patterndb debugging easier. * Added --no-parse option to "pdbtool patternize" in order to read files without syslog-style parsing. Other changes: * Added error messages on DBI initialization failures. * Added systemd socket activation support. Build related changes: * Support for old (e.g. pre 7.1 commonly found in RHEL5) PCRE versions at the cost of an inoperating "newline" regexp flag. * configure now validates flex/bison versions better, as the requirements are more strict starting with syslog-ng 3.2 * Drop the creation of libsyslog-ng-patterndb.so. * "make clean" will properly remove libafsocket.so symlink. Credits: syslog-ng is developed as a community project. All changes and improvements requires effort, and this effort is really appreciated. Writing code, testing changes or simply providing use-cases and information on one's setup will make syslog-ng better. Here are the people, listed in no specific order who made this release possible. Steven Chamberlain Roger Paul Krizak (AMD) Attila Szalay (BalaBit) Matthew Hall Marius Tomaschewski (SUSE) Gergely Nagy (BalaBit) Tom Gundersen Dalibor Toman (Fortech.cz) Corinna Vinschen (RedHat) Balazs Scheidler (BalaBit) Laszlo Boszormenyi (LSC.hu) Arkadiusz Miśkiewicz (PLD Linux) Valentijn Sessink Hendrik Visage Peter Gyongyosi (BalaBit) Zoltan Pallagi (BalaBit) Mishou Michael (US IRS) Thanks for their efforts, it is appreciated. DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

1 0

[PATCH] basicfuncs: Implement a $(substr START LEN STR) template function.
by Gergely Nagy 01 May '11

01 May '11

This implements the $(substr) template function - a fairly primitive one: it does error checking, but there's no error reporting present, therefore invalid function calls will just get ignored. Usage should be straightforward. Reported-By: Jakub Jankowski <shasta(a)toxcorp.com> Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/basicfuncs/basic-funcs.c | 46 ++++++++++++++++++++++++++++++++++++++ 1 files changed, 46 insertions(+), 0 deletions(-) diff --git a/modules/basicfuncs/basic-funcs.c b/modules/basicfuncs/basic-funcs.c index 44b8af3..542c2ec 100644 --- a/modules/basicfuncs/basic-funcs.c +++ b/modules/basicfuncs/basic-funcs.c @@ -4,6 +4,9 @@ #include "filter-expr-parser.h" #include "cfg.h" +#include <stdlib.h> +#include <errno.h> + static void tf_echo(LogMessage *msg, gint argc, GString *argv[], GString *result) { @@ -19,6 +22,48 @@ tf_echo(LogMessage *msg, gint argc, GString *argv[], GString *result) TEMPLATE_FUNCTION_SIMPLE(tf_echo); +static gboolean +tf_substr_parse_int(const gchar *s, long *d) +{ + gchar *endptr; + glong val; + + errno = 0; + val = strtoll(s, &endptr, 10); + + if ((errno == ERANGE && (val == LONG_MAX || val == LONG_MIN)) + || (errno != 0 && val == 0)) + return FALSE; + + if (endptr == s || *endptr != '\0') + return FALSE; + + *d = val; + return TRUE; +} + +static void +tf_substr(LogMessage *msg, gint argc, GString *argv[], GString *result) +{ + glong start, len; + + if (argc != 3) + return; + + if (!tf_substr_parse_int (argv[0]->str, &start)) + return; + if (!tf_substr_parse_int (argv[1]->str, &len)) + return; + + if (start > argv[2]->len || + start + len >= argv[2]->len) + return; + + g_string_append_len (result, argv[2]->str + start - 1, len); +} + +TEMPLATE_FUNCTION_SIMPLE(tf_substr); + typedef struct _TFCondState { FilterExprNode *filter; @@ -158,6 +203,7 @@ static Plugin basicfuncs_plugins[] = TEMPLATE_FUNCTION_PLUGIN(tf_echo, "echo"), TEMPLATE_FUNCTION_PLUGIN(tf_grep, "grep"), TEMPLATE_FUNCTION_PLUGIN(tf_if, "if"), + TEMPLATE_FUNCTION_PLUGIN(tf_substr, "substr"), }; gboolean -- 1.7.2.5

3 8