May 2011 - syslog-ng - lists.balabit.hu

[Bug 106] New: Syslog-ng memory strdup and g_malloc leaks
by bugzilla＠bugzilla.balabit.com 23 Oct '11

23 Oct '11

https://bugzilla.balabit.com/show_bug.cgi?id=106 Summary: Syslog-ng memory strdup and g_malloc leaks Product: syslog-ng Version: 3.2.x Platform: Other OS/Version: Solaris Status: NEW Severity: minor Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: daniel_novak(a)kb.cz Type of the Report: refactorization Estimated Hours: 0.0 We tried to compile a few last git version of syslog-3.2.1(syslog-ng-3.2.git-5f25ce47ab57774f8fe8df17bc96a006e535cb53) as well as 3.2.1 stable release. Everything is running well, but there are several minor memory leaks.We run on Solaris 10/08 on sparc platform.With libumem.so in memory debug mode MDB output is following : > ::findleaks CACHE LEAKED BUFCTL CALLER 00052288 2 00110d98 libc.so.1`strdup+0xc 00052788 1 0007de00 libglib-2.0.so.0.2513.0`g_malloc0+0x64 ---------------------------------------------------------------------- Total 3 buffers, 64 bytes =========================================================================================== > ::findleaks -dvf findleaks: maximum buffers => 4121 findleaks: actual buffers => 3967 mdb: [fef80000, fef84000): couldn't read 16384 bytes at fef80000: no mapping for address findleaks: findleaks: potential pointers => 341574 findleaks: dismissals => 196071 (57.4%) findleaks: misses => 130325 (38.1%) findleaks: dups => 11214 ( 3.2%) findleaks: follows => 3964 ( 1.1%) findleaks: findleaks: elapsed wall time => 0 seconds findleaks: CACHE LEAKED BUFCTL CALLER 00052288 2 00110d98 libc.so.1`strdup+0xc 00052788 1 0007de00 libglib-2.0.so.0.2513.0`g_malloc0+0x64 ---------------------------------------------------------------------- Total 3 buffers, 64 bytes umem_alloc_16 leak: 2 buffers, 16 bytes each, 32 bytes total ADDR BUFADDR TIMESTAMP THREAD CACHE LASTLOG CONTENTS 110d98 1097e0 1b08cf3fb0c96c 1 52288 43af4 0 libumem.so.1`umem_cache_alloc+0x210 libumem.so.1`umem_alloc+0x60 libumem.so.1`malloc+0x28 libc.so.1`strdup+0xc libsyslog-ng.so.0.0.0`cfg_lexer_lookup_keyword+0xf0 libsyslog-ng.so.0.0.0`_cfg_lexer_lex+0x18a4 libsyslog-ng.so.0.0.0`cfg_lexer_lex+0x284 libsyslog-ng.so.0.0.0`main_lex+0x1c libsyslog-ng.so.0.0.0`main_parse+0x7a4 libsyslog-ng.so.0.0.0`cfg_parser_parse+0xcc libsyslog-ng.so.0.0.0`cfg_run_parser+0x1ac libsyslog-ng.so.0.0.0`cfg_read_config+0x98 initial_init+0x5c main+0x244 _start+0x5c umem_alloc_32 leak: 1 buffer, 32 bytes ADDR BUFADDR TIMESTAMP THREAD CACHE LASTLOG CONTENTS 7de00 77b98 1b08cf406e21b0 1 52788 373ec 0 libumem.so.1`umem_cache_alloc+0x210 libumem.so.1`umem_alloc+0x60 libumem.so.1`malloc+0x28 libumem.so.1`calloc+0x58 libglib-2.0.so.0.2513.0`g_malloc0+0x64 libsyslog-ng.so.0.0.0`serialize_file_archive_new+0x1c libsyslog-ng.so.0.0.0`persist_state_load+0x54 libsyslog-ng.so.0.0.0`persist_state_start+0x34 libsyslog-ng.so.0.0.0`cfg_initial_init+0x28 initial_init+0xbc main+0x244 _start+0x5c ========================================================================= findleaks: using cached results (-f will force a full run) CACHE LEAKED BUFCTL CALLER 00052288 2 00110d98 libc.so.1`strdup+0xc 00052788 1 0007de00 libglib-2.0.so.0.2513.0`g_malloc0+0x64 ---------------------------------------------------------------------- Total 3 buffers, 64 bytes > 00110d98::umem_verify mdb: cache 110d98 () does not have redzone checking enabled > 00052288::umem_verify Summary for cache 'umem_alloc_16' clean > 00052788::umem_verify Summary for cache 'umem_alloc_32' clean ============================================================================= -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 4

[Bug 104] New: Add libtool option -module
by bugzilla＠bugzilla.balabit.com 23 Oct '11

23 Oct '11

https://bugzilla.balabit.com/show_bug.cgi?id=104 Summary: Add libtool option -module Product: syslog-ng Version: 3.2.x Platform: All OS/Version: All Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: vinschen(a)redhat.com Type of the Report: bug Estimated Hours: 0.0 Created an attachment (id=30) --> (https://bugzilla.balabit.com/attachment.cgi?id=30) Patch to add libtool option -module Libtool 2.2.6 and later provides an option -module which should be used when creating shared libraries which are supposed to be used as modules which get loaded via dlopen. See the "Building modules to dlopen" chapter in `info libtool'. For Cygwin the additional benefit is that the modules are not copied into a parallel "bin" directory at installation time, which is the default otherwise to accommodate the DLL search algorithm on Windows. The attached patch just adds the -module option to each of the Makefile.am files in the modules subdir. Corinna -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

2 4

[Bug 120] New: afsql. c afsql_dd_format_* routines should include table in persist_name string
by bugzilla＠bugzilla.balabit.com 23 Oct '11

23 Oct '11

https://bugzilla.balabit.com/show_bug.cgi?id=120 Summary: afsql.c afsql_dd_format_* routines should include table in persist_name string Product: syslog-ng Version: 3.2.x Platform: Other URL: https://lists.balabit.hu/pipermail/syslog-ng/2011- February/015913.html OS/Version: Other Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: jtk(a)cymru.com Type of the Report: bug Estimated Hours: 0.0 Created an attachment (id=36) --> (https://bugzilla.balabit.com/attachment.cgi?id=36) include table name in persist_name As originally discovered in the syslog-ng list post entitled "[syslog-ng] duplicate config elements in sql destinations" by John Kristoff on 2011-02-11, syslog-ng will complain, when multiple destinations all point to the same database, but different tables with the message: Internal error, duplicate configuration elements refer to the same persistent config; name='afsql_dd(pgsql,127.0.0.1,,bar)' Balazs responded describing the potential problems with counters being swapped and a fix that appears to work in testing. A Patch for syslog-ng 3.2.4 is attached, but similar patch should work for earlier versions as indicated by Balazs' posting. Thank you, John -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3

[PATCH] afmongodb: Support system-installed libmongo-client at configure time.
by Gergely Nagy 30 Sep '11

30 Sep '11

Modify configure.in to recognise a suitable version (>= 0.1.0) of the libmongo-client library, and use it instead of the internal version if requested so. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- configure.in | 10 ++++++---- 1 files changed, 6 insertions(+), 4 deletions(-) diff --git a/configure.in b/configure.in index e5bf174..e7038fb 100644 --- a/configure.in +++ b/configure.in @@ -28,6 +28,7 @@ IVYKIS_MIN_VERSION="0.18" JSON_C_MIN_VERSION="0.7" JSON_GLIB_MIN_VERSION="0.12" PCRE_MIN_VERSION="6.1" +LMC_MIN_VERSION="0.1.0" dnl *************************************************************************** dnl Initial setup @@ -743,12 +744,13 @@ if test "x$with_libmongo_client" = "xinternal"; then AC_MSG_WARN([Internal libmongo-client sources not found in modules/afmongodb/libmongo-client]) with_libmongo_client="no" fi -else - AC_MSG_WARN([Don't know how to locate system installed libmongo-client.]) - with_libmongo_client="no" +elif test "x$with_libmongo_client" = "xsystem"; then + PKG_CHECK_MODULES(LIBMONGO, libmongo-client >= $LMC_MIN_VERSION,,with_libmongo_client="no") fi - +if test "x$with_libmongo_client" = "xno"; then + enable_mongodb="no" +fi dnl *************************************************************************** dnl misc features to be enabled -- 1.7.2.5

4 14

[Bug 14] New: Maintain symlinks to current and previous log output files...
by bugzilla＠bugzilla.balabit.com 28 Sep '11

28 Sep '11

https://bugzilla.balabit.com/show_bug.cgi?id=14 Summary: Maintain symlinks to current and previous log output files... Product: syslog-ng Version: 2.0.x Platform: PC URL: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=218688 OS/Version: Linux Status: NEW Severity: minor Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: sasa(a)debian.org Type of the Report: enhancement Estimated Hours: 0.0 >From bugs.debian.org: I'm hoping that upstream might consider adding functionality to syslog-ng to allow it to track current and previous log files and maintain symlinks to them, as a user option. This replicates one of the options that I (and others) find useful in cronolog. This has been previously discussed on the syslog-ng mailing list, though apparently nothing fully came of it. The archive of that discussion is at: https://lists.balabit.hu/pipermail/syslog-ng/2002-June/003452.html Obviously, it's probably somewhat of a low priority, but it would be nice. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

3.3b1: "EOF occurred while idle"
by Nix 22 Jul '11

22 Jul '11

I'm seeing a lot of these: May 29 18:59:48 fold err: POLLERR occurred while idle; fd='25' [...] May 29 19:04:13 fold err: EOF occurred while idle; fd='25' (It also just coredumped, but this led to an autoreboot and the log was not preserved across the reboot :( ) -- NULL && (void)

2 7

syslog-ng 3.3.0beta1 has been released
by devel＠balabit.hu 19 Jul '11

19 Jul '11

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.3.0beta1 SUMMARY : new stable release DATE : May 24, 2011 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (3.3.0beta1) has been released. For latest fixes in the 3.3.x branch you are recommended to upgrade to this version. CHANGES: 3.3.0beta1 Fri, 13 May 2011 13:17:12 +0200 Highlights: =========== This release marks the feature freeze for the upcoming 3.3 version of syslog-ng, development continues on the newly opened 3.4 branch, while this one only receives bugfixes in the future. db-parser(): ============ * db-parser() has a new option called inject-mode() that can be used to specify where synthetic messages generated by syslog-ng itself should be injected. In 3.2 these appeared in the internal() source, in 3.3 they'll be generated from within db-parser(). * "pdbtool patternize": added --no-parse option sql(): ====== * Make the number of retries runtime configurable in case an INSERT is rejected by the database using the new retry-sql-inserts() option. * Added support for using the "default" value for columns, by specifying "default" as the column value. This can be used for auto-incrementing fields. * Explicit commits were adapted to use the proper BEGIN TRANSACTION command on MS SQL and Oracle. * Oracle doesn't like overly long index names, and since this limit was easily reached by syslog-ng, use md5 to compress the index name. * Removed the defaults for columns(), values() and indexes() as it's almost impossible to come up with a set of defaults that works on all databases. * DBI initialization errors are reported earlier, during startup, to make their detection easier. file(): ======= * The owner(), group(), perm(), dir_owner(), dir_group() and dir_perm() can be specified without an argument which tells syslog-ng to avoid changing file permissions even if the globals are set. mongodb(): ========== * The name of the collection was changed to be a simple string, instead of a template, to make implementing bulk inserts much easier. * Added support for the value-pairs() option, that makes it trivial to add fields to records dynamically. $(format-json): =============== * A new module named tfjson was created which implements a $(format-json) template function, capable of exporting the syslog-ng message model as a properly marked up JSON object. * The plugin supports both json-c and json-glib as backend implementations. Performance changes: ==================== * The LogQueue component was restructured for better scalability. This component is responsible to connect source and destination drivers as they work in separate threads, and as such it plays an important role in influencing the overall performance of syslog-ng. Loggen: ======= * Added support for reading files in multi-threaded mode (each thread sending a copy of that file, instead of mixing lines). * Added support for IPv6. Other changes: ============== * The list of plugins loaded by default became build-time configurable using the --default-modules configure switch. The same command line option can be used to override the same at runtime. * --version now includes information on the list of available modules, and --module-registry displays even more information on every loadable module. * Introduced another plugin possibility, inner-source and inner-dest plugin types allow the extension any kind of source or destination driver. * Use the newly introduced "cap-syslog" capability with kernels post-2.6.38. Build changes: ============== * A new shared library was introduced named libsyslog-ng-crypto.so that contains all crypto related code shared between plugins. * Added pkg-config files to be used in -dev packages in order to make it possible to build external modules. * "make uninstall" now properly uninstalls config files and SCL. Note that it removes everything without checking that they were changed locally. Bugfixes: ========= * The program destination caused a segmentation fault during startup, this was fixed. * Fixed syslog() style framing over TLS or tcp, which caused the syslog-ng server to shut connections down prematurely. * Fixed syslog() client code, which may cause invalid framing to be emitted on contended network connections. * Fixed a possibly failed assert in the flow control code in reaction to a broken source connection. * Fixed processing flush_timeout() for destinations, in earlier alpha versions flush_timeout() never expired. * Some platforms return EINVAL for writev() calls with zero I/O buffers, causing file output to fail. Make sure syslog-ng never generates such calls. * Fixed read behaviour for special files (like /proc/kmsg on Linux, /dev/klog for FreeBSD), as epoll/kqueue is unable to handle these. * Fixed write behaviour for special files (like /dev/console) as epoll/kqueue must be used even though syslog-ng doesn't use those for regular files. * Fixed a premature connection closure when the destination is unable to accept data (e.g. EAGAIN is returned from send()). This was easily reproduced by enabling SSL, as SSL is reporting the same condition when the negotiation is running and the client is expecting the response from a server. * Fixed a 100% CPU usage case in the SSL server case. * Fixed a possible segmentation fault during runtime (use-after-free problem which could cause segfaults). * Fixed a segmentation fault at exit when the tcp() destination couldn't connect to the server. * Don't attempt to remember the file position for files that are read using follow-freq(0), e.g. /dev/klog, /proc/kmsg and similar files. * Fixed a db-parser() correllation bug, sometimes timers were not expired as they should have. [3.2 port] * Accept three forms of the catch-all log statement flag: catch-all, catchall and catch_all since the documentation was wrong and it differed from the actual implementation for ages. * PCRE: fixed a potential resource hog triggered by PCRE 8.12 and "global" replacements. * loggen: make sure loggen sends out all data by calling shutdown() on the output socket. * loggen: handle SIGPIPE signals which caused loggen to exit prematurely when connections broke. * loggen: fixed a potential division by zero when reporting the message rate for unsuccessful connections. * Fixed a number of compilation warnings triggered by gcc 4.6 Credits: ======== syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: * Martin Holste * Balazs Scheidler (BalaBit) * Gergely Nagy (BalaBit) * Andy Ruch * Peter Czanik (BalaBit) * Viktor Juhasz (BalaBit) * Attila Nagy (BalaBit) * Andras Miczki (BalaBit) * György Fischhof (BalaBit) * Tamás Pál (BalaBit) * Bálint Kovács (BalaBit) * Hendrik Visage * Peter Gyöngyösi (BalaBit) * Mishou Michael (IRS) * Matthew Hall * Sándor Gellér (Morgan Stanley) * Micah Anderson * Jose Oliveira (Fedora) * Serge Hallyn (Ubuntu) DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

2 1

[PATCH 0/N]: Various fixes from the syslog-ng team
by Gergely Nagy 14 Jul '11

14 Jul '11

Hi! A few selected patches from the syslog-ng team at BalaBit will follow this e-mail. This here mostly serves as an umberalla mail, under which I can shovel the rest, so they end up like a neat little thread. The patches I am about to send are available on the team/4.1-sprint5/fixes branch of my git tree. Special thanks to Viktor Juhasz <jviktor(a)balabit.hu> of the syslog-ng team for doing the bulk of the work, and assembling the list of patches that can be safely sent to the list. Another batch will be hitting the list a bit later, but unlike this first, which I expect to be safely appliable, the second batch will need a more through review. I'll comment on the whys when those patches are posted. I expect more patches from the team to follow, once the features and fixes are in a good enough shape. (In the meantime, I'll try to browse through their tree to see if any other bugfix can be isolated) -- |8]

3 23

get destination(s) based on log facility and severity
by Dejan Muhamedagic 08 Jun '11

08 Jun '11

Hi, We have a kind of machinery which collects syslog logs from several hosts. Currently it works by sending a message using logger(1) to the given facility.severity and then looking up the message in log files in /var/log and a few other directories (using grep). I'm looking for a more robust and faster way to discover the destination. Would that be possible with syslog-ng? And which is the optimal way to achieve that? An extra syslog-ng option? An extra binary? Or perhaps using logger(1)? Cheers, Dejan

3 10

[RFC] value-pairs and key rewriting
by Gergely Nagy 05 Jun '11

05 Jun '11

Hi! Now that value-pairs() is in 3.3, it's time to dig up an idea Bazsi and I were discussing way back when we first talked about value-pairs: a way to change the keys in a value-pairs set, without the need to explicitly specify them all using pair(). It's actually easier to explain this by explaining the need behind this feature: there's the MongoDB destination, and by default, SDATA goes under the "sdata" key, somewhat like this: { "sdata": { "test": "value" } } Now, if I'd rather have those values under, say "sd", I can't do that with the current driver, because I can't tell value-pairs() that "sdata" should be mapped to "sd" instead. The best I can do, is exclude ".SDATA.*", and either use "$SDATA", and post-process it, or list all the .SDATA.* keys explicitly. Neither of which is good enough. So, I propose that we should have a way to remedy this problem, and this remedy should be called "rekey()". The way I imagine it, is something like this: value-pairs ( scope("selected-macros" "nv-pairs") rekey( regexp("^\.SDATA\.(.*)" "sd.$1") prefix(".secevt.*" "events") prefix("[A-Z]*" "syslog.") ) ) This would do the following: - Any key that begins with ".SDATA." will have that part replaced with "sd." - Keys matching ".secevt.*" (shell glob, not regexp) will be prefixed with "events". Thus ".secevt.verdict" would become "events.secevt.verdict". - Keys that are all uppercase would be prefixed with "syslog.", thus "HOST" would become "syslog.HOST" The transformations would be applied to the raw set of keys, in the order they're listed in the configuration file. Initially, regexp() and prefix() would be implemented only, with the possibility of adding more, if the need arises. This would also solve another problem I encountered recently: if the value-pairs() result set contains both "$SDATA" and "$SDATA.*" (which is the case if one specifies scope("selected-macros" "nv-pairs") and the incoming message has structured data), then we'll have a key conflict in the MongoDB destination, because internally "foo.bar" gets translated to (using JSON notation): { "foo": { "bar": ... } } Now, in the case of SDATA, this translates to something like the following: { SDATA: "[foo=bar]", // $SDATA SDATA: { "foo": "bar" // $SDATA.foo } } This is because the MongoDB destination strips the leading dot at the moment (because that would be invalid too), and we end up with conflicting types: one string, and one object. The driver does not support overriding right now, so this is a problem. I could, of course, change the driver to replace the dot with an underscore, but that would be costier than the current stripping, and would still be ugly, in my opinion. It's much nicer to allow the users to rewrite the keys instead, or prefix them. That's about how far I got with thinking for now. Critique, comments and ideas would be most appreciated. (PS: This is, of course, strictly 3.4 material, as 3.3 is in a feature freeze) -- |8]

3 6