- syslog-ng - lists.balabit.hu

mismatching size/inode information
by Gbe gbe 07 Oct '11

07 Oct '11

Hi everyone ! I have this message in my logs repeated several times and for different files : « syslog-ng[964]: The current log file has a mismatching size/inode information, restarting from the beginning; filename='/var/log/squid3/access.log' » This message comes from this part of the code in src/logreader.c : 644 if (cur_inode && 645 cur_inode == log_reader_get_inode(self) && 646 cur_size <= log_reader_get_size(self)) 647 { 648 /* ok, the stored state matches the current file */ 649 log_reader_update_pos(self, cur_pos); 650 return; 651 } 652 else 653 { 654 /* the stored state does not match the current file */ 655 msg_notice("The current log file has a mismatching size/inode information, restarting from the beginning", 656 evt_tag_str("filename", self->follow_filename), 657 NULL); 658 goto error; 659 } Does this mean I may have a hardware problem? Hard drive not working properly, file system damaged, … Google hasn't been really helpful so far. Greg.

2 4

syslog-ng 3.3.1 released
by Balazs Scheidler 06 Oct '11

06 Oct '11

Hi, I'm proud to announce that syslog-ng 3.3.1 has been released. It is an important release in a number of ways: - syslog-ng architecture was completely revamped to scale properly in today's multi-core environments. I've seen it process 800000 messages / second in some scenarios. - following the licensing change in 3.2, it incorporates contributions (code, testing & feedback) from 54 individuals. In total, the development of 3.3 took about a year, producing 580 patches and the following diffstat: 294 files changed, 20314 insertions(+), 7624 deletions(-) That compared to the total number of lines in the git repository (81804 lines) is a significant change over 3.2 See the syslog-ng ohloh page for more statistics: https://www.ohloh.net/p/syslog-ng For the list of features and changes, please see the NEWS entry at the end of this email. You can download the source code from: https://www.balabit.com/network-security/syslog-ng/opensource-logging-syste… A number of distributions already carry 3.3.1 packages thanks for our faithful package maintainers. You can also get the source code from git, at http://git.balabit.hu/ I'd like to thank all contributors who have helped to make syslog-ng to be what it is today (collected from the git commit summaries): Andras Mitzki <micek(a)balabit.hu> Andy Ruch <adruch2002(a)yahoo.com> Arkadiusz Miśkiewicz <arekm(a)maven.pl> Attila Nagy <naat(a)balabit.hu> Balazs Nemeth Balazs Scheidler <bazsi(a)balabit.hu> Balint Kovacs <balint.kovacs(a)balabit.hu> Brett Edgar <brett.edgar(a)truedigitalsecurity.com> Corinna Vinschen <vinschen(a)redhat.com> Costa Farber <costaf(a)wix.com> Craig Bell Dalibor Toman <dtoman(a)fortech.cz> Dave Reisner <dreisner(a)archlinux.org> Eric Berggren Eric Duda <ericduda(a)yahoo.com> Fischhof Gyorgy <george(a)balabit.hu> Gergely Nagy <algernon(a)balabit.hu> Jakub Jankowski <shasta(a)toxcorp.com> Jose Oliveira <jpo(a)di.uminho.pt> Juhasz Viktor <jviktor(a)balabit.hu> Kevin Guthrie <Kevin.Guthrie(a)itron.com> Laszlo Boszormenyi <gcs(a)lsc.hu> LEBRETON Philippe Lennart Poettering <lennart(a)poettering.net> Marius Tomaschewski <mt(a)suse.de> Martin Holste <mcholste(a)gmail.com> Marton Illes <marci(a)balabit.hu> Matthew Hall Matthew Hall <mhall(a)mhcomputing.net> Micah Anderson <micah(a)riseup.net> Michael <codejodler(a)gmx.ch> Mishou Michael <Michael.Mishou(a)csirc.irs.gov> Nat Baldwin Nix <nix(a)esperi.org.uk> Owen Mann Pallagi Zoltan <pzolee(a)balabit.hu> Paul Krizak <paul.krizak(a)amd.com> Peter "CzP" Czanik <czanik(a)balabit.hu> Peter Eisenlohr Peter Gyongyosi <gyp(a)balabit.hu> Peter Gyorko <gyorkop(a)balabit.hu> Remy Zandwijk <remy(a)luckyhands.nl> Roger <gaudet(a)verizon.net> Sandor Geller <Sandor.Geller(a)morganstanley.com> Serge Hallyn <serge.hallyn(a)ubuntu.com> Steven Chamberlain <steven(a)pyro.eu.org> SZALAY Attila <sasa(a)balabit.hu> Tamas Pal <folti(a)balabit.hu> Tevesz Andras <ghost(a)balabit.hu> Tom Gundersen <teg(a)jklm.no> Valentijn Sessink <valentyn(a)blub.net> Viktor Juhasz <jviktor(a)balabit.hu> Zhengxiang Pan Zoltan Pallagi <pzolee(a)balabit.hu> The list of changes are below (copied verbatim from the NEWS file in the source code): 3.3.1 Sat, 01 Oct 2011 13:17:56 +0200 This is the first stable release in the 3.3 series adding a number of features compared to 3.2: - multi-core/CPU scaling: the new multi-threaded architecture allows syslog-ng to scale into the 800k msg/sec region. - MongoDB support: using MongoDB instead of SQL is faster and allows better representation of log data. - JSON support: using the $(format-json) template function it is now possible to construct JSON (JavaScript Object Notation) output for log messages. JSON is a commonly used format for exchanging information between information systems, providing a rich and extensible format for structured information. - A number of enhancements all over the place: SQL, patterndb. The most important changes in this release: - The default ports have changed. syslog-ng is using the standard RFC allocated ports in the syslog() driver. - The meaning for log-iw-size() was changed for tcp() and unix-stream() drivers. That value is _evenly_ allocated to all possible connections starting with this release, whereas previously that window was shared between actual connections. Changes since 3.3.0beta2: system() source: ================ * Added support for Debian/kFreeBSD internal() source: ================== * Added support for the tags() option pdbtool test: ============= * Added support for testing a specific rule, instead of the complete patterndb file. * Added support for match debugging with the --debug and --color-out command line options. pdbtool dictionary: =================== * New pdbtool command that displays the name-value pairs that are set by any of the rules in a patterndb XML file. $(indent-multi-line): ===================== * This template function was added to make it possible to write multi-line log messages into a file. The first line is written like a regular message, subsequent lines are indented with a tab, just like RFC822. Other features & changes: ========================= * At stats-level(3) syslog-ng keeps track how much messages get tagged with individual tags. * Added support for pad_size() option for destinations, effectively padding the output string to a fixed size. This can be used on HP-UX to send messages to the local syslogd daemon, as that expects messages to be padded to 2048 bytes. * The command-line parser of the value-pairs functionality (used by $(format-json) for instance, was changed to use an empty scope by default. Bugfixes: ========= * Fixed spoof-source support as it didn't have proper thread synchronization in the previous beta versions, causing memory leaks and crashes if spoof-source was enabled on a destination and threading is used. * Fixed a MongoDB and SQL race conditions causing stalls and failed assertions. * Fixed a possible stall of syslog-ng caused by pipe() destinations without readers (e.g. /dev/xconsole). The root cause was a flipped flow-control state: pipes have become flow-controlled even if the user didn't request it. On the other hand files have become non-flow-controlled, even though they always should be. * Fixed possible crashes during reload or shutdown. * Fixes for several, significant memory leaks: - Fixed a major memory leak, effectively causing all messages to be leaked when message contents are changed on some of the log statements (with a rewrite rule or a parser). - Fixed a memory leak in file destinations caused by the leakage of the associated queue when the given file is closed (by time-reap or otherwise). - Fixed a memory leak in the server protocol handling code, leaking a small amount of memory every time a connection is closed. - Fixed a memory leak in the log source code causing some memory to be leaked for each connection when processing a reload. - PCRE matcher has leaked its state, causing a small leak on reloads. - The configuration parser contained leaks also causing leaks when parsing the configuration file (e.g. on reloads). - The persist state handling code has leaked some memory each time the persist state file was processed (startup + reloads). * Fixed thread synchronization when registering/deregistering stats counters. Previously these were performed without caring about threads, effectively the root cause for various ill effects from invalid counters to crashes. * Fixed a timestamp parsing problems affecting timestamps with month values between and including September .. December. * Fixed a timezone representation issue for zones that have half an hour offsets from GMT in the negative direction. * Fixed pdbtool patternize to generate proper UUIDs if openssl was detected. Previously it worked as if openssl was never detected. * Fixed a possible race on MARK message timing, which could cause MARK messages to appear somewhat inconsistently. * Fixed the generation of [meta sequenceId] SDATA field. * Fixed handling of zero-sized messages which have caused a crash due to NULL deref. * The processing of included directories was changed not to include hidden files. * Reloading syslog-ng didn't reinitialize the name resolver causing etc/resolv.conf changes not to be recognized even if syslog-ng was reloaded. A res_init() call was added to match earlier behaviour and UNIX best practice. * Fixed program-override() clash with the 'store-legacy-msghdr' reader flag, which has become default in syslog-ng 3.1. If storing the legacy msghdr is enabled, overwriting the value for $PROGRAM was not reflected in the output, since instead of reconstructing it from the parsed values, syslog-ng always used what was originally in the input. If the value for $PROGRAM changes, that automatically disables the use of the 'store-legacy-msghdr' flag. * Fixed CAP_SYSLOG detection to also detect if either the kernel, or libcap is lacking some required functionality. * Fixed the handling of messages generated by AIX, which include a "message forwarded for" header that syslog-ng failed to parse properly. * Fixed proper size limitation for the data structure used to hold name-value pairs of a log message. Previously, if this structure would go over 262140 bytes syslog-ng crashed because of an unhandled integer overflow. Build changes: ============== * It is possible to request the use of the system-installed libmongo-client instead of the bundled one. At least 0.1.3 is required. * The bundled libmongo-client was updated to 0.1.4, plus some minor patches (tagged in the git.balabit.hu git repo with syslog-ng-3.3-ref2) * autogen.sh: automatically check the availability of git before trying to pull the bundled libraries from git.balabit.hu * systemd unit file redirects syslog-ng startup messages to /dev/null without a better place. * Introduced a pkg-config file for syslog-ng.pc to make it easier to build 3rd party modules for syslog-ng. * The syslog-ng version number is added to libsyslog-ng.so, which goes to $prefix/lib to indicate that binary compatibility is not guaranteed between syslog-ng versions. * Added --without-compile-date option to remove the compilation date from the binary, as SUSE Linux checks if recompilation changes the binary to detect if dependent packages need to be rebuilt or not. * It is now possible to build against libsystemd-daemon instead of using the bundled sources. Other changes: ============== * The --seed command line option has become a no-op, syslog-ng tries to detect the availability of the random seed automatically. The command line option of the syslog-ng binary is present without doing anything, but it was removed from other utilities. Credits: ======== syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: * Attila Nagy (BalaBit) * Balazs Scheidler (BalaBit) * Balint Kovacs (BalaBit) * Costa Farber (wix.com) * Eric Duda * Gergely Nagy (BalaBit) * Jakub Jankowski (SuperHost.pl) * Kevin Guthrie (itron) * Lennart Poettering (RedHat) * Martin Holste * Marton Illes (BalaBit) * Peter Czanik (BalaBit) * Peter Eisenlohr * Peter Gyorko (BalaBit) * Sandor Geller (Morgan Stanley) * Tamas Pal (BalaBit) * Viktor Juhasz (BalaBit) -- Bazsi

2 1

[Bug 137] New: [debian #642479]: should print a warning at startup if not configured to log anything
by bugzilla＠bugzilla.balabit.com 06 Oct '11

06 Oct '11

https://bugzilla.balabit.com/show_bug.cgi?id=137 Summary: [debian #642479]: should print a warning at startup if not configured to log anything Product: syslog-ng Version: unspecified Platform: PC URL: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=642479 OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: algernon(a)balabit.hu Type of the Report: --- Estimated Hours: 0.0 The following report was filed in the Debian BTS against the syslog-ng package: So I was frustrated for a while by why "/var/log/messages", etc., all had a timestamp of a few months ago (which is when I last installed or reconfigured Syslog-NG on this host). The attached file does work; I'm pretty sure the problem was that I had a whole page of "log(...)" statements with no "source(...)" option and no "catchall" flag. If the start-script can prove by static analysis that it's impossible for any messages to match and be written to any destination, it should print a warning, not just start silently. In fact, a "log(...)" statement with no "source(...)" option and no "catchall" flag should probably trigger a warning. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 0

SQL support under cygwin?
by Fekete Robert 05 Oct '11

05 Oct '11

Hi, I assume that the SQL destination is not enabled in the cygwin package of syslog-ng. Is that correct? Someone is trying to use syslog-ng on Windows to log into a database, and I suspect that is why he fails (http://www.syslog.org/forum/index.php?topic=1224.0) Can anyone confirm that? Thank you in advance. Regards, Robert

2 2

Question about PE edition disk-based buffering
by Clayton Dukes 04 Oct '11

04 Oct '11

Hi Folks, We have a team doing testing on the PE edition and they are finding that the timeout for the failover trigger takes 60 seconds: It was observed that it takes anytime between 10 to 60 seconds to detect connection is broken. We found out it by running syslog-ng in debug mode. I guess it relies on connection timeout. This 60 second is large interval compared to the default interval that we have (15 sec). My question: Is there a way to tune this parameter down to 15 seconds in the config? ______________________________________________________________ Clayton Dukes ______________________________________________________________

2 2

RFC: An option to not warn when a module's not found.
by Gergely Nagy 04 Oct '11

04 Oct '11

Hi! Recently I stumbled upon an issue while packaging syslog-ng 3.3 modules for Debian: for convenience's sake, whenever a module package gets installed, it places a config file in /etc/syslog-ng/conf.d/, and all files from that directory are included into the main configuration. This makes it possible that one does not have to add explicit @module options to his config when he wants to use a module he installed: it's already there. This works well, and there's no problem here. Except that when one removes a module package, Debian leaves the config file on the system. They get removed only when the package is purged. This is understandable, and there's good reasons behind the distinction between remove and purge. However, this results in syslog-ng trying to load a removed module whenever it starts up or is reloaded, which emits a warning, which will result in a mail from cron after logrotates, among other things. So I'd like to introduce a command-line option that would result in these warnings NOT getting printed. So if a module is not found, even if it was explicitly loaded, with the command-line option set, syslog-ng wouldn't print a warning. This would mean I can use this flag in the init script and in the logrotate config aswell: if the module is loaded, but not used, there's no harm in ignoring its inexistence. If it would be used, syslog-ng will throw an error later anyway. I'll prepare a patch that adds this sometime this week, but I wanted to give a heads up, and ask for opinions too. If there's anyone who sees a problem with this approach, or has a better idea, please let me know! -- |8]

1 0

Syslog-ng under cygwin
by pedro castro 04 Oct '11

04 Oct '11

Does anyone know the step by step to compile syslog-ng under cygwin? I know that inside cygwin there is a syslog-ng package but it does not have spoof-source option enable so as documentation indicate I need to compile. any ideas? Thanks a lot

2 1

Error in documentation on syslog() source port
by Patrick H. 03 Oct '11

03 Oct '11

The syslog-ng admin guide for 3.2 says the default port for syslog() sources is 514. This isnt the observed behavior. When a syslog() source is configured without specifying the port, it listens on 601

3 4

syslog-ng Digest, Vol 73, Issue 37
by Kristen Eisenberg 02 Oct '11

02 Oct '11

Someone must have used a so-called "smart" editor to edit this file, since it seems to have used fancy quotes on the console line. #destination console_all { file("/dev/tty12"); }; destination console_all { file(“/dev/console”); }; #destination loghost { udp("loghost" port(999)); }; Notice how they are different from those on the line above? Word processors often replace quotes with matching pairs like that automatically; only plain text editors should be used to edit config files like this. Check for this kind of thing with a hex dumper or 'cat -vet': $ cat frank #destination console_all { file("/dev/tty12"); }; destination console_all { file("/dev/console"); }; #destination loghost { udp("loghost" port(999)); }; Kristen Eisenberg Billige Flüge Marketing GmbH Emanuelstr. 3, 10317 Berlin Deutschland Telefon: +49 (33) 5310967 Email: utebachmeier at gmail.com Site: http://flug.airego.de- Billige Flüge vergleichen

1 0

[PATCH] afmongodb: Support system-installed libmongo-client at configure time.
by Gergely Nagy 30 Sep '11

30 Sep '11

Modify configure.in to recognise a suitable version (>= 0.1.0) of the libmongo-client library, and use it instead of the internal version if requested so. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- configure.in | 10 ++++++---- 1 files changed, 6 insertions(+), 4 deletions(-) diff --git a/configure.in b/configure.in index e5bf174..e7038fb 100644 --- a/configure.in +++ b/configure.in @@ -28,6 +28,7 @@ IVYKIS_MIN_VERSION="0.18" JSON_C_MIN_VERSION="0.7" JSON_GLIB_MIN_VERSION="0.12" PCRE_MIN_VERSION="6.1" +LMC_MIN_VERSION="0.1.0" dnl *************************************************************************** dnl Initial setup @@ -743,12 +744,13 @@ if test "x$with_libmongo_client" = "xinternal"; then AC_MSG_WARN([Internal libmongo-client sources not found in modules/afmongodb/libmongo-client]) with_libmongo_client="no" fi -else - AC_MSG_WARN([Don't know how to locate system installed libmongo-client.]) - with_libmongo_client="no" +elif test "x$with_libmongo_client" = "xsystem"; then + PKG_CHECK_MODULES(LIBMONGO, libmongo-client >= $LMC_MIN_VERSION,,with_libmongo_client="no") fi - +if test "x$with_libmongo_client" = "xno"; then + enable_mongodb="no" +fi dnl *************************************************************************** dnl misc features to be enabled -- 1.7.2.5

4 14