August 2011 - syslog-ng - lists.balabit.hu

snmptrapd to syslog-ng 3.1
by Smart, Dan 20 Aug '11

20 Aug '11

My interest is in network device syslog and traps. I'm trying to receive traps, and then process them in Simple Event Correlator (SEC). I've got SEC working fine with standard remote syslog. After reading everything I could find, I found a discussion from 2008 about losing the source hostname when sending the trap to syslog. I'm trying the source program method, and eliminating multi-line traps. As I understand that syslog-ng is looking for Standard Out from the program, I specified -f in snmptrapd to stop forking, and -Lo to send output to standard output. I'm getting nothing in my d_debug file. Any suggestions? There is also a web page with a filter and rewrite recipe for traps. Not sure why I need this if I am sending the trap directly to SEC. See https://lists.balabit.hu/pipermail/syslog-ng/2008-November/012200.html And http://bazsi.blogs.balabit.com/2008/11/syslog-ng-3-0-and-snmp-traps/ -=Dan=- ========= syslog-ng.conf ================= # options { long_hostnames(off); use_dns(yes); use_fqdn(no); keep_hostname(yes); owner("root"); group("adm"); perm(0640); stats_freq(0); bad_hostname("^gconfd$"); ts_format(iso); flush_lines(100); log_fetch_limit(100); log_fifo_size(2048); dir_perm(0755); }; source s_program { program("/usr/sbin/snmptrapd -a -f -Lo --disableAuthorization=yes", flags(no-multi-line)); }; destination d_debug { file("/var/log/syslog-ng-debug" owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes)); }; destination d_sec { program("/usr/local/bin/sec -input=\"-\" -conf=/usr/local/etc/sec.conf" flags(no-multi-line) ); }; log { source(s_program); destination(d_sec); destination(d_debug); flags(flow-control); };

2 6

Store syslog occurrence frequency instead of adding all of them to the DB
by Marcos Tang 19 Aug '11

19 Aug '11

Hi, I am using Syslog-NG which stores the Syslog to a remote MySQL server for central logging. Now, I am running into a situation a lot of records inside the MySQL database is "almost" exactly the same, just except the timestamp is different. For example, I get "file system is full" every minute for a host and this message repeats itself periodically. In order to better manage the MySQL database, do you have any suggestions how I can reduce the number of overall records inside the DB, but add the occurrence frequency instead? For example, syslog message "file system is full" occurs every minutes on the system. There will be 24 x 60 "file system is full" records on my MySQL DB after 1 day. If no one fix it, I will get 10 x 24 x 60 "file system is full" records for 10 hosts having the same problem for 1 day. Can those records being "processed" some how and when I search the MySQL DB, I only see ONE record list the following only? Total occurrenceMessage content ===================== 14,400File system is full If I can do that, it will definitely reduce a log of space of my DB and speed up the query. It seems this is a database problem as well. Please let me know your idea at anytime. Regards, Marcos

5 7

openSUSE 3.3beta2 packages, nice article, poll
by Peter Czanik 19 Aug '11

19 Aug '11

Hello, I just sent out the syslog-ng newsletter a few days ago, but there are already fresh news. But first of all: If you got 1 minute, please fill our syslog-ng performance survey at https://www.surveymonkey.com/s/6ZQDVH6 There was a new release of syslog-ng OSE 3.3 last weekend, it's now at beta2. FreeBSD ports was the first to add support for it, and now here are the openSUSE packages for testing: http://download.opensuse.org/repositories/home:/czanik:/syslog-ng33/ or if more interested in package sources, then check https://build.opensuse.org/project/show?project=home%3Aczanik%3Asyslog-ng33 where these packages were built. Systemd support is being reworked in openSUSE, so please stick to sysvinit (which is the default) when testing these packages. There is one known problem affecting functionality: mongodb support is broken. It's already fixed in git, so it will be part of the next release, or if there is a request for it, I can add the patch to the beta2 package. I just found a nice introductory article about syslog-ng at Linux Journal, which describes the basics and also some more interesting topics and provides good explanations. You can read it at http://www.linuxjournal.com/content/creating-centralized-syslog-server It was written on Debian, but most information should apply to any Linux distribution. Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

2 1

mongodb stats problem
by Peter Czanik 17 Aug '11

17 Aug '11

Hello, I just ran into this problem while checking Google Alerts about syslog-ng: http://www.quora.com/Error-starting-syslog-ng-with-mongo-module Is it addressed with the latest patch in git, or it's something new? Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

2 1

syslog-ng Insider - August 2011
by Peter Czanik 17 Aug '11

17 Aug '11

Dear syslog-ng users, This is the 5th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com <mailto:documentation@balabit.com> FEATURED NEWS Please test 3.3 beta2! -------------------- Version 3.3 is a major step in syslog-ng's development, as it is now fully multi-threaded, and is also the first release with considerable amount of community developed features. A new beta was released this weekend! It received many changes and fixes since the beta1 release. According to internal testing, it works nice and stable, but we already received some external problem reports. So your feedback is very valuable, especially if you could test it in real world situations with logs and configurations we could never imagine ourselves. As threading is a major new feature, which is not enabled by default, please try it by adding “threaded(yes)” to your options in syslog-ng.conf Please download sources from http://www.balabit.com/downloads/files?path=/syslog-ng/sources/3.3.0beta2 ! For a list of changes and fixes check the announcment at http://lists.balabit.hu/pipermail/syslog-ng-announce/2011-August/000117.html If you use FreeBSD, ports is already updated to beta2: http://www.freshports.org/sysutils/syslog-ng3-devel/ Documentation is also available: http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guid… Windows in focus ---------------- The syslog-ng application is often used in a Windows environment. There is now an ongoing blog series about syslog clients for Windows. BalaBit also released some patterns for Windows. The patterns are available from http://czanik.blogs.balabit.com/2011/07/patterns-for-windows-server-2008/ Blogs about syslog clients: http://czanik.blogs.balabit.com/2011/07/eventlog-to-syslog/ http://czanik.blogs.balabit.com/2011/07/snare/ http://czanik.blogs.balabit.com/2011/07/some-more-applications-to-forward-w… To be continued... Performance ----------- Both syslog-ng OSE and PE are now multi threaded, which brings performance to extreme. On the test machine HDD and multiple gigabit Ethernet lines were the limiting factor, not syslog-ng. The test were done using PE, but OSE performance should be similar: http://pzolee.blogs.balabit.com/2011/07/do-you-want-to-process-800-000-mess… BalaBit call the features and technologies aimed compliance at a high performance HSRL (High Spead Reliable Logging). POLL Please take a minute to answer three syslog-ng related questions at https://www.surveymonkey.com/s/6ZQDVH6 OTHER SHORT NEWS * A few more words on CVE-2011-1951: http://bazsi.blogs.balabit.com/2011/07/on-cve-2011-1951-bug-or-security-iss… * Logs as a movie: http://algernon.blogs.balabit.com/2011/06/twelve-days-of-log/ * ELSA updated: http://ossectools.blogspot.com/2011/07/elsa-vmware-appliance-available.html NEW RELEASES * syslog-ng OSE 3.3 beta2 was released: http://lists.balabit.hu/pipermail/syslog-ng-announce/2011-August/000117.html ARCHIVE http://insider.blogs.balabit.com/ -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

internal() messages looping back on PostgreSQL restart
by Aaron 17 Aug '11

17 Aug '11

I have syslog-ng logging to a remote PostgreSQL database and to a local log file as well. If at any point the Postgres database restarts and syslog-ng happens to attempt to log to the database for the brief period that Postgres is unavailable, my local messages are flooded with internal() looping back messages similar to this: syslog-ng[31587]: syslog-ng internal() messages are looping back, preventing loop by suppressing further messages; recurse_count='3' syslog-ng[31587]: syslog-ng internal() messages are looping back, preventing loop by suppressing further messages; recurse_count='3' syslog-ng[31587]: syslog-ng internal() messages are looping back, preventing loop by suppressing further messages; recurse_count='2' My guess is that syslog-ng panics when it is unable to connect to the PostgreSQL database. The problem is that syslog-ng never re-establishes the connection, and writes several thousands of the above messages to the local log file each second, which can become a little problematic resource-wise. In order to make syslog-ng behave again, I need to manually restart the service, otherwise it would continue to flood my local messages and never reconnect to the database. I thought it might be a problem with my configuration, and I tried using the time_reopen() option but that seemed to have no effect. Is this perhaps a bug? Syslog-ng version: 3.2.4

2 1

[PATCH] afmongodb: Add stats locking.
by Gergely Nagy 17 Aug '11

17 Aug '11

Lock / unlock the stats when registering the counters. Reported-by: Costa Farber <costaf(a)wix.com> Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/afmongodb/afmongodb.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/modules/afmongodb/afmongodb.c b/modules/afmongodb/afmongodb.c index f82f10e..9e155ed 100644 --- a/modules/afmongodb/afmongodb.c +++ b/modules/afmongodb/afmongodb.c @@ -439,12 +439,14 @@ afmongodb_dd_init(LogPipe *s) self->queue = log_dest_driver_acquire_queue(&self->super, afmongodb_dd_format_persist_name(self)); + stats_lock(); stats_register_counter(0, SCS_MONGODB | SCS_DESTINATION, self->super.super.id, afmongodb_dd_format_stats_instance(self), SC_TYPE_STORED, &self->stored_messages); stats_register_counter(0, SCS_MONGODB | SCS_DESTINATION, self->super.super.id, afmongodb_dd_format_stats_instance(self), SC_TYPE_DROPPED, &self->dropped_messages); + stats_unlock(); log_queue_set_counters(self->queue, self->stored_messages, self->dropped_messages); afmongodb_dd_start_thread(self); -- 1.7.0.4

2 1

Frontend viewing logfiles
by Volker Lieder 17 Aug '11

17 Aug '11

Hi there, we have installed syslog-ng-pe with file encryption. Is there a webfrontend where we can search through the encrypted logfiles? e.g i have to set the cert-key-file in a config and then i can search the logfiles via webfrontend. Any ideas? Regards, Volker Lieder -- Mit freundlichen Grüßen/Best regards Volker Lieder System Administrator mail volker(a)its-lieder.de ITS-Lieder Kleinlindenerstrasse 3 35398 Giessen / Germany http://www.its-lieder.de UST-ID: DE260836372 ========================================================== Jegliche Stellungnahmen und Meinungen dieser E-Mail sind alleine die des Autors und nicht notwendigerweise die der Firma. Falls erforderlich, können Sie eine gesonderte schriftliche Bestätigung anfordern. Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the company. If verification is required please request a hard-copy version.

3 4

syslog-ng 3.3.0beta2 has been released
by devel＠balabit.hu 17 Aug '11

17 Aug '11

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.3.0beta2 SUMMARY : new stable release DATE : Aug 15, 2011 ------------------------------------------------------------------------------ DESCRIPTION: A new version of syslog-ng Open Source Edition (3.3.0beta2) has been released. CHANGES: 3.3.0beta2 Fri, 12 Aug 2011 09:00:17 +0200 Highlights ========== This release has started using the officially assigned port numbers for RFC5424 based log transports. syslog-ng will properly use the old defaults if the configuration @version indicates an old config. This release contains all identified bug fixes published by the BalaBit syslog-ng team in their syslog-ng PE 4.1 release, additional features will be integrated into syslog-ng 3.4 as 3.3 is already feature-frozen. See more details on the list of bugs fixed in the "bugfixes" section below. This release got a "beta" tag instead of a "release-candidate" because of the level of public testing it has received. However all currently known bugs are believed to be fixed and this should be much better than beta1 was. Bugfixes: ========= * Fixed a memory leak in macro based destination files. * Fixed the "dropped" and "stored" counters for the SQL destination, which were swapped previously. * Fixed file sources to properly start the source file from the beginning when restoring the persistent state failed. * Fixed BSD timestamp parsing, that caused an extra syslog header to be added to messages which had a single digit day-of-the-month (e.g. the first ten days for every month). * Fixed a possible crasher at exit, caused by the destinations being flushed and the associated sources already cleaned up. * Fixed a destination specific memory leak, which caused some hundred bytes to be leaked when the configuration was reloaded. * Set O_CLOEXEC for child program pipes, in order to avoid inheriting those to other children. * Allow RFC5424 structured data blocks to have no parameters (e.g. "[foo]", which only has an SDID and no SDPARAM). * Fixed the value of the $PRI macro which expanded to an empty string if the priority was 0 (= kern.emerg) * Fixed a race conditions in template compilation and expansion, that could cause segmentation faults and other nastiness. * Fixed a race condition in macro based file destinations, which caused crashes when multiple sources wrote into the same file, when the filename expanded to the same name from the multiple connections. (e.g. using $PROGRAM in the filename and receiving messages from different hosts running the same programs). * Fixed a possible segmentation fault in certificate validation, when the peer certificate contains an X.509 certificate purpose extension. * Try to load modules with the .a extension on AIX. * Fixed a possible crash in the socket source with optional(yes). * Made stats counters and their registration thread safe, which either caused crashes if stats_level() > 1 or miscalculated values due to parallel threads accessing the same counters. * Fixed a possible crash for unknown or mistyped source flags. * Fixed an fd and memory leak for opened destination connections. * Fixed possible crashes in SIGHUP handling in case syslog-ng is busy processing messages. * Fixed a failed assertion in persistent-state handling, triggered when multiple file sources are busy processing messages and recording their current file position. * Fixed SCL block expansion, when passing the empty value for a parameter. * The final flush that happens right before syslog-ng exits honoured the throttle() value. Since this is an attempt to squeeze out everything we have to the destination, it's best to ignore throttle in this case. * Fixed a possible use-after-free and other crash causing problems in file macro based destinations. * Fixed a massive memory leak of all processed messages, which were triggered by a change past v3.3.0beta1, but was found in a git snapshot. * Fixed keep-alive(yes) behaviour for destination drivers. * Messages generated by the db-parser() will be marked as local messages. * Implemented locking callbacks for OpenSSL, without which enabling SSL and threading mode almost guaranteed a crash in case multiple connections were running SSL. * Fixed an issue with the SQL destination, which can cause stalls in message processing. Build changes: ============== * Added --with-module-path configure option that specifies the default search path for modules. The help for the older --with-module-dir option was clarified, it indicates where syslog-ng Makefiles install modules. * The configure script automatically adds -pthread to the CC command line. Hopefully this is indeed supported on all platforms (worked on the ones I've tried), and avoids having to play dirty tricks with macros like _REENTRANT and _THREAD_SAFE. * Link unit test programs against -livykis in case we're using a system-installed ivykis, rather than our private copy. * Fixed compile issues on Solaris. * Updated to a newer libmongo-client (v0.1.3) * Updated to a newer ivykis version (v0.19 with syslog-ng specific patches) Performance improvements: ========================= * Use the GSlice allocator for objects allocated in the fast path. * Avoid another use of sprintf in the LogWriter fastpath. * Use per-thread caches for time related functions and the DNS cache. * Use the string representation of IP addresses in the DNS cache, as inet_ntop() performs much worse than the cache. Other changes: ============== * Functional tests: properly detect the presence of sqlite to do the sql() testing. * The minimum permissible value for the log-iw-size was changed to 100 from 10, as the per-connection window calculation has changed. Earlier all connections shared the same window, starting with 3.3, the window is distributed evenly to all connections, to avoid starving (e.g. one connection starves the rest by using up the window space) and to avoid thread synchronization in the message processing fast path. * Ported per-facility, per-severity, per-program and per-host counters from the PE version. * Other systemd unit files. Credits: ======== syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: * Dave Reisner (ArchLinux) * Nix (esperi.org.uk) * Hendrik Visage * Viacheslav Biriukov * Andras Mitzki (BalaBit) * Attila Nagy (BalaBit) * Gyorgy Fischhof (BalaBit) * Viktor Juhasz (BalaBit) * Balazs Scheidler (BalaBit) * Peter Czanik (BalaBit) * Gergely Nagy (BalaBit) * Peter Gyorko (BalaBit) * Tamas Pal (BalaBit) * Zoltan Pallagi (BalaBit) * Robert Fekete (BalaBit) DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

3 2

Symantec SCSP & Syslog-ng
by Hendrik Visage 16 Aug '11

16 Aug '11

Hi there, Anybody else using this combination? (Perhaps even on HP/UX??) Hendrik

1 0