September 2013 - syslog-ng - lists.balabit.hu

[Bug 169] New: Feature Request - reconnect network also resolve dns name
by bugzilla＠bugzilla.balabit.com 10 Oct '13

10 Oct '13

https://bugzilla.balabit.com/show_bug.cgi?id=169 Summary: Feature Request - reconnect network also resolve dns name Product: syslog-ng Version: 3.3.x Platform: Other OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 When a network connection is dropped the destination name should be resolved via DNS prior to the reconnect. Every couple of years to renew the hardware supporting our core syslog service we move the IP name to a new host. The desired process is; 1. Build a new host and ensure that syslog-ng is running correctly. 2. Change the DNS for our central syslog server to the address of the new server. 3. Restart the syslog-ng service on the old syslog server. 4. All of the clients will loose the connection to the old syslog server. 5. All of the clients will reconnect to the *IP Name* in their configuration files. 6. Monitor the old syslog server to determine any hosts that have not made the transition. What really happened in step 5 was that all of the client reconnected to the old syslog server :-( To accomplish the switch over I had to visit 450 servers and manually restart the syslog service on the client. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5

[Bug 252] New: Syslog-NG crashes with destination using unix-dgram() driver
by bugzilla＠bugzilla.balabit.com 09 Oct '13

09 Oct '13

https://bugzilla.balabit.com/show_bug.cgi?id=252 Summary: Syslog-NG crashes with destination using unix-dgram() driver Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: marc.falzon(a)cloudwatt.com Type of the Report: bug Estimated Hours: 0.0 Created an attachment (id=82) --> (https://bugzilla.balabit.com/attachment.cgi?id=82) syslog-ng.conf Hello I'm facing a Syslog-NG crash when trying to use a unix-dgram() driver as log destination (configuration used is attached to this issue): # syslog-ng -Fev Module loaded and initialized successfully; module='system-source' Module loaded and initialized successfully; module='afsocket' Module loaded and initialized successfully; module='affile' Module loaded and initialized successfully; module='afuser' Module loaded and initialized successfully; module='syslogformat' Segmentation fault # Service starts successfully if the log path using the d_local destination is commented. Version used (platform: Linux Ubuntu Precise 12.04): # syslog-ng -V syslog-ng 3.4.2 Installer-Version: 3.4.2 Revision: 3.4.2-1~mhp1.sysvinit~precise [@3b56ec1] (Ubuntu/12.04) Compile-Date: Jun 4 2013 16:52:14 Available-Modules: syslogformat,afprog,basicfuncs,confgen,afsocket-notls,afsocket,afsocket-tls,dbparser,cryptofuncs,csvparser,affile,afuser,system-source Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: on # uname -a Linux p-xxxxxx-0000 3.2.0-51-generic #77-Ubuntu SMP Wed Jul 24 20:18:19 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux # Let me know if you need any more information about my setup and/or configuration. Regards, m. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 12

Fwd: Syslog NG simple filtering not working as expected...
by Paul Hutton 02 Oct '13

02 Oct '13

Initially sent to the wrong address.... Begin forwarded message: > From: Paul Hutton <paul_hutton(a)bigpond.com> > Subject: Syslog NG simple filtering not working as expected... > Date: 26 September 2013 11:23:15 PM AEST > To: syslog-ng-request(a)lists.balabit.hu > > Hi All, > > I am trying to forward network sourced syslogs to a number of destinations depending on a few simple filters, but am immediately coming unstuck. My constraint is that all incoming and outgoing network syslog traffic is udp. My intention in the configuration code below (apart from logging the localhost) is to: > > 1. Forward all incoming network sourced syslogs except those generated by two firewalls to destination Remote_L > 2. Forward incoming network syslogs that are generated by the firewalls to destination Remote_C (currently commented out) > 3. Forward all incoming network syslogs of severity 0, 1 and 2 to destinations Remote_D1 and Remote_D2 (currently commented out) > > In addition all incoming network sourced syslogs are logged locally to destination d_logger and I've added another local file d_debug out of curiosity. > > My problem is that with just the filter enabled at point 1 above and in the code below, the behaviour of syslog-ng is very erratic, beyond any reasonable anticipation of the unreliability of udp as the transport. If I comment out the filter line > > filter (not_firewalls); > > in the code below, then test traffic (none of which is generated by the firewall sources) is received and forwarded correctly as well as logged locally. If I uncomment the filter, then I have seen test traffic forwarded correctly on one occasion, but it is mostly the case that some smaller subset (including none) of the test traffic is forwarded. In this case, the test traffic is also correctly written to the local destination d_debug. That is in the same rule that fails to forward the test traffic. > > I am running 3.2.4 on Solaris 10 09/10 x86 obtained from the Sunfreeware site. > > I have also unashamedly plagiarised configuration code from a couple of posted sources, so if you recognise your work, thanks for putting it out there... > > > Thanks in advance. > > Paul > > > > @version: 3.2 > @include "scl.conf" > options { > stats_level(3); > flush_lines(100); > flush_timeout(10000); > }; > ################################################################ > # Ensure syslog-ng logs this host similarly to syslogd # > ################################################################ > source s_local { > sun-streams("/dev/log" door("/etc/.syslog_door")); > internal(); > }; > # > # Set global network source > # > source src { > udp(ip("0.0.0.0") port (514)); > }; > > ############################################################### > # Local logfiles and remote destinations setup # > ############################################################### > destination syslog { > file("/var/log/syslog"); > }; > destination messages { > file("/var/adm/messages"); > }; > destination loginlog { > file("/var/adm/loginlog"); > }; > # > # Set remote destinations > # > destination Remote_L { > udp ("10.octet.octet.octetL" port (514) ); > }; > destination Remote_D1 { > udp ("10.octet.octet.octetD1" port (514) spoof_source (yes)); > }; > destination Remote_D2 { > udp ("10.octet.octet.octetD2" port (514) spoof_source (yes)); > }; > destination Remote_C { > udp ("10.octet.octet.octetC" port (514) spoof_source (yes)); > }; > # > # Set local destination for all remote syslog input > # > destination d_logger { > file("/var/log/logger.log"); > }; > # > # Add another local debug file to check filters are working > # > destination d_debug { > file("/var/log/debug.log"); > }; > ############################################################## > # Filters for mimicking Solaris syslogd # > # identifying firewall syslogs # > # identifying severity levels 0 to 2 # > ############################################################## > filter f_syslog { > facility(mail) or (facility(daemon) and priority(notice)); > }; > filter f_messages { > priority(err) > or facility(kern) > or (facility(user) and priority(err)) > or (facility(daemon) and priority(notice)) > or (facility(mail) and priority(crit)); > }; > filter f_auth { > facility (auth); > }; > > # > # Set Firewall Filters > # > filter firewalls { > host("10.octet.octet.octetfw1") or host("10.octet.octet.octetfw2"); > }; > filter not_firewalls { > not (host("10.octet.octet.octetfw1") or host("10.octet.octet.octetfw2")); > }; > > # > # Filter Severity Levels 0, 1 and 2 > # > filter f_crit { > level (crit .. emerg); > }; > ############################################################## > # Now do some logging # > ############################################################## > > # > # Log local syslog data appropriately > # > log { > source(s_local); > filter(f_syslog); > destination(syslog); > }; > log { > source(s_local); > filter(f_messages); > destination(messages); > }; > log { > source(s_local); > filter(f_auth); > destination(loginlog); > }; > > # > # Log remotely sourced syslogs of all Severity Levels to remote L system > # but exclude syslogs from the firewalls > # > log { > source (src); > filter (not_firewalls); > destination (Remote_L); > destination (d_debug); > }; > # > # Log remotely sourced syslogs of Severity Levels 0 to 2 to remote D systems > # including syslogs from the firewalls > # > #log { > # source (src); > # filter (f_crit); > # destination (Remote_D1); > # destination (Remote_D2); > #}; > # > # Log remotely sourced syslogs from the two firewalls to remote C system > # > #log { > #source (src); > #filter (firewalls); > # destination (Remote_C); > #}; > # > # Log remotely sourced syslogs to a local file > # > log { > source (src); > destination (d_logger); > }; > > >

4 5

[Bug 241] New: pdbtool test --validate returns success when matching wrong pattern
by bugzilla＠bugzilla.balabit.com 02 Oct '13

02 Oct '13

https://bugzilla.balabit.com/show_bug.cgi?id=241 Summary: pdbtool test --validate returns success when matching wrong pattern Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 pdbtool test --validate pattern.xml only returns a non-zero exit code if test values don't match. If there is a catchall pattern in the pattern.xml database file then any test message that does not match the pattern it belongs to, but does match another pattern will return a zero exit code. The output still shows the wrong .classifier.rule_id Testing message program='mailguard' message='2013-07-03T13:13:38-07:00 local(a)dog.comp.uvic.ca local4.notice mailguard: Could not open /var/mimedefang/sophie.pid' Wrong match name='.classifier.rule_id', value='UN-eb0cfada-af50-48b0-ac1e-de1ffad8fe6c', expected='b826184c-2990-46d4-a777-f87340f1796a' but the return code from the pdbtool is still zero. I would expect any test message that does not match the pattern it belongs to should produce a non-zero return code. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

4 8

syslog-ng leaves a lot of open file handles
by Anton Koldaev 28 Sep '13

28 Sep '13

Hi, I'm using Syslog-NG OSE v.3.3.7-1~mhp1~lucid (Ubuntu Lucid) And I have the following destination file(): file("/u/logs/`app`/${MONTH}${DAY}/${1}/${1}${2}/${LOGSORT.ACCOUNT}.log" Syslog-NG switches to the new file at 23:59:59 every day just fine but for some reason it leaves files for the previous day open: *# date* Wed Jul 24 09:04:19 UTC 2013 *# lsof | grep a/ac/account.log* syslog-ng 30743 root 3351w REG 252,2 72597491 66306075 /u/logs/app/0723/a/ac/account.log (deleted) syslog-ng 30743 root 4896w REG 252,2 17017519 4572052 /u/logs/app/0724/a/ac/account.log And they're being deleted by my rotating script. Reloading syslog-ng using init script or with `kill -HUP` doesn't help - all deleted files are still open by syslog-ng. Global option "time_reap (30);" doesn't seem to help too. Any ideas? -- Best regards, Koldaev Anton

3 6

3.5beta1 openSUSE packages
by Peter Czanik 27 Sep '13

27 Sep '13

Hello, My first experimental and mostly untested beta1 packages for openSUSE are available now at https://build.opensuse.org/package/show/home:czanik:syslog-ng35/syslog-ng More testing and SLES packages will follow next week. Have a nice weekend! Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 2

syslog-ng 3.5.0beta2 has been released
by devel＠balabit.hu 27 Sep '13

27 Sep '13

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.5.0beta2 SUMMARY : new beta release DATE : Sep 27, 2013 ------------------------------------------------------------------------------ DESCRIPTION: A new beta version of syslog-ng Open Source Edition (3.5.0beta2) has been released. Being a beta release, testing is most appreciated, but running it in production is not advised. CHANGES: 3.5.0beta2 Fri, 27 Sep 2013 10:06:52 +0200 This is the second BETA release of the syslog-ng 3.5 development branch, introducing new features and a handful of fixes since the last beta version. While running in production is still not advised, testing is most appreciated. Major Features ============== Redis destination ----------------- Developed during the Google Summer of Code 2013 program, this destination driver makes it possible to easily send commands to a Redis server. For further information and examples, please see the following blog post: http://tichygsoc.blogspot.hu/2013/09/the-road-so-far.html Template type hinting --------------------- In 3.5.0beta1, type hinting was implemented for the MongoDB destination, with this release, it is also available for the $(format-json) template function. Template options honored everywhere ----------------------------------- Until this release, there were situations where template options were ignored, such as filter expressions that use the comparsion operators, regexp based substitutions, incoming templates for parsers, the new value rewrite rules, SMTP values, some of the new drivers (stomp, amqp), and patterndb context-id. These all honor the global template options now, and per-driver options such as frac-digits and local-time-zone are available for drivers which did not have them before, like MongoDB. Bugfixes ======== * The AMQP destination has been fixed, and using it does not lead to a crash anymore. * Several memory leaks in the STOMP destination have been fixed. * The --no-framing option of loggen has been fixed, and works as advertised now. * The in-list() filter ignores empty lines in its input file, to avoid matching on empty values. * Compilation on FreeBSD has been fixed. Credits: ======== syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Balazs Scheidler <bazsi(a)balabit.hu> Gergely Nagy <algernon(a)balabit.hu> Peter Czanik <czanik(a)balabit.hu> Tibor Benke <btibi(a)balabit.hu> Tihamer Petrovics <tihameri(a)gmail.com> Viktor Tusa <tusa(a)balabit.hu> DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

1 0

syslog-ng: redis destination, the road so far
by Tihamér Petrovics 25 Sep '13

25 Sep '13

Hello, I updated my blog with the following post: http://tichygsoc.blogspot.hu/2013/09/the-road-so-far.html I wrote a tutorial about using syslog-ng with redis, attached some sample config file and finally a complex one, that could be a case from everyday. Stay tuned, I intend to maintain the code after gsoc too! All the best, Tihi

1 0

insider 2013-09: syslog-ng 3.5 beta release; incubator project; GSoC updates;
by Peter Czanik 19 Sep '13

19 Sep '13

Dear syslog-ng users, This is the 27th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. FEATURED NEWS syslog-ng 3.5 beta1 is released ------------------------------- The first beta of syslog-ng 3.5 was released today. It has many internal rework and also many user visible new features. Here you can read just the headlines, for a complete list and details please visit the NEWS file ( https://github.com/balabit/syslog-ng-3.5/blob/master/NEWS ) or this blog post: http://asylum.madhouse-project.org/blog/2013/09/05/what-is-cooking-in-syslo… * Stomp destination * Multiline support * Blacklist support * and many more As usual, testing is greatly appreciated as it helps to iron out problems before the final release! Sources are available from git ( https://github.com/balabit/syslog-ng-3.5 ) or as tgz from http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… For third-party binary packages for various Linux and UNIX platforms, visithttp://www.balabit.com/network-security/syslog-ng/opensource-logging-s… .Debian/Ubuntu and openSUSE packages are expected to be available in the coming days. syslog-ng incubator project --------------------------- The syslog-ng module incubator (Incubator henceforth) is a collection of tools and modules for syslog-ng that for one reason or the other, are not part of the official repository. This serves both as a staging ground for experimental modules, and as a repository of plugins that are not aimed at upstream inclusion. It's also an example of a third party syslog-ng module. Sources are available at https://github.com/algernon/syslog-ng-incubator syslog-ng GSoC: finished successfully! -------------------------------------- The coding time for Google Summer of Code isover and both of our students finished their projects successfully. The redis destination is tomerged to syslog-ng 3.5 before the beta2 releaseand the high performance mysql destination is expected to follow it soon.A summary blog post of their work is due to appear next week at http://petrovicsgyula.blogspot.com/ and http://tichygsoc.blogspot.hu/ syslog-ng 3.5 draft documentation --------------------------------- A draft version of syslog-ng 3.5 documentation was also released today. It contains some of the new features. It is available in many forms: * PDF: http://www.balabit.com/support/documentation/syslog-ng-ose-3.5-guides/en/sy… * HTML: http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.5-guid… * Single page HTML: http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.5-guid… SHORT NEWS: * syslogng-web, a simple and fast log viewer: https://czanik.blogs.balabit.com/2013/08/syslog-ng-web-a-simple-and-fast-lo… * how to monitor a directory of log files: http://asylum.madhouse-project.org/blog/2013/09/10/howto-monitor-a-director… NEW RELEASES * syslog-ng 3.5beta1: https://lists.balabit.hu/pipermail/syslog-ng-announce/2013-September/000166… Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com <mailto:documentation@balabit.com> To read this newsletter on-line, visit: http://insider.blogs.balabit.com/ -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

syslog-ng 3.5.0beta1 has been released
by devel＠balabit.hu 19 Sep '13

19 Sep '13

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.5.0beta1 SUMMARY : new beta release DATE : Sep 19, 2013 ------------------------------------------------------------------------------ DESCRIPTION: A new beta version of syslog-ng Open Source Edition (3.5.0beta1) has been released. Being a beta release, testing is most appreciated, but running it in production is not advised. CHANGES: 3.5.0beta1 Thu, 19 Sep 2013 11:03:15 +0200 This is a BETA release of the syslog-ng 3.5 development branch, with a lot of internal rework and even more new features included. Being a beta release, testing is most appreciated, but running it in production is not advised. The release contains all the fixes from the 3.4 branch, the changes below are relative to the 3.4.3 release. Major Features ============== Multi-line support ------------------ A major feature in the 3.5 release is the inclusion of support for multi-line messages, a feature that has been available in syslog-ng PE for a good while. With this, both indented and regexp-based multi-line support becomes available. For further information and examples, see the following blog post: http://asylum.madhouse-project.org/blog/2013/09/05/what-is-cooking-in-syslo… STOMP destination ----------------- Joining the message-queue club, this new destination makes syslog-ng able to send events to any STOMP-capable message queuing server, such as Apache ActiveMQ. For further information about the destination, see this commit message: https://github.com/balabit/syslog-ng-3.5/commit/82d19a4d4ad3f91e8214006f6ea… Template type hinting --------------------- While syslog-ng supported sending events to various datastores and queues for a while now (SQL first, MongoDB, JSON, and AMQP later), even if those supported other types of data, syslog-ng only ever sent strings. With template type hinting, it became possible to tell syslog-ng what type a certain template should be, so that the drivers can use that information in whatever way they see fit. This is currently implemented for the MongoDB driver only. For more information about type hinting and for examples, see the following post: http://asylum.madhouse-project.org/blog/2013/09/05/what-is-cooking-in-syslo… Other features ============== * in-list() filter: this new filter function allows one to easily implement file-based white- and blacklists with a simple syntax: filter f_white { in-list("/path/to/file", value("HOST")); }; * You no longer need to remember how many zeros to put after a big number in the syslog-ng configuration file, you can use unit suffixes, such as: log-fifo-size(2GiB) See the following post for more details: http://asylum.madhouse-project.org/blog/2013/09/05/what-is-cooking-in-syslo… * A set of new string-related template functions are available: $(uppercase STRINGS...), $(lowercase STRINGS...) and $(replace-delimiter DELIMITERS NEW-DELIMITER TEXT). The first two do exactly what their names suggest, while the last one replaces all occurrences of any DELIMITERS within TEXT with the NEW-DELIMITER. * There is also a new $(env VARIABLE...) template function, which looks up the given variables in the environment. This is similar to using backticks in the configuration file, however the name of the environment variable with $(env) may contain macros too. * Support for Linux 3.5+'s /dev/kmsg was added, and the system() source will automatically detect whether to use it over /proc/kmsg. * For every correlated message in patterndb, the ${.classifier.context_id} property is automatically set to the context-id attribute of the matching rule. Incompatible changes ==================== * The replace() transformation function of value-pairs() was renamed to replace-prefix() to better reflect what it actually does. The old name remains as an obsolete alias, however. * The username() and password() options of the MongoDB destination were removed, they never did anything before, either. Miscellaneous changes ===================== * The build system was completely redone, it is much faster, more reliable, and less verbose by default now. * Several systemd-related enhancements were made, including support for notification-based startup. This also means that when systemd support is compiled in, libsystemd-daemon becomes a new dependency. Credits: ======== syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Andras Tim <tia(a)balabit.hu> Anton Koldaev <koldaevav(a)gmail.com> Attila M. Magyar <athos(a)balabit.hu> Attila Nagy <bra(a)fsn.hu> Attila Szalay <sasa(a)balabit.hu> Balazs Scheidler <bazsi(a)balabit.hu> Balint Kovacs <blint(a)balabit.hu> Chris Johnson <chris.johnson3(a)hp.com> Evan Rempel <erempel(a)uvic.ca> Fabien Wernli <cpan(a)faxm0dem.org> Gergely Nagy <algernon(a)balabit.hu> Juhasz Viktor <jviktor(a)balabit.hu> Laszlo Budai <lbudai(a)balabit.hu> Martin <bmartin(a)lavabit.com> Michael Sterrett <mr_bones_(a)gentoo.org> Michal Privoznik <miso.privoznik(a)gmail.com> Paul Dann <pdgiddie+balabit(a)gmail.com> Peter Czanik <czanik(a)balabit.hu> Peter Gyongyosi <gyp(a)balabit.hu> Robert Fekete <frobert(a)balabit.hu> Tamas Pal <folti(a)balabit.hu> Tibor Benke <btibi(a)balabit.hu> Valentijn Sessink <valentijn(a)sessink.nl> Viktor Tusa <tusa(a)balabit.hu> Vincent Brillault <spam(a)lerya.net> DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

1 0