October 2013 - syslog-ng - lists.balabit.hu

insider 2013-10: syslog-ng 3.5 beta releases; syslog-ng PE with Windows support; GSoC summaries;
by Peter Czanik 17 Oct '13

17 Oct '13

Dear syslog-ng users, This is the 28th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. FEATURED NEWS syslog-ng 3.5 beta2 and beta3 are released ------------------------------------------ The second and third beta of syslog-ng 3.5 were released during the past few weeks. These brought mostly bugfixes, but also some new features. Beta2 brought us a new redis destination and extended type hinting. Beta3 finally compiles again on FreeBSD. As usual, testing is greatly appreciated as it helps to iron out problems before the final release! Sources are available from git ( https://github.com/balabit/syslog-ng-3.5 ) or as tgz from http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… For third-party binary packages for various Linux and UNIX platforms, visithttp://www.balabit.com/network-security/syslog-ng/opensource-logging-s… .SLES / openSUSE packages are already available andDebian/Ubuntu packages are expected to be available in the coming days. syslog-ng PE with full Windows support -------------------------------------- The Premium Edition of syslog-ng now offers full Windows support with its latest release, version 5LTS. Until now, only the syslog-ng Agent for Windows could be installed on Windows platforms. With the latest version, syslog-ng Premium Edition can be installed on client but also as server, so even a homogenous Windows environment can benefit from using syslog-ng. Installing a separate Linux machine as relay is no longer necessary in remote offices, as a Windows machine can do the job. Compression support in RLTP was also introduced, which can save valuable bandwidth. Read the product managers blog about the new release: https://gyp.blogs.balabit.com/2013/10/introducing-syslog-ng-premium-edition… syslog-ng GSoC: code merged --------------------------- The syslog-ng GSoC finished a few weeks ago, but it does not mean, that the work is over. The code had to be updated to work with the latest syslog-ng 3.5 sources. The new redis destination is already merged and available as part of the syslog-ng 3.5 beta2 sources. The new mysql destination is heading now tothe “incubator” project and waits there for some additional polish, before it is merged syslog-ng master. A summary blog post of their work is availableat http://petrovicsgyula.blogspot.com/ and http://tichygsoc.blogspot.hu/ A blog post about the syslog-ng GSoC from the mentors (and algernon's) point of view can also be read: https://algernon.blogs.balabit.com/2013/10/onto-the-margin-of-a-summer-of-c… syslog-ng 3.5 documentation is now feature complete --------------------------------------------------- The syslog-ng 3.5 documentation is now feature complete and available on our website: It is available in many forms: * PDF: http://www.balabit.com/support/documentation/syslog-ng-ose-3.5-guides/en/sy… * HTML: http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.5-guid… * Single page HTML: http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.5-guid… SHORT NEWS: * how to use the new in-list() feature: https://czanik.blogs.balabit.com/2013/09/black-cat-white-cat/ * syslog-ng and Riemann are going to be presented at vDay in Budapest: http://vday.hu/program.php * Recent Events Underline the Need for Log Management in Application Security: https://jluby.blogs.balabit.com/2013/10/04/recent-events-underline-the-need… NEW RELEASES * syslog-ng 3.5beta2: https://lists.balabit.hu/pipermail/syslog-ng-announce/2013-October/000168.h… * syslog-ng 3.5beta3: https://lists.balabit.hu/pipermail/syslog-ng-announce/2013-October/000168.h… * syslog-ng PE 5LTS: https://lists.balabit.hu/pipermail/syslog-ng-announce/2013-October/000170.h… * syslog-ng Agent 5.0.2: https://lists.balabit.hu/pipermail/syslog-ng-announce/2013-October/000169.h… * syslog-ng 3.4.4: https://lists.balabit.hu/pipermail/syslog-ng-announce/2013-October/000171.h… Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com <mailto:documentation@balabit.com> To read this newsletter on-line, visit: http://insider.blogs.balabit.com/ -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

[Bug 250] New: Colon does not need escaping in pattern parser @ESTRING@
by bugzilla＠bugzilla.balabit.com 17 Oct '13

17 Oct '13

https://bugzilla.balabit.com/show_bug.cgi?id=250 Summary: Colon does not need escaping in pattern parser @ESTRING@ Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bugzilla.balabit(a)faxm0dem.org Type of the Report: --- Estimated Hours: 0.0 Created an attachment (id=81) --> (https://bugzilla.balabit.com/attachment.cgi?id=81) @ESTRING::::@ vs. @ESTRING:::@ According to the 3.4 documentation (§13.5.1.1) using a ":" (colon) as stop string for the @ESTRING@ parser needs to be escaped using a second ":" (colon). It seems this is not the case: patterns fail when using "::", but match when using ":": $ pdbtool test -v estring_colon_escaped.pdb Testing message program='dummy' message='foo: bar' Wrong match name='.classifier.rule_id', value='', expected='0304be33-4f3e-459e-8888-d86769986c32' Wrong match name='key', value='', expected='foo' Wrong match name='value', value='', expected='bar' $ pdbtool test -v estring_colon.pdb Testing message program='dummy' message='foo: bar' Match name='.classifier.rule_id', value='0304be33-4f3e-459e-8888-d86769986c32', expected='0304be33-4f3e-459e-8888-d86769986c32' Match name='key', value='foo', expected='foo' Match name='value', value='bar', expected='bar' -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

FreeBSD core dump
by Alexandre Biancalana 17 Oct '13

17 Oct '13

Hi list, I'm running syslog-ng 3.4.3 (installed from FreeBSD ports) on FreeBSD 9.1 with self-written patterndb file to parse apache error_log, mod_security and pagespeed errors. The syslog-ng source is httpd error_log file, after starting syslog-ng it core dumps: ERROR:logmsg.c:1303:log_msg_unref: assertion failed: (LOGMSG_REFCACHE_VALUE_TO_REF(old_value) >= 1) Abort trap (core dumped) Let me know if you need more info. Regards, Alexandre

2 16

beta3 packages for openSUSE / SLES
by Peter Czanik 11 Oct '13

11 Oct '13

Hello, My 3.5 beta3 packages for openSUSE and SLES are available at the usual location: https://build.opensuse.org/package/show/home:czanik:syslog-ng35/syslog-ng There was one major change since beta2: redis destination is enabled. Please note, that for SLES you need to use my repo for dependencies. On openSUSE only redis support is not included in the base system from the dependencies. While adding my repo is the easiest way to install packages, it is also possible to download rpm-s and install them, except for syslog-ng-redis. Happy testing! Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

syslog-ng 3.5.0beta3 has been released
by devel＠balabit.hu 11 Oct '13

11 Oct '13

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.5.0beta3 SUMMARY : new beta release DATE : Oct 11, 2013 ------------------------------------------------------------------------------ DESCRIPTION: A new beta version of syslog-ng Open Source Edition (3.5.0beta3) has been released. Being a beta release, testing is most appreciated, but running it in production is not advised. CHANGES: 3.5.0beta3 Fri, 11 Oct 2013 10:21:38 +0200 This is the third BETA release of the syslog-ng 3.5 development branch, with stability fixes and other enhancements. While running in production is still not advised, testing is most appreciated. Incompatible changes ==================== Template escaping ----------------- Template escaping was changed in an incompatible way: previously, both the lexer and the template compiler used the '\' character for escaping, which was confusing. The template compiler uses '$$' to escape the '$' char, and '@@' to escape '@'. If a non-numeric value follows '@', a warning will be printed. Bugfixes ======== * syslog-ng should compile again on non-Linux platforms. * The flush() and fsync() options of the file destination were fixed, and they should work the same way now as they do in 3.3 and 3.4. * The hiredis library should be detected on the default include paths, so one does not necessarily need to specify --with-libhiredis non non-Debian systems. Said option was also made to work. * A memory corruption was fixed in the @STRING@ parser of db-parser(). * Excludes now work properly with value-pairs(), and they do not get ignored if the value to exclude is in the default set. Known Bugs ========== * The afstreams module is broken, does not compile, and does not work. This will be corrected before the final 3.5.1 release. Credits: ======== syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Balazs Scheidler <bazsi(a)balabit.hu> Cy Schubert <Cy.Schubert(a)komquats.com> Evan Rempel <erempel(a)uvic.ca> Gergely Nagy <algernon(a)balabit.hu> Lucas McLane <lucas(a)clicksecurity.com> Oscar Muñoz Peter Czanik <czanik(a)balabit.hu> Sergey Shuman Viktor Juhasz <jviktor(a)balabit.hu> DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

1 0

[Bug 169] New: Feature Request - reconnect network also resolve dns name
by bugzilla＠bugzilla.balabit.com 10 Oct '13

10 Oct '13

https://bugzilla.balabit.com/show_bug.cgi?id=169 Summary: Feature Request - reconnect network also resolve dns name Product: syslog-ng Version: 3.3.x Platform: Other OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 When a network connection is dropped the destination name should be resolved via DNS prior to the reconnect. Every couple of years to renew the hardware supporting our core syslog service we move the IP name to a new host. The desired process is; 1. Build a new host and ensure that syslog-ng is running correctly. 2. Change the DNS for our central syslog server to the address of the new server. 3. Restart the syslog-ng service on the old syslog server. 4. All of the clients will loose the connection to the old syslog server. 5. All of the clients will reconnect to the *IP Name* in their configuration files. 6. Monitor the old syslog server to determine any hosts that have not made the transition. What really happened in step 5 was that all of the client reconnected to the old syslog server :-( To accomplish the switch over I had to visit 450 servers and manually restart the syslog service on the client. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5

[Bug 252] New: Syslog-NG crashes with destination using unix-dgram() driver
by bugzilla＠bugzilla.balabit.com 09 Oct '13

09 Oct '13

https://bugzilla.balabit.com/show_bug.cgi?id=252 Summary: Syslog-NG crashes with destination using unix-dgram() driver Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: marc.falzon(a)cloudwatt.com Type of the Report: bug Estimated Hours: 0.0 Created an attachment (id=82) --> (https://bugzilla.balabit.com/attachment.cgi?id=82) syslog-ng.conf Hello I'm facing a Syslog-NG crash when trying to use a unix-dgram() driver as log destination (configuration used is attached to this issue): # syslog-ng -Fev Module loaded and initialized successfully; module='system-source' Module loaded and initialized successfully; module='afsocket' Module loaded and initialized successfully; module='affile' Module loaded and initialized successfully; module='afuser' Module loaded and initialized successfully; module='syslogformat' Segmentation fault # Service starts successfully if the log path using the d_local destination is commented. Version used (platform: Linux Ubuntu Precise 12.04): # syslog-ng -V syslog-ng 3.4.2 Installer-Version: 3.4.2 Revision: 3.4.2-1~mhp1.sysvinit~precise [@3b56ec1] (Ubuntu/12.04) Compile-Date: Jun 4 2013 16:52:14 Available-Modules: syslogformat,afprog,basicfuncs,confgen,afsocket-notls,afsocket,afsocket-tls,dbparser,cryptofuncs,csvparser,affile,afuser,system-source Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: on # uname -a Linux p-xxxxxx-0000 3.2.0-51-generic #77-Ubuntu SMP Wed Jul 24 20:18:19 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux # Let me know if you need any more information about my setup and/or configuration. Regards, m. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 12

[Bug 254] New: 3.4. 1 patterndb replacing HOST corrupts $HOST and $FULLHOST macros
by bugzilla＠bugzilla.balabit.com 09 Oct '13

09 Oct '13

https://bugzilla.balabit.com/show_bug.cgi?id=254 Summary: 3.4.1 patterndb replacing HOST corrupts $HOST and $FULLHOST macros Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: blocker Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 I am using a patterndb of <pattern>got messages from @STRING:HOST:-.@</pattern> to match the syslog line 2013-10-08T11:01:26-07:00 wolverine.comp.uvic.ca/wolverine.comp.uvic.ca/syslog2.uvic.ca local0.info heartbeat[18869]: got messages from camelotia.comp.uvic.ca so that the syslog line will appears as if it came from camelotia.comp.uvic.ca When this line gets logged to the files, it looks like 2013-10-08T11:01:26-07:00 camelotia.comp.uvic.ca^@heP^@^@^@ local0.info flare-heartbeat[18869]: got messages from camelotia.comp.uvic.ca wher the ^@ are ASCII 0 (control-@) characters. In other words, the HOST macro becomes corrupt. The line should look (preferably) like 2013-10-08T11:01:26-07:00 camelotia.comp.uvic.ca/wolverine.comp.uvic.ca/syslog2.uvic.ca local0.info flare-heartbeat[18869]: got messages from camelotia.comp.uvic.ca or (less preferably) 2013-10-08T11:01:26-07:00 camelotia.comp.uvic.ca local0.info flare-heartbeat[18869]: got messages from camelotia.comp.uvic.ca It isn''t consistent how many ^@ symbols are present. 2013-10-08T11:00:26-07:00 nestor0286.westgrid.uvic.ca^@^@ local0.info flare-heartbeat[18869]: got messages from nestor0286.westgrid.uvic.ca 2013-10-08T11:00:26-07:00 nestor0127.westgrid.uvic.ca^@^@^@^@^@^@ local0.info flare-heartbeat[18869]: got messages from nestor0127.westgrid.uvic.ca 2013-10-08T11:00:26-07:00 hermes0042.westgrid.uvic.ca^@^@ local0.info flare-heartbeat[18869]: got messages from hermes0042.westgrid.uvic.ca 2013-10-08T11:00:26-07:00 hermes0032.westgrid.uvic.ca^@^@ local0.info flare-heartbeat[18869]: got messages from hermes0032.westgrid.uvic.ca 2013-10-08T11:00:26-07:00 nestor0016.westgrid.uvic.ca^@fang local0.info flare-heartbeat[18869]: got messages from nestor0016.westgrid.uvic.ca 2013-10-08T11:00:26-07:00 nestor0027.westgrid.uvic.ca^@^@ local0.info flare-heartbeat[18869]: got messages from nestor0027.westgrid.uvic.ca -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

Fwd: Syslog NG simple filtering not working as expected...
by Paul Hutton 02 Oct '13

02 Oct '13

Initially sent to the wrong address.... Begin forwarded message: > From: Paul Hutton <paul_hutton(a)bigpond.com> > Subject: Syslog NG simple filtering not working as expected... > Date: 26 September 2013 11:23:15 PM AEST > To: syslog-ng-request(a)lists.balabit.hu > > Hi All, > > I am trying to forward network sourced syslogs to a number of destinations depending on a few simple filters, but am immediately coming unstuck. My constraint is that all incoming and outgoing network syslog traffic is udp. My intention in the configuration code below (apart from logging the localhost) is to: > > 1. Forward all incoming network sourced syslogs except those generated by two firewalls to destination Remote_L > 2. Forward incoming network syslogs that are generated by the firewalls to destination Remote_C (currently commented out) > 3. Forward all incoming network syslogs of severity 0, 1 and 2 to destinations Remote_D1 and Remote_D2 (currently commented out) > > In addition all incoming network sourced syslogs are logged locally to destination d_logger and I've added another local file d_debug out of curiosity. > > My problem is that with just the filter enabled at point 1 above and in the code below, the behaviour of syslog-ng is very erratic, beyond any reasonable anticipation of the unreliability of udp as the transport. If I comment out the filter line > > filter (not_firewalls); > > in the code below, then test traffic (none of which is generated by the firewall sources) is received and forwarded correctly as well as logged locally. If I uncomment the filter, then I have seen test traffic forwarded correctly on one occasion, but it is mostly the case that some smaller subset (including none) of the test traffic is forwarded. In this case, the test traffic is also correctly written to the local destination d_debug. That is in the same rule that fails to forward the test traffic. > > I am running 3.2.4 on Solaris 10 09/10 x86 obtained from the Sunfreeware site. > > I have also unashamedly plagiarised configuration code from a couple of posted sources, so if you recognise your work, thanks for putting it out there... > > > Thanks in advance. > > Paul > > > > @version: 3.2 > @include "scl.conf" > options { > stats_level(3); > flush_lines(100); > flush_timeout(10000); > }; > ################################################################ > # Ensure syslog-ng logs this host similarly to syslogd # > ################################################################ > source s_local { > sun-streams("/dev/log" door("/etc/.syslog_door")); > internal(); > }; > # > # Set global network source > # > source src { > udp(ip("0.0.0.0") port (514)); > }; > > ############################################################### > # Local logfiles and remote destinations setup # > ############################################################### > destination syslog { > file("/var/log/syslog"); > }; > destination messages { > file("/var/adm/messages"); > }; > destination loginlog { > file("/var/adm/loginlog"); > }; > # > # Set remote destinations > # > destination Remote_L { > udp ("10.octet.octet.octetL" port (514) ); > }; > destination Remote_D1 { > udp ("10.octet.octet.octetD1" port (514) spoof_source (yes)); > }; > destination Remote_D2 { > udp ("10.octet.octet.octetD2" port (514) spoof_source (yes)); > }; > destination Remote_C { > udp ("10.octet.octet.octetC" port (514) spoof_source (yes)); > }; > # > # Set local destination for all remote syslog input > # > destination d_logger { > file("/var/log/logger.log"); > }; > # > # Add another local debug file to check filters are working > # > destination d_debug { > file("/var/log/debug.log"); > }; > ############################################################## > # Filters for mimicking Solaris syslogd # > # identifying firewall syslogs # > # identifying severity levels 0 to 2 # > ############################################################## > filter f_syslog { > facility(mail) or (facility(daemon) and priority(notice)); > }; > filter f_messages { > priority(err) > or facility(kern) > or (facility(user) and priority(err)) > or (facility(daemon) and priority(notice)) > or (facility(mail) and priority(crit)); > }; > filter f_auth { > facility (auth); > }; > > # > # Set Firewall Filters > # > filter firewalls { > host("10.octet.octet.octetfw1") or host("10.octet.octet.octetfw2"); > }; > filter not_firewalls { > not (host("10.octet.octet.octetfw1") or host("10.octet.octet.octetfw2")); > }; > > # > # Filter Severity Levels 0, 1 and 2 > # > filter f_crit { > level (crit .. emerg); > }; > ############################################################## > # Now do some logging # > ############################################################## > > # > # Log local syslog data appropriately > # > log { > source(s_local); > filter(f_syslog); > destination(syslog); > }; > log { > source(s_local); > filter(f_messages); > destination(messages); > }; > log { > source(s_local); > filter(f_auth); > destination(loginlog); > }; > > # > # Log remotely sourced syslogs of all Severity Levels to remote L system > # but exclude syslogs from the firewalls > # > log { > source (src); > filter (not_firewalls); > destination (Remote_L); > destination (d_debug); > }; > # > # Log remotely sourced syslogs of Severity Levels 0 to 2 to remote D systems > # including syslogs from the firewalls > # > #log { > # source (src); > # filter (f_crit); > # destination (Remote_D1); > # destination (Remote_D2); > #}; > # > # Log remotely sourced syslogs from the two firewalls to remote C system > # > #log { > #source (src); > #filter (firewalls); > # destination (Remote_C); > #}; > # > # Log remotely sourced syslogs to a local file > # > log { > source (src); > destination (d_logger); > }; > > >

4 5

[Bug 241] New: pdbtool test --validate returns success when matching wrong pattern
by bugzilla＠bugzilla.balabit.com 02 Oct '13

02 Oct '13

https://bugzilla.balabit.com/show_bug.cgi?id=241 Summary: pdbtool test --validate returns success when matching wrong pattern Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 pdbtool test --validate pattern.xml only returns a non-zero exit code if test values don't match. If there is a catchall pattern in the pattern.xml database file then any test message that does not match the pattern it belongs to, but does match another pattern will return a zero exit code. The output still shows the wrong .classifier.rule_id Testing message program='mailguard' message='2013-07-03T13:13:38-07:00 local(a)dog.comp.uvic.ca local4.notice mailguard: Could not open /var/mimedefang/sophie.pid' Wrong match name='.classifier.rule_id', value='UN-eb0cfada-af50-48b0-ac1e-de1ffad8fe6c', expected='b826184c-2990-46d4-a777-f87340f1796a' but the return code from the pdbtool is still zero. I would expect any test message that does not match the pattern it belongs to should produce a non-zero return code. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

4 8