- syslog-ng - lists.balabit.hu

[Bug 256] New: rewrite + subst make syslog-ng crash
by bugzilla＠bugzilla.balabit.com 24 Oct '13

24 Oct '13

https://bugzilla.balabit.com/show_bug.cgi?id=256 Summary: rewrite + subst make syslog-ng crash Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: nico(a)rottenbytes.info Type of the Report: bug Estimated Hours: 0.0 Hi, I've been using syslog-ng 3.4.4 (from madhouse-project debian packages) and I'm getting crashes of the daemon on rewrite rules. here is the scheme : rewrite r_services { subst("postfix/.*", "postfix", value("PROGRAM"), flags("global")); subst("nagios.*", "nagios", value("PROGRAM"), flags("global")); subst("jftpgw", "ftp-proxy", value("PROGRAM"), flags("global")); subst("ftp-child", "ftp-proxy", value("PROGRAM"), flags("global")); }; log { source(s_local); source(s_remote); rewrite(r_services); filter(f_services); destination(df_services); flags(final); }; will make the daemon crash. rewrite + set rules are NOT impacted and work fine. Using rewrite + set and a condition will work too for example : rewrite r_services { set("postfix", value("PROGRAM") condition(program("postfix/.*"))); }; works. When the program crashes I get the following error : ERROR:../../../lib/logmsg.c:535:log_msg_set_value_indirect: assertion failed: (!log_msg_is_write_protected(self)) Feel free to contact me if you need more details. Regards -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 6

syslog-ng OSE supported versions, EoL times, etc
by Gergely Nagy 22 Oct '13

22 Oct '13

Hi! We've been thinking about how to approach the problem of supporting syslog-ng OSE versions, the end-of-life of them, and related issues. I've been maintaining syslog-ng branches for about a year and a half now, and the time and effort required to do that is only becoming clear now that I have a reasonable sample size. There was no clear communication about how long each version of syslog-ng OSE would be supported, and when they will reach their end of life, which made it harder for distributions and pretty much everyone else to plan ahead and schedule time for migrating to a new version. We intend to rectify that now, and my plan is outlined below: We would - at any one time - maintain four branches of syslog-ng (including the bleeding edge development branch). The other three would be: a critical-bug-fix-only maintenance release (this is currently 3.3), a recommended stable release (3.4), and the upcoming feature release (3.5beta at the moment). Whenever the development branch gets a beta release, it would automatically become the upcoming feature release, the former feature release would get promoted to stable, and the former stable becomes the maintenance branch. In this scheme, 3.3 would be maintained until 3.6.0beta1, at which point, it would be dropped from the supported set, and 3.4 would take its place, and 3.5 would become the recommended stable release. Of course, it's still hard to plan ahead without dates, but dates are something I cannot give at this point. The best I can do is send an announcement once 3.6 hits the alpha stage, and give a rough estimate on when it would hit beta. That's a few months in advance, and by that time, 3.4 will be out long enough for folk to migrate over, thus 3.3 shouldn't be in much use anymore. With the current pace of development, this means that a syslog-ng OSE release will be maintained for about 2-2.5 years, and I expect it will not go down by much more than half a year, meaning 1.5-2 years of maintenance. And to make it clear what each branch means, let me expand on that a bit: * Maintenance: The former stable branch, which receives critical bug fixes only. People are strongly encouraged to migrate from this to the stable one, but the branch is still supported to some extent, to make it easier for distributions, among others. Critical bugs are security issues, serious regressions, and so on - I reserve the right to decide which bugs to consider critical. Fixes from any other branch are not backported, unless they're critical. Currently the 3.3 branch fills this role. * Stable: The stable branch, receiving all kinds of bug fixes, including smaller fixes backported from newer branches (including the next feature release and the development branch too). It may also receive very small features too, which do not affect the package as a whole: things like the $(format-json) improvement to allow comma-separated scopes, or being able to specify a template for the JSON parser. Currently the 3.4 branch fills this role. * Feature: The next stable branch, which we consider stable enough, but has not received as wide testing as the stable branch yet. This gets the most attention, as the idea is to stabilise it as soon as possible. Other than being newer than the Stable branch, this is treated in exactly the same way. Currently the 3.5 branch fills this role. * Development: The branch where all development happens. While we try to keep it stable, regressions and breakage is more likely here, and new features are not guaranteed to remain the same during the development cycle, so building production on this branch is not advised. New features are developed against this branch, and the Incubator's[1] master branch follows the development branch too. The 3.6 branch will fill this role, once opened. As for the Incubator[1]: it will only have two branches: one for the latest feature release, and another for development. Maintenance and Stable versions of syslog-ng will not be supported in there. [1]: https://github.com/algernon/syslog-ng-incubator The Incubator is a place for experimental modules that are either not ready to be merged into syslog-ng proper, or are kept separate for any other reason. -- |8]

1 0

mongodb stats crash
by Balazs Scheidler 20 Oct '13

20 Oct '13

Hi, I have encountered a fixed crash in 3.5 master in the mongodb() driver, it aborts in stats_unregister_counter() it seems that afmongodb_format_stats_instance() returns different values in init() and deinit(). This is my config: mongodb(value-pairs(pair('foo', "$STAMP")) frac-digits(3)); This is the backtrace: [Thread 0x7ffff7fca700 (LWP 27493) exited] ** ERROR:/home/bazsi/zwa/work/syslog-ng-ose-3.5/syslog-ng-ose/lib/stats.c:302:stats_unregister_counter: assertion failed: (sc && (sc->live_mask & (1 << type)) && &sc->counters[type] == (*counter)) Program received signal SIGABRT, Aborted. 0x00007ffff7275037 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory. (gdb) bt #0 0x00007ffff7275037 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 #1 0x00007ffff7278698 in __GI_abort () at abort.c:90 #2 0x00007ffff788e3b6 in g_assertion_message () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #3 0x00007ffff788e914 in g_assertion_message_expr () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #4 0x00007ffff7b69ef8 in stats_unregister_counter (source=533, id=0x61cd10 "d_local#0", instance=0x7ffff4d26c40 <persist_name.15015> "mongodb,127.0.0.1,27017,syslog,messages", type=SC_TYPE_STORED, counter=0x6e4cd0) at /home/bazsi/zwa/work/syslog-ng-ose-3.5/syslog-ng-ose/lib/stats.c:302 #5 0x00007ffff7b5b838 in log_threaded_dest_driver_deinit_method (s=0x6e4c10) at /home/bazsi/zwa/work/syslog-ng-ose-3.5/syslog-ng-ose/lib/logthrdestdrv.c:162 #6 0x00007ffff7b49014 in log_pipe_deinit (s=0x6e4c10) at /home/bazsi/zwa/work/syslog-ng-ose-3.5/syslog-ng-ose/lib/logpipe.h:268 #7 0x00007ffff7b4afad in cfg_tree_stop (self=0x613f30) at /home/bazsi/zwa/work/syslog-ng-ose-3.5/syslog-ng-ose/lib/cfg-tree.c:1080 #8 0x00007ffff7b44908 in cfg_deinit (cfg=0x613dc0) at /home/bazsi/zwa/work/syslog-ng-ose-3.5/syslog-ng-ose/lib/cfg.c:225 #9 0x00007ffff7b607b1 in main_loop_exit_finish () at /home/bazsi/zwa/work/syslog-ng-ose-3.5/syslog-ng-ose/lib/mainloop.c:646 #10 0x00007ffff7b60407 in main_loop_io_worker_sync_call (func=0x7ffff7b60799 <main_loop_exit_finish>) at /home/bazsi/zwa/work/syslog-ng-ose-3.5/syslog-ng-ose/lib/mainloop.c:470 #11 0x00007ffff7b607d0 in main_loop_exit_timer_elapsed (arg=0x0) at /home/bazsi/zwa/work/syslog-ng-ose-3.5/syslog-ng-ose/lib/mainloop.c:653 #12 0x00007ffff7b96568 in iv_run_timers () from /home/bazsi/zwa/install/syslog-ng-ose-3.5/lib/libsyslog-ng-3.5.0beta3.so #13 0x00007ffff7b99230 in iv_main () from /home/bazsi/zwa/install/syslog-ng-ose-3.5/lib/libsyslog-ng-3.5.0beta3.so #14 0x00007ffff7b60ba7 in main_loop_run () at /home/bazsi/zwa/work/syslog-ng-ose-3.5/syslog-ng-ose/lib/mainloop.c:796 #15 0x0000000000401a67 in main (argc=1, argv=0x7fffffffd648) at /home/bazsi/zwa/work/syslog-ng-ose-3.5/syslog-ng-ose/syslog-ng/main.c:269 (gdb) Stepping through format_stats_instance() at init time: Breakpoint 3, afmongodb_dd_format_stats_instance (d=0x6e4c10) at /home/bazsi/zwa/work/syslog-ng-ose-3.5/syslog-ng-ose/modules/afmongodb/afmongodb.c:189 189 MongoDBDestDriver *self = (MongoDBDestDriver *)d; (gdb) n 192 if (self->port == MONGO_CONN_LOCAL) (gdb) p self->port $1 = 0 (gdb) n 197 "mongodb,%s,%u,%s,%s", self->address, self->port, self->db, self->coll); (gdb) 196 g_snprintf(persist_name, sizeof(persist_name), (gdb) l 191 192 if (self->port == MONGO_CONN_LOCAL) 193 g_snprintf(persist_name, sizeof(persist_name), 194 "mongodb,%s,%s,%s", self->address, self->db, self->coll); 195 else 196 g_snprintf(persist_name, sizeof(persist_name), 197 "mongodb,%s,%u,%s,%s", self->address, self->port, self->db, self->coll); 198 return persist_name; 199 } 200 (gdb) n 197 "mongodb,%s,%u,%s,%s", self->address, self->port, self->db, self->coll); (gdb) 196 g_snprintf(persist_name, sizeof(persist_name), (gdb) 197 "mongodb,%s,%u,%s,%s", self->address, self->port, self->db, self->coll); (gdb) 196 g_snprintf(persist_name, sizeof(persist_name), (gdb) 197 "mongodb,%s,%u,%s,%s", self->address, self->port, self->db, self->coll); (gdb) 196 g_snprintf(persist_name, sizeof(persist_name), (gdb) 198 return persist_name; (gdb) p persist_name $2 = "mongodb,(null),0,syslog,messages", '\000' <repeats 991 times> At deinit time: Breakpoint 3, afmongodb_dd_format_stats_instance (d=0x6e4c10) at /home/bazsi/zwa/work/syslog-ng-ose-3.5/syslog-ng-ose/modules/afmongodb/afmongodb.c:189 189 MongoDBDestDriver *self = (MongoDBDestDriver *)d; (gdb) n 192 if (self->port == MONGO_CONN_LOCAL) (gdb) p self->port $3 = 27017 (gdb) p self->address $4 = (gchar *) 0x61d290 "127.0.0.1" (gdb) n 197 "mongodb,%s,%u,%s,%s", self->address, self->port, self->db, self->coll); (gdb) 196 g_snprintf(persist_name, sizeof(persist_name), (gdb) 197 "mongodb,%s,%u,%s,%s", self->address, self->port, self->db, self->coll); (gdb) 196 g_snprintf(persist_name, sizeof(persist_name), (gdb) 197 "mongodb,%s,%u,%s,%s", self->address, self->port, self->db, self->coll); (gdb) 196 g_snprintf(persist_name, sizeof(persist_name), (gdb) 197 "mongodb,%s,%u,%s,%s", self->address, self->port, self->db, self->coll); (gdb) 196 g_snprintf(persist_name, sizeof(persist_name), (gdb) 198 return persist_name; (gdb) p persist_name $5 = "mongodb,127.0.0.1,27017,syslog,messages", '\000' <repeats 984 times> This patch solves for me, however I'm not 100% certain it's the proper fix and the other LogThreadedDestDriver instances might be affected too. diff --git a/modules/afmongodb/afmongodb.c b/modules/afmongodb/afmongodb.c index 6504a63..b625c2c 100644 --- a/modules/afmongodb/afmongodb.c +++ b/modules/afmongodb/afmongodb.c @@ -516,8 +516,6 @@ afmongodb_dd_init(LogPipe *s) GlobalConfig *cfg = log_pipe_get_config(s); ValuePairsTransformSet *vpts; - if (!log_threaded_dest_driver_init_method(s)) - return FALSE; log_template_options_init(&self->template_options, cfg); @@ -566,6 +564,9 @@ afmongodb_dd_init(LogPipe *s) evt_tag_str("collection", self->coll), NULL); + if (!log_threaded_dest_driver_init_method(s)) + return FALSE; + log_threaded_dest_driver_start(&self->super); return TRUE; In case we're moving this to this point, I'd say that the threaded_dest_driver_start() could be moved into the log_threaded_dest_driver_init_method() function, can't it? Can you have a look? Thanks.

2 6

syslog-ng 3.4.4 has been released
by devel＠balabit.hu 18 Oct '13

18 Oct '13

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.4.4 SUMMARY : new stable release DATE : Oct 17, 2013 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (3.4.4) has been released. For latest fixes in the 3.4.x feature branch you are recommended to upgrade to this version. CHANGES: 3.4.4 Thu, 17 Oct 2013 12:16:40 +0200 This is the third bug-fix release for the 3.4.x series, with important fixes over the 3.4.3 release. Upgrading to this release is highly recommended. Bugfixes ======== * Fixed the log message reference counting in the AMQP destination, thus stopping it from crashing. * Fixed a possible crash in the AMQP destination that could be triggered when the AMQP server was unreachable at the time of a syslog-ng reload or shutdown. * pdbtool no longer segfaults when used with the --module switch [#249] * pdbtool test exits with a return value of 2 if the test value matched a different pattern than expected. [#241] * The inherit-properties attribute was added to the patterndb XSD. [#251] * A memory corruption was fixed in the @STRING@ parser of db-parser(). [#254] * Corrected a mistake in the afsocket module that lead to crashes on reload when unix-stream or unix-dgram destinations were in use. * Excludes now work properly with value-pairs(), and they do not get ignored if the value to exclude is in the default set. * The program() source and destination now correctly kill the started child on reload and shutdown. * The file destination does not use lseek() anymore. * When setting default template options, all random memory junk is cleared from the object first. * The order of header include paths was fixed in the test suite, so it should compile cleanly even in the presence of potentially conflicting headers. Features ======== * The $(format-json) template function, and anything else that may use the same value-pairs command line parser now allow specifying multiple scopes separated by commas (as opposed to having to use the --scope switch multiple times) [#248] Credits: ======== syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Alexandre Biancalana <biancalana(a)gmail.com> Balazs Scheidler <bazsi(a)balabit.hu> Balint Kovacs <blint(a)balabit.hu> Evan Rempel <erempel(a)uvic.ca> Fabien Wernli <cpan(a)faxm0dem.org> Gergely Nagy <algernon(a)balabit.hu> Lucas McLane <lucas(a)clicksecurity.com> Marc Falzon <marc.falzon(a)cloudwatt.com> Michal Privoznik <miso.privoznik(a)gmail.com> Ryan Frederick Valentijn Sessink <valentijn(a)sessink.nl> DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

5 6

ANN: Unofficial Debian/Ubuntu repo switching to 3.4 by default
by Gergely Nagy 17 Oct '13

17 Oct '13

As announced before[1], the syslog-ng Debian/Ubuntu repository[2] I provide will switch to syslog-ng 3.4 as default. This means that if you're using source lines like below, you'll get the 3.4.4 update coming later today. deb http://packages.madhouse-project.org/debian wheezy syslog-ng If you want to remain with 3.3, use the syslog-ng-3.3 component. The 3.3 branch will also be correctly labelled as "maintained", and 3.4 gets promoted to "stable". 3.5 packages will start coming with 3.5.0rc1 due next week. [1]: https://lists.balabit.hu/pipermail/syslog-ng/2013-March/020158.html [2]: http://asylum.madhouse-project.org/projects/debian/ -- |8]

1 0

insider 2013-10: syslog-ng 3.5 beta releases; syslog-ng PE with Windows support; GSoC summaries;
by Peter Czanik 17 Oct '13

17 Oct '13

Dear syslog-ng users, This is the 28th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. FEATURED NEWS syslog-ng 3.5 beta2 and beta3 are released ------------------------------------------ The second and third beta of syslog-ng 3.5 were released during the past few weeks. These brought mostly bugfixes, but also some new features. Beta2 brought us a new redis destination and extended type hinting. Beta3 finally compiles again on FreeBSD. As usual, testing is greatly appreciated as it helps to iron out problems before the final release! Sources are available from git ( https://github.com/balabit/syslog-ng-3.5 ) or as tgz from http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… For third-party binary packages for various Linux and UNIX platforms, visithttp://www.balabit.com/network-security/syslog-ng/opensource-logging-s… .SLES / openSUSE packages are already available andDebian/Ubuntu packages are expected to be available in the coming days. syslog-ng PE with full Windows support -------------------------------------- The Premium Edition of syslog-ng now offers full Windows support with its latest release, version 5LTS. Until now, only the syslog-ng Agent for Windows could be installed on Windows platforms. With the latest version, syslog-ng Premium Edition can be installed on client but also as server, so even a homogenous Windows environment can benefit from using syslog-ng. Installing a separate Linux machine as relay is no longer necessary in remote offices, as a Windows machine can do the job. Compression support in RLTP was also introduced, which can save valuable bandwidth. Read the product managers blog about the new release: https://gyp.blogs.balabit.com/2013/10/introducing-syslog-ng-premium-edition… syslog-ng GSoC: code merged --------------------------- The syslog-ng GSoC finished a few weeks ago, but it does not mean, that the work is over. The code had to be updated to work with the latest syslog-ng 3.5 sources. The new redis destination is already merged and available as part of the syslog-ng 3.5 beta2 sources. The new mysql destination is heading now tothe “incubator” project and waits there for some additional polish, before it is merged syslog-ng master. A summary blog post of their work is availableat http://petrovicsgyula.blogspot.com/ and http://tichygsoc.blogspot.hu/ A blog post about the syslog-ng GSoC from the mentors (and algernon's) point of view can also be read: https://algernon.blogs.balabit.com/2013/10/onto-the-margin-of-a-summer-of-c… syslog-ng 3.5 documentation is now feature complete --------------------------------------------------- The syslog-ng 3.5 documentation is now feature complete and available on our website: It is available in many forms: * PDF: http://www.balabit.com/support/documentation/syslog-ng-ose-3.5-guides/en/sy… * HTML: http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.5-guid… * Single page HTML: http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.5-guid… SHORT NEWS: * how to use the new in-list() feature: https://czanik.blogs.balabit.com/2013/09/black-cat-white-cat/ * syslog-ng and Riemann are going to be presented at vDay in Budapest: http://vday.hu/program.php * Recent Events Underline the Need for Log Management in Application Security: https://jluby.blogs.balabit.com/2013/10/04/recent-events-underline-the-need… NEW RELEASES * syslog-ng 3.5beta2: https://lists.balabit.hu/pipermail/syslog-ng-announce/2013-October/000168.h… * syslog-ng 3.5beta3: https://lists.balabit.hu/pipermail/syslog-ng-announce/2013-October/000168.h… * syslog-ng PE 5LTS: https://lists.balabit.hu/pipermail/syslog-ng-announce/2013-October/000170.h… * syslog-ng Agent 5.0.2: https://lists.balabit.hu/pipermail/syslog-ng-announce/2013-October/000169.h… * syslog-ng 3.4.4: https://lists.balabit.hu/pipermail/syslog-ng-announce/2013-October/000171.h… Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com <mailto:documentation@balabit.com> To read this newsletter on-line, visit: http://insider.blogs.balabit.com/ -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

[Bug 250] New: Colon does not need escaping in pattern parser @ESTRING@
by bugzilla＠bugzilla.balabit.com 17 Oct '13

17 Oct '13

https://bugzilla.balabit.com/show_bug.cgi?id=250 Summary: Colon does not need escaping in pattern parser @ESTRING@ Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bugzilla.balabit(a)faxm0dem.org Type of the Report: --- Estimated Hours: 0.0 Created an attachment (id=81) --> (https://bugzilla.balabit.com/attachment.cgi?id=81) @ESTRING::::@ vs. @ESTRING:::@ According to the 3.4 documentation (§13.5.1.1) using a ":" (colon) as stop string for the @ESTRING@ parser needs to be escaped using a second ":" (colon). It seems this is not the case: patterns fail when using "::", but match when using ":": $ pdbtool test -v estring_colon_escaped.pdb Testing message program='dummy' message='foo: bar' Wrong match name='.classifier.rule_id', value='', expected='0304be33-4f3e-459e-8888-d86769986c32' Wrong match name='key', value='', expected='foo' Wrong match name='value', value='', expected='bar' $ pdbtool test -v estring_colon.pdb Testing message program='dummy' message='foo: bar' Match name='.classifier.rule_id', value='0304be33-4f3e-459e-8888-d86769986c32', expected='0304be33-4f3e-459e-8888-d86769986c32' Match name='key', value='foo', expected='foo' Match name='value', value='bar', expected='bar' -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

FreeBSD core dump
by Alexandre Biancalana 17 Oct '13

17 Oct '13

Hi list, I'm running syslog-ng 3.4.3 (installed from FreeBSD ports) on FreeBSD 9.1 with self-written patterndb file to parse apache error_log, mod_security and pagespeed errors. The syslog-ng source is httpd error_log file, after starting syslog-ng it core dumps: ERROR:logmsg.c:1303:log_msg_unref: assertion failed: (LOGMSG_REFCACHE_VALUE_TO_REF(old_value) >= 1) Abort trap (core dumped) Let me know if you need more info. Regards, Alexandre

2 16

beta3 packages for openSUSE / SLES
by Peter Czanik 11 Oct '13

11 Oct '13

Hello, My 3.5 beta3 packages for openSUSE and SLES are available at the usual location: https://build.opensuse.org/package/show/home:czanik:syslog-ng35/syslog-ng There was one major change since beta2: redis destination is enabled. Please note, that for SLES you need to use my repo for dependencies. On openSUSE only redis support is not included in the base system from the dependencies. While adding my repo is the easiest way to install packages, it is also possible to download rpm-s and install them, except for syslog-ng-redis. Happy testing! Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

syslog-ng 3.5.0beta3 has been released
by devel＠balabit.hu 11 Oct '13

11 Oct '13

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.5.0beta3 SUMMARY : new beta release DATE : Oct 11, 2013 ------------------------------------------------------------------------------ DESCRIPTION: A new beta version of syslog-ng Open Source Edition (3.5.0beta3) has been released. Being a beta release, testing is most appreciated, but running it in production is not advised. CHANGES: 3.5.0beta3 Fri, 11 Oct 2013 10:21:38 +0200 This is the third BETA release of the syslog-ng 3.5 development branch, with stability fixes and other enhancements. While running in production is still not advised, testing is most appreciated. Incompatible changes ==================== Template escaping ----------------- Template escaping was changed in an incompatible way: previously, both the lexer and the template compiler used the '\' character for escaping, which was confusing. The template compiler uses '$$' to escape the '$' char, and '@@' to escape '@'. If a non-numeric value follows '@', a warning will be printed. Bugfixes ======== * syslog-ng should compile again on non-Linux platforms. * The flush() and fsync() options of the file destination were fixed, and they should work the same way now as they do in 3.3 and 3.4. * The hiredis library should be detected on the default include paths, so one does not necessarily need to specify --with-libhiredis non non-Debian systems. Said option was also made to work. * A memory corruption was fixed in the @STRING@ parser of db-parser(). * Excludes now work properly with value-pairs(), and they do not get ignored if the value to exclude is in the default set. Known Bugs ========== * The afstreams module is broken, does not compile, and does not work. This will be corrected before the final 3.5.1 release. Credits: ======== syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Balazs Scheidler <bazsi(a)balabit.hu> Cy Schubert <Cy.Schubert(a)komquats.com> Evan Rempel <erempel(a)uvic.ca> Gergely Nagy <algernon(a)balabit.hu> Lucas McLane <lucas(a)clicksecurity.com> Oscar Muñoz Peter Czanik <czanik(a)balabit.hu> Sergey Shuman Viktor Juhasz <jviktor(a)balabit.hu> DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

1 0

[Bug 169] New: Feature Request - reconnect network also resolve dns name
by bugzilla＠bugzilla.balabit.com 10 Oct '13

10 Oct '13

https://bugzilla.balabit.com/show_bug.cgi?id=169 Summary: Feature Request - reconnect network also resolve dns name Product: syslog-ng Version: 3.3.x Platform: Other OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 When a network connection is dropped the destination name should be resolved via DNS prior to the reconnect. Every couple of years to renew the hardware supporting our core syslog service we move the IP name to a new host. The desired process is; 1. Build a new host and ensure that syslog-ng is running correctly. 2. Change the DNS for our central syslog server to the address of the new server. 3. Restart the syslog-ng service on the old syslog server. 4. All of the clients will loose the connection to the old syslog server. 5. All of the clients will reconnect to the *IP Name* in their configuration files. 6. Monitor the old syslog server to determine any hosts that have not made the transition. What really happened in step 5 was that all of the client reconnected to the old syslog server :-( To accomplish the switch over I had to visit 450 servers and manually restart the syslog service on the client. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5

[Bug 252] New: Syslog-NG crashes with destination using unix-dgram() driver
by bugzilla＠bugzilla.balabit.com 09 Oct '13

09 Oct '13

https://bugzilla.balabit.com/show_bug.cgi?id=252 Summary: Syslog-NG crashes with destination using unix-dgram() driver Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: marc.falzon(a)cloudwatt.com Type of the Report: bug Estimated Hours: 0.0 Created an attachment (id=82) --> (https://bugzilla.balabit.com/attachment.cgi?id=82) syslog-ng.conf Hello I'm facing a Syslog-NG crash when trying to use a unix-dgram() driver as log destination (configuration used is attached to this issue): # syslog-ng -Fev Module loaded and initialized successfully; module='system-source' Module loaded and initialized successfully; module='afsocket' Module loaded and initialized successfully; module='affile' Module loaded and initialized successfully; module='afuser' Module loaded and initialized successfully; module='syslogformat' Segmentation fault # Service starts successfully if the log path using the d_local destination is commented. Version used (platform: Linux Ubuntu Precise 12.04): # syslog-ng -V syslog-ng 3.4.2 Installer-Version: 3.4.2 Revision: 3.4.2-1~mhp1.sysvinit~precise [@3b56ec1] (Ubuntu/12.04) Compile-Date: Jun 4 2013 16:52:14 Available-Modules: syslogformat,afprog,basicfuncs,confgen,afsocket-notls,afsocket,afsocket-tls,dbparser,cryptofuncs,csvparser,affile,afuser,system-source Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: on # uname -a Linux p-xxxxxx-0000 3.2.0-51-generic #77-Ubuntu SMP Wed Jul 24 20:18:19 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux # Let me know if you need any more information about my setup and/or configuration. Regards, m. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 12

[Bug 254] New: 3.4. 1 patterndb replacing HOST corrupts $HOST and $FULLHOST macros
by bugzilla＠bugzilla.balabit.com 09 Oct '13

09 Oct '13

https://bugzilla.balabit.com/show_bug.cgi?id=254 Summary: 3.4.1 patterndb replacing HOST corrupts $HOST and $FULLHOST macros Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: blocker Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 I am using a patterndb of <pattern>got messages from @STRING:HOST:-.@</pattern> to match the syslog line 2013-10-08T11:01:26-07:00 wolverine.comp.uvic.ca/wolverine.comp.uvic.ca/syslog2.uvic.ca local0.info heartbeat[18869]: got messages from camelotia.comp.uvic.ca so that the syslog line will appears as if it came from camelotia.comp.uvic.ca When this line gets logged to the files, it looks like 2013-10-08T11:01:26-07:00 camelotia.comp.uvic.ca^@heP^@^@^@ local0.info flare-heartbeat[18869]: got messages from camelotia.comp.uvic.ca wher the ^@ are ASCII 0 (control-@) characters. In other words, the HOST macro becomes corrupt. The line should look (preferably) like 2013-10-08T11:01:26-07:00 camelotia.comp.uvic.ca/wolverine.comp.uvic.ca/syslog2.uvic.ca local0.info flare-heartbeat[18869]: got messages from camelotia.comp.uvic.ca or (less preferably) 2013-10-08T11:01:26-07:00 camelotia.comp.uvic.ca local0.info flare-heartbeat[18869]: got messages from camelotia.comp.uvic.ca It isn''t consistent how many ^@ symbols are present. 2013-10-08T11:00:26-07:00 nestor0286.westgrid.uvic.ca^@^@ local0.info flare-heartbeat[18869]: got messages from nestor0286.westgrid.uvic.ca 2013-10-08T11:00:26-07:00 nestor0127.westgrid.uvic.ca^@^@^@^@^@^@ local0.info flare-heartbeat[18869]: got messages from nestor0127.westgrid.uvic.ca 2013-10-08T11:00:26-07:00 hermes0042.westgrid.uvic.ca^@^@ local0.info flare-heartbeat[18869]: got messages from hermes0042.westgrid.uvic.ca 2013-10-08T11:00:26-07:00 hermes0032.westgrid.uvic.ca^@^@ local0.info flare-heartbeat[18869]: got messages from hermes0032.westgrid.uvic.ca 2013-10-08T11:00:26-07:00 nestor0016.westgrid.uvic.ca^@fang local0.info flare-heartbeat[18869]: got messages from nestor0016.westgrid.uvic.ca 2013-10-08T11:00:26-07:00 nestor0027.westgrid.uvic.ca^@^@ local0.info flare-heartbeat[18869]: got messages from nestor0027.westgrid.uvic.ca -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

Fwd: Syslog NG simple filtering not working as expected...
by Paul Hutton 02 Oct '13

02 Oct '13

Initially sent to the wrong address.... Begin forwarded message: > From: Paul Hutton <paul_hutton(a)bigpond.com> > Subject: Syslog NG simple filtering not working as expected... > Date: 26 September 2013 11:23:15 PM AEST > To: syslog-ng-request(a)lists.balabit.hu > > Hi All, > > I am trying to forward network sourced syslogs to a number of destinations depending on a few simple filters, but am immediately coming unstuck. My constraint is that all incoming and outgoing network syslog traffic is udp. My intention in the configuration code below (apart from logging the localhost) is to: > > 1. Forward all incoming network sourced syslogs except those generated by two firewalls to destination Remote_L > 2. Forward incoming network syslogs that are generated by the firewalls to destination Remote_C (currently commented out) > 3. Forward all incoming network syslogs of severity 0, 1 and 2 to destinations Remote_D1 and Remote_D2 (currently commented out) > > In addition all incoming network sourced syslogs are logged locally to destination d_logger and I've added another local file d_debug out of curiosity. > > My problem is that with just the filter enabled at point 1 above and in the code below, the behaviour of syslog-ng is very erratic, beyond any reasonable anticipation of the unreliability of udp as the transport. If I comment out the filter line > > filter (not_firewalls); > > in the code below, then test traffic (none of which is generated by the firewall sources) is received and forwarded correctly as well as logged locally. If I uncomment the filter, then I have seen test traffic forwarded correctly on one occasion, but it is mostly the case that some smaller subset (including none) of the test traffic is forwarded. In this case, the test traffic is also correctly written to the local destination d_debug. That is in the same rule that fails to forward the test traffic. > > I am running 3.2.4 on Solaris 10 09/10 x86 obtained from the Sunfreeware site. > > I have also unashamedly plagiarised configuration code from a couple of posted sources, so if you recognise your work, thanks for putting it out there... > > > Thanks in advance. > > Paul > > > > @version: 3.2 > @include "scl.conf" > options { > stats_level(3); > flush_lines(100); > flush_timeout(10000); > }; > ################################################################ > # Ensure syslog-ng logs this host similarly to syslogd # > ################################################################ > source s_local { > sun-streams("/dev/log" door("/etc/.syslog_door")); > internal(); > }; > # > # Set global network source > # > source src { > udp(ip("0.0.0.0") port (514)); > }; > > ############################################################### > # Local logfiles and remote destinations setup # > ############################################################### > destination syslog { > file("/var/log/syslog"); > }; > destination messages { > file("/var/adm/messages"); > }; > destination loginlog { > file("/var/adm/loginlog"); > }; > # > # Set remote destinations > # > destination Remote_L { > udp ("10.octet.octet.octetL" port (514) ); > }; > destination Remote_D1 { > udp ("10.octet.octet.octetD1" port (514) spoof_source (yes)); > }; > destination Remote_D2 { > udp ("10.octet.octet.octetD2" port (514) spoof_source (yes)); > }; > destination Remote_C { > udp ("10.octet.octet.octetC" port (514) spoof_source (yes)); > }; > # > # Set local destination for all remote syslog input > # > destination d_logger { > file("/var/log/logger.log"); > }; > # > # Add another local debug file to check filters are working > # > destination d_debug { > file("/var/log/debug.log"); > }; > ############################################################## > # Filters for mimicking Solaris syslogd # > # identifying firewall syslogs # > # identifying severity levels 0 to 2 # > ############################################################## > filter f_syslog { > facility(mail) or (facility(daemon) and priority(notice)); > }; > filter f_messages { > priority(err) > or facility(kern) > or (facility(user) and priority(err)) > or (facility(daemon) and priority(notice)) > or (facility(mail) and priority(crit)); > }; > filter f_auth { > facility (auth); > }; > > # > # Set Firewall Filters > # > filter firewalls { > host("10.octet.octet.octetfw1") or host("10.octet.octet.octetfw2"); > }; > filter not_firewalls { > not (host("10.octet.octet.octetfw1") or host("10.octet.octet.octetfw2")); > }; > > # > # Filter Severity Levels 0, 1 and 2 > # > filter f_crit { > level (crit .. emerg); > }; > ############################################################## > # Now do some logging # > ############################################################## > > # > # Log local syslog data appropriately > # > log { > source(s_local); > filter(f_syslog); > destination(syslog); > }; > log { > source(s_local); > filter(f_messages); > destination(messages); > }; > log { > source(s_local); > filter(f_auth); > destination(loginlog); > }; > > # > # Log remotely sourced syslogs of all Severity Levels to remote L system > # but exclude syslogs from the firewalls > # > log { > source (src); > filter (not_firewalls); > destination (Remote_L); > destination (d_debug); > }; > # > # Log remotely sourced syslogs of Severity Levels 0 to 2 to remote D systems > # including syslogs from the firewalls > # > #log { > # source (src); > # filter (f_crit); > # destination (Remote_D1); > # destination (Remote_D2); > #}; > # > # Log remotely sourced syslogs from the two firewalls to remote C system > # > #log { > #source (src); > #filter (firewalls); > # destination (Remote_C); > #}; > # > # Log remotely sourced syslogs to a local file > # > log { > source (src); > destination (d_logger); > }; > > >

4 5

[Bug 241] New: pdbtool test --validate returns success when matching wrong pattern
by bugzilla＠bugzilla.balabit.com 02 Oct '13

02 Oct '13

https://bugzilla.balabit.com/show_bug.cgi?id=241 Summary: pdbtool test --validate returns success when matching wrong pattern Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 pdbtool test --validate pattern.xml only returns a non-zero exit code if test values don't match. If there is a catchall pattern in the pattern.xml database file then any test message that does not match the pattern it belongs to, but does match another pattern will return a zero exit code. The output still shows the wrong .classifier.rule_id Testing message program='mailguard' message='2013-07-03T13:13:38-07:00 local(a)dog.comp.uvic.ca local4.notice mailguard: Could not open /var/mimedefang/sophie.pid' Wrong match name='.classifier.rule_id', value='UN-eb0cfada-af50-48b0-ac1e-de1ffad8fe6c', expected='b826184c-2990-46d4-a777-f87340f1796a' but the return code from the pdbtool is still zero. I would expect any test message that does not match the pattern it belongs to should produce a non-zero return code. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

4 8

syslog-ng leaves a lot of open file handles
by Anton Koldaev 28 Sep '13

28 Sep '13

Hi, I'm using Syslog-NG OSE v.3.3.7-1~mhp1~lucid (Ubuntu Lucid) And I have the following destination file(): file("/u/logs/`app`/${MONTH}${DAY}/${1}/${1}${2}/${LOGSORT.ACCOUNT}.log" Syslog-NG switches to the new file at 23:59:59 every day just fine but for some reason it leaves files for the previous day open: *# date* Wed Jul 24 09:04:19 UTC 2013 *# lsof | grep a/ac/account.log* syslog-ng 30743 root 3351w REG 252,2 72597491 66306075 /u/logs/app/0723/a/ac/account.log (deleted) syslog-ng 30743 root 4896w REG 252,2 17017519 4572052 /u/logs/app/0724/a/ac/account.log And they're being deleted by my rotating script. Reloading syslog-ng using init script or with `kill -HUP` doesn't help - all deleted files are still open by syslog-ng. Global option "time_reap (30);" doesn't seem to help too. Any ideas? -- Best regards, Koldaev Anton

3 6

3.5beta1 openSUSE packages
by Peter Czanik 27 Sep '13

27 Sep '13

Hello, My first experimental and mostly untested beta1 packages for openSUSE are available now at https://build.opensuse.org/package/show/home:czanik:syslog-ng35/syslog-ng More testing and SLES packages will follow next week. Have a nice weekend! Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 2

syslog-ng 3.5.0beta2 has been released
by devel＠balabit.hu 27 Sep '13

27 Sep '13

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.5.0beta2 SUMMARY : new beta release DATE : Sep 27, 2013 ------------------------------------------------------------------------------ DESCRIPTION: A new beta version of syslog-ng Open Source Edition (3.5.0beta2) has been released. Being a beta release, testing is most appreciated, but running it in production is not advised. CHANGES: 3.5.0beta2 Fri, 27 Sep 2013 10:06:52 +0200 This is the second BETA release of the syslog-ng 3.5 development branch, introducing new features and a handful of fixes since the last beta version. While running in production is still not advised, testing is most appreciated. Major Features ============== Redis destination ----------------- Developed during the Google Summer of Code 2013 program, this destination driver makes it possible to easily send commands to a Redis server. For further information and examples, please see the following blog post: http://tichygsoc.blogspot.hu/2013/09/the-road-so-far.html Template type hinting --------------------- In 3.5.0beta1, type hinting was implemented for the MongoDB destination, with this release, it is also available for the $(format-json) template function. Template options honored everywhere ----------------------------------- Until this release, there were situations where template options were ignored, such as filter expressions that use the comparsion operators, regexp based substitutions, incoming templates for parsers, the new value rewrite rules, SMTP values, some of the new drivers (stomp, amqp), and patterndb context-id. These all honor the global template options now, and per-driver options such as frac-digits and local-time-zone are available for drivers which did not have them before, like MongoDB. Bugfixes ======== * The AMQP destination has been fixed, and using it does not lead to a crash anymore. * Several memory leaks in the STOMP destination have been fixed. * The --no-framing option of loggen has been fixed, and works as advertised now. * The in-list() filter ignores empty lines in its input file, to avoid matching on empty values. * Compilation on FreeBSD has been fixed. Credits: ======== syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Balazs Scheidler <bazsi(a)balabit.hu> Gergely Nagy <algernon(a)balabit.hu> Peter Czanik <czanik(a)balabit.hu> Tibor Benke <btibi(a)balabit.hu> Tihamer Petrovics <tihameri(a)gmail.com> Viktor Tusa <tusa(a)balabit.hu> DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

1 0

syslog-ng: redis destination, the road so far
by Tihamér Petrovics 25 Sep '13

25 Sep '13

Hello, I updated my blog with the following post: http://tichygsoc.blogspot.hu/2013/09/the-road-so-far.html I wrote a tutorial about using syslog-ng with redis, attached some sample config file and finally a complex one, that could be a case from everyday. Stay tuned, I intend to maintain the code after gsoc too! All the best, Tihi

1 0

insider 2013-09: syslog-ng 3.5 beta release; incubator project; GSoC updates;
by Peter Czanik 19 Sep '13

19 Sep '13

Dear syslog-ng users, This is the 27th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. FEATURED NEWS syslog-ng 3.5 beta1 is released ------------------------------- The first beta of syslog-ng 3.5 was released today. It has many internal rework and also many user visible new features. Here you can read just the headlines, for a complete list and details please visit the NEWS file ( https://github.com/balabit/syslog-ng-3.5/blob/master/NEWS ) or this blog post: http://asylum.madhouse-project.org/blog/2013/09/05/what-is-cooking-in-syslo… * Stomp destination * Multiline support * Blacklist support * and many more As usual, testing is greatly appreciated as it helps to iron out problems before the final release! Sources are available from git ( https://github.com/balabit/syslog-ng-3.5 ) or as tgz from http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… For third-party binary packages for various Linux and UNIX platforms, visithttp://www.balabit.com/network-security/syslog-ng/opensource-logging-s… .Debian/Ubuntu and openSUSE packages are expected to be available in the coming days. syslog-ng incubator project --------------------------- The syslog-ng module incubator (Incubator henceforth) is a collection of tools and modules for syslog-ng that for one reason or the other, are not part of the official repository. This serves both as a staging ground for experimental modules, and as a repository of plugins that are not aimed at upstream inclusion. It's also an example of a third party syslog-ng module. Sources are available at https://github.com/algernon/syslog-ng-incubator syslog-ng GSoC: finished successfully! -------------------------------------- The coding time for Google Summer of Code isover and both of our students finished their projects successfully. The redis destination is tomerged to syslog-ng 3.5 before the beta2 releaseand the high performance mysql destination is expected to follow it soon.A summary blog post of their work is due to appear next week at http://petrovicsgyula.blogspot.com/ and http://tichygsoc.blogspot.hu/ syslog-ng 3.5 draft documentation --------------------------------- A draft version of syslog-ng 3.5 documentation was also released today. It contains some of the new features. It is available in many forms: * PDF: http://www.balabit.com/support/documentation/syslog-ng-ose-3.5-guides/en/sy… * HTML: http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.5-guid… * Single page HTML: http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.5-guid… SHORT NEWS: * syslogng-web, a simple and fast log viewer: https://czanik.blogs.balabit.com/2013/08/syslog-ng-web-a-simple-and-fast-lo… * how to monitor a directory of log files: http://asylum.madhouse-project.org/blog/2013/09/10/howto-monitor-a-director… NEW RELEASES * syslog-ng 3.5beta1: https://lists.balabit.hu/pipermail/syslog-ng-announce/2013-September/000166… Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com <mailto:documentation@balabit.com> To read this newsletter on-line, visit: http://insider.blogs.balabit.com/ -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

syslog-ng 3.5.0beta1 has been released
by devel＠balabit.hu 19 Sep '13

19 Sep '13

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.5.0beta1 SUMMARY : new beta release DATE : Sep 19, 2013 ------------------------------------------------------------------------------ DESCRIPTION: A new beta version of syslog-ng Open Source Edition (3.5.0beta1) has been released. Being a beta release, testing is most appreciated, but running it in production is not advised. CHANGES: 3.5.0beta1 Thu, 19 Sep 2013 11:03:15 +0200 This is a BETA release of the syslog-ng 3.5 development branch, with a lot of internal rework and even more new features included. Being a beta release, testing is most appreciated, but running it in production is not advised. The release contains all the fixes from the 3.4 branch, the changes below are relative to the 3.4.3 release. Major Features ============== Multi-line support ------------------ A major feature in the 3.5 release is the inclusion of support for multi-line messages, a feature that has been available in syslog-ng PE for a good while. With this, both indented and regexp-based multi-line support becomes available. For further information and examples, see the following blog post: http://asylum.madhouse-project.org/blog/2013/09/05/what-is-cooking-in-syslo… STOMP destination ----------------- Joining the message-queue club, this new destination makes syslog-ng able to send events to any STOMP-capable message queuing server, such as Apache ActiveMQ. For further information about the destination, see this commit message: https://github.com/balabit/syslog-ng-3.5/commit/82d19a4d4ad3f91e8214006f6ea… Template type hinting --------------------- While syslog-ng supported sending events to various datastores and queues for a while now (SQL first, MongoDB, JSON, and AMQP later), even if those supported other types of data, syslog-ng only ever sent strings. With template type hinting, it became possible to tell syslog-ng what type a certain template should be, so that the drivers can use that information in whatever way they see fit. This is currently implemented for the MongoDB driver only. For more information about type hinting and for examples, see the following post: http://asylum.madhouse-project.org/blog/2013/09/05/what-is-cooking-in-syslo… Other features ============== * in-list() filter: this new filter function allows one to easily implement file-based white- and blacklists with a simple syntax: filter f_white { in-list("/path/to/file", value("HOST")); }; * You no longer need to remember how many zeros to put after a big number in the syslog-ng configuration file, you can use unit suffixes, such as: log-fifo-size(2GiB) See the following post for more details: http://asylum.madhouse-project.org/blog/2013/09/05/what-is-cooking-in-syslo… * A set of new string-related template functions are available: $(uppercase STRINGS...), $(lowercase STRINGS...) and $(replace-delimiter DELIMITERS NEW-DELIMITER TEXT). The first two do exactly what their names suggest, while the last one replaces all occurrences of any DELIMITERS within TEXT with the NEW-DELIMITER. * There is also a new $(env VARIABLE...) template function, which looks up the given variables in the environment. This is similar to using backticks in the configuration file, however the name of the environment variable with $(env) may contain macros too. * Support for Linux 3.5+'s /dev/kmsg was added, and the system() source will automatically detect whether to use it over /proc/kmsg. * For every correlated message in patterndb, the ${.classifier.context_id} property is automatically set to the context-id attribute of the matching rule. Incompatible changes ==================== * The replace() transformation function of value-pairs() was renamed to replace-prefix() to better reflect what it actually does. The old name remains as an obsolete alias, however. * The username() and password() options of the MongoDB destination were removed, they never did anything before, either. Miscellaneous changes ===================== * The build system was completely redone, it is much faster, more reliable, and less verbose by default now. * Several systemd-related enhancements were made, including support for notification-based startup. This also means that when systemd support is compiled in, libsystemd-daemon becomes a new dependency. Credits: ======== syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Andras Tim <tia(a)balabit.hu> Anton Koldaev <koldaevav(a)gmail.com> Attila M. Magyar <athos(a)balabit.hu> Attila Nagy <bra(a)fsn.hu> Attila Szalay <sasa(a)balabit.hu> Balazs Scheidler <bazsi(a)balabit.hu> Balint Kovacs <blint(a)balabit.hu> Chris Johnson <chris.johnson3(a)hp.com> Evan Rempel <erempel(a)uvic.ca> Fabien Wernli <cpan(a)faxm0dem.org> Gergely Nagy <algernon(a)balabit.hu> Juhasz Viktor <jviktor(a)balabit.hu> Laszlo Budai <lbudai(a)balabit.hu> Martin <bmartin(a)lavabit.com> Michael Sterrett <mr_bones_(a)gentoo.org> Michal Privoznik <miso.privoznik(a)gmail.com> Paul Dann <pdgiddie+balabit(a)gmail.com> Peter Czanik <czanik(a)balabit.hu> Peter Gyongyosi <gyp(a)balabit.hu> Robert Fekete <frobert(a)balabit.hu> Tamas Pal <folti(a)balabit.hu> Tibor Benke <btibi(a)balabit.hu> Valentijn Sessink <valentijn(a)sessink.nl> Viktor Tusa <tusa(a)balabit.hu> Vincent Brillault <spam(a)lerya.net> DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

1 0

[FR] symlink and wildcard character support for unix-dgram()
by Sergey Y. Afonin 16 Sep '13

16 Sep '13

Hello. Is it possible to implement support for symlinks and wildcard character in unix-dgram() function ? For example. Some services can use chroot environment and dev/log can be placed to chroot. Some directories may have a set of symbolic links to various dev/log: # ls -l /etc/syslog.d/* lrwxrwxrwx 1 root root 21 Apr 2 20:36 /etc/syslog.d/bind -> /var/lib/bind/dev/log lrwxrwxrwx 1 root root 22 Feb 19 2010 /etc/syslog.d/klogd -> /var/lib/klogd/dev/log it would be convenient to describe it as unix-dgram("/etc/syslog.d/") -- Regards, Sergey Y. Afonin asy(a)kraft-s.ru

2 4

[PATCH] template_escaping: escaping in templates is modified
by Viktor Juhasz 16 Sep '13

16 Sep '13

From: Juhasz Viktor <jviktor(a)balabit.hu> the '\' is parsed by lexer and the template_compile used this character to escaping. It was confusing. Now, it has to be used $$ for printing literal $ in the result The other modification is the escaping of the '@' if it follows macro or temlate function. if non-numeric value follows the '@', use the '@' as a literal '@', but show a warning message. It has to be used '@@' for printing literal '@' in the result Signed-off-by: Juhasz Viktor <jviktor(a)balabit.hu> --- lib/template/templates.c | 33 +++++++++++++++++++++++----- lib/template/tests/test_template_compile.c | 25 +++++++++++++++++++-- 2 files changed, 50 insertions(+), 8 deletions(-) diff --git a/lib/template/templates.c b/lib/template/templates.c index 436d645..442a7ec 100644 --- a/lib/template/templates.c +++ b/lib/template/templates.c @@ -882,13 +882,27 @@ parse_msg_ref(LogTemplateCompiler *self) if ((*self->cursor) == '@') { self->cursor++; - /* syntax: ${name}@1 to denote the log message index in the correllation state */ - while ((*self->cursor) >= '0' && (*self->cursor) <= '9') + if ((*self->cursor) >= '0' && (*self->cursor) <= '9') { - self->msg_ref += self->msg_ref * 10 + ((*self->cursor) - '0'); - self->cursor++; + /* syntax: ${name}@1 to denote the log message index in the correllation state */ + while ((*self->cursor) >= '0' && (*self->cursor) <= '9') + { + self->msg_ref += self->msg_ref * 10 + ((*self->cursor) - '0'); + self->cursor++; + } + self->msg_ref += 1; + } + else + { + if ((*self->cursor) != '@') + { + msg_warning("Non-numeric correlation state ID found, assuming a literal '@' character. To avoid confusion when using a literal '@' after a macro or template function, write '@@' in the template.", + evt_tag_str("Template", self->template->template), + NULL); + self->cursor--; + } + self->msg_ref = 0; } - self->msg_ref += 1; } } @@ -1171,7 +1185,14 @@ log_template_compiler_process_token(LogTemplateCompiler *self, GError **error) } if (*self->cursor == '\\') { - self->cursor++; + if (cfg_is_config_version_older(self->template->cfg, 0x305)) + { + msg_warning("Template escaping changed in version 3.5. Use '$$' to specify a literal dollar sign instead of '\\$' and remove the escaping of the backslash character when you upgrade your configuration", + evt_tag_str("Template", self->template->template), + NULL); + self->cursor++; + } + } if (*self->cursor) { diff --git a/lib/template/tests/test_template_compile.c b/lib/template/tests/test_template_compile.c index efebfdd..21e1885 100644 --- a/lib/template/tests/test_template_compile.c +++ b/lib/template/tests/test_template_compile.c @@ -203,17 +203,22 @@ static void test_macro_with_invalid_msgref_are_recognized_as_the_top_element_in_the_stack(void) { assert_template_compile("${MESSAGE}(a)gmail.com"); - assert_compiled_template(text = "", default_value = NULL, macro = M_MESSAGE, type = LTE_MACRO, msg_ref = 1); + assert_compiled_template(text = "", default_value = NULL, macro = M_MESSAGE, type = LTE_MACRO, msg_ref = 0); select_next_element(); - assert_compiled_template(text = "gmail.com", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); + assert_compiled_template(text = "@gmail.com", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); } static void test_dollar_prefixed_with_backslash_is_a_literal_dollar(void) { + cfg_set_version(configuration, 0x304); assert_template_compile("Test \\$STRING"); assert_compiled_template(text = "Test $STRING", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); + + cfg_set_version(configuration, 0x305); + assert_template_compile("Test \\$STRING"); + assert_compiled_template(text = "Test \\", default_value = NULL, value_handle = log_msg_get_value_handle("STRING"), type = LTE_VALUE, msg_ref = 0); } static void @@ -250,8 +255,23 @@ test_dollar_with_an_invalid_macro_name_without_braces_is_parsed_as_a_literal_dol static void test_backslash_without_finishing_the_escape_sequence_is_ignored(void) { + cfg_set_version(configuration, 0x304); assert_template_compile("foo\\"); assert_compiled_template(text = "foo", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); + + cfg_set_version(configuration, 0x305); + assert_template_compile("foo\\"); + assert_compiled_template(text = "foo\\", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); +} + +static void +test_double_at_is_a_literal_at(void) +{ + assert_template_compile("${MESSAGE}@@12"); + assert_compiled_template(text = "", default_value = NULL, macro = M_MESSAGE, type = LTE_MACRO, msg_ref = 0); + + select_next_element(); + assert_compiled_template(text = "@12", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); } static void @@ -270,6 +290,7 @@ test_template_compile_macro(void) TEMPLATE_TESTCASE(test_double_dollars_is_a_literal_dollar); TEMPLATE_TESTCASE(test_dollar_with_an_invalid_macro_name_without_braces_is_parsed_as_a_literal_dollar); TEMPLATE_TESTCASE(test_backslash_without_finishing_the_escape_sequence_is_ignored); + TEMPLATE_TESTCASE(test_double_at_is_a_literal_at); } static void -- 1.7.9.5

1 0

Bug report
by Gonzalo Paniagua 15 Sep '13

15 Sep '13

Hi there. I have been having trouble creating a user in bugzilla and wanted to let you guys know about a bug I run into. syslog-ng version: 3.4.3 built from sources syslog-ng -V: syslog-ng 3.4.3 Installer-Version: 3.4.3 Revision: ssh+git://algernon@git.balabit /var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.4#no_branch#64d670f3cbfb90769f3c7f0fdd9c70bb9136ec5b Compile-Date: Sep 11 2013 05:33:29 Available-Modules: confgen,afsocket-tls,csvparser,affile,system-source,syslogformat,afuser,dbparser,afsocket-notls,basicfuncs,cryptofuncs,afsocket,afprog Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-Linux-Caps: off Enable-Pcre: on On rare occasions (I can't reproduce it reliably) syslog-ng segfaults when reloading (syslog-ng-ctl reload). I have a core dump file that gave me this stack trace: #0 0x00007fbd30102131 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007fbd30101d76 in strdup () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x00007fbd2f868b19 in evt_tag_str (tag=0x7fbd309a42f0 "encoding", value=0xffffffffffffffff <Address 0xffffffffffffffff out of bounds>) at evttags.c:76 #3 0x00007fbd3097918c in log_proto_server_options_validate (options=<optimized out>) at logproto-server.c:138 #4 0x00007fbd3097d191 in log_proto_server_validate_options (self=<optimized out>) at logproto-server.h:82 #5 log_reader_init (s=0x14a6a30) at logreader.c:638 #6 0x00007fbd2e9d5b07 in log_pipe_init (cfg=0x0, s=0x14a6a30) at ../../lib/logpipe.h:253 #7 log_pipe_init (cfg=0x0, s=0x14a6a30) at afsocket-source.c:763 #8 afsocket_sc_init (s=0x14a68f0) at afsocket-source.c:149 #9 0x00007fbd2e9d67fd in log_pipe_init (cfg=0x0, s=0x14a68f0) at ../../lib/logpipe.h:253 #10 log_pipe_init (cfg=0x0, s=0x14a68f0) at afsocket-source.c:763 #11 afsocket_sd_init (s=0x1489630) at afsocket-source.c:548 #12 0x00007fbd2e9d9719 in afunix_sd_init (s=<optimized out>) at afunix-source.c:194 #13 0x00007fbd3096d27a in log_pipe_init (cfg=<optimized out>, s=0x1489630) at logpipe.h:253 #14 cfg_tree_start (self=0x1c15c48) at cfg-tree.c:1064 #15 0x00007fbd309685de in cfg_init (cfg=0x1c15ae0) at cfg.c:220 #16 0x00007fbd30984faf in main_loop_reload_config_apply () at mainloop.c:501 #17 0x00007fbd3096d699 in control_connection_reload (self=0x1d09e90, command=<optimized out>) at control.c:150 #18 0x00007fbd3096dae8 in control_connection_io_input (s=0x1d09e90) at control.c:242 #19 control_connection_io_input (s=0x1d09e90) at control.c:172 #20 0x00007fbd309a106c in iv_fd_poll_and_run (st=0x1482580, to=<optimized out>) at iv_fd.c:163 #21 0x00007fbd309a1aac in iv_main () at iv_main_posix.c:117 #22 0x00007fbd309859fb in main_loop_run () at mainloop.c:736 #23 0x000000000040142b in main (argc=1, argv=0x7fffd8212a58) at main.c:267 When at frame #5, I got this: (gdb) print *self->proto->options $19 = {destroy = 0, initialized = -1, encoding = 0xffffffffffffffff <Address 0xffffffffffffffff out of bounds>, convert = 0xffffffffffffffff, max_msg_size = -1, max_buffer_size = 0, init_buffer_size = 0} I could not find anywhere in the code where initialized is set to -1, so I assume this is some kind of memory corruption. Let me know if you need anything else from me. Cheers. -Gonzalo

1 1

[PATCH (3.5)] log_template_compile: function refactorized
by Juhász Viktor 12 Sep '13

12 Sep '13

From: Juhasz Viktor <jviktor(a)balabit.hu> implementation of LogTemplateCompiler object helps to understand the whole compile process and reduces the number of parameters of the functions. Some unit test cases added to the unit test. The next possible step should be, that split the templates.c file into more files. This patch based on the log-template-refactor branch Signed-off-by: Juhasz Viktor <jviktor(a)balabit.hu> --- lib/template/templates.c | 609 ++++++++++++++++------------ lib/template/tests/test_template_compile.c | 16 + 2 files changed, 366 insertions(+), 259 deletions(-) diff --git a/lib/template/templates.c b/lib/template/templates.c index 1454cd2..13abf57 100644 --- a/lib/template/templates.c +++ b/lib/template/templates.c @@ -653,6 +653,13 @@ log_macro_lookup(gchar *macro, gint len) return macro_id; } +typedef struct { + LogTemplate *template; + gchar *cursor; + GString *text; + gint msg_ref; +} LogTemplateCompiler; + enum { LTE_MACRO, @@ -785,43 +792,43 @@ log_template_reset_compiled(LogTemplate *self) } static void -log_template_add_macro_elem(LogTemplate *self, guint macro, GString *text, gchar *default_value, gint msg_ref) +log_template_add_macro_elem(LogTemplateCompiler *self, guint macro, gchar *default_value) { LogTemplateElem *e; e = g_new0(LogTemplateElem, 1); e->type = LTE_MACRO; - e->text_len = text ? text->len : 0; - e->text = text ? g_strndup(text->str, text->len) : NULL; + e->text_len = self->text ? self->text->len : 0; + e->text = self->text ? g_strndup(self->text->str, self->text->len) : NULL; e->macro = macro; e->default_value = default_value; - e->msg_ref = msg_ref; - self->compiled_template = g_list_prepend(self->compiled_template, e); + e->msg_ref = self->msg_ref; + self->template->compiled_template = g_list_prepend(self->template->compiled_template, e); } static void -log_template_add_value_elem(LogTemplate *self, gchar *value_name, gsize value_name_len, GString *text, gchar *default_value, gint msg_ref) +log_template_add_value_elem(LogTemplateCompiler *self, gchar *value_name, gsize value_name_len, gchar *default_value) { LogTemplateElem *e; gchar *dup; e = g_new0(LogTemplateElem, 1); e->type = LTE_VALUE; - e->text_len = text ? text->len : 0; - e->text = text ? g_strndup(text->str, text->len) : NULL; + e->text_len = self->text ? self->text->len : 0; + e->text = self->text ? g_strndup(self->text->str, self->text->len) : NULL; /* value_name is not NUL terminated */ dup = g_strndup(value_name, value_name_len); e->value_handle = log_msg_get_value_handle(dup); g_free(dup); e->default_value = default_value; - e->msg_ref = msg_ref; - self->compiled_template = g_list_prepend(self->compiled_template, e); + e->msg_ref = self->msg_ref; + self->template->compiled_template = g_list_prepend(self->template->compiled_template, e); } /* NOTE: this steals argv if successful */ static gboolean -log_template_add_func_elem(LogTemplate *self, GString *text, gint argc, gchar *argv[], gint msg_ref, GError **error) +log_template_add_func_elem(LogTemplateCompiler *self, gint argc, gchar *argv[], GError **error) { LogTemplateElem *e; Plugin *p; @@ -834,11 +841,11 @@ log_template_add_func_elem(LogTemplate *self, GString *text, gint argc, gchar *a e = g_malloc0(sizeof(LogTemplateElem) + (argc - 1) * sizeof(LogTemplate *)); e->type = LTE_FUNC; - e->text_len = text ? text->len : 0; - e->text = text ? g_strndup(text->str, text->len) : NULL; - e->msg_ref = msg_ref; + e->text_len = self->text ? self->text->len : 0; + e->text = self->text ? g_strndup(self->text->str, self->text->len) : NULL; + e->msg_ref = self->msg_ref; - p = plugin_find(self->cfg, LL_CONTEXT_TEMPLATE_FUNC, argv[0]); + p = plugin_find(self->template->cfg, LL_CONTEXT_TEMPLATE_FUNC, argv[0]); if (!p) { g_set_error(error, LOG_TEMPLATE_ERROR, LOG_TEMPLATE_ERROR_COMPILE, "Unknown template function %s", argv[0]); @@ -846,21 +853,21 @@ log_template_add_func_elem(LogTemplate *self, GString *text, gint argc, gchar *a } else { - e->func.ops = plugin_construct(p, self->cfg, LL_CONTEXT_TEMPLATE_FUNC, argv[0]); + e->func.ops = plugin_construct(p, self->template->cfg, LL_CONTEXT_TEMPLATE_FUNC, argv[0]); } e->func.state = g_malloc0(e->func.ops->size_of_state); /* prepare may modify the argv array: remove and rearrange elements */ memcpy(argv_copy, argv, (argc + 1) * sizeof(argv[0])); - if (!e->func.ops->prepare(e->func.ops, e->func.state, self, argc, argv_copy, error)) + if (!e->func.ops->prepare(e->func.ops, e->func.state, self->template, argc, argv_copy, error)) { e->func.ops->free_state(e->func.state); g_free(e->func.state); goto error; } g_strfreev(argv); - self->compiled_template = g_list_prepend(self->compiled_template, e); + self->template->compiled_template = g_list_prepend(self->template->compiled_template, e); return TRUE; error: @@ -870,283 +877,367 @@ log_template_add_func_elem(LogTemplate *self, GString *text, gint argc, gchar *a return FALSE; } +static LogTemplateCompiler * +log_template_compiler_new(LogTemplate *template) +{ + LogTemplateCompiler *self = g_new0(LogTemplateCompiler, 1); + self->template = log_template_ref(template); + self->cursor = self->template->template; + self->text = g_string_sized_new(32); + return self; +} + static void -parse_msg_ref(gchar **p, gint *msg_ref) +log_template_compiler_free(LogTemplateCompiler *self) { - *msg_ref = 0; - if ((**p) == '@') + log_template_unref(self->template); + g_string_free(self->text, TRUE); + g_free(self); +} + + +static void +parse_msg_ref(LogTemplateCompiler *self) +{ + self->msg_ref = 0; + if ((*self->cursor) == '@') { - (*p)++; + self->cursor++; /* syntax: ${name}@1 to denote the log message index in the correllation state */ - while ((**p) >= '0' && (**p) <= '9') + while ((*self->cursor) >= '0' && (*self->cursor) <= '9') { - (*msg_ref) += (*msg_ref) * 10 + ((**p) - '0'); - (*p)++; + self->msg_ref += self->msg_ref * 10 + ((*self->cursor) - '0'); + self->cursor++; } - *msg_ref += 1; + self->msg_ref += 1; } } -gboolean -log_template_compile(LogTemplate *self, const gchar *template, GError **error) +static void +log_template_fill_compile_error(GError **error, const gchar *error_info, gint error_pos) { - gchar *start, *p; - guint last_macro = M_NONE; - GString *last_text = NULL; - gchar *error_info; - gint error_pos = 0; + g_set_error(error, LOG_TEMPLATE_ERROR, LOG_TEMPLATE_ERROR_COMPILE, "%s, error_pos='%d'", error_info, error_pos); +} - g_return_val_if_fail(error == NULL || *error == NULL, FALSE); +static void +log_template_append_and_increment(LogTemplateCompiler *self, GString *text) +{ + g_string_append_c(text, *self->cursor); + self->cursor++; +} + +static gint +log_template_get_macro_length(gchar *start, gchar *end, gchar **token) +{ + gint result = 0; + gchar *colon = memchr(start, ':', end - start - 1); + if (colon) + { + result = colon - start; + *token = colon < end ? colon + 1 : NULL; + } + else + { + result = end - start - 1; + *token = NULL; + } + return result; +} - log_template_reset_compiled(self); - if (self->template) - g_free(self->template); - self->template = g_strdup(template); - p = self->template; - while (*p) +static gchar * +log_template_compiler_get_default_value(LogTemplateCompiler *self, gchar *token) +{ + g_assert(token); + if (*token != '-') { - if (last_text == NULL) - last_text = g_string_sized_new(32); - if (*p == '\\') + return NULL; + } + return g_strndup(token + 1, self->cursor - token - 2); +} + +static inline gboolean +is_macro_name(gchar c) +{ + return (c >= 'A' && c <= 'Z') || (c >= 'a' && c <= 'z') || (c == '_') || (c >= '0' && c <= '9'); +} + +#define STEP_BY_TRUE(p, x) while(x) p++; + +static void +log_template_compiler_add_elem(LogTemplateCompiler *self, gchar *start, gint macro_len, gchar *default_value) +{ + gint macro = log_macro_lookup(start, macro_len); + if (macro == M_NONE) + { + log_template_add_value_elem(self, start, macro_len, default_value); + } + else + { + log_template_add_macro_elem(self, macro, default_value); + } +} + +static gboolean +log_template_compiler_process_braced_template(LogTemplateCompiler *self, GError **error) +{ + gint macro_len; + gchar *default_value = NULL; + gchar *start; + gchar *end; + gchar *token; + + self->cursor++; + start = self->cursor; + end = strchr(self->cursor, '}'); + + if (!end) + { + log_template_fill_compile_error(error, "Invalid macro, '}' is missing", strlen(self->template->template)); + return FALSE; + } + + self->cursor = end + 1; + macro_len = log_template_get_macro_length(start, self->cursor, &token); + if (token) + { + default_value = log_template_compiler_get_default_value(self, token); + if (!default_value) { - p++; - if (*p) + log_template_fill_compile_error(error, "Unknown substitution function", token - self->template->template); + return FALSE; + } + } + parse_msg_ref(self); + log_template_compiler_add_elem(self, start, macro_len, default_value); + return TRUE; +} + +static gboolean +log_template_compiler_add_quoted_string(LogTemplateCompiler *self, gboolean is_top_level, GString *result) +{ + gchar *quote = self->cursor; + gchar *end_of_quote = strchr(quote + 1, *quote); + if (!end_of_quote) + { + return FALSE; + } + self->cursor = end_of_quote + 1; + if (is_top_level) + { + /* skip the quote in top-level and don't skip in expressions enclosed in parens */ + quote++; + } + else + { + end_of_quote++; + } + g_string_append_len(result, quote, end_of_quote - quote); + return TRUE; +} + +static gboolean +log_template_compiler_process_arg_list(LogTemplateCompiler *self, GPtrArray *result) +{ + GString *arg_buf = g_string_sized_new(32); + gint parens = 1; + self->cursor++; + + while (*self->cursor && *self->cursor == ' ') + self->cursor++; + + while(*self->cursor) + { + if (*self->cursor == '\\') + { + self->cursor++; + } + else if (*self->cursor == '(') + { + parens++; + } + else if (*self->cursor == ')') + { + parens--; + if (parens == 0) { - g_string_append_c(last_text, *p); - p++; + break; } } - else if (*p == '$') + else if (*self->cursor == '"' || *self->cursor == '\'') { - gboolean finished = FALSE; - p++; - /* macro reference */ - if (*p == '{') + if (!log_template_compiler_add_quoted_string(self, parens == 1, arg_buf)) { - gchar *colon, *fncode; - gint macro_len; - gchar *default_value = NULL; - gint msg_ref = 0; + g_ptr_array_add(result, NULL); + g_string_free(arg_buf, TRUE); + return FALSE; + } + continue; + } + else if (parens == 1 && (*self->cursor == ' ' || *self->cursor == '\t')) + { + g_ptr_array_add(result, g_strndup(arg_buf->str, arg_buf->len)); + g_string_truncate(arg_buf, 0); + while (*self->cursor && (*self->cursor == ' ' || *self->cursor == '\t')) + self->cursor++; + continue; + } + log_template_append_and_increment(self, arg_buf); + } + if (arg_buf->len > 0) + { + g_ptr_array_add(result, g_strndup(arg_buf->str, arg_buf->len)); + } + g_ptr_array_add(result, NULL); + g_string_free(arg_buf, TRUE); + return *self->cursor == ')'; +} - p++; - start = p; - while (*p && *p != '}') - p++; +static gboolean +log_template_compiler_process_template_function(LogTemplateCompiler *self, GError **error) +{ + GPtrArray *strv = g_ptr_array_new(); - if (!*p) - { - error_pos = p - self->template; - error_info = "Invalid macro, '}' is missing"; - goto error; - } + if (!log_template_compiler_process_arg_list(self, strv)) + { + log_template_fill_compile_error(error, "Invalid template function reference, missing function name or inbalanced '('", self->cursor - self->template->template); + goto error; + } + self->cursor++; - p++; + parse_msg_ref(self); + if (!log_template_add_func_elem(self, strv->len - 1, (gchar **) strv->pdata, error)) + { + goto error; + } + g_ptr_array_free(strv, FALSE); + return TRUE; +error: + g_strfreev((gchar **)strv->pdata); + g_ptr_array_free(strv, FALSE); + return FALSE; +} - colon = memchr(start, ':', p - start - 1); - if (colon) - { - macro_len = colon - start; - fncode = colon < p ? colon + 1 : NULL; - } - else - { - macro_len = p - start - 1; - fncode = NULL; - } +static void +log_template_compiler_process_unbraced_template(LogTemplateCompiler *self) +{ + gchar *start = self->cursor; + gint macro_len; + do + { + self->cursor++; + } + while (is_macro_name(*self->cursor)); + macro_len = self->cursor - start; + log_template_compiler_add_elem(self, start, macro_len, NULL); +} - if (fncode) - { - if (*fncode == '-') - { - default_value = g_strndup(fncode + 1, p - fncode - 2); - } - else - { - error_pos = fncode - self->template; - error_info = "Unknown substitution function"; - goto error; - } - } - parse_msg_ref(&p, &msg_ref); - last_macro = log_macro_lookup(start, macro_len); - if (last_macro == M_NONE) - { - /* this was not a known macro, take it as a "value" reference */ - log_template_add_value_elem(self, start, macro_len, last_text, default_value, msg_ref); - } - else - { - log_template_add_macro_elem(self, last_macro, last_text, default_value, msg_ref); - } - finished = TRUE; - } - /* template function */ - else if (*p == '(') - { - gchar quote = 0; - gint parens = 1; - GPtrArray *strv; - GString *arg_buf; - gint msg_ref; - - strv = g_ptr_array_new(); - arg_buf = g_string_sized_new(32); - - p++; - start = p; - /* skip white space */ - while (*p && *p == ' ') - p++; - - g_string_truncate(arg_buf, 0); - while (*p) - { - if (!quote) - { - if (*p == '\\') - { - p++; - } - else if (*p == '"' || *p == '\'') - { - quote = *p; - if (parens == 1) - { - /* skip the quote in top-level and don't skip in expressions enclosed in parens */ - p++; - continue; - } - } - else if (parens == 1 && (*p == ' ' || *p == '\t')) - { - g_ptr_array_add(strv, g_strndup(arg_buf->str, arg_buf->len)); - g_string_truncate(arg_buf, 0); - while (*p && (*p == ' ' || *p == '\t')) - p++; - continue; - } - else if (*p == '(') - { - parens++; - } - else if (*p == ')') - { - parens--; - if (parens == 0) - break; - } - } - else - { - if (quote == *p) - { - /* end of string */ - quote = 0; - if (parens == 1) - { - p++; - continue; - } - } - } - if (*p) - { - g_string_append_c(arg_buf, *p); - p++; - } - } +static gboolean +log_template_compiler_process_value(LogTemplateCompiler *self, GError **error) +{ + gboolean finished = FALSE; + gchar p; + self->cursor++; + p = *self->cursor; + /* macro reference */ + if (p == '{') + { + if (!log_template_compiler_process_braced_template(self, error)) + { + return FALSE; + } + finished = TRUE; + } + /* template function */ + else if (p == '(') + { + if (!log_template_compiler_process_template_function(self, error)) + { + return FALSE; + } + finished = TRUE; + } + /* unbraced macro */ + else if (is_macro_name(p)) + { + log_template_compiler_process_unbraced_template(self); + finished = TRUE; + } + /* escaped value with dollar */ + else + { + if (p != '$') + { + g_string_append_c(self->text, '$'); + } + if (p) + { + log_template_append_and_increment(self, self->text); + } + } + if (finished) + { + g_string_truncate(self->text, 0); + } + return TRUE; +} - if (arg_buf->len) - g_ptr_array_add(strv, g_strndup(arg_buf->str, arg_buf->len)); - g_ptr_array_add(strv, NULL); - if (*p != ')') - { - g_string_free(arg_buf, TRUE); - g_ptr_array_foreach(strv, (GFunc) g_free, NULL); - g_ptr_array_free(strv, TRUE); +gboolean +log_template_compiler_process_token(LogTemplateCompiler *self, GError **error) +{ + self->msg_ref = 0; + if (*self->cursor == '$') + { + return log_template_compiler_process_value(self, error); + } + if (*self->cursor == '\\') + { + self->cursor++; + } + if (*self->cursor) + { + log_template_append_and_increment(self, self->text); + } + return TRUE; +} - error_pos = p - self->template; - error_info = "Invalid template function reference, missing function name or inbalanced '('"; - goto error; - } - p++; - parse_msg_ref(&p, &msg_ref); - if (!log_template_add_func_elem(self, last_text, strv->len - 1, (gchar **) strv->pdata, msg_ref, error)) - { - g_ptr_array_foreach(strv, (GFunc) g_free, NULL); - g_ptr_array_free(strv, TRUE); - g_string_free(arg_buf, TRUE); - goto error_set; - } - g_ptr_array_free(strv, FALSE); - g_string_free(arg_buf, TRUE); - finished = TRUE; - } - else if ((*p >= 'A' && *p <= 'Z') || (*p >= 'a' && *p <= 'z') || (*p == '_') || (*p >= '0' && *p <= '9')) - { - start = p; - do - { - p++; - } - while ((*p >= 'A' && *p <= 'Z') || (*p >= 'a' && *p <= 'z') || (*p == '_') || (*p >= '0' && *p <= '9')); - last_macro = log_macro_lookup(start, p - start); - if (last_macro == M_NONE) - { - /* this was not a known macro, take it as a "value" reference */ - log_template_add_value_elem(self, start, p-start, last_text, NULL, 0); - } - else - { - log_template_add_macro_elem(self, last_macro, last_text, NULL, 0); - } - finished = TRUE; - } - else if (*p == '$') - { - g_string_append_c(last_text, '$'); - p++; - } - else - { - g_string_append_c(last_text, '$'); - g_string_append_c(last_text, *p); - p++; - } - if (finished) - { - if (last_text) - g_string_free(last_text, TRUE); - last_macro = M_NONE; - last_text = NULL; - } - } - else +static gboolean +log_template_compiler_compile(LogTemplateCompiler *self, GError **error) +{ + while(*self->cursor) + { + if (!log_template_compiler_process_token(self, error)) { - g_string_append_c(last_text, *p); - p++; + log_template_reset_compiled(self->template); + g_string_sprintf(self->text, "error in template: %s", self->template->template); + log_template_add_macro_elem(self, M_NONE, NULL); + return FALSE; } } - if (last_macro != M_NONE || last_text) + if (self->text->len) { - log_template_add_macro_elem(self, last_macro, last_text, NULL, 0); - g_string_free(last_text, TRUE); + log_template_add_macro_elem(self, M_NONE, NULL); } - self->compiled_template = g_list_reverse(self->compiled_template); + self->template->compiled_template = g_list_reverse(self->template->compiled_template); return TRUE; +} - error: - g_set_error(error, LOG_TEMPLATE_ERROR, LOG_TEMPLATE_ERROR_COMPILE, "%s, error_pos='%d'", error_info, error_pos); +gboolean +log_template_compile(LogTemplate *self, const gchar *template, GError **error) +{ + LogTemplateCompiler *compiler; + gboolean result; + + g_return_val_if_fail(error == NULL || *error == NULL, FALSE); - error_set: log_template_reset_compiled(self); - if (!last_text) - last_text = g_string_sized_new(0); - g_string_sprintf(last_text, "error in template: %s", self->template); - log_template_add_macro_elem(self, M_NONE, last_text, NULL, 0); - g_string_free(last_text, TRUE); - return FALSE; + if (self->template) + g_free(self->template); + self->template = g_strdup(template); + compiler = log_template_compiler_new(self); + result = log_template_compiler_compile(compiler, error); + log_template_compiler_free(compiler); + return result; } void diff --git a/lib/template/tests/test_template_compile.c b/lib/template/tests/test_template_compile.c index 5fe243e..efebfdd 100644 --- a/lib/template/tests/test_template_compile.c +++ b/lib/template/tests/test_template_compile.c @@ -334,11 +334,19 @@ test_simple_template_function_with_additional_text(void) } static void +test_qouted_string_in_name_template_function(void) +{ + assert_template_compile("$(he\"ll\"o)"); + assert_compiled_template(text = "", default_value = NULL, func.ops = hello_construct(&hello_plugin, configuration, LL_CONTEXT_TEMPLATE_FUNC, "hello"), type = LTE_FUNC, msg_ref = 0); +} + +static void test_template_compile_func(void) { TEMPLATE_TESTCASE(test_simple_template_function); TEMPLATE_TESTCASE(test_complicated_template_function); TEMPLATE_TESTCASE(test_simple_template_function_with_additional_text); + TEMPLATE_TESTCASE(test_qouted_string_in_name_template_function); } static void @@ -372,6 +380,13 @@ test_template_function_bad2(void) } static void +test_template_function_bad3(void) +{ + assert_failed_template_compile("$(hello \"This is an unclosed quoted string)", "Invalid template function reference, missing function name or inbalanced '(', error_pos='8'"); + assert_compiled_template(text = "error in template: $(hello \"This is an unclosed quoted string)", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); +} + +static void test_unknown_function(void) { assert_failed_template_compile("$(unknown function)", "Unknown template function unknown"); @@ -385,6 +400,7 @@ test_template_compile_negativ_tests(void) TEMPLATE_TESTCASE(test_invalid_subst); TEMPLATE_TESTCASE(test_template_function_bad1); TEMPLATE_TESTCASE(test_template_function_bad2); + TEMPLATE_TESTCASE(test_template_function_bad3); TEMPLATE_TESTCASE(test_unknown_function); } -- 1.7.9.5

2 1