- syslog-ng - lists.balabit.hu

blocking on /dev/log
by Micah Anderson 12 Sep '13

12 Sep '13

Hello, I recently ugpraded a server to Debian Wheezy, which means I get 3.3.5-4 of syslog-ng. I did my best to upgrade the config to the 3.3 format, and then started it up and it seemed to be working fine. So I started providing services there, but under load the system seemed to be paused. Load is quite low on the system, but what was happening was logins via dovecot or ssh were pausing for a very long time. Typically, this indicates a DNS issue, but I quickly ruled that out. Eventually, I was reduced to using strace and I found that strace was pausing for a really long time trying to write a log message to file descriptor 4, and looking at what that FD was, I found: [pid 9051] connect(4, {sa_family=AF_FILE, path="/dev/log"}, 110) = 0 I stopped syslog-ng, and then voila, everything was working as it should be. I tried to increase my log_fifo_size(2048); to be 4096, but that didn't help, and so I started searching around and I found a similar bug in 2009 that was fixed[0]. I figured that bug was properly fixed, but maybe something came back, or something is wrong in my config. I noticed in the bug log that one of the configuration options that they had I did not, so I tried adding a max-connections argument to the unix-stream for the /dev/log: unix-stream("/dev/log" max-connections(2048)); so far that seems to have made things function ok, although the syslog-ng startup message produces a warning that has me puzzled: WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='12', new_log_iw_size='100', min_log_fifo_size='204800' So... I'm wondering, is this a bug in syslog-ng, or a bug in my config? Thanks for any help, or suggestions of things to try! Here is my config: @version: 3.3 # First, set some global options. options { # default is zero flush_lines(25); # default is 10000(10s) flush_timeout(20000); use_dns(no); use_fqdn(no); owner("root"); group("adm"); perm(0640); stats_freq(0); time_reopen(10); time_reap(360); create_dirs(yes); log_fifo_size(4096); }; ######################## # Sources ######################## source s_all { internal(); unix-stream("/dev/log" max-connections(2048)); file("/proc/kmsg" program_override("kernel")); }; ######################## # Rewrites ######################## rewrite r_ip { subst('(25[0-5]|2[0-4][0-9]|[0-1]?[0-9]?[0-9])([\.\-](25[0-5]|2[0-4][0-9]|[0-1]?[0-9]?[0-9])){3}', "0.0.0.0", value("MESSAGE"), type("pcre"), flags("global" "nobackref")); }; ######################## # Filters ######################## filter f_dbg { level(debug); }; filter f_info { level(info); }; filter f_notice { level(notice); }; filter f_warn { level(warn); }; filter f_err { level(err); }; filter f_crit { level(crit .. emerg); }; filter f_debug { level(debug) and not facility(auth, authpriv, news, mail); }; filter f_error { level(err .. emerg) ; }; filter f_cron { facility(cron) and not filter(f_debug); }; filter f_local { facility(local0, local1, local3, local4, local5, local6, local7) and not filter(f_debug); }; filter f_syslog3 { not facility(auth, authpriv, mail) and not filter(f_debug); }; filter f_cother { level(debug, info, notice, warn) or facility(daemon, mail); }; filter f_console { level(warn .. emerg); }; ######################## # Destinations ######################## ############################################################ ## apache.log filter f_apache { program("apache") and level(info); }; destination d_apache { file("/var/log/apache/access.log"); }; log { source(s_all); filter(f_apache); rewrite(r_ip); destination(d_apache); flags(final); }; filter f_apache_err { program("apache") and level(err); }; destination d_apache_err { file("/var/log/apache/error.log"); }; log { source(s_all); filter(f_apache_err); rewrite(r_ip); destination(d_apache_err); flags(final); }; ############################################################ ## auth.log filter f_auth { facility(auth, authpriv); }; destination df_auth { file("/var/log/auth.log"); }; log { source(s_all); filter(f_auth); destination(df_auth); }; ############################################################ ## clamav.log filter f_clamav { program("^clamav-milter|clamd"); }; destination d_clamav { file("/var/log/clamav.log"); }; log { source(s_all); filter(f_clamav); destination(d_clamav); flags(final); }; ############################################################ ## courier.log filter f_courier { program("courier|imap|pop"); }; destination d_courier { file("/var/log/courier.log"); }; log { source(s_all); filter(f_courier); rewrite(r_ip); destination(d_courier); flags(final); }; ############################################################ ## dovecot.log filter f_dovecot { program("dovecot.*") and not message("^lda"); }; destination d_dovecot { file("/var/log/dovecot/dovecot.log", group("dovecot")); }; log { source(s_all); filter(f_dovecot); rewrite(r_ip); destination(d_dovecot); flags(final); }; ############################################################ ## dovecot-lda.log filter f_dovecot-lda { program("dovecot.*") and message("^lda"); }; destination d_dovecot-lda { file("/var/log/dovecot/dovecot-lda.log", group("dovecot")); }; log { source(s_all); filter(f_dovecot-lda); rewrite(r_ip); destination(d_dovecot-lda); flags(final); }; ############################################################ ## daemon.log filter f_daemon { facility(daemon); }; destination df_daemon { file("/var/log/daemon.log"); }; log { source(s_all); filter(f_daemon); rewrite(r_ip); destination(df_daemon); }; ############################################################ ## errors.log filter f_errors { level(warn,err,crit,alert,emerg) and not facility(auth,authpriv,daemon,mail,user,kern,local1); }; destination df_errors { file("/var/log/errors.log"); }; log { source(s_all); filter(f_errors); destination(df_errors); }; ############################################################ ## emergencies filter f_emerg { level(emerg); }; destination du_all { usertty("*"); }; log { source(s_all); filter(f_emerg); destination(du_all); }; ############################################################ ## kern.log filter f_kern { facility(kern); }; destination df_kern { file("/var/log/kern.log"); }; log { source(s_all); filter(f_kern); destination(df_kern); }; ############################################################ ## ldap.log filter f_ldap { program("slapd"); }; destination d_ldap { file("/var/log/ldap.log"); }; log { source(s_all); filter(f_ldap); destination(d_ldap); flags(final); }; ############################################################ ## maildrop.log filter f_maildrop { program("^maildrop"); }; destination d_maildrop { file("/var/log/maildrop.log"); }; log { source(s_all); filter(f_maildrop); destination(d_maildrop); flags(final); }; ############################################################ ## mail.log filter f_mail { facility(mail) and not facility(local1) and not program("^postfix/"); }; destination df_mail { file("/var/log/mail.log"); }; log { source(s_all); filter(f_mail); destination(df_mail); }; ############################################################ ## messages.log filter f_messages { level(debug,info,notice) and not facility(auth,authpriv,daemon,mail,user,kern,local1); }; destination df_messages { file("/var/log/messages.log"); }; log { source(s_all); filter(f_messages); destination(df_messages); }; ############################################################ ## perdition.log filter f_perdition { program("perdition"); }; destination d_perdition { file("/var/log/perdition.log"); }; log { source(s_all); filter(f_perdition); destination(d_perdition); flags(final); }; ############################################################ ## postfix.log filter f_postfix { program("^postfix/"); }; # We increase the flush lines/timeout since postfix writes so much destination d_postfix { file("/var/log/postfix.log" flush_lines(500) flush_timeout(60000)); }; log { source(s_all); filter(f_postfix); rewrite(r_ip); destination(d_postfix); flags(final); }; ############################################################ ## rails logging filter f_rails { program("rails"); }; destination d_rails { file("/var/log/rails.log"); }; log { source(s_all); filter(f_rails); destination(d_rails); flags(final); }; ############################################################ ## user.log filter f_user { facility(user) and not facility(local1); }; destination df_user { file("/var/log/user.log"); }; log { source(s_all); filter(f_user); destination(df_user); }; # Some `catch-all' logfiles. # destination d_debug { file("/var/log/debug"); }; destination d_error { file("/var/log/error"); }; destination d_messages { file("/var/log/messages"); }; # The root's console. # destination d_console { usertty("root"); }; # Virtual console. # destination d_console_all { file("/dev/tty10"); }; 0. https://bugzilla.balabit.com/show_bug.cgi?id=48

3 2

[PATCH (3.5) 0/6]: Type hinting support
by Gergely Nagy 06 Sep '13

06 Sep '13

The following patchset introduces type hinting support, for mongodb and format-json for the time being, along with a couple of cleanups done prior to the introduction. The code passes unit and func tests, and is the result of a pair programming session by Balazs Scheidler <bazsi(a)balabit.hu> and myself, I rearranged the result and split it up into individual, mostly self-contained commits. To use type hinting with value-pairs: value-pair(pair("SOME-INTEGER" int("1234")) pair("SOME-DATETIME" datetime("$UNIXTIME"))); Simiarly, with format-json: template("$(format-json some-integer=int(1234) some-datetime=datetime($UNIXTIME))\n") Further documentation will follow later, most likely in a form of a blog post.

2 7

[Bug 247] New: FEATURE: add context-id value pair
by bugzilla＠bugzilla.balabit.com 06 Sep '13

06 Sep '13

https://bugzilla.balabit.com/show_bug.cgi?id=247 Summary: FEATURE: add context-id value pair Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bugzilla.balabit(a)faxm0dem.org Type of the Report: --- Estimated Hours: 0.0 It would be great to add the context-id to the message from the patterndb, e.g. ".classifier.context-id" or something. For one it would greatly help troubleshooting. Thanks -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

2 11

Problems with rewrite set and template functions...
by Johnson, Chris (HP TippingPoint Roseville) 05 Sep '13

05 Sep '13

Hi all, I've come across a problem when using the rewrite set function with a template function. I've created a custom template function 'audit-TPTI-to-Email' and use it in a rewrite: rewrite r_audit_EMail { set("$(audit-TPTI-to-EMail ${MSG})", value("MSG")); }; Then call it: filter f_audit_pgm{program("AUDIT-*" type("glob"));}; log { source(s_local); filter(f_audit_pgm); log { destination(d_logID_02); }; log { rewrite(r_audit_EMail); rewrite(r_quote_newlines); destination(d_logID_13); }; flags(final); }; Everything work fine. Then if I add another call to rewrite (i.e. add a second email destination): filter f_audit_pgm{program("AUDIT-*" type("glob"));}; log { source(s_local); filter(f_audit_pgm); log { destination(d_logID_02); }; log { rewrite(r_audit_EMail); rewrite(r_quote_newlines); destination(d_logID_13); }; log { rewrite(r_audit_EMail); rewrite(r_quote_newlines); destination(d_logID_14); }; flags(final); }; Syslog-ng crashes with a segfault. I've narrowed in down to any template function (just to make sure *I* wasn't screwing something up in my custom function): rewrite r_echo { set("$(echo $PROGRAM)" value("PROGRAM")); }; destination d_test1{ file("/var/log/test1.log"); }; destination d_test2{ file("/var/log/test2.log"); }; log { source(s_local); log { rewrite(r_echo); destination(d_test1); }; log { rewrite(r_echo); destination(d_test2); }; }; The backtrace: Backtrace: /usr/local/lib/libsyslog-ng-3.3.3.so(plugin_find+0x39)[0x7f3eb76ff019] /usr/local/lib/libsyslog-ng-3.3.3.so(log_template_compile+0x84f)[0x7f3eb7703baf] /usr/local/lib/libsyslog-ng-3.3.3.so(log_rewrite_set_new+0x99)[0x7f3eb76f3349] /usr/local/lib/libsyslog-ng-3.3.3.so[0x7f3eb76f3371] /usr/local/lib/libsyslog-ng-3.3.3.so(log_center_init_pipe_line+0x35d)[0x7f3eb76dfecd] /usr/local/lib/libsyslog-ng-3.3.3.so(log_center_init_pipe_line+0xd2)[0x7f3eb76dfc42] /usr/local/lib/libsyslog-ng-3.3.3.so(log_center_init+0x56)[0x7f3eb76e0226] /usr/local/lib/libsyslog-ng-3.3.3.so(cfg_init+0xb0)[0x7f3eb76e1530] /usr/local/lib/libsyslog-ng-3.3.3.so(main_loop_init+0x11b)[0x7f3eb76f9abb] /usr/local/sbin/syslog-ng(main+0x11f)[0x40168f] /lib/libc.so.6(__libc_start_main+0xe6)[0x7f3eb6240126] /usr/local/sbin/syslog-ng[0x401379] I threw in some debug statements: LogRewrite * log_rewrite_set_new(const gchar *new_value) { fprintf(stderr, "%s('%s'):\n", __FUNCTION__, new_value); Plugin * plugin_find(GlobalConfig *cfg, gint plugin_type, const gchar *plugin_name) { fprintf(stderr, "%s(%p, %d, '%s'): '\n", __FUNCTION__, cfg, plugin_type, plugin_name); Which showed that the 'cfg' pointer is null when rewrite is called the second time: log_rewrite_set_new('$(echo $PROGRAM)'): plugin_find(0x60e210, 13, 'echo'): ' plugin_find: plugin->name = 'sys-to-EMail' plugin_find: plugin->name = 'audit-TPTI-to-EMail' plugin_find: plugin->name = 'quar-TPTI-to-EMail' plugin_find: plugin->name = 'quar-TPTI-to-CEF' plugin_find: plugin->name = 'tab-to-bar' plugin_find: plugin->name = 'tab-to-semicolon' plugin_find: plugin->name = 'tab-to-comma' plugin_find: plugin->name = 'to-upper-case' plugin_find: plugin->name = 'to-lower-case' plugin_find: plugin->name = 'ipv4-to-int' plugin_find: plugin->name = 'log-session-seqnum' plugin_find: plugin->name = 'indent-multi-line' plugin_find: plugin->name = 'if' plugin_find: plugin->name = 'grep' plugin_find: plugin->name = 'echo' plugin_find(0x60e210, 2, 'file'): ' [...] log_rewrite_set_new('$(echo $PROGRAM)'): plugin_find((nil), 13, 'echo'): ' *** Segmentation fault Sooo, my questions are: Is this expected behavior? Has this been patched already? Is there another way I can call a custom function to reformat the message field on a destination-by-destination basis? Thanks, Chris ---------------------------------------- Christopher Johnson chris.johnson3(a)hp.com<mailto:chris.johnson3@hp.com> HP Software - Security Product Group (916) 785-2817 ----------------------------------------

3 5

[PATCH] filter: Implement an in-list() filter
by Gergely Nagy 05 Sep '13

05 Sep '13

The in-list filter allows us to do simple file-based black- or whitelisting. The syntax is: filter f_whitelist { in-list("/path/to/file.list", value("PROGRAM")); }; Implementing a blacklist is as simple as negating the in-list() filter. The filter works by pulling in the whole file (one entry per line) and putting it into a tree structure, and comparing against that on eval. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- Makefile.am | 2 + lib/filter/Makefile.am | 2 + lib/filter/filter-expr-grammar.ym | 21 ++++++- lib/filter/filter-expr-parser.c | 1 + lib/filter/filter-in-list.c | 95 +++++++++++++++++++++++++++++++ lib/filter/filter-in-list.h | 33 +++++++++++ lib/filter/tests/Makefile.am | 13 ++++- lib/filter/tests/test.list | 1 + lib/filter/tests/test_filters_in_list.c | 53 +++++++++++++++++ 9 files changed, 215 insertions(+), 6 deletions(-) create mode 100644 lib/filter/filter-in-list.c create mode 100644 lib/filter/filter-in-list.h create mode 100644 lib/filter/tests/test.list create mode 100644 lib/filter/tests/test_filters_in_list.c diff --git a/Makefile.am b/Makefile.am index 265de8d..345ee76 100644 --- a/Makefile.am +++ b/Makefile.am @@ -1,6 +1,8 @@ SUBDIRS = AM_MAKEFLAGS = --no-print-directory +AM_TESTS_ENVIRONMENT = top_srcdir="$(top_srcdir)" + ACLOCAL_AMFLAGS = -I m4 --install EXTRA_DIST = ${BUILT_SOURCES} VERSION \ diff --git a/lib/filter/Makefile.am b/lib/filter/Makefile.am index 7425491..aeb4f2b 100644 --- a/lib/filter/Makefile.am +++ b/lib/filter/Makefile.am @@ -2,6 +2,7 @@ filter_headers = \ lib/filter/filter-expr.h \ lib/filter/filter-op.h \ lib/filter/filter-cmp.h \ + lib/filter/filter-in-list.h \ lib/filter/filter-tags.h \ lib/filter/filter-netmask.h \ lib/filter/filter-call.h \ @@ -14,6 +15,7 @@ filter_sources = \ lib/filter/filter-expr.c \ lib/filter/filter-op.c \ lib/filter/filter-cmp.c \ + lib/filter/filter-in-list.c \ lib/filter/filter-tags.c \ lib/filter/filter-netmask.c \ lib/filter/filter-call.c \ diff --git a/lib/filter/filter-expr-grammar.ym b/lib/filter/filter-expr-grammar.ym index 1d4fec4..a41932b 100644 --- a/lib/filter/filter-expr-grammar.ym +++ b/lib/filter/filter-expr-grammar.ym @@ -1,5 +1,5 @@ /* - * Copyright (c) 2002-2011 BalaBit IT Ltd, Budapest, Hungary + * Copyright (c) 2002-2013 BalaBit IT Ltd, Budapest, Hungary * Copyright (c) 1998-2011 Balázs Scheidler * * This library is free software; you can redistribute it and/or @@ -34,6 +34,7 @@ #include "filter/filter-netmask.h" #include "filter/filter-op.h" #include "filter/filter-cmp.h" +#include "filter/filter-in-list.h" #include "filter/filter-tags.h" #include "filter/filter-call.h" #include "filter/filter-re.h" @@ -55,7 +56,7 @@ FilterRE *last_re_filter; /* INCLUDE_DECLS */ -%token KW_PROGRAM +%token KW_PROGRAM KW_IN_LIST %left ';' %left KW_OR @@ -96,7 +97,21 @@ filter_simple_expr | KW_FILTER '(' string ')' { $$ = filter_call_new($3, configuration); free($3); } | KW_NETMASK '(' string ')' { $$ = filter_netmask_new($3); free($3); } | KW_TAGS '(' string_list ')' { $$ = filter_tags_new($3); } - | KW_PROGRAM '(' string + | KW_IN_LIST '(' string string ')' + { + const gchar *p = $4; + if (p[0] == '$') + { + msg_warning("Value references in filters should not use the '$' prefix, those are only needed in templates", + evt_tag_str("value", $4), + NULL); + p++; + } + $$ = filter_in_list_new($3, p); + free($3); + free($4); + } + | KW_PROGRAM '(' string { last_re_filter = (FilterRE *) filter_re_new(LM_V_PROGRAM); } diff --git a/lib/filter/filter-expr-parser.c b/lib/filter/filter-expr-parser.c index c2c2e7b..781220a 100644 --- a/lib/filter/filter-expr-parser.c +++ b/lib/filter/filter-expr-parser.c @@ -55,6 +55,7 @@ static CfgLexerKeyword filter_expr_keywords[] = { { "match", KW_MATCH }, { "netmask", KW_NETMASK }, { "tags", KW_TAGS, 0x0301 }, + { "in_list", KW_IN_LIST, 0x0305 }, { "type", KW_TYPE, 0x0300 }, { "value", KW_VALUE, 0x0300 }, diff --git a/lib/filter/filter-in-list.c b/lib/filter/filter-in-list.c new file mode 100644 index 0000000..6736196 --- /dev/null +++ b/lib/filter/filter-in-list.c @@ -0,0 +1,95 @@ +/* + * Copyright (c) 2013 BalaBit IT Ltd, Budapest, Hungary + * Copyright (c) 2013 Gergely Nagy <algernon(a)balabit.hu> + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA + * + * As an additional exemption you are allowed to compile & link against the + * OpenSSL libraries as published by the OpenSSL project. See the file + * COPYING for details. + * + */ + +#include "filter-in-list.h" +#include "logmsg.h" +#include "misc.h" + +#include <stdlib.h> +#include <stdio.h> + +typedef struct _FilterInList +{ + FilterExprNode super; + NVHandle value_handle; + GTree *tree; +} FilterInList; + +static gboolean +filter_in_list_eval(FilterExprNode *s, LogMessage **msgs, gint num_msg) +{ + FilterInList *self = (FilterInList *)s; + LogMessage *msg = msgs[0]; + const gchar *value; + gssize len = 0; + + value = log_msg_get_value(msg, self->value_handle, &len); + APPEND_ZERO(value, value, len); + + return (g_tree_lookup(self->tree, value) != NULL) ^ s->comp; +} + +static void +filter_in_list_free(FilterExprNode *s) +{ + FilterInList *self = (FilterInList *)s; + + g_tree_unref(self->tree); +} + +FilterExprNode * +filter_in_list_new(const gchar *list_file, const gchar *property) +{ + FilterInList *self; + FILE *stream; + size_t n; + gchar *line = NULL; + + stream = fopen(list_file, "r"); + if (!stream) + { + msg_error("Error opening in-list filter list file", + evt_tag_str("file", list_file), + evt_tag_errno("errno", errno), + NULL); + return NULL; + } + + self = g_new0(FilterInList, 1); + filter_expr_node_init(&self->super); + self->value_handle = log_msg_get_value_handle(property); + self->tree = g_tree_new((GCompareFunc) strcmp); + + while (getline(&line, &n, stream) != -1) + { + line[strlen(line) - 1] = '\0'; + g_tree_insert(self->tree, line, GINT_TO_POINTER(1)); + line = NULL; + } + fclose(stream); + + self->super.eval = filter_in_list_eval; + self->super.free_fn = filter_in_list_free; + return &self->super; +} diff --git a/lib/filter/filter-in-list.h b/lib/filter/filter-in-list.h new file mode 100644 index 0000000..9cf3c0d --- /dev/null +++ b/lib/filter/filter-in-list.h @@ -0,0 +1,33 @@ +/* + * Copyright (c) 2013 BalaBit IT Ltd, Budapest, Hungary + * Copyright (c) 2013 Gergely Nagy <algernon(a)balabit.hu> + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA + * + * As an additional exemption you are allowed to compile & link against the + * OpenSSL libraries as published by the OpenSSL project. See the file + * COPYING for details. + * + */ + +#ifndef FILTER_IN_LIST_H_INCLUDED +#define FILTER_IN_LIST_H_INCLUDED + +#include "filter-expr.h" + +FilterExprNode *filter_in_list_new(const gchar *list_file, + const gchar *property); + +#endif diff --git a/lib/filter/tests/Makefile.am b/lib/filter/tests/Makefile.am index 2b1e11a..3d72441 100644 --- a/lib/filter/tests/Makefile.am +++ b/lib/filter/tests/Makefile.am @@ -1,13 +1,20 @@ lib_filter_tests_TESTS = \ - lib/filter/tests/test_filters + lib/filter/tests/test_filters \ + lib/filter/tests/test_filters_in_list check_PROGRAMS += ${lib_filter_tests_TESTS} lib_filter_tests_test_filters_CFLAGS = $(TEST_CFLAGS) \ -I${top_srcdir}/lib/filter/tests - lib_filter_tests_test_filters_LDADD = $(TEST_LDADD) \ $(PREOPEN_SYSLOGFORMAT) - lib_filter_tests_test_filters_SOURCES = \ lib/filter/tests/test_filters.c + +lib_filter_tests_test_filters_in_list_CFLAGS = $(TEST_CFLAGS) \ + -I${top_srcdir}/lib/filter/tests +lib_filter_tests_test_filters_in_list_LDADD = $(TEST_LDADD) \ + $(PREOPEN_SYSLOGFORMAT) + +EXTRA_DIST += \ + lib/filter/tests/test.list diff --git a/lib/filter/tests/test.list b/lib/filter/tests/test.list new file mode 100644 index 0000000..dca8351 --- /dev/null +++ b/lib/filter/tests/test.list @@ -0,0 +1 @@ +test-program diff --git a/lib/filter/tests/test_filters_in_list.c b/lib/filter/tests/test_filters_in_list.c new file mode 100644 index 0000000..9844fa2 --- /dev/null +++ b/lib/filter/tests/test_filters_in_list.c @@ -0,0 +1,53 @@ +#include <stdlib.h> +#include <glib.h> + +#include "cfg.h" +#include "messages.h" +#include "syslog-names.h" +#include "logmsg.h" +#include "apphook.h" +#include "plugin.h" +#include "filter/filter-in-list.h" + +#include "testutils.h" + +static MsgFormatOptions parse_options; + +static LogMessage * +create_test_message (void) +{ + const gchar *log_message = "<15>Sep 4 15:03:55 localhost test-program[3086]: some random message"; + + return log_msg_new(log_message, strlen(log_message), NULL, &parse_options); +} + +int +main(int argc G_GNUC_UNUSED, char *argv[] G_GNUC_UNUSED) +{ + LogMessage *msg; + FilterExprNode *filter_node; + char *top_srcdir = getenv("top_srcdir"); + gchar *list_file; + + app_startup(); + + configuration = cfg_new(0x0305); + plugin_load_module("syslogformat", configuration, NULL); + msg_format_options_defaults(&parse_options); + msg_format_options_init(&parse_options, configuration); + + assert_not_null(top_srcdir, "The $top_srcdir environment variable MUST NOT be empty!"); + + list_file = g_strdup_printf("%s/lib/filter/tests/test.list", top_srcdir); + filter_node = filter_in_list_new(list_file, "PROGRAM"); + assert_not_null(filter_node, "Constructing an in-list filter"); + g_free (list_file); + + msg = create_test_message(); + + assert_true(filter_expr_eval(filter_node, msg), "in-list filter matches"); + + app_shutdown(); + + return 0; +} -- 1.7.10.4

2 2

[PATCH (3.5)] Add support for number unit suffixes
by Gergely Nagy 04 Sep '13

04 Sep '13

There are a number of cases in the configuration where one needs to set a size (think buffer sizes), and counting zeroes is extremely annoying and unintuitive. For this reason, we introduce number unit suffixes for kilo-, mega- and giga-: K, M, G, respectively (case-insensitive, optionally followed by a 'b' or 'B', for byte). The parser also understands KiB, MiB, etc to indicate base-two units. Anywhere where you can specify a number, you'll be able to specify one with a unit suffix from now on: log-fifo-size(200M) A function to parse longs already existed in basic-funcs (tf_parse_int), which was moved to lib/parse-number.c now, and enhanced to recognise the unit suffixes too. This also closes #28. Signed-off-by: Balazs Scheidler <bazsi(a)balabit.hu> Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- lib/Makefile.am | 2 + lib/cfg-lex.l | 10 ++- lib/parse-number.c | 161 ++++++++++++++++++++++++++++++++++++ lib/parse-number.h | 9 ++ lib/tests/Makefile.am | 8 +- lib/tests/test_parse_number.c | 118 ++++++++++++++++++++++++++ modules/basicfuncs/basic-funcs.c | 21 +---- modules/basicfuncs/numeric-funcs.c | 4 +- modules/basicfuncs/str-funcs.c | 4 +- 9 files changed, 310 insertions(+), 27 deletions(-) create mode 100644 lib/parse-number.c create mode 100644 lib/parse-number.h create mode 100644 lib/tests/test_parse_number.c diff --git a/lib/Makefile.am b/lib/Makefile.am index f7eb421..5814596 100644 --- a/lib/Makefile.am +++ b/lib/Makefile.am @@ -63,6 +63,7 @@ pkginclude_HEADERS += \ lib/ml-batched-timer.h \ lib/msg-format.h \ lib/nvtable.h \ + lib/parse-number.h \ lib/persist-state.h \ lib/plugin.h \ lib/plugin-types.h \ @@ -139,6 +140,7 @@ lib_libsyslog_ng_la_SOURCES = \ lib/ml-batched-timer.c \ lib/msg-format.c \ lib/nvtable.c \ + lib/parse-number.c \ lib/persist-state.c \ lib/plugin.c \ lib/poll-events.c \ diff --git a/lib/cfg-lex.l b/lib/cfg-lex.l index 7b1a81d..b923a7c 100644 --- a/lib/cfg-lex.l +++ b/lib/cfg-lex.l @@ -27,7 +27,7 @@ #include "cfg-lexer.h" #include "cfg-grammar.h" #include "messages.h" -#include "misc.h" +#include "parse-number.h" #include <string.h> #include <strings.h> @@ -160,7 +160,13 @@ word [^ \#'"\{\}\\;\n\t,|\.@:] (-|\+)?{digit}+\.{digit}+ { yylval->fnum = strtod(yytext, NULL); return LL_FLOAT; } 0x{xdigit}+ { yylval->num = strtoll(yytext + 2, NULL, 16); return LL_NUMBER; } 0{odigit}+ { yylval->num = strtoll(yytext + 1, NULL, 8); return LL_NUMBER; } -(-|\+)?{digit}+ { yylval->num = strtoll(yytext, NULL, 10); return LL_NUMBER; } +(-|\+)?{digit}+(M|m|G|g|k|K)?(i|I)?(b|B)? { + if (!parse_number_with_suffix(yytext, &yylval->num)) + { + YY_FATAL_ERROR("Invalid number specification"); + } + return LL_NUMBER; + } ({word}+(\.)?)*{word}+ { return cfg_lexer_lookup_keyword(yyextra, yylval, yylloc, yytext); } $ | $ | diff --git a/lib/parse-number.c b/lib/parse-number.c new file mode 100644 index 0000000..b9b1be5 --- /dev/null +++ b/lib/parse-number.c @@ -0,0 +1,161 @@ +#include "parse-number.h" + +#include <string.h> +#include <errno.h> +#include <stdlib.h> + +static gboolean +_valid_unit(const gchar unit_char) +{ + return (unit_char == 'B' || unit_char == 'b'); +} + +static gboolean +_valid_exponent(const gchar exponent_char) +{ + gchar e = exponent_char; + + return e == 'k' || e == 'K' || + e == 'm' || e == 'M' || + e == 'g' || e == 'G'; +} + +static gboolean +_parse_suffix(const gchar *suffix, gchar *exponent_char, gchar *base_char, gchar *unit_char) +{ + gint suffix_len; + + suffix_len = strlen(suffix); + if (suffix_len > 3) + return FALSE; + if (suffix_len == 0) + return TRUE; + + if (suffix_len == 3) + { + *exponent_char = suffix[0]; + *base_char = suffix[1]; + *unit_char = suffix[2]; + } + else if (suffix_len == 2) + { + *exponent_char = suffix[0]; + if (_valid_unit(suffix[1])) + *unit_char = suffix[1]; + else + *base_char = suffix[1]; + } + else if (suffix_len == 1) + { + if (_valid_exponent(suffix[0])) + *exponent_char = suffix[0]; + else if (_valid_unit(suffix[0])) + *unit_char = suffix[0]; + else + return FALSE; + } + return TRUE; +} + +static gboolean +_determine_multiplier(gchar base_char, glong *multiplier) +{ + if (base_char == 0) + *multiplier = 1000; + else if (base_char == 'I' || base_char == 'i') + *multiplier = 1024; + else + return FALSE; + return TRUE; +} + +static gboolean +_validate_unit(gchar unit_char) +{ + if (unit_char && !_valid_unit(unit_char)) + return FALSE; + return TRUE; +} + +static gboolean +_process_exponent(gchar exponent_char, glong *d, glong multiplier) +{ + switch (exponent_char) + { + case 'G': + case 'g': + (*d) *= multiplier; + case 'm': + case 'M': + (*d) *= multiplier; + case 'K': + case 'k': + (*d) *= multiplier; + case 0: + return TRUE; + default: + return FALSE; + } +} + +static gboolean +_process_suffix(const gchar *suffix, glong *d) +{ + gchar exponent_char = 0, base_char = 0, unit_char = 0; + glong multiplier = 0; + + if (!_parse_suffix(suffix, &exponent_char, &base_char, &unit_char)) + return FALSE; + + if (!_determine_multiplier(base_char, &multiplier)) + return FALSE; + + if (!_validate_unit(unit_char)) + return FALSE; + + if (!_process_exponent(exponent_char, d, multiplier)) + return FALSE; + + return TRUE; +} + +static gboolean +_parse_number(const gchar *s, gchar **endptr, long *d) +{ + glong val; + + errno = 0; + val = strtoll(s, endptr, 10); + + if ((errno == ERANGE && (val == LONG_MAX || val == LONG_MIN)) + || (errno != 0 && val == 0)) + return FALSE; + + if (*endptr == s) + return FALSE; + + *d = val; + return TRUE; +} + +gboolean +parse_number(const gchar *s, glong *d) +{ + gchar *endptr; + + if (!_parse_number(s, &endptr, d)) + return FALSE; + if (*endptr) + return FALSE; + return TRUE; +} + +gboolean +parse_number_with_suffix(const gchar *s, glong *d) +{ + gchar *endptr; + + if (!_parse_number(s, &endptr, d)) + return FALSE; + return _process_suffix(endptr, d); +} diff --git a/lib/parse-number.h b/lib/parse-number.h new file mode 100644 index 0000000..e8e0c52 --- /dev/null +++ b/lib/parse-number.h @@ -0,0 +1,9 @@ +#ifndef PARSE_NUMBER_H_INCLUDED +#define PARSE_NUMBER_H_INCLUDED + +#include "syslog-ng.h" + +gboolean parse_number_with_suffix(const gchar *str, glong *result); +gboolean parse_number(const gchar *str, glong *result); + +#endif diff --git a/lib/tests/Makefile.am b/lib/tests/Makefile.am index 6bead1d..ad5cd26 100644 --- a/lib/tests/Makefile.am +++ b/lib/tests/Makefile.am @@ -1,7 +1,8 @@ lib_tests_TESTS = \ lib/tests/test_log_message \ lib/tests/test_cfg_lexer_subst \ - lib/tests/test_type_hints + lib/tests/test_type_hints \ + lib/tests/test_parse_number check_PROGRAMS += ${lib_tests_TESTS} @@ -19,3 +20,8 @@ lib_tests_test_type_hints_CFLAGS = \ $(TEST_CFLAGS) lib_tests_test_type_hints_LDADD = \ $(TEST_LDADD) + +lib_tests_test_parse_number_CFLAGS = \ + $(TEST_CFLAGS) +lib_tests_test_parse_number_LDADD = \ + $(TEST_LDADD) diff --git a/lib/tests/test_parse_number.c b/lib/tests/test_parse_number.c new file mode 100644 index 0000000..5895005 --- /dev/null +++ b/lib/tests/test_parse_number.c @@ -0,0 +1,118 @@ +#include <stdlib.h> + +#include "testutils.h" +#include "parse-number.h" + +static void +assert_parse_with_suffix(const gchar *str, long expected) +{ + long n; + gboolean res; + + res = parse_number_with_suffix(str, &n); + + assert_gboolean(res, TRUE, "Parsing (w/ suffix) %s failed", str); + assert_gint64(n, expected, "Parsing (w/ suffix) %s failed", str); +} + +static void +assert_parse_with_suffix_fails(const gchar *str) +{ + long n; + gboolean res; + + res = parse_number_with_suffix(str, &n); + assert_gboolean(res, FALSE, "Parsing (w/ suffix) %s succeeded, while expecting failure", str); +} + +static void +assert_parse(const gchar *str, long expected) +{ + long n; + gboolean res; + + res = parse_number(str, &n); + + assert_gboolean(res, TRUE, "Parsing (w/o suffix) %s failed", str); + assert_gint64(n, expected, "Parsing (w/o suffix) %s failed", str); +} + +static void +assert_parse_fails(const gchar *str) +{ + long n; + gboolean res; + + res = parse_number(str, &n); + + assert_gboolean(res, FALSE, "Parsing (w/o suffix) %s succeeded, while expecting failure", str); +} + +static void +test_simple_numbers_are_parsed_properly(void) +{ + assert_parse_with_suffix("1234", 1234); + assert_parse_with_suffix("+1234", 1234); + assert_parse_with_suffix("-1234", -1234); + + assert_parse("1234", 1234); +} + +static void +test_exponent_suffix_is_parsed_properly(void) +{ + assert_parse_with_suffix("1K", 1000); + assert_parse_with_suffix("1k", 1000); + assert_parse_with_suffix("1m", 1000 * 1000); + assert_parse_with_suffix("1M", 1000 * 1000); + assert_parse_with_suffix("1G", 1000 * 1000 * 1000); + assert_parse_with_suffix("1g", 1000 * 1000 * 1000); +} + +static void +test_byte_units_are_accepted(void) +{ + assert_parse_with_suffix("1b", 1); + assert_parse_with_suffix("1B", 1); + assert_parse_with_suffix("1Kb", 1000); + assert_parse_with_suffix("1kB", 1000); + assert_parse_with_suffix("1mb", 1000 * 1000); + assert_parse_with_suffix("1MB", 1000 * 1000); + assert_parse_with_suffix("1Gb", 1000 * 1000 * 1000); + assert_parse_with_suffix("1gB", 1000 * 1000 * 1000); +} + +static void +test_base2_is_selected_by_an_i_modifier(void) +{ + assert_parse_with_suffix("1Kib", 1024); + assert_parse_with_suffix("1kiB", 1024); + assert_parse_with_suffix("1Ki", 1024); + assert_parse_with_suffix("1kI", 1024); + assert_parse_with_suffix("1mib", 1024 * 1024); + assert_parse_with_suffix("1MiB", 1024 * 1024); + assert_parse_with_suffix("1Gib", 1024 * 1024 * 1024); + assert_parse_with_suffix("1giB", 1024 * 1024 * 1024); +} + +static void +test_invalid_formats_are_not_accepted(void) +{ + assert_parse_with_suffix_fails("1234Z"); + assert_parse_with_suffix_fails("1234kZ"); + assert_parse_with_suffix_fails("1234kdZ"); + assert_parse_with_suffix_fails("1234kiZ"); + assert_parse_fails("1234kiZ"); +} + +int +main(int argc, char *argv[]) +{ + test_simple_numbers_are_parsed_properly(); + test_exponent_suffix_is_parsed_properly(); + test_byte_units_are_accepted(); + test_base2_is_selected_by_an_i_modifier(); + test_invalid_formats_are_not_accepted(); + + return 0; +} diff --git a/modules/basicfuncs/basic-funcs.c b/modules/basicfuncs/basic-funcs.c index cb73b02..523f6cf 100644 --- a/modules/basicfuncs/basic-funcs.c +++ b/modules/basicfuncs/basic-funcs.c @@ -26,6 +26,7 @@ #include "filter/filter-expr.h" #include "filter/filter-expr-parser.h" #include "cfg.h" +#include "parse-number.h" #include "str-format.h" #include "plugin-types.h" @@ -33,26 +34,6 @@ #include <errno.h> #include <string.h> -static gboolean -tf_parse_int(const gchar *s, long *d) -{ - gchar *endptr; - glong val; - - errno = 0; - val = strtoll(s, &endptr, 10); - - if ((errno == ERANGE && (val == LONG_MAX || val == LONG_MIN)) - || (errno != 0 && val == 0)) - return FALSE; - - if (endptr == s || *endptr != '\0') - return FALSE; - - *d = val; - return TRUE; -} - /* in order to avoid having to declare all construct functions, we * include them all here. If it causes compilation times to increase * drastically, we should probably make them into separate compilation diff --git a/modules/basicfuncs/numeric-funcs.c b/modules/basicfuncs/numeric-funcs.c index 6bd00ef..9125c3c 100644 --- a/modules/basicfuncs/numeric-funcs.c +++ b/modules/basicfuncs/numeric-funcs.c @@ -32,7 +32,7 @@ tf_num_parse(gint argc, GString *argv[], return FALSE; } - if (!tf_parse_int(argv[0]->str, n)) + if (!parse_number_with_suffix(argv[0]->str, n)) { msg_debug("Parsing failed, template function's first argument is not a number", evt_tag_str("function", func_name), @@ -40,7 +40,7 @@ tf_num_parse(gint argc, GString *argv[], return FALSE; } - if (!tf_parse_int(argv[1]->str, m)) + if (!parse_number_with_suffix(argv[1]->str, m)) { msg_debug("Parsing failed, template function's first argument is not a number", evt_tag_str("function", func_name), diff --git a/modules/basicfuncs/str-funcs.c b/modules/basicfuncs/str-funcs.c index 6c8ae3c..060795d 100644 --- a/modules/basicfuncs/str-funcs.c +++ b/modules/basicfuncs/str-funcs.c @@ -79,14 +79,14 @@ tf_substr(LogMessage *msg, gint argc, GString *argv[], GString *result) return; /* get offset position from second argument */ - if (!tf_parse_int(argv[1]->str, &start)) { + if (!parse_number(argv[1]->str, &start)) { msg_error("$(substr) parsing failed, start could not be parsed", evt_tag_str("start", argv[1]->str), NULL); return; } /* if we were called with >2 arguments, third was desired length */ if (argc > 2) { - if (!tf_parse_int(argv[2]->str, &len)) { + if (!parse_number(argv[2]->str, &len)) { msg_error("$(substr) parsing failed, length could not be parsed", evt_tag_str("length", argv[2]->str), NULL); return; } -- 1.7.10.4

1 0

Re: [syslog-ng] [PATCH] Add support for number unit suffixes
by Balazs Scheidler 04 Sep '13

04 Sep '13

Hi, I've received your patch via the merge-queue/3.5 branch: > There are a number of cases in the configuration where one needs to set > a size (think buffer sizes), and counting zeroes is extremely annoying > and unintuitive. For this reason, we introduce number unit suffixes for > kilo-, mega- and giga-: K, M, G, respectively (case-insensitive, > optionally followed by a 'b' or 'B', for byte). > > Anywhere where you can specify a number, you'll be able to specify one > with a unit suffix from now on: > > log-fifo-size(200M) > > A function to parse longs already existed in basic-funcs (tf_parse_int), > which was moved to lib/misc.c now, and enhanced to recognise the unit > suffixes too. > > This also closes #28. > > Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> > I really like the change as it definitely seems useful, however I'd have some things I'd like changed, and I've coded this up in a branch named numeric-suffixes. Could you please fold the patches and resubmit them? Thanks. This is the patch I had in mind commit 3fb2cfbbefe5b3d8f2952e807aa40a7b207a7a68 Author: Balazs Scheidler <bazsi(a)balabit.hu> Date: Tue Jul 30 16:37:31 2013 +0200 parse-number: follow up patch to number-units functionality This patch improves the patch series in a number of ways: - moves the number parsing to its separate module - removes the conflicting rule from the lexer, which would never match - accepts 'i' modifier in suffixes to indicate base2 units - string related functions don't get the suffixes functionality - extends the test suite, and adds test functions to indicate the intent of various asserts - test-num-parse is moved under lib/tests This patch should be folded back into the base set before integration. diff --git a/lib/Makefile.am b/lib/Makefile.am index d16f755..44ee107 100644 --- a/lib/Makefile.am +++ b/lib/Makefile.am @@ -62,6 +62,7 @@ pkginclude_HEADERS += \ lib/ml-batched-timer.h \ lib/msg-format.h \ lib/nvtable.h \ + lib/parse-number.h \ lib/persist-state.h \ lib/plugin.h \ lib/plugin-types.h \ @@ -135,6 +136,7 @@ lib_libsyslog_ng_la_SOURCES = \ lib/ml-batched-timer.c \ lib/msg-format.c \ lib/nvtable.c \ + lib/parse-number.c \ lib/persist-state.c \ lib/plugin.c \ lib/pragma-parser.c \ diff --git a/lib/cfg-lex.l b/lib/cfg-lex.l index ed61c6f..b923a7c 100644 --- a/lib/cfg-lex.l +++ b/lib/cfg-lex.l @@ -27,7 +27,7 @@ #include "cfg-lexer.h" #include "cfg-grammar.h" #include "messages.h" -#include "misc.h" +#include "parse-number.h" #include <string.h> #include <strings.h> @@ -160,14 +160,13 @@ word [^ \#'"\{\}\\;\n\t,|\.@:] (-|\+)?{digit}+\.{digit}+ { yylval->fnum = strtod(yytext, NULL); return LL_FLOAT; } 0x{xdigit}+ { yylval->num = strtoll(yytext + 2, NULL, 16); return LL_NUMBER; } 0{odigit}+ { yylval->num = strtoll(yytext + 1, NULL, 8); return LL_NUMBER; } -(-|\+)?{digit}+(M|m|G|g|k|K)?(b|B)? { - if (!num_parse_int(yytext, &yylval->num)) +(-|\+)?{digit}+(M|m|G|g|k|K)?(i|I)?(b|B)? { + if (!parse_number_with_suffix(yytext, &yylval->num)) { YY_FATAL_ERROR("Invalid number specification"); } return LL_NUMBER; } -(-|\+)?{digit}+ { yylval->num = strtoll(yytext, NULL, 10); return LL_NUMBER; } ({word}+(\.)?)*{word}+ { return cfg_lexer_lookup_keyword(yyextra, yylval, yylloc, yytext); } $ | $ | diff --git a/lib/misc.c b/lib/misc.c index 2d95c78..7a5fad6 100644 --- a/lib/misc.c +++ b/lib/misc.c @@ -635,48 +635,3 @@ utf8_escape_string(const gchar *str, gssize len) return res; } - -gboolean -num_parse_int(const gchar *s, long *d) -{ - gchar *endptr; - glong val; - - errno = 0; - val = strtoll(s, &endptr, 10); - - if ((errno == ERANGE && (val == LONG_MAX || val == LONG_MIN)) - || (errno != 0 && val == 0)) - return FALSE; - - if (endptr == s) - return FALSE; - - switch (*endptr) - { - case 'G': - case 'g': - val *= 1000; - case 'm': - case 'M': - val *= 1000; - case 'K': - case 'k': - val *= 1000; - endptr++; - break; - case '\0': - break; - default: - return FALSE; - } - - if (*endptr == 'b' || *endptr == 'B') - endptr++; - - if (*endptr != '\0') - return FALSE; - - *d = val; - return TRUE; -} diff --git a/lib/parse-number.c b/lib/parse-number.c new file mode 100644 index 0000000..b9b1be5 --- /dev/null +++ b/lib/parse-number.c @@ -0,0 +1,161 @@ +#include "parse-number.h" + +#include <string.h> +#include <errno.h> +#include <stdlib.h> + +static gboolean +_valid_unit(const gchar unit_char) +{ + return (unit_char == 'B' || unit_char == 'b'); +} + +static gboolean +_valid_exponent(const gchar exponent_char) +{ + gchar e = exponent_char; + + return e == 'k' || e == 'K' || + e == 'm' || e == 'M' || + e == 'g' || e == 'G'; +} + +static gboolean +_parse_suffix(const gchar *suffix, gchar *exponent_char, gchar *base_char, gchar *unit_char) +{ + gint suffix_len; + + suffix_len = strlen(suffix); + if (suffix_len > 3) + return FALSE; + if (suffix_len == 0) + return TRUE; + + if (suffix_len == 3) + { + *exponent_char = suffix[0]; + *base_char = suffix[1]; + *unit_char = suffix[2]; + } + else if (suffix_len == 2) + { + *exponent_char = suffix[0]; + if (_valid_unit(suffix[1])) + *unit_char = suffix[1]; + else + *base_char = suffix[1]; + } + else if (suffix_len == 1) + { + if (_valid_exponent(suffix[0])) + *exponent_char = suffix[0]; + else if (_valid_unit(suffix[0])) + *unit_char = suffix[0]; + else + return FALSE; + } + return TRUE; +} + +static gboolean +_determine_multiplier(gchar base_char, glong *multiplier) +{ + if (base_char == 0) + *multiplier = 1000; + else if (base_char == 'I' || base_char == 'i') + *multiplier = 1024; + else + return FALSE; + return TRUE; +} + +static gboolean +_validate_unit(gchar unit_char) +{ + if (unit_char && !_valid_unit(unit_char)) + return FALSE; + return TRUE; +} + +static gboolean +_process_exponent(gchar exponent_char, glong *d, glong multiplier) +{ + switch (exponent_char) + { + case 'G': + case 'g': + (*d) *= multiplier; + case 'm': + case 'M': + (*d) *= multiplier; + case 'K': + case 'k': + (*d) *= multiplier; + case 0: + return TRUE; + default: + return FALSE; + } +} + +static gboolean +_process_suffix(const gchar *suffix, glong *d) +{ + gchar exponent_char = 0, base_char = 0, unit_char = 0; + glong multiplier = 0; + + if (!_parse_suffix(suffix, &exponent_char, &base_char, &unit_char)) + return FALSE; + + if (!_determine_multiplier(base_char, &multiplier)) + return FALSE; + + if (!_validate_unit(unit_char)) + return FALSE; + + if (!_process_exponent(exponent_char, d, multiplier)) + return FALSE; + + return TRUE; +} + +static gboolean +_parse_number(const gchar *s, gchar **endptr, long *d) +{ + glong val; + + errno = 0; + val = strtoll(s, endptr, 10); + + if ((errno == ERANGE && (val == LONG_MAX || val == LONG_MIN)) + || (errno != 0 && val == 0)) + return FALSE; + + if (*endptr == s) + return FALSE; + + *d = val; + return TRUE; +} + +gboolean +parse_number(const gchar *s, glong *d) +{ + gchar *endptr; + + if (!_parse_number(s, &endptr, d)) + return FALSE; + if (*endptr) + return FALSE; + return TRUE; +} + +gboolean +parse_number_with_suffix(const gchar *s, glong *d) +{ + gchar *endptr; + + if (!_parse_number(s, &endptr, d)) + return FALSE; + return _process_suffix(endptr, d); +} diff --git a/lib/parse-number.h b/lib/parse-number.h new file mode 100644 index 0000000..e8e0c52 --- /dev/null +++ b/lib/parse-number.h @@ -0,0 +1,9 @@ +#ifndef PARSE_NUMBER_H_INCLUDED +#define PARSE_NUMBER_H_INCLUDED + +#include "syslog-ng.h" + +gboolean parse_number_with_suffix(const gchar *str, glong *result); +gboolean parse_number(const gchar *str, glong *result); + +#endif diff --git a/lib/tests/Makefile.am b/lib/tests/Makefile.am index 7117983..4d88daa 100644 --- a/lib/tests/Makefile.am +++ b/lib/tests/Makefile.am @@ -1,4 +1,8 @@ -lib_tests_TESTS = lib/tests/test_log_message lib/tests/test_cfg_lexer_subst +lib_tests_TESTS = \ + lib/tests/test_log_message \ + lib/tests/test_cfg_lexer_subst \ + lib/tests/test_parse_number + check_PROGRAMS += ${lib_tests_TESTS} lib_tests_test_log_message_CFLAGS = \ @@ -10,3 +14,8 @@ lib_tests_test_cfg_lexer_subst_CFLAGS = \ $(TEST_CFLAGS) lib_tests_test_cfg_lexer_subst_LDADD = \ $(TEST_LDADD) + +lib_tests_test_parse_number_CFLAGS = \ + $(TEST_CFLAGS) +lib_tests_test_parse_number_LDADD = \ + $(TEST_LDADD) diff --git a/lib/tests/test_parse_number.c b/lib/tests/test_parse_number.c new file mode 100644 index 0000000..5895005 --- /dev/null +++ b/lib/tests/test_parse_number.c @@ -0,0 +1,118 @@ +#include <stdlib.h> + +#include "testutils.h" +#include "parse-number.h" + +static void +assert_parse_with_suffix(const gchar *str, long expected) +{ + long n; + gboolean res; + + res = parse_number_with_suffix(str, &n); + + assert_gboolean(res, TRUE, "Parsing (w/ suffix) %s failed", str); + assert_gint64(n, expected, "Parsing (w/ suffix) %s failed", str); +} + +static void +assert_parse_with_suffix_fails(const gchar *str) +{ + long n; + gboolean res; + + res = parse_number_with_suffix(str, &n); + assert_gboolean(res, FALSE, "Parsing (w/ suffix) %s succeeded, while expecting failure", str); +} + +static void +assert_parse(const gchar *str, long expected) +{ + long n; + gboolean res; + + res = parse_number(str, &n); + + assert_gboolean(res, TRUE, "Parsing (w/o suffix) %s failed", str); + assert_gint64(n, expected, "Parsing (w/o suffix) %s failed", str); +} + +static void +assert_parse_fails(const gchar *str) +{ + long n; + gboolean res; + + res = parse_number(str, &n); + + assert_gboolean(res, FALSE, "Parsing (w/o suffix) %s succeeded, while expecting failure", str); +} + +static void +test_simple_numbers_are_parsed_properly(void) +{ + assert_parse_with_suffix("1234", 1234); + assert_parse_with_suffix("+1234", 1234); + assert_parse_with_suffix("-1234", -1234); + + assert_parse("1234", 1234); +} + +static void +test_exponent_suffix_is_parsed_properly(void) +{ + assert_parse_with_suffix("1K", 1000); + assert_parse_with_suffix("1k", 1000); + assert_parse_with_suffix("1m", 1000 * 1000); + assert_parse_with_suffix("1M", 1000 * 1000); + assert_parse_with_suffix("1G", 1000 * 1000 * 1000); + assert_parse_with_suffix("1g", 1000 * 1000 * 1000); +} + +static void +test_byte_units_are_accepted(void) +{ + assert_parse_with_suffix("1b", 1); + assert_parse_with_suffix("1B", 1); + assert_parse_with_suffix("1Kb", 1000); + assert_parse_with_suffix("1kB", 1000); + assert_parse_with_suffix("1mb", 1000 * 1000); + assert_parse_with_suffix("1MB", 1000 * 1000); + assert_parse_with_suffix("1Gb", 1000 * 1000 * 1000); + assert_parse_with_suffix("1gB", 1000 * 1000 * 1000); +} + +static void +test_base2_is_selected_by_an_i_modifier(void) +{ + assert_parse_with_suffix("1Kib", 1024); + assert_parse_with_suffix("1kiB", 1024); + assert_parse_with_suffix("1Ki", 1024); + assert_parse_with_suffix("1kI", 1024); + assert_parse_with_suffix("1mib", 1024 * 1024); + assert_parse_with_suffix("1MiB", 1024 * 1024); + assert_parse_with_suffix("1Gib", 1024 * 1024 * 1024); + assert_parse_with_suffix("1giB", 1024 * 1024 * 1024); +} + +static void +test_invalid_formats_are_not_accepted(void) +{ + assert_parse_with_suffix_fails("1234Z"); + assert_parse_with_suffix_fails("1234kZ"); + assert_parse_with_suffix_fails("1234kdZ"); + assert_parse_with_suffix_fails("1234kiZ"); + assert_parse_fails("1234kiZ"); +} + +int +main(int argc, char *argv[]) +{ + test_simple_numbers_are_parsed_properly(); + test_exponent_suffix_is_parsed_properly(); + test_byte_units_are_accepted(); + test_base2_is_selected_by_an_i_modifier(); + test_invalid_formats_are_not_accepted(); + + return 0; +} diff --git a/modules/basicfuncs/basic-funcs.c b/modules/basicfuncs/basic-funcs.c index c519d86..26fa8bd 100644 --- a/modules/basicfuncs/basic-funcs.c +++ b/modules/basicfuncs/basic-funcs.c @@ -26,7 +26,7 @@ #include "filter/filter-expr.h" #include "filter/filter-expr-parser.h" #include "cfg.h" -#include "misc.h" +#include "parse-number.h" #include "str-format.h" #include "plugin-types.h" diff --git a/modules/basicfuncs/numeric-funcs.c b/modules/basicfuncs/numeric-funcs.c index 39e4d83..9125c3c 100644 --- a/modules/basicfuncs/numeric-funcs.c +++ b/modules/basicfuncs/numeric-funcs.c @@ -32,7 +32,7 @@ tf_num_parse(gint argc, GString *argv[], return FALSE; } - if (!num_parse_int(argv[0]->str, n)) + if (!parse_number_with_suffix(argv[0]->str, n)) { msg_debug("Parsing failed, template function's first argument is not a number", evt_tag_str("function", func_name), @@ -40,7 +40,7 @@ tf_num_parse(gint argc, GString *argv[], return FALSE; } - if (!num_parse_int(argv[1]->str, m)) + if (!parse_number_with_suffix(argv[1]->str, m)) { msg_debug("Parsing failed, template function's first argument is not a number", evt_tag_str("function", func_name), diff --git a/modules/basicfuncs/str-funcs.c b/modules/basicfuncs/str-funcs.c index b0bcdd4..060795d 100644 --- a/modules/basicfuncs/str-funcs.c +++ b/modules/basicfuncs/str-funcs.c @@ -79,14 +79,14 @@ tf_substr(LogMessage *msg, gint argc, GString *argv[], GString *result) return; /* get offset position from second argument */ - if (!num_parse_int(argv[1]->str, &start)) { + if (!parse_number(argv[1]->str, &start)) { msg_error("$(substr) parsing failed, start could not be parsed", evt_tag_str("start", argv[1]->str), NULL); return; } /* if we were called with >2 arguments, third was desired length */ if (argc > 2) { - if (!num_parse_int(argv[2]->str, &len)) { + if (!parse_number(argv[2]->str, &len)) { msg_error("$(substr) parsing failed, length could not be parsed", evt_tag_str("length", argv[2]->str), NULL); return; } diff --git a/tests/unit/Makefile.am b/tests/unit/Makefile.am index d5c8ac6..8841834 100644 --- a/tests/unit/Makefile.am +++ b/tests/unit/Makefile.am @@ -15,8 +15,7 @@ tests_unit_TESTS = \ tests/unit/test_logwriter \ tests/unit/test_zone \ tests/unit/test_persist_state \ - tests/unit/test_value_pairs \ - tests/unit/test_num_parse + tests/unit/test_value_pairs check_PROGRAMS += \ ${tests_unit_TESTS} @@ -81,10 +80,6 @@ tests_unit_test_persist_state_LDADD = \ tests_unit_test_value_pairs_LDADD = \ $(TEST_LDADD) $(unit_test_extra_modules) -tests_unit_test_num_parse_CFLAGS = ${TEST_CFLAGS} -tests_unit_test_num_parse_LDADD = \ - $(TEST_LDADD) - CLEANFILES += \ test_values.persist \ test_values.persist- diff --git a/tests/unit/test_num_parse.c b/tests/unit/test_num_parse.c deleted file mode 100644 index ab75d9e..0000000 --- a/tests/unit/test_num_parse.c +++ /dev/null @@ -1,37 +0,0 @@ -#include <stdlib.h> - -#include "testutils.h" -#include "misc.h" - -static void -assert_num_parse (const gchar *str, long expected) -{ - long n; - gboolean res; - - res = num_parse_int(str, &n); - - assert_gboolean(res, TRUE, "Parsing %s failed", str); - assert_gint64(n, expected, "Parsing %s failed", str); -} - -int -main(int argc, char *argv[]) -{ - assert_num_parse("1234", 1234); - assert_num_parse("+1234", 1234); - assert_num_parse("-1234", -1234); - - assert_num_parse("1K", 1000); - assert_num_parse("1k", 1000); - assert_num_parse("1kB", 1000); - assert_num_parse("1Kb", 1000); - - assert_num_parse("1m", 1000 * 1000); - assert_num_parse("1M", 1000 * 1000); - - assert_num_parse("1G", 1000 * 1000 * 1000); - assert_num_parse("1g", 1000 * 1000 * 1000); - - return 0; -}

3 2

[Bug 251] New: Missing property 'inherit-properties' from tag 'action/ message' in patterndb.xsd
by bugzilla＠bugzilla.balabit.com 04 Sep '13

04 Sep '13

https://bugzilla.balabit.com/show_bug.cgi?id=251 Summary: Missing property 'inherit-properties' from tag 'action/message' in patterndb.xsd Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bugzilla.balabit(a)faxm0dem.org Type of the Report: --- Estimated Hours: 0.0 The property 'inherit-properties' from tag 'action/message' seems to be missing from patterndb-4.xsd, resulting in parse errors when running 'pdbtool --validate'. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5

Handling correctly HOST value on the client side
by Christophe Brocas 02 Sep '13

02 Sep '13

Hello, I have a log management infrastructure with many servers sending logs to centralized servers. Currently, on the server side, I use the HOST field present in the log messages to store log messages in separate files (like <path>/<host>/messages). Of course, due to inconsistent content of the HOST value in the log messages, I have several log file locations for a single client. Bad point. My question : I want to fix the problem at the source and so, set a single unique value in the HOST field of every log message sent by each client. What is your preferred way to address this problem on the client side ? Do you use use_fqdn() global option ? If yes, how it works ? A reverse DNS call ? What is the value returned if a PTR value is not set in the DNS zone for the IP of the host ? Thank you for the reading and for your answer :) Christophe ***************************************************** "Le contenu de ce courriel et ses eventuelles pièces jointes sont confidentiels. Ils s'adressent exclusivement à la personne destinataire. Si cet envoi ne vous est pas destiné, ou si vous l'avez reçu par erreur, et afin de ne pas violer le secret des correspondances, vous ne devez pas le transmettre à d'autres personnes ni le reproduire. Merci de le renvoyer à l'émetteur et de le détruire. Attention : L'Organisme de l'émetteur du message ne pourra être tenu responsable de l'altération du présent courriel. Il appartient au destinataire de vérifier que les messages et pièces jointes reçus ne contiennent pas de virus. Les opinions contenues dans ce courriel et ses éventuelles pièces jointes sont celles de l'émetteur. Elles ne reflètent pas la position de l'Organisme sauf s'il en est disposé autrement dans le présent courriel." ******************************************************

2 3

[PATCH (3.5)] test_template_compile v2
by Juhász Viktor 30 Aug '13

30 Aug '13

From: Juhasz Viktor <jviktor(a)balabit.hu> test_template_compile unit test is refactorized The assert macros are merged into the assert_compile_template macro This macro fill the expected_elem global variable and checks the current_elem. Therefore we don't need the local expected_element variables and calling the fill_expected_template_element macro This patch based on log-template-refactor branch Signed-off-by: Juhasz Viktor <jviktor(a)balabit.hu> --- tests/unit/test_template_compile.c | 171 +++++++++--------------------------- 1 file changed, 41 insertions(+), 130 deletions(-) diff --git a/tests/unit/test_template_compile.c b/tests/unit/test_template_compile.c index fd6fd50..9378e34 100644 --- a/tests/unit/test_template_compile.c +++ b/tests/unit/test_template_compile.c @@ -26,22 +26,6 @@ Plugin hello_plugin = TEMPLATE_FUNCTION_PLUGIN(hello, "hello"); } \ assert_gint(current_elem->msg_ref, expected.msg_ref, ASSERTION_ERROR("Bad compiled template msg_ref")); -#define assert_value_element(expected) \ - assert_gint(current_elem->type, LTE_VALUE, ASSERTION_ERROR("Bad compiled template type")); \ - assert_common_element(expected); \ - assert_gint(current_elem->value_handle, expected.value_handle, ASSERTION_ERROR("Bad compiled template macro")); - -#define assert_macro_element(expected) \ - assert_gint(current_elem->type, LTE_MACRO, ASSERTION_ERROR("Bad compiled template type")); \ - assert_common_element(expected); \ - assert_gint(current_elem->macro, expected.macro, ASSERTION_ERROR("Bad compiled template macro")); - -#define assert_func_element(expected) \ - assert_gint(current_elem->type, LTE_FUNC, ASSERTION_ERROR("Bad compiled template type")); \ - assert_common_element(expected); \ - assert_gpointer(current_elem->func.ops, expected.func.ops, ASSERTION_ERROR("Bad compiled template macro")); - - #define fill_expected_template_element(element, text, default_value, spec, type, msg_ref) {\ element.text; \ element.default_value; \ @@ -49,6 +33,13 @@ Plugin hello_plugin = TEMPLATE_FUNCTION_PLUGIN(hello, "hello"); element.type; \ element.msg_ref; } +#define assert_compiled_template(text, default_value, spec, type, msg_ref) \ + fill_expected_template_element(expected_elem, text, default_value, spec, type, msg_ref); \ + assert_gint((current_elem->type), (expected_elem.type), ASSERTION_ERROR("Bad compiled template type")); \ + assert_common_element(expected_elem); \ + if ((expected_elem.type) == LTE_MACRO) assert_gint(current_elem->macro, expected_elem.macro, ASSERTION_ERROR("Bad compiled template macro")); \ + if ((expected_elem.type) == LTE_VALUE) assert_gint(current_elem->value_handle, expected_elem.value_handle, ASSERTION_ERROR("Bad compiled template macro")); \ + if ((expected_elem.type) == LTE_FUNC) assert_gpointer(current_elem->func.ops, expected_elem.func.ops, ASSERTION_ERROR("Bad compiled template macro")); \ #define TEMPLATE_TESTCASE(x, ...) do { template_testcase_begin(#x, #__VA_ARGS__); x(__VA_ARGS__); template_testcase_end(); } while(0) @@ -75,6 +66,7 @@ Plugin hello_plugin = TEMPLATE_FUNCTION_PLUGIN(hello, "hello"); static LogTemplate *template; static GList *current_elem_list; static LogTemplateElem *current_elem; +static LogTemplateElem expected_elem; static void select_current_element(void) @@ -121,163 +113,119 @@ assert_failed_template_compile(const gchar *template_string, const gchar *expect static void test_simple_string_literal(void) { - LogTemplateElem expected_element; gchar *text = "Test String"; assert_template_compile(text); - fill_expected_template_element(expected_element, text = text, default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); - assert_macro_element(expected_element); + assert_compiled_template(text = text, default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); } static void test_simple_macro(void) { - LogTemplateElem expected_element; - assert_template_compile("${MESSAGE}"); - fill_expected_template_element(expected_element, text = "", default_value = NULL, macro = M_MESSAGE, type = LTE_MACRO, msg_ref = 0); - assert_macro_element(expected_element); + assert_compiled_template(text = "", default_value = NULL, macro = M_MESSAGE, type = LTE_MACRO, msg_ref = 0); } static void test_macro_and_text(void) { - LogTemplateElem expected_element; assert_template_compile("${MESSAGE}test value"); - fill_expected_template_element(expected_element, text = "", default_value = NULL, macro = M_MESSAGE, type = LTE_MACRO, msg_ref = 0); - assert_macro_element(expected_element); + assert_compiled_template(text = "", default_value = NULL, macro = M_MESSAGE, type = LTE_MACRO, msg_ref = 0); select_next_element(); - fill_expected_template_element(expected_element, text = "test value", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); - assert_macro_element(expected_element); + assert_compiled_template(text = "test value", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); } static void test_macro_without_braces(void) { - LogTemplateElem expected_element; - assert_template_compile("$MESSAGE"); - fill_expected_template_element(expected_element, text = "", default_value = NULL, macro = M_MESSAGE, type = LTE_MACRO, msg_ref = 0); - assert_macro_element(expected_element); + assert_compiled_template(text = "", default_value = NULL, macro = M_MESSAGE, type = LTE_MACRO, msg_ref = 0); } static void test_macro_name_without_braces_are_terminated_with_non_identifier_characters(void) { - LogTemplateElem expected_element; - /* macro names consist of [A-Z0-9_] */ assert_template_compile("$MESSAGE test value"); - fill_expected_template_element(expected_element, text = "", default_value = NULL, macro = M_MESSAGE, type = LTE_MACRO, msg_ref = 0); - assert_macro_element(expected_element); + assert_compiled_template(text = "", default_value = NULL, macro = M_MESSAGE, type = LTE_MACRO, msg_ref = 0); select_next_element(); - fill_expected_template_element(expected_element, text = " test value", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); - assert_macro_element(expected_element); + assert_compiled_template(text = " test value", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); } static void test_macro_without_at_records_that_no_msgref_was_present_by_msgref_zero(void) { - LogTemplateElem expected_element; - assert_template_compile("${MESSAGE}"); - fill_expected_template_element(expected_element, text = "", default_value = NULL, macro = M_MESSAGE, type = LTE_MACRO, msg_ref = 0); - assert_macro_element(expected_element); + assert_compiled_template(text = "", default_value = NULL, macro = M_MESSAGE, type = LTE_MACRO, msg_ref = 0); } static void test_macro_with_at_references_a_single_msg_in_the_context_stack_by_setting_msgref(void) { - LogTemplateElem expected_element; - assert_template_compile("${MESSAGE}@0"); - fill_expected_template_element(expected_element, text = "", default_value = NULL, macro = M_MESSAGE, type = LTE_MACRO, msg_ref = 1); - assert_macro_element(expected_element); + assert_compiled_template(text = "", default_value = NULL, macro = M_MESSAGE, type = LTE_MACRO, msg_ref = 1); assert_template_compile("${MESSAGE}@1"); - fill_expected_template_element(expected_element, text = "", default_value = NULL, macro = M_MESSAGE, type = LTE_MACRO, msg_ref = 2); - assert_macro_element(expected_element); + assert_compiled_template(text = "", default_value = NULL, macro = M_MESSAGE, type = LTE_MACRO, msg_ref = 2); } static void test_macro_with_invalid_msgref_are_recognized_as_the_top_element_in_the_stack(void) { - LogTemplateElem expected_element; - assert_template_compile("${MESSAGE}(a)gmail.com"); - fill_expected_template_element(expected_element, text = "", default_value = NULL, macro = M_MESSAGE, type = LTE_MACRO, msg_ref = 1); - assert_macro_element(expected_element); + assert_compiled_template(text = "", default_value = NULL, macro = M_MESSAGE, type = LTE_MACRO, msg_ref = 1); select_next_element(); - fill_expected_template_element(expected_element, text = "gmail.com", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); - assert_macro_element(expected_element); + assert_compiled_template(text = "gmail.com", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); } static void test_dollar_prefixed_with_backslash_is_a_literal_dollar(void) { - LogTemplateElem expected_element; - assert_template_compile("Test \\$STRING"); - fill_expected_template_element(expected_element, text = "Test $STRING", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); - assert_macro_element(expected_element); + assert_compiled_template(text = "Test $STRING", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); } static void test_colon_dash_in_braces_is_parsed_as_default_value(void) { - LogTemplateElem expected_element; - assert_template_compile("${MESSAGE:-default value}"); - fill_expected_template_element(expected_element, text = "", default_value = "default value", macro = M_MESSAGE, type = LTE_MACRO, msg_ref = 0); - assert_macro_element(expected_element); + assert_compiled_template(text = "", default_value = "default value", macro = M_MESSAGE, type = LTE_MACRO, msg_ref = 0); assert_template_compile("${MESSAGE:-}"); - fill_expected_template_element(expected_element, text = "", default_value = "", macro = M_MESSAGE, type = LTE_MACRO, msg_ref = 0); - assert_macro_element(expected_element); + assert_compiled_template(text = "", default_value = "", macro = M_MESSAGE, type = LTE_MACRO, msg_ref = 0); } static void test_double_dollars_is_a_literal_dollar() { - LogTemplateElem expected_element; - assert_template_compile("$$VALUE_NAME"); - fill_expected_template_element(expected_element, text = "$VALUE_NAME", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); - assert_macro_element(expected_element); + assert_compiled_template(text = "$VALUE_NAME", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); assert_template_compile("$${VALUE_NAME}"); - fill_expected_template_element(expected_element, text = "${VALUE_NAME}", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); - assert_macro_element(expected_element); + assert_compiled_template(text = "${VALUE_NAME}", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); } static void test_dollar_with_an_invalid_macro_name_without_braces_is_parsed_as_a_literal_dollar(void) { - LogTemplateElem expected_element; - assert_template_compile("$:VALUE_NAME"); - fill_expected_template_element(expected_element, text = "$:VALUE_NAME", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); - assert_macro_element(expected_element); + assert_compiled_template(text = "$:VALUE_NAME", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); assert_template_compile("$"); - fill_expected_template_element(expected_element, text = "$", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); - assert_macro_element(expected_element); + assert_compiled_template(text = "$", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); } static void test_backslash_without_finishing_the_escape_sequence_is_ignored(void) { - LogTemplateElem expected_element; - assert_template_compile("foo\\"); - fill_expected_template_element(expected_element, text = "foo", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); - assert_macro_element(expected_element); + assert_compiled_template(text = "foo", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); } static void @@ -301,41 +249,29 @@ test_template_compile_macro(void) static void test_simple_value(void) { - LogTemplateElem expected_element; - assert_template_compile("${VALUE_NAME}"); - fill_expected_template_element(expected_element, text = "", default_value = NULL, value_handle = log_msg_get_value_handle("VALUE_NAME"), type = LTE_VALUE, msg_ref = 0); - assert_value_element(expected_element); + assert_compiled_template(text = "", default_value = NULL, value_handle = log_msg_get_value_handle("VALUE_NAME"), type = LTE_VALUE, msg_ref = 0); } static void test_value_without_braces(void) { - LogTemplateElem expected_element; - assert_template_compile("$VALUE_NAME"); - fill_expected_template_element(expected_element, text = "", default_value = NULL, value_handle = log_msg_get_value_handle("VALUE_NAME"), type = LTE_VALUE, msg_ref = 0); - assert_value_element(expected_element); + assert_compiled_template(text = "", default_value = NULL, value_handle = log_msg_get_value_handle("VALUE_NAME"), type = LTE_VALUE, msg_ref = 0); } static void test_backslash_within_braces_is_taken_literally(void) { - LogTemplateElem expected_element; - assert_template_compile("${VALUE\\}NAME}"); - fill_expected_template_element(expected_element, text = "", default_value = NULL, value_handle = log_msg_get_value_handle("VALUE\\"), type = LTE_VALUE, msg_ref = 0); - assert_value_element(expected_element); + assert_compiled_template(text = "", default_value = NULL, value_handle = log_msg_get_value_handle("VALUE\\"), type = LTE_VALUE, msg_ref = 0); } static void test_value_name_can_be_the_empty_string_when_referenced_using_braces(void) { - LogTemplateElem expected_element; - assert_template_compile("${}"); - fill_expected_template_element(expected_element, text = "", default_value = NULL, value_handle = log_msg_get_value_handle(""), type = LTE_VALUE, msg_ref = 0); - assert_value_element(expected_element); + assert_compiled_template(text = "", default_value = NULL, value_handle = log_msg_get_value_handle(""), type = LTE_VALUE, msg_ref = 0); } static void @@ -350,35 +286,25 @@ test_template_compile_value() static void test_simple_template_function(void) { - LogTemplateElem expected_element; - assert_template_compile("$(hello)"); - fill_expected_template_element(expected_element, text = "", default_value = NULL, func.ops = hello_construct(&hello_plugin, configuration, LL_CONTEXT_TEMPLATE_FUNC, "hello"), type = LTE_FUNC, msg_ref = 0); - assert_func_element(expected_element); + assert_compiled_template(text = "", default_value = NULL, func.ops = hello_construct(&hello_plugin, configuration, LL_CONTEXT_TEMPLATE_FUNC, "hello"), type = LTE_FUNC, msg_ref = 0); } static void test_complicated_template_function(void) { - LogTemplateElem expected_element; - assert_template_compile("$( hello \\tes\t\t\t value(xyz) \"value with spaces\" 'test value with spa\"ces')@2"); - fill_expected_template_element(expected_element, text = "", default_value = NULL, func.ops = hello_construct(&hello_plugin, configuration, LL_CONTEXT_TEMPLATE_FUNC, "hello"), type = LTE_FUNC, msg_ref = 3); - assert_func_element(expected_element); + assert_compiled_template(text = "", default_value = NULL, func.ops = hello_construct(&hello_plugin, configuration, LL_CONTEXT_TEMPLATE_FUNC, "hello"), type = LTE_FUNC, msg_ref = 3); } static void test_simple_template_function_with_additional_text(void) { - LogTemplateElem expected_element; - assert_template_compile("$(hello)test value"); - fill_expected_template_element(expected_element, text = "", default_value = NULL, func.ops = hello_construct(&hello_plugin, configuration, LL_CONTEXT_TEMPLATE_FUNC, "hello"), type = LTE_FUNC, msg_ref = 0); - assert_func_element(expected_element); + assert_compiled_template(text = "", default_value = NULL, func.ops = hello_construct(&hello_plugin, configuration, LL_CONTEXT_TEMPLATE_FUNC, "hello"), type = LTE_FUNC, msg_ref = 0); select_next_element(); - fill_expected_template_element(expected_element, text = "test value", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); - assert_macro_element(expected_element); + assert_compiled_template(text = "test value", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); } static void @@ -392,53 +318,38 @@ test_template_compile_func(void) static void test_invalid_macro(void) { - LogTemplateElem expected_element; - assert_failed_template_compile("${MESSAGE", "Invalid macro, '}' is missing, error_pos='9'"); - fill_expected_template_element(expected_element, text = "error in template: ${MESSAGE", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); - assert_macro_element(expected_element); + assert_compiled_template(text = "error in template: ${MESSAGE", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); } static void test_invalid_subst(void) { - LogTemplateElem expected_element; - assert_failed_template_compile("${MESSAGE:1}", "Unknown substitution function, error_pos='10'"); - fill_expected_template_element(expected_element, text = "error in template: ${MESSAGE:1}", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); - assert_macro_element(expected_element); + assert_compiled_template(text = "error in template: ${MESSAGE:1}", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); } static void test_template_function_bad1(void) { - LogTemplateElem expected_element; - assert_failed_template_compile("$( hello \\tes\t\t\t value(xyz \"value with spaces\" 'test value with spa\"ces')", "Invalid template function reference, missing function name or inbalanced '(', error_pos='73'"); - fill_expected_template_element(expected_element, text = "error in template: $( hello \\tes\t\t\t value(xyz \"value with spaces\" 'test value with spa\"ces')", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); - assert_macro_element(expected_element); + assert_compiled_template(text = "error in template: $( hello \\tes\t\t\t value(xyz \"value with spaces\" 'test value with spa\"ces')", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); } static void test_template_function_bad2(void) { - LogTemplateElem expected_element; - assert_failed_template_compile("$( hello \\tes\t\t\t value xyz \"value with spaces\" 'test value with spa\"ces'", "Invalid template function reference, missing function name or inbalanced '(', error_pos='72'"); - fill_expected_template_element(expected_element, text = "error in template: $( hello \\tes\t\t\t value xyz \"value with spaces\" 'test value with spa\"ces'", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); - assert_macro_element(expected_element); + assert_compiled_template(text = "error in template: $( hello \\tes\t\t\t value xyz \"value with spaces\" 'test value with spa\"ces'", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); } static void test_unknown_function(void) { - LogTemplateElem expected_element; - assert_failed_template_compile("$(unknown function)", "Unknown template function unknown"); - fill_expected_template_element(expected_element, text = "error in template: $(unknown function)", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); - assert_macro_element(expected_element); + assert_compiled_template(text = "error in template: $(unknown function)", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); } static void -- 1.7.9.5

2 1

syslog conf (off topic)
by Mike Siedelberg 29 Aug '13

29 Aug '13

Hello, this is a bit off topic but I can't find a good syslog list anymore. This has to do with the old original syslog, sorry :-). I like syslog-ng and use it, but we also have AIX hosts to deal with. One of our applications programmers supposedly found this remote syslog config on a site on how to comply with basic security. I can't seem to convince folks here about why this is wrong. When this config is modeled we do get events. Please comment on the correctnes of the syslog.conf lines below. As you can see, all the remote lines point to the same syslog collector. Many thanks. If there is a more appropriate list to go to, please advise, thanks. auth.info @arcsl1 local7.info @arcsl1 local6.info @arcsl1 auth.* @arcsl1 authpriv.* @arcsl1 local7.* @arcsl1

2 1

How to use parsed results from program pattern?
by Daniel Walter 29 Aug '13

29 Aug '13

Hi folks, the documentation (https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.4-gui…) states multiple times that program patterns and message patterns work the same way. "You can also use parsers in the program pattern if needed, and use the parsed results later. For example: <pattern>postfix\@ESTRING:.postfix.component:[@</pattern>" I try to achieve exactly this behaviour. But my foobar.pdb (see attachment or http://pastebin.com/aZKMKmkc) seems not to work. I tested it with pdbtool: pdbtool -v test foobar.pdb Testing message program='imapd(foo)' message='connect from 192.168.2.179 (192.168.2.179)' Match name='.classifier.rule_id', value='foobaz', expected='foobaz' Match name='IPA', value='192.168.2.179', expected='192.168.2.179' Match name='IPB', value='192.168.2.179', expected='192.168.2.179' Wrong match name='FOOA', value='', expected='foo' Wrong match name='FOOB', value='', expected='foo' What I am doing wrong? How can I use the parsed results from program pattern later? Tested with syslog version: syslog-ng 3.4.3 Installer-Version: 3.4.3 Revision: ssh+git://algernon@git.balabit/var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.4#no_branch#64d670f3cbfb90769f3c7f0fdd9c70bb9136ec5b Compile-Date: Aug 27 2013 17:55:45 Available-Modules: afsocket-tls,cryptofuncs,affile,afsocket,syslogformat,dbparser,afsocket-notls,basicfuncs,json-plugin,system-source,afmongodb,afamqp,afprog,afuser,confgen,csvparser Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: on Enable-Pcre: on And: syslog-ng 3.3.9 Installer-Version: 3.3.9 Revision: 3.3.9-1 (/) Compile-Date: May 26 2013 11:36:00 Default-Modules: affile,afprog,afsocket,afuser,afsql,basicfuncs,csvparser,dbparser,syslogformat Available-Modules: afsocket-notls,basicfuncs,confgen,afsocket,convertfuncs,syslogformat,afuser,csvparser,dbparser,affile,afsocket-tls,afprog Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: on Best Regards, Daniel

2 1

Re: [syslog-ng] patterndb and intrusion prevention
by Valentijn Sessink 29 Aug '13

29 Aug '13

Hi Florian, Thank you for your explanation. You are using the "program" destination, which works fine. The problem I am experiencing only shows up when using "file" output, as there seems to be a problem with lseek in combination with the proc filesystem. Best regards, Valentijn -- Mobile - hence terse.

2 1

patterndb and intrusion prevention
by Matt Zagrabelny 28 Aug '13

28 Aug '13

Greetings, In 2011, Bazsi posted an entry [1] about a fellow named Valentijn Sessink doing some interesting work with patterndb and the iptables xt_recent module. I am mostly there (following Valentijn's example), but am having issues. Specifically, it does not seem that syslog-ng is writing the IPs to the xt_recent proc "file". Syslog-ng is writing them to a regular log file, though: # cat /proc/net/xt_recent/syslogblock # other-system$ ssh doug@system other-system$ # cat /var/log/syslogblock 2013-07-30 16:26:42 Failed password for invalid user doug from 131.212.109.58 port 37867 ssh2 +131.212.109.58 2013-07-30 16:26:42 Failed password for invalid user doug from 131.212.109.58 port 37867 ssh2 +131.212.109.58 2013-07-30 16:45:26 Failed none for invalid user doug from 131.212.109.58 port 37956 ssh2 +131.212.109.58 # cat /proc/net/xt_recent/syslogblock # I can add the host by hand to the xt_recent file: # echo '+131.212.109.58' > /proc/net/xt_recent/syslogblock # echo '+131.212.109.58' > /proc/net/xt_recent/syslogblock # echo '+131.212.109.58' > /proc/net/xt_recent/syslogblock # cat /proc/net/xt_recent/syslogblock src=131.212.109.58 ttl: 0 last_seen: 5445045124 oldest_pkt: 3 5445044388, 5445044844, 5445045124 The above is the expected output. Here are my configs: $ cat /etc/syslog-ng/conf.d/ssh-failure-iptables-recent-block.conf # a destination; we have a dual destination here, a file to see the # blocked hosts, and an iptables-destination in /proc to block them. destination d_syslogblock { file( "/proc/net/xt_recent/syslogblock" template("+${usracct.device}\n") ); file( "/var/log/syslogblock" template("$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC $MSG +${usracct.device}\n") ); }; # a parser for the pattern-DB we made in step 1 parser pattern_db { db_parser( file("/var/lib/syslog-ng/patterndb.xml") ); }; # a filter to filter the parser results filter f_syslogblock { tags("secevt") and match( "REJECT" value("secevt.verdict") ); }; # and finally, the log itself: log { source(s_src); parser(pattern_db); filter(f_syslogblock); destination(d_syslogblock); };

3 5

[Bug 155] New: Compilation failure in 3.3.3: the Darwin linker does not accept "--whole-archive"
by bugzilla＠bugzilla.balabit.com 26 Aug '13

26 Aug '13

https://bugzilla.balabit.com/show_bug.cgi?id=155 Summary: Compilation failure in 3.3.3: the Darwin linker does not accept "--whole-archive" Product: syslog-ng Version: 3.3.x Platform: Macintosh OS/Version: Mac OS Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: jacknagel(a)gmail.com Type of the Report: bug Estimated Hours: 0.0 Trying to compile 3.3.3 on Mac OS X 10.6.8/XCode 4.2, I get this: libtool: link: /usr/bin/llvm-gcc -std=gnu99 -dynamiclib -o .libs/libsyslog-ng-3.3.3.dylib [...] ld: unknown option: --whole-archive collect2: ld returned 1 exit status some cursory googling reveals that the Darwin linker does not accept "--whole-archive". I'd be happy to help test a patch. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 11

[Bug 173] New: Compile issues with syslog-ng_3.3.4.tar.gz
by bugzilla＠bugzilla.balabit.com 26 Aug '13

26 Aug '13

https://bugzilla.balabit.com/show_bug.cgi?id=173 Summary: Compile issues with syslog-ng_3.3.4.tar.gz Product: syslog-ng Version: 3.3.x Platform: Other OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: mark.ulmer(a)apollogrp.edu Type of the Report: bug Estimated Hours: 0.0 Bazsi, I've been working to get a version of 3.3.x compiled on our RHEL 5.6 server for quite some time. I finally have found part of what I think is an issue that might need to be resolved on my side. I don't know. So the latest eventlog-0.2.12 compiles, makes and installs fine. Then in compiling syslog-ng-3.3.4 I get the following error: checking sanity checking Glib headers... no configure: error: Glib headers inconsistent with current compiler setting. You might be using 32 bit Glib with a 64 bit compiler, check PKG_CONFIG_PATH I see in the config.log it's looing for include files in /usr/include/glib-2.0/glib-2.0. The path on my server only has one level glib-2.0. I've tried work arounds and this still gets replicated to lower modules with the same duplicated path level. Go you have a suggestion for me to get past this? Thanks in advance for your assistance. Mark Ulmer Phoenix, AZ, USA -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

4 9

[Bug 248] New: template parsing does not split ", " argument of --scope
by bugzilla＠bugzilla.balabit.com 26 Aug '13

26 Aug '13

https://bugzilla.balabit.com/show_bug.cgi?id=248 Summary: template parsing does not split "," argument of --scope Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bugzilla.balabit(a)faxm0dem.org Type of the Report: --- Estimated Hours: 0.0 As stated in paragraph 2.8.4.(Structuring macros, metadata, and other value-pairs) of the documentation, in example 2.1. (Using the value-pairs() option) it should be possible to pass multiple coma separated values to a template. However, this seems to be broken: Error parsing config, Error compiling template (Error parsing value-pairs: unknown scope all-nv-pairs,rfc5423) in /opt/etc/syslog-ng.conf at line 44, column 14: template("$(format-json --scope all-nv-pairs,rfc5423 )\\n"); ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The workaround is to specify '--scope $scope' multiple times -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

2 5

[Bug 249] New: pdbtool segfaults when using --module switch
by bugzilla＠bugzilla.balabit.com 26 Aug '13

26 Aug '13

https://bugzilla.balabit.com/show_bug.cgi?id=249 Summary: pdbtool segfaults when using --module switch Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bugzilla.balabit(a)faxm0dem.org Type of the Report: --- Estimated Hours: 0.0 Core was generated by `/opt/bin/pdbtool test --validate --color-out /tmp/collectd.pdb -d --module test'. Program terminated with signal 11, Segmentation fault. #0 0x00007f528910ca5f in plugin_load_module (module_name=0x159d8a0 "test", cfg=0x0, args=0x0) at plugin.c:346 346 if (cfg->lexer) -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3

[Bug 246] New: Enhancement: add support for array values
by bugzilla＠bugzilla.balabit.com 26 Aug '13

26 Aug '13

https://bugzilla.balabit.com/show_bug.cgi?id=246 Summary: Enhancement: add support for array values Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bugzilla.balabit(a)faxm0dem.org Type of the Report: --- Estimated Hours: 0.0 It would be great to support arrays in addition to name-value pairs and tags. This would for instance allow for a @TAGS array, instead of the space-separated $TAGS string, which you need to parse on the remote end when sending json format to an application. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5

syslog-ng to receive remote messages from sysklogd and rsyslog?
by subin 25 Aug '13

25 Aug '13

Hi, I have syslog-ng v3.1.3-3 on Debian 6 listening on a network interface. 1. Is it possible/compatible for syslog-ng to receive messages over network from sysklogd(v1.4.1-17) and rsyslog(v3.18.6-4)? 2. In the event of syslog-ng getting restarted (part of logrotate scripts) would the messages be queued up on the remote syslog services? Or would there be any loss of log-data? Thanks.

2 2

[PATCH (3.5) 1/3] tests: Add a testcase for log_template_compile function
by Juhász Viktor 24 Aug '13

24 Aug '13

test_template_compile: new unit test implemented. Main goal of this test is to help the refactoring the log_template_compile monster function Signed-off-by: Juhasz Viktor <jviktor(a)balabit.hu> --- tests/unit/Makefile.am | 5 + tests/unit/test_template_compile.c | 320 ++++++++++++++++++++++++++++++++++++ 2 files changed, 325 insertions(+) diff --git a/tests/unit/Makefile.am b/tests/unit/Makefile.am index 8841834..af5730f 100644 --- a/tests/unit/Makefile.am +++ b/tests/unit/Makefile.am @@ -8,6 +8,7 @@ tests_unit_TESTS = \ tests/unit/test_serialize \ tests/unit/test_msgparse \ tests/unit/test_template \ + tests/unit/test_template_compile \ tests/unit/test_template_speed \ tests/unit/test_dnscache \ tests/unit/test_findcrlf \ @@ -56,6 +57,10 @@ tests_unit_test_template_LDADD = \ $(unit_test_extra_modules) \ $(PREOPEN_BASICFUNCS) +tests_unit_test_template_compile_CFLAGS = $(TEST_CFLAGS) +tests_unit_test_template_compile_LDADD = \ + $(TEST_LDADD) $(unit_test_extra_modules) + tests_unit_test_template_speed_LDADD = \ $(TEST_LDADD) $(unit_test_extra_modules) diff --git a/tests/unit/test_template_compile.c b/tests/unit/test_template_compile.c new file mode 100644 index 0000000..23fed4f --- /dev/null +++ b/tests/unit/test_template_compile.c @@ -0,0 +1,320 @@ +#include "templates.c" +#include "logmsg.h" +#include "testutils.h" +#include "cfg.h" +#include "plugin.h" + +#define SIMPLE_STRING "Test String" +#define SIMPLE_MACRO "${MESSAGE}" +#define SIMPLE_MACRO_UNBRACE "$MESSAGE" +#define SIMPLE_MACRO_UNBRACE_ADD_TEXT "$MESSAGE test value" +#define SIMPLE_MACRO_MSGREF "${MESSAGE}@1" +#define SIMPLE_MACRO_INVALID_REF "${MESSAGE}(a)gmail.com" +#define SIMPLE_MACRO_ADD_TEXT "${MESSAGE}test value" +#define ESCAPED_CHAR "Test \\$STRING" +#define VALID_SUBST "${MESSAGE:-default value}" +#define TRICKY_VALUE "$$VALUE_NAME" +#define TRICKY_VALUE_2 "$:VALUE_NAME" +#define TRICKY_VALUE_3 "$${VALUE_NAME}" +#define TRICKY_VALUE_4 "\\" +#define TRICKY_VALUE_5 "$" +#define TRICKY_VALUE_6 "${MESSAGE:-}" +#define TRICKY_VALUE_7 "${}" + +#define INVALID_MACRO "${MESSAGE" +#define INVALID_SUBST "${MESSAGE:1}" + +#define SIMPLE_VALUE "${VALUE_NAME}" +#define SIMPLE_VALUE_UNBRACE "$VALUE_NAME" +#define SIMPLE_VALUE_ESCAPED "${VALUE\\}NAME}" + +#define SIMPLE_TEMPLATE_FUNCTION "$(hello)" +#define SIMPLE_TEMPLATE_FUNCTION_WITH_ADDITIONAL_TEXT "$(hello)test value" +#define COMPLICATED_TEMPLATE_FUNCTION "$( hello \\tes\t\t\t value(xyz) \"value with spaces\" 'test value with spa\"ces')@2" +#define COMPLICATED_TEMPLATE_FUNCTION_BAD_1 "$( hello \\tes\t\t\t value(xyz \"value with spaces\" 'test value with spa\"ces')" +#define COMPLICATED_TEMPLATE_FUNCTION_BAD_2 "$( hello \\tes\t\t\t value xyz \"value with spaces\" 'test value with spa\"ces'" + +#define SIMPLE_UNKNOWN_FUNCTION "$(unknown function)" + +static void +hello(LogMessage *msg, int argc, GString *argv[], GString *result) +{ + return; +} + +TEMPLATE_FUNCTION_SIMPLE(hello); +Plugin hello_plugin = TEMPLATE_FUNCTION_PLUGIN(hello, "hello"); + + +GlobalConfig *configuration; +static void +hide_internal_message(LogMessage *msg) +{ + log_msg_unref(msg); + return; +} + +#define assert_common_element(actual, expected) \ + assert_string(actual.text, expected.text, ASSERTION_ERROR("Bad compiled template text")); \ + if (expected.default_value) \ + { \ + assert_string(actual.default_value, expected.default_value, ASSERTION_ERROR("Bad compiled template default value")); \ + } \ + else \ + { \ + assert_gpointer(actual.default_value, NULL, ASSERTION_ERROR("Bad compiled template default value")); \ + } \ + assert_gint(actual.msg_ref, expected.msg_ref, ASSERTION_ERROR("Bad compiled template msg_ref")); + +#define assert_value_element(actual, expected) \ + assert_gint(actual.type, LTE_VALUE, ASSERTION_ERROR("Bad compiled template type")); \ + assert_common_element(actual, expected); \ + assert_gint(actual.value_handle, expected.value_handle, ASSERTION_ERROR("Bad compiled template macro")); + +#define assert_macro_element(actual, expected) \ + assert_gint(actual.type, LTE_MACRO, ASSERTION_ERROR("Bad compiled template type")); \ + assert_common_element(actual, expected); \ + assert_gint(actual.macro, expected.macro, ASSERTION_ERROR("Bad compiled template macro")); + +#define assert_func_element(actual, expected) \ + assert_gint(actual.type, LTE_FUNC, ASSERTION_ERROR("Bad compiled template type")); \ + assert_common_element(actual, expected); \ + assert_gpointer(actual.func.ops, expected.func.ops, ASSERTION_ERROR("Bad compiled template macro")); + +#define assert_bad_element(actual, expected, expected_error_message) \ + assert_string(err->message, expected_error_message, ASSERTION_ERROR("Bad error message")); \ + assert_macro_element(actual, expected) + + +#define assert_template_compile(template_string) \ + assert_true(log_template_compile(template, template_string, &err), ASSERTION_ERROR("Can't compile template")); \ + assert_string(template->template, template_string, ASSERTION_ERROR("Bad stored template")); + +#define assert_failed_template_compile(template_string) assert_false(log_template_compile(template, template_string, &err), ASSERTION_ERROR("Can compile bad template")); + +#define fill_expected_template_element(element, text, default_value, spec, type, msg_ref) {\ + element.text; \ + element.default_value; \ + element.spec; \ + element.type; \ + element.msg_ref; } + + + +void +test_template_compile_macro() +{ + GError *err = NULL; + LogTemplateElem *element; + LogTemplateElem expected_element; + LogTemplate *template = log_template_new(configuration, NULL); + + assert_template_compile(SIMPLE_STRING); + element = (LogTemplateElem *)template->compiled_template->data; + fill_expected_template_element(expected_element, text = SIMPLE_STRING, default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); + assert_macro_element(element[0], expected_element); + + assert_template_compile(SIMPLE_MACRO); + element = (LogTemplateElem *)template->compiled_template->data; + fill_expected_template_element(expected_element, text = "", default_value = NULL, macro = M_MESSAGE, type = LTE_MACRO, msg_ref = 0); + assert_macro_element(element[0], expected_element); + + assert_template_compile(SIMPLE_MACRO_ADD_TEXT); + element = (LogTemplateElem *)template->compiled_template->data; + fill_expected_template_element(expected_element, text = "", default_value = NULL, macro = M_MESSAGE, type = LTE_MACRO, msg_ref = 0); + assert_macro_element(element[0], expected_element); + element = (LogTemplateElem *)template->compiled_template->next->data; + fill_expected_template_element(expected_element, text = "test value", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); + assert_macro_element(element[0], expected_element); + + assert_template_compile(SIMPLE_MACRO_UNBRACE); + element = (LogTemplateElem *)template->compiled_template->data; + fill_expected_template_element(expected_element, text = "", default_value = NULL, macro = M_MESSAGE, type = LTE_MACRO, msg_ref = 0); + assert_macro_element(element[0], expected_element); + + assert_template_compile(SIMPLE_MACRO_UNBRACE_ADD_TEXT); + element = (LogTemplateElem *)template->compiled_template->data; + fill_expected_template_element(expected_element, text = "", default_value = NULL, macro = M_MESSAGE, type = LTE_MACRO, msg_ref = 0); + assert_macro_element(element[0], expected_element); + element = (LogTemplateElem *)template->compiled_template->next->data; + fill_expected_template_element(expected_element, text = " test value", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); + assert_macro_element(element[0], expected_element); + + assert_template_compile(SIMPLE_MACRO_MSGREF); + element = (LogTemplateElem *)template->compiled_template->data; + fill_expected_template_element(expected_element, text = "", default_value = NULL, macro = M_MESSAGE, type = LTE_MACRO, msg_ref = 2); + assert_macro_element(element[0], expected_element); + + assert_template_compile(SIMPLE_MACRO_INVALID_REF); + element = (LogTemplateElem *)template->compiled_template->data; + fill_expected_template_element(expected_element, text = "", default_value = NULL, macro = M_MESSAGE, type = LTE_MACRO, msg_ref = 1); + assert_macro_element(element[0], expected_element); + element = (LogTemplateElem *)template->compiled_template->next->data; + fill_expected_template_element(expected_element, text = "gmail.com", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); + assert_macro_element(element[0], expected_element); + + assert_template_compile(ESCAPED_CHAR); + element = (LogTemplateElem *)template->compiled_template->data; + fill_expected_template_element(expected_element, text = "Test $STRING", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); + assert_macro_element(element[0], expected_element); + + assert_template_compile(VALID_SUBST); + element = (LogTemplateElem *)template->compiled_template->data; + fill_expected_template_element(expected_element, text = "", default_value = "default value", macro = M_MESSAGE, type = LTE_MACRO, msg_ref = 0); + assert_macro_element(element[0], expected_element); + + assert_template_compile(TRICKY_VALUE); + element = (LogTemplateElem *)template->compiled_template->data; + fill_expected_template_element(expected_element, text = "$VALUE_NAME", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); + assert_macro_element(element[0], expected_element); + + assert_template_compile(TRICKY_VALUE_2); + element = (LogTemplateElem *)template->compiled_template->data; + fill_expected_template_element(expected_element, text = "$:VALUE_NAME", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); + assert_macro_element(element[0], expected_element); + + assert_template_compile(TRICKY_VALUE_3); + element = (LogTemplateElem *)template->compiled_template->data; + fill_expected_template_element(expected_element, text = "${VALUE_NAME}", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); + assert_macro_element(element[0], expected_element); + + assert_template_compile(TRICKY_VALUE_4); + element = (LogTemplateElem *)template->compiled_template->data; + fill_expected_template_element(expected_element, text = "", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); + assert_macro_element(element[0], expected_element); + + assert_template_compile(TRICKY_VALUE_5); + element = (LogTemplateElem *)template->compiled_template->data; + fill_expected_template_element(expected_element, text = "$", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); + assert_macro_element(element[0], expected_element); + + assert_template_compile(TRICKY_VALUE_6); + element = (LogTemplateElem *)template->compiled_template->data; + fill_expected_template_element(expected_element, text = "", default_value = "", macro = M_MESSAGE, type = LTE_MACRO, msg_ref = 0); + assert_macro_element(element[0], expected_element); + + log_template_unref(template); +} + +void +test_template_compile_value() +{ + GError *err = NULL; + LogTemplateElem *element; + LogTemplateElem expected_element; + LogTemplate *template = log_template_new(configuration, NULL); + + assert_template_compile(SIMPLE_VALUE); + element = (LogTemplateElem *)template->compiled_template->data; + fill_expected_template_element(expected_element, text = "", default_value = NULL, value_handle = log_msg_get_value_handle("VALUE_NAME"), type = LTE_VALUE, msg_ref = 0); + assert_value_element(element[0], expected_element); + + assert_template_compile(SIMPLE_VALUE_UNBRACE); + element = (LogTemplateElem *)template->compiled_template->data; + fill_expected_template_element(expected_element, text = "", default_value = NULL, value_handle = log_msg_get_value_handle("VALUE_NAME"), type = LTE_VALUE, msg_ref = 0); + assert_value_element(element[0], expected_element); + + assert_template_compile(SIMPLE_VALUE_ESCAPED); + element = (LogTemplateElem *)template->compiled_template->data; + fill_expected_template_element(expected_element, text = "", default_value = NULL, value_handle = log_msg_get_value_handle("VALUE\\"), type = LTE_VALUE, msg_ref = 0); + assert_value_element(element[0], expected_element); + + assert_template_compile(TRICKY_VALUE_7); + element = (LogTemplateElem *)template->compiled_template->data; + fill_expected_template_element(expected_element, text = "", default_value = NULL, value_handle = log_msg_get_value_handle(""), type = LTE_VALUE, msg_ref = 0); + assert_value_element(element[0], expected_element); + + log_template_unref(template); +} + +void +test_template_compile_func() +{ + GError *err = NULL; + LogTemplateElem *element; + LogTemplateElem expected_element; + LogTemplate *template = log_template_new(configuration, NULL); + plugin_register(configuration, &hello_plugin, 1); + + assert_template_compile(SIMPLE_TEMPLATE_FUNCTION); + element = (LogTemplateElem *)template->compiled_template->data; + fill_expected_template_element(expected_element, text = "", default_value = NULL, func.ops = hello_construct(&hello_plugin, configuration, LL_CONTEXT_TEMPLATE_FUNC, "hello"), type = LTE_FUNC, msg_ref = 0); + assert_func_element(element[0], expected_element); + + assert_template_compile(COMPLICATED_TEMPLATE_FUNCTION); + element = (LogTemplateElem *)template->compiled_template->data; + fill_expected_template_element(expected_element, text = "", default_value = NULL, func.ops = hello_construct(&hello_plugin, configuration, LL_CONTEXT_TEMPLATE_FUNC, "hello"), type = LTE_FUNC, msg_ref = 3); + assert_func_element(element[0], expected_element); + + assert_template_compile(SIMPLE_TEMPLATE_FUNCTION_WITH_ADDITIONAL_TEXT); + element = (LogTemplateElem *)template->compiled_template->data; + fill_expected_template_element(expected_element, text = "", default_value = NULL, func.ops = hello_construct(&hello_plugin, configuration, LL_CONTEXT_TEMPLATE_FUNC, "hello"), type = LTE_FUNC, msg_ref = 0); + assert_func_element(element[0], expected_element); + element = (LogTemplateElem *)template->compiled_template->next->data; + fill_expected_template_element(expected_element, text = "test value", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); + assert_macro_element(element[0], expected_element); + + log_template_unref(template); +} + +void +test_template_compile_negativ_tests() +{ + GError *err = NULL; + LogTemplateElem *element; + LogTemplateElem expected_element; + LogTemplate *template = log_template_new(configuration, NULL); + + assert_failed_template_compile(INVALID_MACRO); + element = (LogTemplateElem *)template->compiled_template->data; + fill_expected_template_element(expected_element, text = "error in template: ${MESSAGE", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); + assert_bad_element(element[0], expected_element, "Invalid macro, '}' is missing, error_pos='9'"); + g_clear_error(&err); + + + assert_failed_template_compile(INVALID_SUBST); + element = (LogTemplateElem *)template->compiled_template->data; + fill_expected_template_element(expected_element, text = "error in template: ${MESSAGE:1}", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); + assert_bad_element(element[0], expected_element, "Unknown substitution function, error_pos='10'"); + g_clear_error(&err); + + + assert_failed_template_compile(COMPLICATED_TEMPLATE_FUNCTION_BAD_1); + element = (LogTemplateElem *)template->compiled_template->data; + fill_expected_template_element(expected_element, text = "error in template: $( hello \\tes\t\t\t value(xyz \"value with spaces\" 'test value with spa\"ces')", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); + assert_bad_element(element[0], expected_element, "Invalid template function reference, missing function name or inbalanced '(', error_pos='73'"); + g_clear_error(&err); + + assert_failed_template_compile(COMPLICATED_TEMPLATE_FUNCTION_BAD_2); + element = (LogTemplateElem *)template->compiled_template->data; + fill_expected_template_element(expected_element, text = "error in template: $( hello \\tes\t\t\t value xyz \"value with spaces\" 'test value with spa\"ces'", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); + assert_bad_element(element[0], expected_element, "Invalid template function reference, missing function name or inbalanced '(', error_pos='72'"); + g_clear_error(&err); + + assert_failed_template_compile(SIMPLE_UNKNOWN_FUNCTION); + element = (LogTemplateElem *)template->compiled_template->data; + fill_expected_template_element(expected_element, text = "error in template: $(unknown function)", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); + assert_bad_element(element[0], expected_element, "Unknown template function unknown"); + g_clear_error(&err); + + log_template_unref(template); +} + +int main(int argc, char **argv) +{ + msg_init(FALSE); + configuration = cfg_new(0x305); + log_msg_registry_init(); + log_template_global_init(); + + msg_set_post_func(hide_internal_message); + + test_template_compile_macro(); + test_template_compile_value(); + test_template_compile_func(); + test_template_compile_negativ_tests(); + + log_msg_registry_deinit(); + msg_deinit(); + return 0; +} --

3 2

[PATCH (3.5) 3/3] template escaping changed
by Juhász Viktor 23 Aug '13

23 Aug '13

template_escaping: template and correlation index escaping changed. correlation index escaping introduced, because it's a correct use-case, that somebody want to use @ character after template, in this case use '@@' after the template e.g.: ${USERNAME}(a)gmail.com caused bad behaviour and there wasn't any possible solution to reach the correct behaviour from now the correct usage is: ${USERNAME}@@gmail.com The template $ escaping changed from '\$' to '$$'\ because it was very confusing that the lexer could process '\' and the template_compile processed '\' too. >From now, the correct usage if you want to use $ in the output string is: $$HELLO instead of \$HELLO Signed-off-by: Juhasz Viktor <jviktor(a)balabit.hu> --- lib/templates.c | 38 +++++++++++++++++++++++++++--------- tests/unit/test_template_compile.c | 38 +++++++++++++++++++++++++++++++++--- 2 files changed, 64 insertions(+), 12 deletions(-) diff --git a/lib/templates.c b/lib/templates.c index 6cd90ee..a34667a 100644 --- a/lib/templates.c +++ b/lib/templates.c @@ -871,19 +871,33 @@ log_template_add_func_elem(LogTemplate *self, GString *text, gint argc, gchar *a } static void -parse_msg_ref(gchar **p, gint *msg_ref) +parse_msg_ref(gchar *template_string, gchar **p, gint *msg_ref) { *msg_ref = 0; if ((**p) == '@') { (*p)++; - /* syntax: ${name}@1 to denote the log message index in the correllation state */ - while ((**p) >= '0' && (**p) <= '9') + if ((**p) >= '0' && (**p) <= '9') { - (*msg_ref) += (*msg_ref) * 10 + ((**p) - '0'); - (*p)++; + /* syntax: ${name}@1 to denote the log message index in the correllation state */ + while ((**p) >= '0' && (**p) <= '9') + { + (*msg_ref) += (*msg_ref) * 10 + ((**p) - '0'); + (*p)++; + } + *msg_ref += 1; + } + else + { + if ((**p) != '@') + { + msg_warning("Non-numeric correlation state ID found, assuming a literal '@' character. To avoid confusion when using a literal '@' after a macro or template function, write '@@' in the templ + evt_tag_str("Template", template_string), + NULL); + (*p)--; + } + *msg_ref = 0; } - *msg_ref += 1; } } @@ -978,7 +992,7 @@ log_template_process_braced_template(LogTemplate *self, gchar **p, GString *text return FALSE; } } - parse_msg_ref(p, &msg_ref); + parse_msg_ref(self->template, p, &msg_ref); log_template_add_elem(self, start, macro_len, default_value, text, msg_ref); return TRUE; } @@ -1091,7 +1105,7 @@ log_template_process_template_function(LogTemplate *self, gchar **p, GString *te goto error; } (*p)++; - parse_msg_ref(p, &msg_ref); + parse_msg_ref(self->template, p, &msg_ref); if (!log_template_add_func_elem(self, text, strv->len - 1, (gchar **) strv->pdata, msg_ref, error)) { goto error; @@ -1180,7 +1194,13 @@ log_template_process_character(LogTemplate *self, gchar **p, GString **text, GEr } if (**p == '\\') { - (*p)++; + if (cfg_is_config_version_older(self->cfg, 0x0305)) + { + msg_warning("Template escaping changed in syslog-ng-ose 3.5. Use '$$' to specify a literal dollar sign instead of '\\$' and remove the escaping of the backslash character when you upgrade your c + evt_tag_str("Template", self->template), + NULL); + (*p)++; + } } if (**p) { diff --git a/tests/unit/test_template_compile.c b/tests/unit/test_template_compile.c index 23fed4f..f73f653 100644 --- a/tests/unit/test_template_compile.c +++ b/tests/unit/test_template_compile.c @@ -10,6 +10,7 @@ #define SIMPLE_MACRO_UNBRACE_ADD_TEXT "$MESSAGE test value" #define SIMPLE_MACRO_MSGREF "${MESSAGE}@1" #define SIMPLE_MACRO_INVALID_REF "${MESSAGE}(a)gmail.com" +#define SIMPLE_MACRO_ESCAPED_REF "${MESSAGE}@@12" #define SIMPLE_MACRO_ADD_TEXT "${MESSAGE}test value" #define ESCAPED_CHAR "Test \\$STRING" #define VALID_SUBST "${MESSAGE:-default value}" @@ -20,6 +21,7 @@ #define TRICKY_VALUE_5 "$" #define TRICKY_VALUE_6 "${MESSAGE:-}" #define TRICKY_VALUE_7 "${}" +#define TRICKY_VALUE_8 "\\tsomething ${MESSAGE}" #define INVALID_MACRO "${MESSAGE" #define INVALID_SUBST "${MESSAGE:1}" @@ -36,6 +38,9 @@ #define SIMPLE_UNKNOWN_FUNCTION "$(unknown function)" +#define CURRENT_VERSION 0x0305 +#define PREVIOUS_VERSION 0x0304 + static void hello(LogMessage *msg, int argc, GString *argv[], GString *result) { @@ -147,17 +152,32 @@ test_template_compile_macro() assert_template_compile(SIMPLE_MACRO_INVALID_REF); element = (LogTemplateElem *)template->compiled_template->data; - fill_expected_template_element(expected_element, text = "", default_value = NULL, macro = M_MESSAGE, type = LTE_MACRO, msg_ref = 1); + fill_expected_template_element(expected_element, text = "", default_value = NULL, macro = M_MESSAGE, type = LTE_MACRO, msg_ref = 0); + assert_macro_element(element[0], expected_element); + element = (LogTemplateElem *)template->compiled_template->next->data; + fill_expected_template_element(expected_element, text = "@gmail.com", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); + assert_macro_element(element[0], expected_element); + + assert_template_compile(SIMPLE_MACRO_ESCAPED_REF); + element = (LogTemplateElem *)template->compiled_template->data; + fill_expected_template_element(expected_element, text = "", default_value = NULL, macro = M_MESSAGE, type = LTE_MACRO, msg_ref = 0); assert_macro_element(element[0], expected_element); element = (LogTemplateElem *)template->compiled_template->next->data; - fill_expected_template_element(expected_element, text = "gmail.com", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); + fill_expected_template_element(expected_element, text = "@12", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); assert_macro_element(element[0], expected_element); + cfg_set_version(template->cfg, PREVIOUS_VERSION); assert_template_compile(ESCAPED_CHAR); element = (LogTemplateElem *)template->compiled_template->data; fill_expected_template_element(expected_element, text = "Test $STRING", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); assert_macro_element(element[0], expected_element); + cfg_set_version(template->cfg, CURRENT_VERSION); + assert_template_compile(ESCAPED_CHAR); + element = (LogTemplateElem *)template->compiled_template->data; + fill_expected_template_element(expected_element, text = "Test \\", default_value = NULL, value_handle=log_msg_get_value_handle("STRING"), type = LTE_VALUE, msg_ref = 0); + assert_value_element(element[0], expected_element); + assert_template_compile(VALID_SUBST); element = (LogTemplateElem *)template->compiled_template->data; fill_expected_template_element(expected_element, text = "", default_value = "default value", macro = M_MESSAGE, type = LTE_MACRO, msg_ref = 0); @@ -178,11 +198,18 @@ test_template_compile_macro() fill_expected_template_element(expected_element, text = "${VALUE_NAME}", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); assert_macro_element(element[0], expected_element); + cfg_set_version(template->cfg, PREVIOUS_VERSION); assert_template_compile(TRICKY_VALUE_4); element = (LogTemplateElem *)template->compiled_template->data; fill_expected_template_element(expected_element, text = "", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); assert_macro_element(element[0], expected_element); + cfg_set_version(template->cfg, CURRENT_VERSION); + assert_template_compile(TRICKY_VALUE_4); + element = (LogTemplateElem *)template->compiled_template->data; + fill_expected_template_element(expected_element, text = "\\", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); + assert_macro_element(element[0], expected_element); + assert_template_compile(TRICKY_VALUE_5); element = (LogTemplateElem *)template->compiled_template->data; fill_expected_template_element(expected_element, text = "$", default_value = NULL, macro = M_NONE, type = LTE_MACRO, msg_ref = 0); @@ -193,6 +220,11 @@ test_template_compile_macro() fill_expected_template_element(expected_element, text = "", default_value = "", macro = M_MESSAGE, type = LTE_MACRO, msg_ref = 0); assert_macro_element(element[0], expected_element); + assert_template_compile(TRICKY_VALUE_8); + element = (LogTemplateElem *)template->compiled_template->data; + fill_expected_template_element(expected_element, text = "\\tsomething ", default_value = NULL, macro = M_MESSAGE, type = LTE_MACRO, msg_ref = 0); + assert_macro_element(element[0], expected_element); + log_template_unref(template); } @@ -303,7 +335,7 @@ test_template_compile_negativ_tests() int main(int argc, char **argv) { msg_init(FALSE); - configuration = cfg_new(0x305); + configuration = cfg_new(CURRENT_VERSION); log_msg_registry_init(); log_template_global_init(); --

1 0

[PATCH (3.5) 2/3] log_template_compile function refactorized
by Juhász Viktor 23 Aug '13

23 Aug '13

templates: log_template_compile refactored. It was a monster function. It was very hard to maintain and develop, so before the template escaping changes it was useful to refactor this huge function. Signed-off-by: Juhasz Viktor <jviktor(a)balabit.hu> --- lib/templates.c | 554 +++++++++++++++++++++++++++++++------------------------ 1 file changed, 315 insertions(+), 239 deletions(-) diff --git a/lib/templates.c b/lib/templates.c index e084c1a..6cd90ee 100644 --- a/lib/templates.c +++ b/lib/templates.c @@ -887,15 +887,314 @@ parse_msg_ref(gchar **p, gint *msg_ref) } } +static void +log_template_fill_compile_error(GError **error, const gchar *error_info, gint error_pos) +{ + g_set_error(error, LOG_TEMPLATE_ERROR, LOG_TEMPLATE_ERROR_COMPILE, "%s, error_pos='%d'", error_info, error_pos); +} + +static void +log_template_append_and_increment(gchar **p, GString *text) +{ + g_string_append_c(text, **p); + (*p)++; +} + +static gint +log_template_get_macro_length(gchar *start, gchar *end, gchar **token) +{ + gint result = 0; + gchar *colon = memchr(start, ':', end - start - 1); + if (colon) + { + result = colon - start; + *token = colon < end ? colon + 1 : NULL; + } + else + { + result = end - start - 1; + *token = NULL; + } + return result; +} + + +static gchar * +log_template_parse_default_value(gchar *macro_end, gchar *token) +{ + g_assert(token); + if (*token != '-') + { + return NULL; + } + return g_strndup(token + 1, macro_end - token - 2); +} + +#define IS_MACRO_NAME(c) (c >= 'A' && c <= 'Z') || (c >= 'a' && c <= 'z') || (c == '_') || (c >= '0' && c <= '9') + +#define STEP_BY_TRUE(p, x) while(x) p++; + +static void +log_template_add_elem(LogTemplate *self, gchar *start, gint macro_len, gchar *default_value, GString *text, gint msg_ref) +{ + gint macro = log_macro_lookup(start, macro_len); + if (macro == M_NONE) + { + log_template_add_value_elem(self, start, macro_len, text, default_value, msg_ref); + } + else + { + log_template_add_macro_elem(self, macro, text, default_value, msg_ref); + } +} + +static gboolean +log_template_process_braced_template(LogTemplate *self, gchar **p, GString *text, GError **error) +{ + gint macro_len; + gint msg_ref = 0; + gchar *default_value = NULL; + gchar *start; + gchar *token; + + (*p)++; + start = *p; + *p = strchr(*p, '}'); + + if (!*p) + { + log_template_fill_compile_error(error, "Invalid macro, '}' is missing", strlen(self->template)); + return FALSE; + } + + (*p)++; + macro_len = log_template_get_macro_length(start, *p, &token); + if (token) + { + default_value = log_template_parse_default_value(*p, token); + if (!default_value) + { + log_template_fill_compile_error(error, "Unknown substitution function", token - self->template); + return FALSE; + } + } + parse_msg_ref(p, &msg_ref); + log_template_add_elem(self, start, macro_len, default_value, text, msg_ref); + return TRUE; +} + +static gboolean +log_template_get_unquoted_part(gchar **p, GPtrArray *strv, gint *parens, gchar *quote, GString *arg_buf) +{ + gboolean result = TRUE; + if (**p == '\\') + { + (*p)++; + } + else if (**p == '(') + { + (*parens)++; + } + else if (**p == ')') + { + (*parens)--; + } + else if (**p == '"' || **p == '\'') + { + *quote = **p; + if (*parens == 1) + { + /* skip the quote in top-level and don't skip in expressions enclosed in parens */ + (*p)++; + result = FALSE; + } + } + else if (*parens == 1 && (**p == ' ' || **p == '\t')) + { + g_ptr_array_add(strv, g_strndup(arg_buf->str, arg_buf->len)); + g_string_truncate(arg_buf, 0); + while (**p && (**p == ' ' || **p == '\t')) + (*p)++; + result = FALSE; + } + if (*parens == 0) + { + result = FALSE; + } + return result; +} + +static gboolean +log_template_get_quoted_part(gchar **p, gint parens, gchar *quote) +{ + gboolean result = TRUE; + if (*quote == **p) + { + /* end of string */ + *quote = 0; + if (parens == 1) + { + (*p)++; + result = FALSE; + } + } + return result; +} + +static void +log_template_parse_template_function(gchar **p, GPtrArray *strv) +{ + gchar quote = 0; + gint parens = 1; + GString *arg_buf; + arg_buf = g_string_sized_new(32); + (*p)++; + gboolean add_char_to_string; + /* skip white space */ + while (**p && **p == ' ') + (*p)++; + + while (**p && parens != 0) + { + add_char_to_string = TRUE; + if (!quote) + { + add_char_to_string = log_template_get_unquoted_part(p, strv, &parens, &quote, arg_buf); + } + else + { + add_char_to_string = log_template_get_quoted_part(p, parens, &quote); + } + if (**p && add_char_to_string) + { + log_template_append_and_increment(p, arg_buf); + } + } + + if (arg_buf->len) + g_ptr_array_add(strv, g_strndup(arg_buf->str, arg_buf->len)); + g_ptr_array_add(strv, NULL); + g_string_free(arg_buf, TRUE); +} + +static gboolean +log_template_process_template_function(LogTemplate *self, gchar **p, GString *text, GError **error) +{ + GPtrArray *strv = g_ptr_array_new(); + gint msg_ref; + + log_template_parse_template_function(p, strv); + + if (**p != ')') + { + log_template_fill_compile_error(error, "Invalid template function reference, missing function name or inbalanced '('", *p - self->template); + goto error; + } + (*p)++; + parse_msg_ref(p, &msg_ref); + if (!log_template_add_func_elem(self, text, strv->len - 1, (gchar **) strv->pdata, msg_ref, error)) + { + goto error; + } + g_ptr_array_free(strv, FALSE); + return TRUE; +error: + g_strfreev((gchar **)strv->pdata); + g_ptr_array_free(strv, FALSE); + return FALSE; +} + +static void +log_template_process_unbraced_template(LogTemplate *self, gchar **p, GString *text) +{ + gchar *start = *p; + gint macro_len; + do + { + (*p)++; + } + while (IS_MACRO_NAME(**p)); + macro_len = *p - start; + log_template_add_elem(self, start, macro_len, NULL, text, 0); +} + +static gboolean +log_template_process_value(LogTemplate *self, gchar **p, GString **text, GError **error) +{ + gboolean finished = FALSE; + (*p)++; + /* macro reference */ + if (**p == '{') + { + if (!log_template_process_braced_template(self, p, *text, error)) + { + return FALSE; + } + finished = TRUE; + } + /* template function */ + else if (**p == '(') + { + if (!log_template_process_template_function(self, p, *text, error)) + { + return FALSE; + } + finished = TRUE; + } + /* unbraced macro */ + else if (IS_MACRO_NAME(**p)) + { + log_template_process_unbraced_template(self, p, *text); + finished = TRUE; + } + /* escaped value with dollar */ + else + { + if (**p != '$') + { + g_string_append_c(*text, '$'); + } + if (**p) + { + log_template_append_and_increment(p, *text); + } + } + if (finished) + { + if (*text) + g_string_free(*text, TRUE); + *text = NULL; + } + return TRUE; +} + +gboolean +log_template_process_character(LogTemplate *self, gchar **p, GString **text, GError **error) +{ + if (*text == NULL) + *text = g_string_sized_new(32); + + if (**p == '$') + { + return log_template_process_value(self, p, text, error); + } + if (**p == '\\') + { + (*p)++; + } + if (**p) + { + log_template_append_and_increment(p, *text); + } + return TRUE; +} + gboolean log_template_compile(LogTemplate *self, const gchar *template, GError **error) { - gchar *start, *p; - guint last_macro = M_NONE; + gchar *p; GString *last_text = NULL; - gchar *error_info; - gint error_pos = 0; - + g_return_val_if_fail(error == NULL || *error == NULL, FALSE); log_template_reset_compiled(self); @@ -903,250 +1202,27 @@ log_template_compile(LogTemplate *self, const gchar *template, GError **error) g_free(self->template); self->template = g_strdup(template); p = self->template; - + while (*p) { - if (last_text == NULL) - last_text = g_string_sized_new(32); - if (*p == '\\') + if (!log_template_process_character(self, &p, &last_text, error)) { - p++; - if (*p) - { - g_string_append_c(last_text, *p); - p++; - } - } - else if (*p == '$') - { - gboolean finished = FALSE; - p++; - /* macro reference */ - if (*p == '{') - { - gchar *colon, *fncode; - gint macro_len; - gchar *default_value = NULL; - gint msg_ref = 0; - - p++; - start = p; - while (*p && *p != '}') - p++; - - if (!*p) - { - error_pos = p - self->template; - error_info = "Invalid macro, '}' is missing"; - goto error; - } - - p++; - - colon = memchr(start, ':', p - start - 1); - if (colon) - { - macro_len = colon - start; - fncode = colon < p ? colon + 1 : NULL; - } - else - { - macro_len = p - start - 1; - fncode = NULL; - } - - if (fncode) - { - if (*fncode == '-') - { - default_value = g_strndup(fncode + 1, p - fncode - 2); - } - else - { - error_pos = fncode - self->template; - error_info = "Unknown substitution function"; - goto error; - } - } - parse_msg_ref(&p, &msg_ref); - last_macro = log_macro_lookup(start, macro_len); - if (last_macro == M_NONE) - { - /* this was not a known macro, take it as a "value" reference */ - log_template_add_value_elem(self, start, macro_len, last_text, default_value, msg_ref); - } - else - { - log_template_add_macro_elem(self, last_macro, last_text, default_value, msg_ref); - } - finished = TRUE; - } - /* template function */ - else if (*p == '(') - { - gchar quote = 0; - gint parens = 1; - GPtrArray *strv; - GString *arg_buf; - gint msg_ref; - - strv = g_ptr_array_new(); - arg_buf = g_string_sized_new(32); - - p++; - start = p; - /* skip white space */ - while (*p && *p == ' ') - p++; - - g_string_truncate(arg_buf, 0); - while (*p) - { - if (!quote) - { - if (*p == '\\') - { - p++; - } - else if (*p == '"' || *p == '\'') - { - quote = *p; - if (parens == 1) - { - /* skip the quote in top-level and don't skip in expressions enclosed in parens */ - p++; - continue; - } - } - else if (parens == 1 && (*p == ' ' || *p == '\t')) - { - g_ptr_array_add(strv, g_strndup(arg_buf->str, arg_buf->len)); - g_string_truncate(arg_buf, 0); - while (*p && (*p == ' ' || *p == '\t')) - p++; - continue; - } - else if (*p == '(') - { - parens++; - } - else if (*p == ')') - { - parens--; - if (parens == 0) - break; - } - } - else - { - if (quote == *p) - { - /* end of string */ - quote = 0; - if (parens == 1) - { - p++; - continue; - } - } - } - if (*p) - { - g_string_append_c(arg_buf, *p); - p++; - } - } - - if (arg_buf->len) - g_ptr_array_add(strv, g_strndup(arg_buf->str, arg_buf->len)); - g_ptr_array_add(strv, NULL); - if (*p != ')') - { - g_string_free(arg_buf, TRUE); - g_ptr_array_foreach(strv, (GFunc) g_free, NULL); - g_ptr_array_free(strv, TRUE); - - error_pos = p - self->template; - error_info = "Invalid template function reference, missing function name or inbalanced '('"; - goto error; - } - p++; - parse_msg_ref(&p, &msg_ref); - if (!log_template_add_func_elem(self, last_text, strv->len - 1, (gchar **) strv->pdata, msg_ref, error)) - { - g_ptr_array_foreach(strv, (GFunc) g_free, NULL); - g_ptr_array_free(strv, TRUE); - g_string_free(arg_buf, TRUE); - goto error_set; - } - g_ptr_array_free(strv, FALSE); - g_string_free(arg_buf, TRUE); - finished = TRUE; - } - else if ((*p >= 'A' && *p <= 'Z') || (*p >= 'a' && *p <= 'z') || (*p == '_') || (*p >= '0' && *p <= '9')) - { - start = p; - do - { - p++; - } - while ((*p >= 'A' && *p <= 'Z') || (*p >= 'a' && *p <= 'z') || (*p == '_') || (*p >= '0' && *p <= '9')); - last_macro = log_macro_lookup(start, p - start); - if (last_macro == M_NONE) - { - /* this was not a known macro, take it as a "value" reference */ - log_template_add_value_elem(self, start, p-start, last_text, NULL, 0); - } - else - { - log_template_add_macro_elem(self, last_macro, last_text, NULL, 0); - } - finished = TRUE; - } - else if (*p == '$') - { - g_string_append_c(last_text, '$'); - p++; - } - else - { - g_string_append_c(last_text, '$'); - g_string_append_c(last_text, *p); - p++; - } - if (finished) - { - if (last_text) - g_string_free(last_text, TRUE); - last_macro = M_NONE; - last_text = NULL; - } - } - else - { - g_string_append_c(last_text, *p); - p++; + log_template_reset_compiled(self); + if (!last_text) + last_text = g_string_sized_new(0); + g_string_sprintf(last_text, "error in template: %s", self->template); + log_template_add_macro_elem(self, M_NONE, last_text, NULL, 0); + g_string_free(last_text, TRUE); + return FALSE; } } - if (last_macro != M_NONE || last_text) + if (last_text) { - log_template_add_macro_elem(self, last_macro, last_text, NULL, 0); + log_template_add_macro_elem(self, M_NONE, last_text, NULL, 0); g_string_free(last_text, TRUE); } self->compiled_template = g_list_reverse(self->compiled_template); return TRUE; - - error: - g_set_error(error, LOG_TEMPLATE_ERROR, LOG_TEMPLATE_ERROR_COMPILE, "%s, error_pos='%d'", error_info, error_pos); - - error_set: - log_template_reset_compiled(self); - if (!last_text) - last_text = g_string_sized_new(0); - g_string_sprintf(last_text, "error in template: %s", self->template); - log_template_add_macro_elem(self, M_NONE, last_text, NULL, 0); - g_string_free(last_text, TRUE); - return FALSE; } --

1 0

syslogng-web: a simple and fast log viewer
by Peter Czanik 23 Aug '13

23 Aug '13

Hello, It's always nice to learn about projects building on syslog-ng. "syslogng-web" is one of them, a simple and fast log viewer for syslog-ng logs collected into mongodb. You can read more about it at https://czanik.blogs.balabit.com/2013/08/syslog-ng-web-a-simple-and-fast-lo… If you know about any other projects with syslog-ng at their core, please let me know so I can check it out and spread the word about it! Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

[Bug 245] New: syslog-ng 3.4.3 / 3.4.2 add bogus characters to /var/log/ messages
by bugzilla＠bugzilla.balabit.com 17 Aug '13

17 Aug '13

https://bugzilla.balabit.com/show_bug.cgi?id=245 Summary: syslog-ng 3.4.3 / 3.4.2 add bogus characters to /var/log/messages Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: himbeere(a)meine-oma.de Type of the Report: bug Estimated Hours: 0.0 Hello. Suddenly my syslog-ng adds bogus characters to /var/log/messages like this: Aug 16 21:40:56 hn2 syslog-ng[2009]: syslog-ng shutting down; version='3.4.2' Aug 16 21:41:44 hn2 syslog-ng[1994]: syslog-ng starting up; version='3.4.3' Aug 16 21:41:44 hn2 syslog-ng[1994]: Internal error, alarm_set() called while an alarm is still active; Aug 16 21:41:44 ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ kernel: [ 0.000000] Initializing cgroup subsys cpuset Aug 16 21:41:44 hn2 kernel: [ 0.000000] Initializing cgroup subsys cpu Aug 16 21:41:44 hn2 kernel: [ 0.000000] Initializing cgroup subsys cpuacct Aug 16 21:41:44 hn2 kernel: [ 0.000000] Linux version 3.10.7 (root@hn2) (gcc version 4.6.3 (Gentoo 4.6.3 p1.13, pie-0.5.2) ) #2 SMP Fri Aug 16 21:17:31 CEST 2013 This leads to an unreadable file. Well i can open the file with vi and then it looks okay. But with less it looks like this: 00000000 41 75 67 20 31 36 20 32 31 3a 32 37 3a 34 36 20 |Aug 16 21:27:46 | 00000010 68 6e 32 20 73 79 73 6c 6f 67 2d 6e 67 5b 32 38 |hn2 syslog-ng[28| 00000020 38 31 5d 3a 20 73 79 73 6c 6f 67 2d 6e 67 20 73 |81]: syslog-ng s| 00000030 74 61 72 74 69 6e 67 20 75 70 3b 20 76 65 72 73 |tarting up; vers| 00000040 69 6f 6e 3d 27 33 2e 34 2e 32 27 0a 41 75 67 20 |ion='3.4.2'.Aug | 00000050 31 36 20 32 31 3a 32 38 3a 30 33 20 68 6e 32 20 |16 21:28:03 hn2 | 00000060 73 68 75 74 64 6f 77 6e 5b 32 38 39 35 5d 3a 20 |shutdown[2895]: | 00000070 73 68 75 74 74 69 6e 67 20 64 6f 77 6e 20 66 6f |shutting down fo| 00000080 72 20 73 79 73 74 65 6d 20 72 65 62 6f 6f 74 0a |r system reboot.| 00000090 41 75 67 20 31 36 20 32 31 3a 32 38 3a 30 33 20 |Aug 16 21:28:03 | 000000a0 68 6e 32 20 69 6e 69 74 3a 20 53 77 69 74 63 68 |hn2 init: Switch| 000000b0 69 6e 67 20 74 6f 20 72 75 6e 6c 65 76 65 6c 3a |ing to runlevel:| 000000c0 20 36 0a 41 75 67 20 31 36 20 32 31 3a 32 38 3a | 6.Aug 16 21:28:| Is there any solution to this? Thanks and best regards t. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2