December 2013 - syslog-ng - lists.balabit.hu

Output to file in /proc
by Valentijn Sessink 13 Dec '13

13 Dec '13

Hello list, As most of the functionality of "sshguard", "fail2ban" and others is in syslog-ng, I'm trying to use syslog-ng to add entries to my Linux kernel based firewall (iptables). I'm almost ready to implement one, but I have a question about undeliverable log lines: what happens if a certain file() destination blocks? Will syslog-ng retry to deliver the same message, or will it lose the message? What I'm doing is using the "ipt_recent" module; with this module, one can check if a certain IP address was seen before. For example, the following rule will block you if you were seen 5 or more times ("hitcount 5") in the "violators" list last minute ("seconds 60): iptables -A INPUT -m recent \ --rcheck --hitcount 5 --seconds 60 --name violators -j DROP The fun part is, that you can add IP adresses by stuffing them into a file in /proc: echo '+127.0.0.1' > /proc/net/xt_recent/violators ... will add 127.0.0.1 to the "violators" with the current time. However: echo 'whatever' > /proc/net/xt_recent/violators ... gives an error: Invalid argument (22) As far as I can see, syslog-ng will not try again to deliver the same message; but is this by design? I.e. can I trust syslog-ng to not "block" because of a single malformed IP address? Because then, an implementation of sshguard could be simply swapped with a simple: destination df_ipt_recent { file ("/proc/net/xt_recent/blocker" template("+${usracct.device}\n") ); }; filter f_ipt_recent { tags("secevt") and match("REJECT" value("secevt.verdict")); }; log { source (s_all); parser(pattern_db); filter(f_addtofirewall);destination(df_addtofirewall);}; ... which would be cool! Best regards, Valentijn

6 26

syslog-ng 3.5.1 - question about flags(final)...
by Johnson, Chris (HP TippingPoint Roseville) 10 Dec '13

10 Dec '13

Hello all, I'm in the process of upgrading from version 3.3.9 to 3.5.1 and have a question about how the 'flags(final);' is working in 3.5.1. In 3.3.9, I use the following structure of imbedded log statements: ################################################################################ # Service ipsec # filter f_ipsec_pgm{program("IPSEC-*" type("glob")) or program("IKE-*" type("glob")) or program("CHARON-*" type("glob")) or program("charon-*" type("glob")); }; filter f_ipsec_lvl_01{level(warning..emerg)}; filter f_ipsec_lvl_02{level(info..emerg)}; log { source(s_local); filter(f_ipsec_pgm); log { filter(f_ipsec_lvl_01); destination(d_logID_11); }; log { filter(f_ipsec_lvl_02); rewrite(r_quote_newlines); destination(d_logID_13); }; flags(final); }; In this case log messages of the 'correct' program would further be filtered on their severity level. * Info level messages would only be sent to 'd_logID_13' * Warning level messages (and above) would be sent to BOTH 'd_logID_11' and 'd_logID_13'. * Debug level messages would be discarded. Under 3.5.1, the 'filtering ' stops after it matches once: * Warning messages (and above) are only sent to 'd_logID_11' and NOT 'd_logID_13'. * Info messages are still (correctly) being sent only to 'd_logID_13'. If I remove (or comment out) the 'flags(final);' statement, messages are filtered correctly (i.e. the way I *want* them to be filtered :)) except that they also are being processed by all the following log statements and are being caught in my final filter of 'program("*" type("glob"))'. NOTE: yes, I know that I could use 'flags(fallback)' in my final filter, but that would still have every message processed by every log filter, and I would like to avoid that. So, what would be the correct way to set up my log statement to re-create the 3.3 behavior? Thanks, Chris ---------------------------------------- Christopher Johnson chris.johnson3(a)hp.com<mailto:chris.johnson3@hp.com> HP Software - Security Product Group (916) 785-2817 ----------------------------------------

3 7

[review request] 3.6queue/f/stats-reset
by Balazs Scheidler 10 Dec '13

10 Dec '13

Hi, I'm about to integrate a stats-counter related change and prior to doing that I'd like to solicit feedback. There are both user-visible changes and developer visible changes, feedback on both is sought. Code related comments are preferred on github, functionality comments are best sent via email in response to this one. Here's the branch: https://github.com/balabit/syslog-ng/tree/3.6queue/f/stats-reset Thanks in advance. User-visible changes: ===================== syslog-ng is keeping track on a number of statistics counters that can be used to track what is happening. The detail level of these counters is controlled by the stats-level() global option, which defaults to zero. syslog-ng reports these counters periodically every stats-freq() seconds, or preferably on-demand by invoking "syslog-ng-ctl stats" on the command line. At stats-level(3) a number of "dynamic" counters are enabled, which track the number of messages on a per $HOST, per $PROGRAM and per $HOST_FROM basis. This means that you can follow how many messages a given $PROGRAM is generating in a period. The issue with dynamic counters is that they can consume a lot of memory if the rate of new $PRORGRAM/$HOST values is high. Previously the cleanup happened in a limited way at configuration reload (e.g. SIGHUP) time, which is not really useful for controlling memory usage by this feature. The current patch series introduces a stats-lifetime() option and syslog-ng starts inspecting counters every once in a while and prune those that are updated earlier than stats-lifetime(). Not-so visible changes ====================== I have fixed a few bugs and renamed a function to make it easier to understand. Details are in the git history. Balazs Scheidler (7): logsource: use stats_instant_inc_dynamic_counter() for HOST counters logsource: track time for PROGRAM counters mainloop: remove old orphanage based stats cleanup stats: add stats_prune_old_counters() stats-timer: implement stats_lifetime option cfg: add stats-lifetime() global option stats: rename stats_instance_inc_dynamic_counter() to stats_register_and_increment_dynamic_counter() lib/cfg-grammar.y | 16 ++++++++------- lib/cfg-parser.c | 1 + lib/cfg.c | 3 ++- lib/cfg.h | 1 + lib/logsource.c | 15 +++----------- lib/mainloop.c | 1 - lib/stats-timer.c | 61 +++++++++++++++++++++++++++++++++++++++---------------- lib/stats-timer.h | 2 +- lib/stats.c | 52 +++++++++++++++++++++++++++++++++++++++-------- lib/stats.h | 7 +++---- 10 files changed, 108 insertions(+), 51 deletions(-) -- Bazsi

6 12

[review request] please check
by Balazs Scheidler 10 Dec '13

10 Dec '13

Hi, As a number of things depend on the logtransport.c split, I'd like to merge that separately. https://github.com/balabit/syslog-ng/pull/36 The branch itself is really just shuffling around files, basically the monolitic logtransport.c that contained 4 classes were split into separate files under the lib/transport subdirectory. Thanks in advance. Bazsi

2 1

[git] 3.6queue prefix in branches changed to 3.6
by Balazs Scheidler 07 Dec '13

07 Dec '13

Hi, For consistency, I've renamed all "3.6queue" prefixes in branch names to 3.6. This makes them possible to push all "pending" branches with a single push call and is consistent with the branches aimed at different major releases. I might remove the prefix altogether in the future (e.g. using no prefix for development branches destined at the latest master), but as I'm working a number of things in parallel for 3.6, I prefer to keep them as one bunch for now. -- Bazsi

1 0

Trouble with microseconds
by Clayton Dukes 06 Dec '13

06 Dec '13

Hi, I have a system configured to microseconds logging, but for most of entries it's always zero - actually the only program which logs it correctly is the syslog-ng itself. @1386239689.000000 13 blah @1386239859.947724 45 syslog-ng shutting down; version='3.5.2' @1386239860.304194 45 syslog-ng starting up; version='3.5.2' @1386239860.337325 45 EOF on control channel, closing connection; @1386239865.000000 13 blah Anyone have an idea what I'm doing wrong? Here's the config: destination d_mydest{ program("/somepath" template("@${UNIXTIME:--}\t${PRI:--}\t${MSGONLY}\n") flags(no_multi_line) flush_timeout(100) frac_digits(6) ); }; ______________________________________________________________ Clayton Dukes ______________________________________________________________

2 2

syslog - sql error
by vijay 03 Dec '13

03 Dec '13

HI All, Iam using syslog-ng3.2.5 rpm in centos x86_64 environment.Past days it is working well.Suddenly iam seeing a problem in writing to mysql Error running SQL query; type='mysql', host='192.168.9.68', port='3306', user='cygnet', database='CygNetDB', error='1072: Key column \'date\' doesn\'t exist in table', query='CREATE INDEX SYSLOGS_20131130_date_idx ON SYSLOGS_20131130 (date)' Error adding missing index; table='SYSLOGS_20131130', column='date' Error running SQL query; type='mysql', host='192.168.9.68', port='3306', user='cygnet', database='CygNetDB', error='1170: BLOB/TEXT column \'facility\' used in key specification without a key length', query='CREATE INDEX SYSLOGS_20131130_facility_idx ON SYSLOGS_20131130 (facility)' Error adding missing index; table='SYSLOGS_20131130', column='facility' vijay

2 1

[PATCH] rewrite_rule_clone: unit tests for clone of rewrite rules are implemented
by Juhász Viktor 02 Dec '13

02 Dec '13

These tests evince some problems about clone implementation. The clone implementations didn't increased the reference of the condition, and it leaks the template which is used by the constructors (set, subst). These errors are corrected in this patch. Signed-off-by: Juhász Viktor <jviktor(a)balabit.hu> --- lib/rewrite/Makefile.am | 2 + lib/rewrite/rewrite-set-tag.c | 2 +- lib/rewrite/rewrite-set.c | 4 +- lib/rewrite/rewrite-subst.c | 4 +- lib/rewrite/tests/Makefile.am | 9 ++++ lib/rewrite/tests/test_clone.c | 96 ++++++++++++++++++++++++++++++++++++++++++ libtest/test_case.h | 37 ++++++++++++++++ 7 files changed, 149 insertions(+), 5 deletions(-) create mode 100644 lib/rewrite/tests/Makefile.am create mode 100644 lib/rewrite/tests/test_clone.c create mode 100644 libtest/test_case.h diff --git a/lib/rewrite/Makefile.am b/lib/rewrite/Makefile.am index b111935..bc6547f 100644 --- a/lib/rewrite/Makefile.am +++ b/lib/rewrite/Makefile.am @@ -21,3 +21,5 @@ BUILT_SOURCES += \ lib/rewrite/rewrite-expr-grammar.h EXTRA_DIST += lib/rewrite/rewrite-expr-grammar.ym + +include lib/rewrite/tests/Makefile.am diff --git a/lib/rewrite/rewrite-set-tag.c b/lib/rewrite/rewrite-set-tag.c index 45d3cbf..154973d 100644 --- a/lib/rewrite/rewrite-set-tag.c +++ b/lib/rewrite/rewrite-set-tag.c @@ -56,7 +56,7 @@ log_rewrite_set_tag_clone(LogPipe *s) log_rewrite_init(&cloned->super); cloned->super.super.clone = log_rewrite_set_tag_clone; cloned->super.process = log_rewrite_set_tag_process; - cloned->super.condition = self->super.condition; + cloned->super.condition = self->super.condition ? filter_expr_ref(self->super.condition) : NULL; cloned->tag_id = self->tag_id; cloned->value = self->value; diff --git a/lib/rewrite/rewrite-set.c b/lib/rewrite/rewrite-set.c index 5b117ff..de42c8d 100644 --- a/lib/rewrite/rewrite-set.c +++ b/lib/rewrite/rewrite-set.c @@ -56,9 +56,9 @@ log_rewrite_set_clone(LogPipe *s) LogRewriteSet *self = (LogRewriteSet *) s; LogRewriteSet *cloned; - cloned = (LogRewriteSet *) log_rewrite_set_new(log_template_ref(self->value_template)); + cloned = (LogRewriteSet *) log_rewrite_set_new(self->value_template); cloned->super.value_handle = self->super.value_handle; - cloned->super.condition = self->super.condition; + cloned->super.condition = self->super.condition ? filter_expr_ref(self->super.condition) : NULL; return &cloned->super.super; } diff --git a/lib/rewrite/rewrite-subst.c b/lib/rewrite/rewrite-subst.c index f16afae..385fb82 100644 --- a/lib/rewrite/rewrite-subst.c +++ b/lib/rewrite/rewrite-subst.c @@ -101,10 +101,10 @@ log_rewrite_subst_clone(LogPipe *s) LogRewriteSubst *self = (LogRewriteSubst *) s; LogRewriteSubst *cloned; - cloned = (LogRewriteSubst *) log_rewrite_subst_new(log_template_ref(self->replacement)); + cloned = (LogRewriteSubst *) log_rewrite_subst_new(self->replacement); cloned->matcher = log_matcher_ref(self->matcher); cloned->super.value_handle = self->super.value_handle; - cloned->super.condition = self->super.condition; + cloned->super.condition = self->super.condition ? filter_expr_ref(self->super.condition) : NULL; return &cloned->super.super; } diff --git a/lib/rewrite/tests/Makefile.am b/lib/rewrite/tests/Makefile.am new file mode 100644 index 0000000..c5f740d --- /dev/null +++ b/lib/rewrite/tests/Makefile.am @@ -0,0 +1,9 @@ +lib_rewrite_tests_TESTS = lib/rewrite/tests/test_clone + +check_PROGRAMS += ${lib_rewrite_tests_TESTS} + +lib_rewrite_tests_test_clone_CFLAGS = $(TEST_CFLAGS) \ + -I${top_srcdir}/lib/rewrite/tests + +lib_rewrite_tests_test_clone_LDADD = $(TEST_LDADD) \ + $(PREOPEN_SYSLOGFORMAT) diff --git a/lib/rewrite/tests/test_clone.c b/lib/rewrite/tests/test_clone.c new file mode 100644 index 0000000..8bb6d90 --- /dev/null +++ b/lib/rewrite/tests/test_clone.c @@ -0,0 +1,96 @@ +#include "apphook.h" +#include "plugin.h" +#include "testutils.h" +#include "lib/rewrite/rewrite-expr.h" +#include "lib/rewrite/rewrite-set.h" +#include "lib/rewrite/rewrite-subst.h" +#include "lib/rewrite/rewrite-set-tag.h" +#include "lib/filter/filter-cmp.h" +#include "test_case.h" + +typedef struct _RewriteTestCase { + TestCase super; + LogRewrite *test_set_rewrite; + LogRewrite *test_subst_rewrite; + LogRewrite *test_tag_rewrite; +} RewriteTestCase; + +void +rewrite_testcase_setup(TestCase *s) +{ + RewriteTestCase *self = (RewriteTestCase *)s; + configuration = cfg_new(0x305); + + FilterExprNode *set_condition = fop_cmp_new(log_template_new(configuration, "OP1"), log_template_new(configuration, "OP2"), 10520); + FilterExprNode *subst_condition = fop_cmp_new(log_template_new(configuration, "OP1"), log_template_new(configuration, "OP2"), 10520); + FilterExprNode *set_tag_condition = fop_cmp_new(log_template_new(configuration, "OP1"), log_template_new(configuration, "OP2"), 10520); + + LogTemplate *test_template = log_template_new(configuration, "TEST"); + + self->test_set_rewrite = log_rewrite_set_new(test_template); + self->test_subst_rewrite = log_rewrite_subst_new(test_template); + + self->test_tag_rewrite = log_rewrite_set_tag_new("test_tag", FALSE); + + log_rewrite_set_condition(self->test_set_rewrite, set_condition); + log_rewrite_set_condition(self->test_subst_rewrite, subst_condition); + log_rewrite_set_condition(self->test_tag_rewrite, set_tag_condition); + + log_rewrite_subst_set_matcher(self->test_subst_rewrite, log_matcher_posix_re_new()); + + log_template_unref(test_template); +} + +void +rewrite_testcase_teardown(TestCase *s) +{ + RewriteTestCase *self = (RewriteTestCase *)s; + log_pipe_unref((LogPipe *)self->test_subst_rewrite); + log_pipe_unref((LogPipe *)self->test_set_rewrite); + log_pipe_unref((LogPipe *)self->test_tag_rewrite); + cfg_free(configuration); +} + +void +test_clone(LogRewrite *rule) +{ + LogPipe *cloned_rule = log_pipe_clone((LogPipe *)rule); + assert_guint32(rule->condition->ref_cnt, 2, ASSERTION_ERROR("Bad reference number of condition")); + log_pipe_unref(cloned_rule); + assert_guint32(rule->condition->ref_cnt, 1, ASSERTION_ERROR("Bad reference number of condition")); +} + +void +test_reference_on_set_condition_cloned(TestCase *s) +{ + RewriteTestCase *self = (RewriteTestCase *) s; + test_clone(self->test_set_rewrite); +} + +void +test_reference_on_subst_condition_cloned(TestCase *s) +{ + RewriteTestCase *self = (RewriteTestCase *) s; + test_clone(self->test_subst_rewrite); +} + +void +test_reference_on_tag_condition_cloned(TestCase *s) +{ + RewriteTestCase *self = (RewriteTestCase *) s; + test_clone(self->test_tag_rewrite); +} + +int +main(int argc, char **argv) +{ + app_startup(); + putenv("TZ=MET-1METDST"); + tzset(); + + start_grabbing_messages(); + RUN_TESTCASE(RewriteTestCase, rewrite_testcase, test_reference_on_set_condition_cloned); + RUN_TESTCASE(RewriteTestCase, rewrite_testcase, test_reference_on_subst_condition_cloned); + RUN_TESTCASE(RewriteTestCase, rewrite_testcase, test_reference_on_tag_condition_cloned); + stop_grabbing_messages(); +} diff --git a/libtest/test_case.h b/libtest/test_case.h new file mode 100644 index 0000000..4a49c9a --- /dev/null +++ b/libtest/test_case.h @@ -0,0 +1,37 @@ +/* + * test_case.h + * + * Created on: Nov 27, 2013 + * Author: jviktor + */ + +#ifndef TEST_CASE_H_ +#define TEST_CASE_H_ + +typedef struct _TestCase TestCase; + +struct _TestCase { + void (*setup)(TestCase *self); + void (*run)(TestCase *self); + void (*teardown)(TestCase *self); +}; + + +static void +run_test_case(TestCase *tc) +{ + if (tc->setup) tc->setup(tc); + tc->run(tc); + if (tc->teardown) tc->teardown(tc); +} + +#define RUN_TESTCASE(class_name, test_case_prefix, test_function) { \ + class_name *tc = g_new0(class_name, 1); \ + tc->super.setup = test_case_prefix ## _setup; \ + tc->super.teardown = test_case_prefix ## _teardown; \ + tc->super.run = test_function;\ + run_test_case(&tc->super); \ + g_free(tc); \ +} + +#endif /* TEST_CASE_H_ */ -- 1.8.3.2

2 1

[Bug 264] New: docbug: all-hard-macros does not exist in value-pairs
by bugzilla＠bugzilla.balabit.com 02 Dec '13

02 Dec '13

https://bugzilla.balabit.com/show_bug.cgi?id=264 Summary: docbug: all-hard-macros does not exist in value-pairs Product: syslog-ng Version: 3.5.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bugzilla.balabit(a)faxm0dem.org Type of the Report: --- Estimated Hours: 0.0 using this yields "Error parsing config, Error compiling template (Error parsing value-pairs: unknown scope all-hard-macros)" Indeed, the key "all-hard-macros" is not being referenced in 'lib/value-pairs.c' -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

Syslog-ng OSE Question.
by Malkin, Vlad 02 Dec '13

02 Dec '13

Hi, Is it possible to have syslog-ng client to invoke an external (to it) software to acknowledge when each individual log statement is sent to the server? Background: logs should not be lost. The system needs to store logs internally until they are transmitted to the server (use of TCP transport provides a level of reliable delivery). After each log statement is transmitted by syslog-ng client, it needs to be removed from internal store to preserve allocated space. Thank you for your help! --Vlad

5 6