Hi, please ignore my previous message, talking to Bazsi IRL revealed that I had an incorrect understanding of the context-timeout in correlation. In fact the context-timeout is updated every time the message matches a pattern, so having the context-timeout set to 0 in the last rule has the exact same effect. I had a feeling that this feature was designed into it somehow, just didn't know how :) Balint On 09/21/2011 12:07 PM, Balint Kovacs wrote:
Hi,
I'm trying to use patterndb correlation to find matching firewall connection startup and ending log messages and emit a consolidated message. I found that with high-volume load, the memory usage of syslog-ng climbs rapidly and it would be beneficial to be able to kick already ended events from the memory. Please find a patch below (against 3.4) that does this, I thought the easy place for the user would be the <action> part of the ending rule, this way it can even be specified when the context should be ended.
Balint
From: Balint Kovacs<blint@balabit.hu> Date: Wed, 21 Sep 2011 11:46:30 +0200 Subject: [PATCH] correlation: add action to explicitly end context
If there is a message that can be associated with the end of a context, the below action can explicitly end it, reducing the memory footprint.
Example: <action> <end-context /> </action>
Signed-off-by: Balint Kovacs<blint@balabit.hu> --- modules/dbparser/patterndb-int.h | 3 ++- modules/dbparser/patterndb.c | 22 ++++++++++++++++++++++ 2 files changed, 24 insertions(+), 1 deletions(-)
diff --git a/modules/dbparser/patterndb-int.h b/modules/dbparser/patterndb-int.h index 0434847..eff77e6 100644 --- a/modules/dbparser/patterndb-int.h +++ b/modules/dbparser/patterndb-int.h @@ -115,7 +115,8 @@ enum enum { RAC_NONE, - RAC_MESSAGE + RAC_MESSAGE, + RAC_EXPIRE };
/* a rule may contain one or more actions to be performed */ diff --git a/modules/dbparser/patterndb.c b/modules/dbparser/patterndb.c index 343ef6d..72e97e0 100644 --- a/modules/dbparser/patterndb.c +++ b/modules/dbparser/patterndb.c @@ -577,6 +577,18 @@ pdb_rule_run_actions(PDBRule *self, gint trigger, PatternDB *db, PDBContext *con emit(genmsg, TRUE, emit_data); log_msg_unref(genmsg); break; + case RAC_EXPIRE: + if (context) + { + msg_debug("Expiring patterndb correllation context as directed by action", + evt_tag_str("last_rule", context->rule->rule_id), + evt_tag_long("remaining_context_count", g_hash_table_size(context->db->state)), + NULL); + g_hash_table_remove(context->db->state, &context->key); + timer_wheel_del_timer(db->timer_wheel, context->timer); + pdb_context_unref(context); + } + break; default: g_assert_not_reached(); break; @@ -921,6 +933,16 @@ pdb_loader_start_element(GMarkupParseContext *context, const gchar *element_name state->current_action->content_type = RAC_MESSAGE; state->current_message =&state->current_action->content.message; } + else if (strcmp(element_name, "end-context") == 0) + { + if (!state->in_action) + { + *error = g_error_new(1, 0, "Unexpected<end-context> element, it must be inside an action"); + return; + } + state->current_action->content_type = RAC_EXPIRE; + state->current_message =&state->current_action->content.message; + } }
void