[syslog-ng] [PATCH] correlation: explicit context timeout action

[Balint Kovacs]

Hi,

please ignore my previous message, talking to Bazsi IRL revealed that I 
had an incorrect understanding of the context-timeout in correlation. In 
fact the context-timeout is updated every time the message matches a 
pattern, so having the context-timeout set to 0 in the last rule has the 
exact same effect. I had a feeling that this feature was designed into 
it somehow, just didn't know how :)

Balint

On 09/21/2011 12:07 PM, Balint Kovacs wrote:
> Hi,
>
> I'm trying to use patterndb correlation to find matching firewall
> connection startup and ending log messages and emit a consolidated
> message. I found that with high-volume load, the memory usage of
> syslog-ng climbs rapidly and it would be beneficial to be able to kick
> already ended events from the memory. Please find a patch below (against
> 3.4) that does this, I thought the easy place for the user would be the
> <action>  part of the ending rule, this way it can even be specified when
> the context should be ended.
>
> Balint
>
> From: Balint Kovacs<blint at balabit.hu>
> Date: Wed, 21 Sep 2011 11:46:30 +0200
> Subject: [PATCH] correlation: add action to explicitly end context
>
> If there is a message that can be associated with the end of a context,
> the below action can explicitly end it, reducing the memory footprint.
>
> Example:
> <action>
> <end-context />
> </action>
>
> Signed-off-by: Balint Kovacs<blint at balabit.hu>
> ---
>    modules/dbparser/patterndb-int.h |    3 ++-
>    modules/dbparser/patterndb.c     |   22 ++++++++++++++++++++++
>    2 files changed, 24 insertions(+), 1 deletions(-)
>
> diff --git a/modules/dbparser/patterndb-int.h
> b/modules/dbparser/patterndb-int.h
> index 0434847..eff77e6 100644
> --- a/modules/dbparser/patterndb-int.h
> +++ b/modules/dbparser/patterndb-int.h
> @@ -115,7 +115,8 @@ enum
>    enum
>    {
>      RAC_NONE,
> -  RAC_MESSAGE
> +  RAC_MESSAGE,
> +  RAC_EXPIRE
>    };
>
>    /* a rule may contain one or more actions to be performed */
> diff --git a/modules/dbparser/patterndb.c b/modules/dbparser/patterndb.c
> index 343ef6d..72e97e0 100644
> --- a/modules/dbparser/patterndb.c
> +++ b/modules/dbparser/patterndb.c
> @@ -577,6 +577,18 @@ pdb_rule_run_actions(PDBRule *self, gint trigger,
> PatternDB *db, PDBContext *con
>                      emit(genmsg, TRUE, emit_data);
>                      log_msg_unref(genmsg);
>                      break;
> +                case RAC_EXPIRE:
> +                  if (context)
> +                    {
> +                      msg_debug("Expiring patterndb correllation
> context as directed by action",
> +                                 evt_tag_str("last_rule",
> context->rule->rule_id),
> +
> evt_tag_long("remaining_context_count",
> g_hash_table_size(context->db->state)),
> +                                 NULL);
> +                      g_hash_table_remove(context->db->state,
> &context->key);
> +                      timer_wheel_del_timer(db->timer_wheel,
> context->timer);
> +                      pdb_context_unref(context);
> +                    }
> +                  break;
>                    default:
>                      g_assert_not_reached();
>                      break;
> @@ -921,6 +933,16 @@ pdb_loader_start_element(GMarkupParseContext
> *context, const gchar *element_name
>          state->current_action->content_type = RAC_MESSAGE;
>          state->current_message =&state->current_action->content.message;
>        }
> +  else if (strcmp(element_name, "end-context") == 0)
> +    {
> +      if (!state->in_action)
> +        {
> +          *error = g_error_new(1, 0, "Unexpected<end-context>  element,
> it must be inside an action");
> +          return;
> +        }
> +      state->current_action->content_type = RAC_EXPIRE;
> +      state->current_message =&state->current_action->content.message;
> +    }
>    }
>
>    void