[syslog-ng] Message correlation, everything needs an "@"? And a grep problem.

[Balazs Scheidler]

On Sun, 2011-02-20 at 14:43 +0100, Balazs Scheidler wrote:
> Hi,
> 
> On Fri, 2011-02-11 at 16:48 +0100, Valentijn Sessink wrote:
> > Hello list,
> > 
> > After playing with correlation for a while, here are a few remarks on
> > it's workings. I have played with syslog-ng 3.2.2, built from source on
> > an Ubuntu 10.04 machine.
> > 
> > For some reason, the "@1", "@2"... etcetera specifiers needed to be all
> > over the place to be able to see any variable at all. For example, the
> > rule below describes the last message of a Postfix log trail. Note that
> > even the context-id is specified as "${queueid}@1". All other references
> > also have an "@" sign. Without the @, the variables seem just empty.
> > 
> > <rule id='X' context-id="${queueid}@1" context-scope="host"
> > provider="openoffice" class="maildelivery">
> >  <patterns>
> >     <pattern>@ESTRING:queueid: @removed</pattern>
> >  </patterns>
> >  <actions>
> >     <action>
> >        <message>
> >           <values>
> >              <value name="MESSAGE">${queueid}@1 mail
> > to=&lt;${toaddress}@2&gt; from=&lt;${pffrom}@3&gt; connection
> > from=${pfremotehost}@5[${pfremoteip}@5] msgid=&lt;${msgid}@4&gt;
> > orig_to=&lt;${origto}@2&gt; delay=${deliverystatus}@2</value>
> >           </values>
> >        </message>
> >     </action>
> >  </actions>
> > </rule>
> > 
> > Is this how it's meant to be?
> 
> hmm.. no, @1 should be the one before the current message. I've just
> tried to reproduce this with 3.3 but it works fine.
> 
> ... seems to be work OK in 3.2 too. how did you reproduce this exactly?

I looked into this problem, and now I know the root cause. In case of
the main body of the rule, @1 is the predecessor message and @0 is the
current one.

However within <actions>, the message being created is also added to the
context temporarily, e.g. the numbers are shifted up

I don't remember my original intention, it was probably just for
symmetry's sake, by this patch:

$ git show 058e8499
commit 058e8499e2d2513b1da7fccf694b60f4b25dfbaf
Author: Balazs Scheidler <bazsi at balabit.hu>
Date:   Mon Oct 11 14:50:10 2010 +0200

    patterndb: temporarily add the message being generated to the
context

diff --git a/modules/dbparser/patterndb.c b/modules/dbparser/patterndb.c
index 14e511b..b4c05a4 100644
--- a/modules/dbparser/patterndb.c
+++ b/modules/dbparser/patterndb.c
@@ -521,7 +521,9 @@ pdb_rule_run_actions(PDBRule *self, gint trigger, PatternDB *db, PDBContext *con
                       g_assert_not_reached();
                       break;
                     }
+                  g_ptr_array_add(context->messages, genmsg);
                   pdb_message_apply(&action->content.message, context, genmsg, buffer);
+                  g_ptr_array_remove_index_fast(context->messages, context->messages->len - 1);
                   emit(genmsg, TRUE, emit_data);
                   break;
                 default:


This behaviour makes it possible to reference name-value pairs of the
_current_ message (e.g. the one being generated), but that's usually
empty (except for the default fields HOST, PROGRAM & PID depending on
how the context was filled).

Documentation wise, this is bad behaviour, as the documentation contains
the one without '@1' values, but there are two releases in the 3.2
branch which behave as you have found out.

I'd say that not that many people are probably using this feature right
now, so I'd go for reverting that patch.

Are there any other opinions?

-- 
Bazsi