[syslog-ng] [RFC] value-pairs and key rewriting

[Gergely Nagy]

Ladies and Gentlemen, welcome to the latest issue of a tired mouse's
brain dump! (Also known as "Quid scriptura, parva mus?")

> A while ago, I posted a proposal about key rewriting for
> value-pairs. Today I'm happy to announce that I have some half-baked
> code, and a couple of ideas on how to proceed.

Good news, everyone! The code's far better baked now! But we still need
a little bit of dressing here and there to make it not only delicious,
but attractive to the eyes aswell.

A new class of object-like things were introduced, the root of whom is
the ValuePairsTransform structure:

It has a match, a transform and a destroy function, and a match_str
property. But I plan to get rid of match_str, and move it to the
appropriate sub-classes instead - it just so happens that I sat down to
write this memo before doing that.

Anyway, the ValuePairsTransform family of objects are at the core of the
key rewriting functionality: whenever the value-pairs framework would
insert a key into the final set, it will transform them using a list of
transformers. It will first call the appropriate object's ->match()
function, which decides whether a particular key is interesting or
not. If it is, we quickly transform it with ->transform().

And in the very end, we free up all memory with ->destroy().

One thing of note about the transform function, is that it MUST return a
const gchar * - and it is its responsibility to free that at ->destroy()
time, and not earlier. If it needs freeing at all, that is.

I might relax this, and add support for explicitly freeing the cached
values, so that once we're done processing a message, we can free the
associated cached data. Though, care was taken to design the cache so
that it will not eat all that much memory.

The reason it must return a const gchar * is because I wanted to allow
the key to be a borrowed pointer: the shift() transform function for
example takes advantage of this.

Now, ValuePairsTransform, and its descendants live in
lib/vptransform.[ch]. They were moved out of value-pairs.[ch] as part of
the cleanup process.

Apart from baking the code into something that starts to resemble a
delicious cake, a few more ingredients were added too! Well, one. But
it's a start, and I'm not a cook anyway...

This is the replace() transform function, with which one can select a
prefix to match on, and replace it with another.

Thus, we can do this:

value-pairs (
 scope("selected-macros" "nv-pairs")
 rekey(
   replace("." "_")
 )
);

And that will replace leading dots with an underscore. This can then
replace a similar thing in the current mongodb destination, giving more
freedom to the administrator, and making the destination driver's code
simpler aswell.

This is all available on the vp/rekey branch of my git tree at
git://git.balabit.hu/algernon/syslog-ng-3.3.git - for the brave and
adventurous, who are not afraid of mice.

For fun and profit, we can do some interesting transformation chains
now:

value-pairs(
  scope("everything")
  rekey(
   add-prefix(".secevt" "events")
   add-prefix(".classifier" "syslog-ng")
   shift(".sdata.*" 1)
   replace("." "_")
  )
)

This will turn a key like ".secevt.verdict" into
"events.secevt.verdict"; a key like ".classifier.rule_id" to
"syslog-ng.classifier.rule_id"; everything under ".sdata" will be moved
to the "sdata" namespace, and the rest of the keys that begin with a
dot, will begin with an underscore instead.

Funky, isn't it?

Performance should be a bit better than the last time, but there's still
room for improvement: I see a few possible ways to get rid of the ugly
self->cache hash tables I'm using in some of the transformers.

But that will be the topic of another brain dump some other time. For
now, lets rejoice that the cake is almost fully baked now, the code's a
lot cleaner, replace() is done, and there's very little left to do!

-- 
|8]