[syslog-ng] value-pairs and sdata

[Gergely Nagy]

Balazs Scheidler <bazsi at balabit.hu> writes:

>> In the long run, I think that key rewriting is the way forward (but
>> that's 3.4 material), and if we do go that way, I don't see much point
>> in introducing a conflicting idea in 3.3.
>> 
> I think key rewriting doesn't conflict with my idea to use '_' as the
> initial character in these cases.

[..snip...]

I'm convinced.

> Here's what I propose:
>   * let's use the underscore for now, key rewrite functions will come
> soon anyway.
>   * let's solve the performance concerns by doing the initial dot ->
> underscore translation in libmongo-client, which is in a better position
> to do that, it really doesn't make sense to use initial dots for any
> future libmongo-client users anyway. That way we can do it without
> having to copy the value name.

We've talked about this earlier, and I spent the night brooding about
this, and while it would be easy to do this in libmongo-client, I'm
reluctant to do so, because it feels too much like second guessing the
user. LMC is reasonably simple, and the lower level functions
syslog-ng's afmongodb uses from it are like the bare metal. I really
wouldn't want to add transformations like this there.

On the other hand, recent libmongo-client has a higher-level interface,
which does a lot of second guessing, and the .->_ transformation might
go there (afmongodb will be adapted to use these higher level functions
at some point, btw). But even then, I'd rather return with an error and
let the library user handle it in any way they like.

Therefore I think this should be done on the syslog-ng side. The
performance loss wouldn't be much bigger than doing the same thing in
libmongo-client: it's not cheaper to do it there (not with the current
code, anyway).

> In the way forward, I'd like to create a rewrite plugin (in the
> syslog-ng sense) that would transform a log message using value-pairs
> syntax. This way such translations would become possible _before_ a
> message would hit the destinations.

This sounds intriguing.

-- 
|8]