[syslog-ng] RFC: Applying transformations to a whole log message

[Gergely Nagy]

Martin Holste <mcholste at gmail.com> writes:

> This is definitely something that's needed, but I'm a bit concerned
> with the complexity.  I want to propose another idea, which is just
> off the top of my head:  What if something like the program()
> destination can be used to do the message transformations so that your
> favorite script or C program can be used inline as a log preprocessor
> as well as a destination.

That would make it necessary to serialize LogMessages, pass it to the
program, then deserialize it - which would be costy, and that's
something I can already do: I can send JSON to a program, and set up my
system to get JSON back, parse it and be happy.

It's not efficient, and requires a separate program running. It's much
much faster if some of these things can be done *inside* syslog-ng.

It may not suite every need that is possible, but it covers a large set,
and I hope to make it so that adding new functionality would be very,
very easy.

> The reason I think this could be helpful is
> that then you can re-use utility scripts and code you already have
> laying around without having to learn the new system.

That's already possible with a little glue-code. It could be made
simpler, so that you could use program() as a kind of pipe, and that's
something that might be worth exploring, but it's not a replacement for
what I wish to do with map{}.

> Granted, in a lot of cases, the proposed built-in system would be
> fairly straightforward, but for advanced usage, like tying in with
> external databases, it could be very helpful to have the ability to
> offload the transforming to an arbitrary script or program.  I think
> the challenge would be with latency and potential queue clogging, but
> that can be managed.

Indeed. This would be another useful feature, perhaps even easier to
implement than the map{} stuff I proposed, but it has its disadvantages
(speed & efficiency for one).

-- 
|8]