Martin Holste <mcholste@gmail.com> writes:
This is definitely something that's needed, but I'm a bit concerned with the complexity. I want to propose another idea, which is just off the top of my head: What if something like the program() destination can be used to do the message transformations so that your favorite script or C program can be used inline as a log preprocessor as well as a destination.
That would make it necessary to serialize LogMessages, pass it to the program, then deserialize it - which would be costy, and that's something I can already do: I can send JSON to a program, and set up my system to get JSON back, parse it and be happy. It's not efficient, and requires a separate program running. It's much much faster if some of these things can be done *inside* syslog-ng. It may not suite every need that is possible, but it covers a large set, and I hope to make it so that adding new functionality would be very, very easy.
The reason I think this could be helpful is that then you can re-use utility scripts and code you already have laying around without having to learn the new system.
That's already possible with a little glue-code. It could be made simpler, so that you could use program() as a kind of pipe, and that's something that might be worth exploring, but it's not a replacement for what I wish to do with map{}.
Granted, in a lot of cases, the proposed built-in system would be fairly straightforward, but for advanced usage, like tying in with external databases, it could be very helpful to have the ability to offload the transforming to an arbitrary script or program. I think the challenge would be with latency and potential queue clogging, but that can be managed.
Indeed. This would be another useful feature, perhaps even easier to implement than the map{} stuff I proposed, but it has its disadvantages (speed & efficiency for one). -- |8]