[syslog-ng] configuring a redis syslog-ng destination

[Bill Anderson]

On Nov 26, 2010, at 2:22 PM, Jacob Vorreuter wrote:

> I'd like to configure syslog-ng to send logs to redis via a unix socket.  This would require implementing the redis protocol as a template or rewrite rule.  The only trick is getting the size in bytes of the message being processed.  Is there a macro for this or some other way of determining message size?
> 
> The protocol would look like this (pretending there existed a macro, MSG_SIZE):
> 
> *3            (number of arguments)
> $5            (size in bytes of argument 1)
> RPUSH         (operation)
> $4            (size in bytes of argument 2)
> logs          (key)
> $${MSG_SIZE}  (size in bytes of argument 3)
> $MSG          (value)
> 
> or as a string:
> 
> *3\r\n$5\r\nRPUSH\r\n$4\r\nlogs\r\n$${MSG_SIZE}\r\n$MSG\r\n
> 
> and finally as a destination with template:
> 
> destination d_redis { unix-stream("/tmp/redis.sock") template("*3\r\n$5\r\nRPUSH\r\n$4\r\nlogs\r\n$${MSG_SIZE}\r\n$MSG\r\n"); };
> 
> Does anyone have suggestions of how to accomplish this?


I would first prototype the pattern by writing a Python script and py-redis that took everything in on the command line and relayed it in. It'd be pretty basic and simple. You could then determine the message size in Python. Of course, you could also do it in C, C++, perl, etc.. In fact, I've got it in my backlog to write one in the next couple of weeks for a logging relay system; though it will be in c++.

You could even do it via socket and have the Python script determine message size (though I am not sure why you'd need it). If you wrote it it in C w/hiredis or c++ with the redis-cpp library it shouldn't be that much more work to make it into a plugin I would think. That way you wouldn't need to implement the actual protocol and would have the communication aspect worked out before tinkering inside of syslog-ng plugin world. Note: I've not made any plugins (yet) so I'm not saying it is difficult, just advocating doing one step at a time.

Cheers,
Bill