[syslog-ng] syslog-ng to elasticsearch ?

[Daniel Maher]

On Thu, 2011-04-28 at 10:19 -0500, Martin Holste wrote:
> Logging with any expensive mechanism like HTTP posts will be
> problematic at logging rates over a hundred or so per second unless
> the messages can be batched.  Even then, HTTP may not be viable.  In
> any case, definitely start with an external program until you're sure
> that the backend you're logging to is doing what you want it do before
> worrying about natively logging from syslog-ng.

Yes, this was troubling me as well.  My draft proposal envisioned using
an AMQP provider to ensure that the queues are retained, though this is
clearly outside of the realm of syslog-ng specifically.

Something like :
syslog-ng -> amqp.rb -> RabbitMQ -> inserter.rb -> ES

Just thinking out loud here : an AMQP driver for syslog-ng could be
highly useful for a variety of potential environments, including (but
not limited to) this sort of end game...

-- 
Daniel Maher
« can't talk, too busy calculating computrons. »