[syslog-ng] syslog-ng to elasticsearch ?

[Gergely Nagy]

Daniel Maher <dmaher at milestonelab.com> writes:

> On Thu, 2011-04-28 at 10:19 -0500, Martin Holste wrote:
>> Logging with any expensive mechanism like HTTP posts will be
>> problematic at logging rates over a hundred or so per second unless
>> the messages can be batched.  Even then, HTTP may not be viable.  In
>> any case, definitely start with an external program until you're sure
>> that the backend you're logging to is doing what you want it do before
>> worrying about natively logging from syslog-ng.
>
> Yes, this was troubling me as well.  My draft proposal envisioned using
> an AMQP provider to ensure that the queues are retained, though this is
> clearly outside of the realm of syslog-ng specifically.
>
> Something like :
> syslog-ng -> amqp.rb -> RabbitMQ -> inserter.rb -> ES
>
> Just thinking out loud here : an AMQP driver for syslog-ng could be
> highly useful for a variety of potential environments, including (but
> not limited to) this sort of end game...

AMQP (and sometime later afterwards, 0MQ) drivers are on my TODO list,
and with a bit of luck, I'll be able to present something useful within
a month or two, depending on how fast I can proceed with my other
obligations.

(I have a proof of concepct 0MQ destination lying on my development
system, but it's bleeding from a thousand wounds, including a couple of
stupid design errors)

But, as always, if someone feels up to the challenge, I'll happily
assist to get this moving forward faster.

-- 
|8]