Daniel Maher <dmaher@milestonelab.com> writes:
On Thu, 2011-04-28 at 10:19 -0500, Martin Holste wrote:
Logging with any expensive mechanism like HTTP posts will be problematic at logging rates over a hundred or so per second unless the messages can be batched. Even then, HTTP may not be viable. In any case, definitely start with an external program until you're sure that the backend you're logging to is doing what you want it do before worrying about natively logging from syslog-ng.
Yes, this was troubling me as well. My draft proposal envisioned using an AMQP provider to ensure that the queues are retained, though this is clearly outside of the realm of syslog-ng specifically.
Something like : syslog-ng -> amqp.rb -> RabbitMQ -> inserter.rb -> ES
Just thinking out loud here : an AMQP driver for syslog-ng could be highly useful for a variety of potential environments, including (but not limited to) this sort of end game...
AMQP (and sometime later afterwards, 0MQ) drivers are on my TODO list, and with a bit of luck, I'll be able to present something useful within a month or two, depending on how fast I can proceed with my other obligations. (I have a proof of concepct 0MQ destination lying on my development system, but it's bleeding from a thousand wounds, including a couple of stupid design errors) But, as always, if someone feels up to the challenge, I'll happily assist to get this moving forward faster. -- |8]