syslog-ng to elasticsearch ?
Hello, I am curious to know if anybody has tried (or even better, succeeded) in sending logs from Syslog-NG direct to Elasticsearch ? I see that there is already a driver for MongoDB, which is along the same lines... My first idea was to send all the incoming logs to a named pipe, then have a small script read from the pipe and trigger calls to ES for each line (not unlike the many SQL howtos out there). I suspect that this would work, but if anybody has any ideas which are more elegant, I would love to hear about it. Thanks, and have a great day ! -- Daniel Maher « can't talk, too busy calculating computrons. »
Daniel Maher <dmaher@milestonelab.com> writes:
Hello,
I am curious to know if anybody has tried (or even better, succeeded) in sending logs from Syslog-NG direct to Elasticsearch ? I see that there is already a driver for MongoDB, which is along the same lines...
My first idea was to send all the incoming logs to a named pipe, then have a small script read from the pipe and trigger calls to ES for each line (not unlike the many SQL howtos out there). I suspect that this would work, but if anybody has any ideas which are more elegant, I would love to hear about it.
I only had a quick look at what ElasticSearch is about, and so far, it looks like you pretty much only need to format whatever you want to send to JSON, and send it over HTTP. If that's the case, then if you're sticking to stable versions of syslog-ng, your best bet is to use an external program, that receives data from syslog-ng, formats it appropriately, and forwards it to ES (either via a pipe, or via the program() destination). However, there is a patch available that implements a $(format_json) template function, which makes things a little easier (but you still need an external program to forward the messages, it's just the formatting that is done on syslog-ng side in this case). In a couple of days, whenever I get around to fix a few blocking issues, I'll make a syslog-ng-module-collection package available, which will contain the $(format_json) template function, and you won't need to patch syslog-ng itself (it will require a very recent [git HEAD at the moment :P] version of syslog-ng 3.3 though). I do have plans to add a simple http client to syslog-ng (as that would be useful for a couchdb destination aswell), which could be used together with $(format_json). But that's not currently available, and will be a while until I get there. That would be the most elegant solution, I believe, and it really isn't all that hard to write, either. I'd be happy to assist if someone would like to tackle this route and code up something like this. -- |8]
Logging with any expensive mechanism like HTTP posts will be problematic at logging rates over a hundred or so per second unless the messages can be batched. Even then, HTTP may not be viable. In any case, definitely start with an external program until you're sure that the backend you're logging to is doing what you want it do before worrying about natively logging from syslog-ng. On Thu, Apr 28, 2011 at 10:06 AM, Gergely Nagy <algernon@balabit.hu> wrote:
Daniel Maher <dmaher@milestonelab.com> writes:
Hello,
I am curious to know if anybody has tried (or even better, succeeded) in sending logs from Syslog-NG direct to Elasticsearch ? I see that there is already a driver for MongoDB, which is along the same lines...
My first idea was to send all the incoming logs to a named pipe, then have a small script read from the pipe and trigger calls to ES for each line (not unlike the many SQL howtos out there). I suspect that this would work, but if anybody has any ideas which are more elegant, I would love to hear about it.
I only had a quick look at what ElasticSearch is about, and so far, it looks like you pretty much only need to format whatever you want to send to JSON, and send it over HTTP.
If that's the case, then if you're sticking to stable versions of syslog-ng, your best bet is to use an external program, that receives data from syslog-ng, formats it appropriately, and forwards it to ES (either via a pipe, or via the program() destination).
However, there is a patch available that implements a $(format_json) template function, which makes things a little easier (but you still need an external program to forward the messages, it's just the formatting that is done on syslog-ng side in this case). In a couple of days, whenever I get around to fix a few blocking issues, I'll make a syslog-ng-module-collection package available, which will contain the $(format_json) template function, and you won't need to patch syslog-ng itself (it will require a very recent [git HEAD at the moment :P] version of syslog-ng 3.3 though).
I do have plans to add a simple http client to syslog-ng (as that would be useful for a couchdb destination aswell), which could be used together with $(format_json). But that's not currently available, and will be a while until I get there.
That would be the most elegant solution, I believe, and it really isn't all that hard to write, either. I'd be happy to assist if someone would like to tackle this route and code up something like this.
-- |8] ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Thu, 2011-04-28 at 10:19 -0500, Martin Holste wrote:
Logging with any expensive mechanism like HTTP posts will be problematic at logging rates over a hundred or so per second unless the messages can be batched. Even then, HTTP may not be viable. In any case, definitely start with an external program until you're sure that the backend you're logging to is doing what you want it do before worrying about natively logging from syslog-ng.
Yes, this was troubling me as well. My draft proposal envisioned using an AMQP provider to ensure that the queues are retained, though this is clearly outside of the realm of syslog-ng specifically. Something like : syslog-ng -> amqp.rb -> RabbitMQ -> inserter.rb -> ES Just thinking out loud here : an AMQP driver for syslog-ng could be highly useful for a variety of potential environments, including (but not limited to) this sort of end game... -- Daniel Maher « can't talk, too busy calculating computrons. »
Daniel Maher <dmaher@milestonelab.com> writes:
On Thu, 2011-04-28 at 10:19 -0500, Martin Holste wrote:
Logging with any expensive mechanism like HTTP posts will be problematic at logging rates over a hundred or so per second unless the messages can be batched. Even then, HTTP may not be viable. In any case, definitely start with an external program until you're sure that the backend you're logging to is doing what you want it do before worrying about natively logging from syslog-ng.
Yes, this was troubling me as well. My draft proposal envisioned using an AMQP provider to ensure that the queues are retained, though this is clearly outside of the realm of syslog-ng specifically.
Something like : syslog-ng -> amqp.rb -> RabbitMQ -> inserter.rb -> ES
Just thinking out loud here : an AMQP driver for syslog-ng could be highly useful for a variety of potential environments, including (but not limited to) this sort of end game...
AMQP (and sometime later afterwards, 0MQ) drivers are on my TODO list, and with a bit of luck, I'll be able to present something useful within a month or two, depending on how fast I can proceed with my other obligations. (I have a proof of concepct 0MQ destination lying on my development system, but it's bleeding from a thousand wounds, including a couple of stupid design errors) But, as always, if someone feels up to the challenge, I'll happily assist to get this moving forward faster. -- |8]
participants (3)
-
Daniel Maher
-
Gergely Nagy
-
Martin Holste