Daniel Maher <dmaher@milestonelab.com> writes:
Hello,
I am curious to know if anybody has tried (or even better, succeeded) in sending logs from Syslog-NG direct to Elasticsearch ? I see that there is already a driver for MongoDB, which is along the same lines...
My first idea was to send all the incoming logs to a named pipe, then have a small script read from the pipe and trigger calls to ES for each line (not unlike the many SQL howtos out there). I suspect that this would work, but if anybody has any ideas which are more elegant, I would love to hear about it.
I only had a quick look at what ElasticSearch is about, and so far, it looks like you pretty much only need to format whatever you want to send to JSON, and send it over HTTP. If that's the case, then if you're sticking to stable versions of syslog-ng, your best bet is to use an external program, that receives data from syslog-ng, formats it appropriately, and forwards it to ES (either via a pipe, or via the program() destination). However, there is a patch available that implements a $(format_json) template function, which makes things a little easier (but you still need an external program to forward the messages, it's just the formatting that is done on syslog-ng side in this case). In a couple of days, whenever I get around to fix a few blocking issues, I'll make a syslog-ng-module-collection package available, which will contain the $(format_json) template function, and you won't need to patch syslog-ng itself (it will require a very recent [git HEAD at the moment :P] version of syslog-ng 3.3 though). I do have plans to add a simple http client to syslog-ng (as that would be useful for a couchdb destination aswell), which could be used together with $(format_json). But that's not currently available, and will be a while until I get there. That would be the most elegant solution, I believe, and it really isn't all that hard to write, either. I'd be happy to assist if someone would like to tackle this route and code up something like this. -- |8]