[syslog-ng] syslog-ng to elasticsearch ?

[Gergely Nagy]

Daniel Maher <dmaher at milestonelab.com> writes:

> Hello,
>
> I am curious to know if anybody has tried (or even better, succeeded) in
> sending logs from Syslog-NG direct to Elasticsearch ?  I see that there
> is already a driver for MongoDB, which is along the same lines...
> 
> My first idea was to send all the incoming logs to a named pipe, then
> have a small script read from the pipe and trigger calls to ES for each
> line (not unlike the many SQL howtos out there).  I suspect that this
> would work, but if anybody has any ideas which are more elegant, I would
> love to hear about it.

I only had a quick look at what ElasticSearch is about, and so far, it
looks like you pretty much only need to format whatever you want to send
to JSON, and send it over HTTP.

If that's the case, then if you're sticking to stable versions of
syslog-ng, your best bet is to use an external program, that receives
data from syslog-ng, formats it appropriately, and forwards it to ES
(either via a pipe, or via the program() destination).

However, there is a patch available that implements a $(format_json)
template function, which makes things a little easier (but you still
need an external program to forward the messages, it's just the
formatting that is done on syslog-ng side in this case). In a couple of
days, whenever I get around to fix a few blocking issues, I'll make a
syslog-ng-module-collection package available, which will contain the
$(format_json) template function, and you won't need to patch syslog-ng
itself (it will require a very recent [git HEAD at the moment :P]
version of syslog-ng 3.3 though).

I do have plans to add a simple http client to syslog-ng (as that would
be useful for a couchdb destination aswell), which could be used
together with $(format_json). But that's not currently available, and
will be a while until I get there.

That would be the most elegant solution, I believe, and it really isn't
all that hard to write, either. I'd be happy to assist if someone would
like to tackle this route and code up something like this.

-- 
|8]