[syslog-ng] syslog-ng to elasticsearch ?

[Martin Holste]

Logging with any expensive mechanism like HTTP posts will be
problematic at logging rates over a hundred or so per second unless
the messages can be batched.  Even then, HTTP may not be viable.  In
any case, definitely start with an external program until you're sure
that the backend you're logging to is doing what you want it do before
worrying about natively logging from syslog-ng.

On Thu, Apr 28, 2011 at 10:06 AM, Gergely Nagy <algernon at balabit.hu> wrote:
> Daniel Maher <dmaher at milestonelab.com> writes:
>
>> Hello,
>>
>> I am curious to know if anybody has tried (or even better, succeeded) in
>> sending logs from Syslog-NG direct to Elasticsearch ?  I see that there
>> is already a driver for MongoDB, which is along the same lines...
>>
>> My first idea was to send all the incoming logs to a named pipe, then
>> have a small script read from the pipe and trigger calls to ES for each
>> line (not unlike the many SQL howtos out there).  I suspect that this
>> would work, but if anybody has any ideas which are more elegant, I would
>> love to hear about it.
>
> I only had a quick look at what ElasticSearch is about, and so far, it
> looks like you pretty much only need to format whatever you want to send
> to JSON, and send it over HTTP.
>
> If that's the case, then if you're sticking to stable versions of
> syslog-ng, your best bet is to use an external program, that receives
> data from syslog-ng, formats it appropriately, and forwards it to ES
> (either via a pipe, or via the program() destination).
>
> However, there is a patch available that implements a $(format_json)
> template function, which makes things a little easier (but you still
> need an external program to forward the messages, it's just the
> formatting that is done on syslog-ng side in this case). In a couple of
> days, whenever I get around to fix a few blocking issues, I'll make a
> syslog-ng-module-collection package available, which will contain the
> $(format_json) template function, and you won't need to patch syslog-ng
> itself (it will require a very recent [git HEAD at the moment :P]
> version of syslog-ng 3.3 though).
>
> I do have plans to add a simple http client to syslog-ng (as that would
> be useful for a couchdb destination aswell), which could be used
> together with $(format_json). But that's not currently available, and
> will be a while until I get there.
>
> That would be the most elegant solution, I believe, and it really isn't
> all that hard to write, either. I'd be happy to assist if someone would
> like to tackle this route and code up something like this.
>
> --
> |8]
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>