[syslog-ng] Elasticsearch destination and time-zone info

[Evan Rempel]

We are running this in mode "node" on a three node cluster running in vmware. It does not handle the load yet :-(
There is a bottle neck from syslog-ng to produce a json stream of more than about 10,000 messages per second. Right now we are kind of surviving
just due to the in memory buffering of syslog-ng.

I don't actually run the elasticsearch cluster, but am getting more involved all of the time.

We are in the process of setting up an elasticsearch cluster with the following

2 nodes used in node mode to ingest the data from syslog-ng. This could scale out when I get my roundRobin transport code in place.

3 nodes with storage, so this is the real elasticsearch cluster

1 node running kibana.

With this setup we will be able to determine where the bottle necks are and then address them as needed.



I am working on a piece of code that will round robin the data that syslog-ng sends it (program destination)
so I can set up something like

filter f_persecond { match("XX") value("$SEC") };

log { filter(f_persecond) destination(d_round_robin) };
...
log { filter(f_persecond) destination(d_round_robin) };


for each value of $SEC.

This will give syslog-ng 60 threads by which to make json objects, which can then be done 10,000 per core on the syslog server. So this would scale to 200,000+ message per second on a 24 core box, and
evenly load the ingestion nodes of the elasticsearch cluster.


I'll let the list know when I get more details.



On 09/29/2015 12:24 PM, Fabien Wernli wrote:
> Hi Evan,
>
> On Tue, Sep 29, 2015 at 09:13:40AM -0700, Evan Rempel wrote:
>> We are now feeding a steady 5,000 messages per second into elasticsearch with spikes into the 30,000 messages per second.
>> All the right indexes and all of the soft macros parsed by the syslog-ng patterndb.
> Good to hear! Do you use transport or node client mode?
> Also, it would be great if you could share some details about your
> Elasticsearch cluster architecture (number of nodes, shards, replicas, etc.)
>
> Thanks!
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>


-- 
Evan Rempel                                      erempel at uvic.ca
Senior Systems Administrator                        250.721.7691
Data Centre Services, University Systems, University of Victoria