We are running this in mode "node" on a three node cluster running in vmware. It does not handle the load yet :-( There is a bottle neck from syslog-ng to produce a json stream of more than about 10,000 messages per second. Right now we are kind of surviving just due to the in memory buffering of syslog-ng. I don't actually run the elasticsearch cluster, but am getting more involved all of the time. We are in the process of setting up an elasticsearch cluster with the following 2 nodes used in node mode to ingest the data from syslog-ng. This could scale out when I get my roundRobin transport code in place. 3 nodes with storage, so this is the real elasticsearch cluster 1 node running kibana. With this setup we will be able to determine where the bottle necks are and then address them as needed. I am working on a piece of code that will round robin the data that syslog-ng sends it (program destination) so I can set up something like filter f_persecond { match("XX") value("$SEC") }; log { filter(f_persecond) destination(d_round_robin) }; ... log { filter(f_persecond) destination(d_round_robin) }; for each value of $SEC. This will give syslog-ng 60 threads by which to make json objects, which can then be done 10,000 per core on the syslog server. So this would scale to 200,000+ message per second on a 24 core box, and evenly load the ingestion nodes of the elasticsearch cluster. I'll let the list know when I get more details. On 09/29/2015 12:24 PM, Fabien Wernli wrote:
Hi Evan,
On Tue, Sep 29, 2015 at 09:13:40AM -0700, Evan Rempel wrote:
We are now feeding a steady 5,000 messages per second into elasticsearch with spikes into the 30,000 messages per second. All the right indexes and all of the soft macros parsed by the syslog-ng patterndb. Good to hear! Do you use transport or node client mode? Also, it would be great if you could share some details about your Elasticsearch cluster architecture (number of nodes, shards, replicas, etc.)
Thanks!
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Evan Rempel erempel@uvic.ca Senior Systems Administrator 250.721.7691 Data Centre Services, University Systems, University of Victoria