OK - latest update and a request for recommendations (help actually :-) I have syslog-ng receiving logs from the network, parsing them with patterndb using fairly complex parsing consisting of 5 patterns parsing Bluecoat proxy logs into their respective fields. Here is one example: <pattern>@STRING:PROXY.TIME::@ @NUMBER:PROXY.TIME_TAKEN@ @IPv4:PROXY.C_IP@ @NUMBER:PROXY.SC_STATUS@ @STRING:PROXY.S_ACTION:_@ @NUMBER:PROXY.SC_BYTES@ @NUMBER:PROXY.CS_BYTES@ @STRING:PROXY.CS_METHOD@ @STRING:PROXY.CS_URI_SCHEME:-@ @STRING:PROXY.CS_HOST:_-.@ @NUMBER:PROXY.CS_URI_PORT:-@ @ESTRING:PROXY.CS_URI_PATH: @@ESTRING:PROXY.CS_URI_EQUERY: @@STRING:PROXY.CS_USERNAME:-$@ @STRING:PROXY.CS_AUTH__GROUP:-_@ @STRING:PROXY.S_SUPPLIER_NAME:_-.@ @ESTRING:PROXY.CONTENT_TYPE: @@ESTRING:PROXY.REFERRER: @@QSTRING:PROXY.USER_AGENT:"@ @ESTRING:PROXY.FILTER_RESULT: @@QSTRING:PROXY.CS_CATEGORIES:"@ @STRING:PROXY.X_VIRUS_ID:-@ @IPv4:PROXY.S_IP@</pattern> This is using the Perl Search::Elasticsearch module running on syslog-ng-3.5.6 with the incubator adding mod-perl support. It is being sent to elasticsearch, and I can build basic Kibana dashboards to start analyzing the logs. So far so good. Now the issue is performance. I am sending roughly ~5000 EPS to the syslog-ng instance running patterndb, but only able to "sustain" less than 1000 to elasticsearch (oddly, ES seems to start receiving at ~5000 EPS, and within an hour or less, drops to ~1000) I have tried a number of things, including running a second ES node and letting syslog-ng "round robin" with no luck at all. ES tuning has included locking 16G of memory per ES instance, and setting indices.memory.index_buffer_size: 50% syslog-ng tuning was limited to setting "threaded(yes)" but since I only have a single source and destination, I didn't expect much from this. I *did* notice that when I increased max_count from 256 to 1024 that syslog-ng memory usage dropped dramatically (it had been around 8GB and now has been holding around 100m !) but the overall performance has not improved (much). I feel like I must not be looking in the right area, since syslog-ng stats show a huge drop rate ( 60 - 80 % !!) and also, the "network" source shows absolutely nothing (zero). SourceName;SourceId;SourceInstance;State;Type;Number source;s_network;;a;processed;0 center;;received;a;processed;5 destination;d_elasticsearch;;a;processed;633968 src.internal;s_local#2;;a;processed;5 src.internal;s_local#2;;a;stamp;1414527594 center;;queued;a;processed;633988 dst.none;d_elasticsearch#0;perl,/usr/local/share/include/scl/es-perl/Elasticsearch.pm,SyslogNG::Elasticsearch::init,SyslogNG::Elasticsearch::queue_daily,SyslogNG::Elasticsearch::deinit;a;dropped;511475 dst.none;d_elasticsearch#0;perl,/usr/local/share/include/scl/es-perl/Elasticsearch.pm,SyslogNG::Elasticsearch::init,SyslogNG::Elasticsearch::queue_daily,SyslogNG::Elasticsearch::deinit;a;stored;10000 src.none;;;a;processed;0 src.none;;;a;stamp;0 global;payload_reallocs;;a;processed;870904 global;sdata_updates;;a;processed;0 destination;d_local;;a;processed;20 global;msg_clones;;a;processed;0 source;s_local;;a;processed;5 Any help on where to look next would be greatly appreciated!! Thanks all. Jim