OK - latest update and a request for recommendations (help actually :-)

I have syslog-ng receiving logs from the network, parsing them with patterndb using fairly complex parsing consisting of 5 patterns parsing Bluecoat proxy logs into their respective fields. Here is one example:

          <pattern>@STRING:PROXY.TIME::@ @NUMBER:PROXY.TIME_TAKEN@ @IPv4:PROXY.C_IP@ @NUMBER:PROXY.SC_STATUS@ @STRING:PROXY.S_ACTION:_@ @NUMBER:PROXY.SC_BYTES@ @NUMBER:PROXY.CS_BYTES@ @STRING:PROXY.CS_METHOD@ @STRING:PROXY.CS_URI_SCHEME:-@ @STRING:PROXY.CS_HOST:_-.@ @NUMBER:PROXY.CS_URI_PORT:-@ @ESTRING:PROXY.CS_URI_PATH: @@ESTRING:PROXY.CS_URI_EQUERY: @@STRING:PROXY.CS_USERNAME:-$@ @STRING:PROXY.CS_AUTH__GROUP:-_@ @STRING:PROXY.S_SUPPLIER_NAME:_-.@ @ESTRING:PROXY.CONTENT_TYPE: @@ESTRING:PROXY.REFERRER: @@QSTRING:PROXY.USER_AGENT:"@ @ESTRING:PROXY.FILTER_RESULT: @@QSTRING:PROXY.CS_CATEGORIES:"@ @STRING:PROXY.X_VIRUS_ID:-@ @IPv4:PROXY.S_IP@</pattern>

This is using the Perl Search::Elasticsearch module running on syslog-ng-3.5.6 with the incubator adding mod-perl support.

It is being sent to elasticsearch, and I can build basic Kibana dashboards to start analyzing the logs.

So far so good.

Now the issue is performance. I am sending roughly ~5000 EPS to the syslog-ng instance running patterndb, but only able to "sustain" less than 1000 to elasticsearch (oddly, ES seems to start receiving at ~5000 EPS, and within an hour or less, drops to ~1000)

I have tried a number of things, including running a second ES node and letting syslog-ng "round robin" with no luck at all.

ES tuning has included locking 16G of memory per ES instance, and setting indices.memory.index_buffer_size: 50%

syslog-ng tuning was limited to setting "threaded(yes)" but since I only have a single source and destination, I didn't expect much from this.

I *did* notice that when I increased max_count from 256 to 1024 that syslog-ng memory usage dropped dramatically (it had been around 8GB and now has been holding around 100m !) but the overall performance has not improved (much).

I feel like I must not be looking in the right area, since syslog-ng stats show a huge drop rate (
60 - 80 % !!) and also, the "network" source shows absolutely nothing (zero).

SourceName;SourceId;SourceInstance;State;Type;Number
source;s_network;;a;processed;0
center;;received;a;processed;5
destination;d_elasticsearch;;a;processed;633968
src.internal;s_local#2;;a;processed;5
src.internal;s_local#2;;a;stamp;1414527594
center;;queued;a;processed;633988
dst.none;d_elasticsearch#0;perl,/usr/local/share/include/scl/es-perl/Elasticsearch.pm,SyslogNG::Elasticsearch::init,SyslogNG::Elasticsearch::queue_daily,SyslogNG::Elasticsearch::deinit;a;dropped;511475
dst.none;d_elasticsearch#0;perl,/usr/local/share/include/scl/es-perl/Elasticsearch.pm,SyslogNG::Elasticsearch::init,SyslogNG::Elasticsearch::queue_daily,SyslogNG::Elasticsearch::deinit;a;stored;10000
src.none;;;a;processed;0
src.none;;;a;stamp;0
global;payload_reallocs;;a;processed;870904
global;sdata_updates;;a;processed;0
destination;d_local;;a;processed;20
global;msg_clones;;a;processed;0
source;s_local;;a;processed;5

Any help on where to look next would be greatly appreciated!!

Thanks all.
Jim