<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
OK - latest update and a request for recommendations (help actually
:-)<br>
<br>
I have syslog-ng receiving logs from the network, parsing them with
patterndb using fairly complex parsing consisting of 5 patterns
parsing Bluecoat proxy logs into their respective fields. Here is
one example:<br>
<br>
<pre wrap=""><font face="Courier New, Courier, monospace"> <pattern>@STRING:PROXY.TIME::@ @NUMBER:PROXY.TIME_TAKEN@ @IPv4:PROXY.C_IP@ @NUMBER:PROXY.SC_STATUS@ @STRING:PROXY.S_ACTION:_@ @NUMBER:PROXY.SC_BYTES@ @NUMBER:PROXY.CS_BYTES@ @STRING:PROXY.CS_METHOD@ @STRING:PROXY.CS_URI_SCHEME:-@ @STRING:PROXY.CS_HOST:_-.@ @NUMBER:PROXY.CS_URI_PORT:-@ @ESTRING:PROXY.CS_URI_PATH: @@ESTRING:PROXY.CS_URI_EQUERY: @@STRING:PROXY.CS_USERNAME:-$@ @STRING:PROXY.CS_AUTH__GROUP:-_@ @STRING:PROXY.S_SUPPLIER_NAME:_-.@ @ESTRING:PROXY.CONTENT_TYPE: @@ESTRING:PROXY.REFERRER: @@QSTRING:PROXY.USER_AGENT:"@ @ESTRING:PROXY.FILTER_RESULT: @@QSTRING:PROXY.CS_CATEGORIES:"@ @STRING:PROXY.X_VIRUS_ID:-@ @IPv4:PROXY.S_IP@</pattern></font>
</pre>
<br>
This is using the Perl Search::Elasticsearch module running on
syslog-ng-3.5.6 with the incubator adding mod-perl support.<br>
<br>
It is being sent to elasticsearch, and I can build basic Kibana
dashboards to start analyzing the logs.<br>
<br>
So far so good.<br>
<br>
Now the issue is performance. I am sending roughly ~5000 EPS to the
syslog-ng instance running patterndb, but only able to "sustain"
less than 1000 to elasticsearch (oddly, ES seems to start receiving
at ~5000 EPS, and within an hour or less, drops to ~1000)<br>
<br>
I have tried a number of things, including running a second ES node
and letting syslog-ng "round robin" with no luck at all.<br>
<br>
ES tuning has included locking 16G of memory per ES instance, and
setting indices.memory.index_buffer_size: 50%<br>
<br>
syslog-ng tuning was limited to setting "threaded(yes)" but since I
only have a single source and destination, I didn't expect much from
this.<br>
<br>
I *did* notice that when I increased max_count from 256 to 1024 that
syslog-ng memory usage dropped dramatically (it had been around 8GB
and now has been holding around 100m !) but the overall performance
has not improved (much).<br>
<br>
I feel like I must not be looking in the right area, since syslog-ng
stats show a huge drop rate (<br>
60 - 80 % !!) and also, the "network" source shows absolutely
nothing (zero).<br>
<br>
<pre wrap="">
<font face="Courier New, Courier, monospace">SourceName;SourceId;SourceInstance;State;Type;Number
source;s_network;;a;processed;0
center;;received;a;processed;5
destination;d_elasticsearch;;a;processed;633968
src.internal;s_local#2;;a;processed;5
src.internal;s_local#2;;a;stamp;1414527594
center;;queued;a;processed;633988
dst.none;d_elasticsearch#0;perl,/usr/local/share/include/scl/es-perl/Elasticsearch.pm,SyslogNG::Elasticsearch::init,SyslogNG::Elasticsearch::queue_daily,SyslogNG::Elasticsearch::deinit;a;dropped;511475
dst.none;d_elasticsearch#0;perl,/usr/local/share/include/scl/es-perl/Elasticsearch.pm,SyslogNG::Elasticsearch::init,SyslogNG::Elasticsearch::queue_daily,SyslogNG::Elasticsearch::deinit;a;stored;10000
src.none;;;a;processed;0
src.none;;;a;stamp;0
global;payload_reallocs;;a;processed;870904
global;sdata_updates;;a;processed;0
destination;d_local;;a;processed;20
global;msg_clones;;a;processed;0
source;s_local;;a;processed;5
</font>
</pre>
<pre wrap=""><font face="sans-serif">Any help on where to look next would be greatly appreciated!!</font>
</pre>
<br>
Thanks all.<br>
Jim<br>
<br>
<br>
<br>
</body>
</html>