[syslog-ng] Elasticsearch destination
jrhendri at roadrunner.com
Tue Oct 28 21:36:19 CET 2014
OK - latest update and a request for recommendations (help actually :-)
I have syslog-ng receiving logs from the network, parsing them with
patterndb using fairly complex parsing consisting of 5 patterns parsing
Bluecoat proxy logs into their respective fields. Here is one example:
<pattern>@STRING:PROXY.TIME::@ @NUMBER:PROXY.TIME_TAKEN@ @IPv4:PROXY.C_IP@ @NUMBER:PROXY.SC_STATUS@ @STRING:PROXY.S_ACTION:_@ @NUMBER:PROXY.SC_BYTES@ @NUMBER:PROXY.CS_BYTES@ @STRING:PROXY.CS_METHOD@ @STRING:PROXY.CS_URI_SCHEME:-@ @STRING:PROXY.CS_HOST:_-.@ @NUMBER:PROXY.CS_URI_PORT:-@ @ESTRING:PROXY.CS_URI_PATH: @@ESTRING:PROXY.CS_URI_EQUERY: @@STRING:PROXY.CS_USERNAME:-$@ @STRING:PROXY.CS_AUTH__GROUP:-_@ @STRING:PROXY.S_SUPPLIER_NAME:_-.@ @ESTRING:PROXY.CONTENT_TYPE: @@ESTRING:PROXY.REFERRER: @@QSTRING:PROXY.USER_AGENT:"@ @ESTRING:PROXY.FILTER_RESULT: @@QSTRING:PROXY.CS_CATEGORIES:"@ @STRING:PROXY.X_VIRUS_ID:-@ @IPv4:PROXY.S_IP@</pattern>
This is using the Perl Search::Elasticsearch module running on
syslog-ng-3.5.6 with the incubator adding mod-perl support.
It is being sent to elasticsearch, and I can build basic Kibana
dashboards to start analyzing the logs.
So far so good.
Now the issue is performance. I am sending roughly ~5000 EPS to the
syslog-ng instance running patterndb, but only able to "sustain" less
than 1000 to elasticsearch (oddly, ES seems to start receiving at ~5000
EPS, and within an hour or less, drops to ~1000)
I have tried a number of things, including running a second ES node and
letting syslog-ng "round robin" with no luck at all.
ES tuning has included locking 16G of memory per ES instance, and
setting indices.memory.index_buffer_size: 50%
syslog-ng tuning was limited to setting "threaded(yes)" but since I only
have a single source and destination, I didn't expect much from this.
I *did* notice that when I increased max_count from 256 to 1024 that
syslog-ng memory usage dropped dramatically (it had been around 8GB and
now has been holding around 100m !) but the overall performance has not
I feel like I must not be looking in the right area, since syslog-ng
stats show a huge drop rate (
60 - 80 % !!) and also, the "network" source shows absolutely nothing
Any help on where to look next would be greatly appreciated!!
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the syslog-ng