[syslog-ng] Elasticsearch destination

Fri Oct 31 04:55:09 CET 2014

Hi,

If the 3rd option is the slowest, then this seems to be related to the
syslog-ng perl module or Elasticsearch.pm.

I've just checked, the syslog-ng perl module does a value-pairs evaluation
and sends the results to the perl function as a hash. This is not the
speediest thing (it'd be better to export the underlying C structure as an
object to Perl, but should still cope with much more than 2k/sec).

I'm wondering what I could do to help. I'm following this thread, but as I
lack the ES experience I don't have the same environment that you do.

If you could use some kind of profiling (like perf for instance) and had
the associated debug symbols in at least syslog-ng (and preferably also in
perl), we should easily pinpoint the issue. Setting up perf and symbols is
easy if your distro supports it, but is a big hassle if it doesn't.

My experience with perf is on Ubuntu, but I heard it's better in Fedora.
Which distro are you using?

This is the outline what you'd have to do in order to perform profiling:
- don't strip syslog-ng (and neither syslog-ng-incubator) after compilation
and use -g in CFLAGS, syslog-ng doesn't do this in its build script, but
.rpm/.deb packaging usually does
- you can verify this by running file <path-to-binary>

- install symbols for syslog-ng dependencies (these are the dbgsyms
packages in ubuntu,
https://wiki.ubuntu.com/DebuggingProgramCrash#Debug_Symbol_Packages)
- run perf record -g  "syslog-ng command line"
- reproduce the load
- run perf report

You'll see what parts uses the most CPU in the result. Or you can send it
here for analysis.

HTH
Bazsi

On Wed, Oct 29, 2014 at 2:50 PM, Jim Hendrick <jrhendri at roadrunner.com>
wrote:

> Thank you sir!
>
>   At least this is not unique to my testing (not sure that's actually
> *good* news :-)
>
> I will try and reproduce some comparable baselines using a couple setups
> I have tried:
>
> 1) proxy-syslog --> syslog-ng --> redis --> logstash+grok --> logstash
> --> elasticsearch
>     This was essentially following a basic set of instructions just to
> make sure I could reproduce them.
>
> 2) proxy-syslog --> syslog-ng+patterndb+format-json --> redis -->
> logstash --> elasticsearch
>     This moved the pattern matching and conversion to json out to the
> edge, leaving redis & logstash since they worked well at feeding
> elasticsearch.
>
> 3) proxy-syslog --> syslog-ng+patterndb+Elasticsearch.pm --> elasticsearch
>     This seemed the simplest & most promising.
>
> I have not tried all three with the same load, so I cannot definitively
> say one is better, but my subjective feel is that #3 was actually the
> slowest. I suspect something with the way the data is being sent to
> elasticsearch but I do not know whether it is an issue with the perl
> module itself or somehow in the way the data is being sent to
> elasticsearch (indexing, etc.)
>
> My overall thought is (still) that parsing at each syslog-ng server with
> no middleman should be fastest, since as you scale to more syslog-ng
> servers you are distributing the pattern matching load.
>
> I am still not sure if a broker (redis, rabbitmq, etc.) will help as
> long as elasticsearch can accept the data fast enough.
>
> Thanks for the feedback - I will certainly post whatever I come up with
> in the next day or so.
>
> Jim
>
>
>
> On 10/29/2014 09:29 AM, Fabien Wernli wrote:
> > Hi Jim,
> >
> > On Tue, Oct 28, 2014 at 04:36:19PM -0400, Jim Hendrick wrote:
> >> Now the issue is performance. I am sending roughly ~5000 EPS to the
> >> syslog-ng instance running patterndb, but only able to "sustain" less
> than
> >> 1000 to elasticsearch (oddly, ES seems to start receiving at ~5000 EPS,
> and
> >> within an hour or less, drops to ~1000)
> > I've got a similar workload, and seeing drops too.
> > When EPS is below 2k/s, usually syslog-ng copes. When it goes above, I
> can
> > see drops. Enabling flow-control seems to help from the syslog-ng
> > perspective (no drops in `syslog-ng-ctl stats`) but when I look at
> protocol
> > counters in the Linux kernel, the drops can be seen as "InErrors" (I'm
> using
> > UDP). I'm a little lost when trying to interpret the effect of syslog-ng
> > tuneables.
> >
> >> I have tried a number of things, including running a second ES node and
> >> letting syslog-ng "round robin" with no luck at all.
> > We're doing that by specifying the `nodes` key in Elasticsearch.pm:
> > according to its documentation [1] this should ensure
> Search::Elasticsearch
> > makes use of load-balancing. This seems to work as intended, when
> checking
> > the bandwidth between syslog-ng and all ES nodeS.
> >
> > When looking at the statistics of my nodes, they seem to be hitting no
> > bottleneck whatsoever:
> >
> > * load is between 0 and 2 (8 cores total)
> > * writes average around 50/s with peaks around 150  (6+P RAID 10k SAS)
> > * reads are ridiculous
> > * heap usage is around 75% (of 24g)
> > * interface rx ~500k/s
> > * elasticsearch index rate ~500/s
> >
> >> ES tuning has included locking 16G of memory per ES instance, and
> setting
> >> indices.memory.index_buffer_size: 50%
> > We're using 'index_buffer_size: 30%' and 'ES_HEAP_SIZE=24g' on our 6 ES
> > nodes. max_size is 256 in syslog-ng/Elasticsearch.pm
> >
> > What we're currently doing (or planning) to try to investigate:
> >
> > 1. micro-benchmark the CPAN module to see if we can go above 2k/s
> > 2. improve the statistics gathered by collectd-elasticsearch [2]
> > 3. write a dummy ES server which only does some
> >    accounting but throws data away, in order to do some benchmarking.
> > 4. compare python, lua and perl implementations
> > 5. tune various syslog-ng parameters
> > 6. use some MQ implementation between ES and syslog-ng
> > 7. use TCP instead of UDP for incoming syslog
> >
> > I realize this won't help you much, but may be of interest so we can
> channel
> > our common research. I'll be meeting with some syslog-ng experts very
> soon,
> > and I am convinced I'll come back with many options to improve the
> > situation.
> >
> > Cheers
> >
> > [1]
> http://search.cpan.org/~drtech/Search-Elasticsearch-1.14/lib/Search/Elasticsearch.pm#nodes
> > [2] https://github.com/phobos182/collectd-elasticsearch
> >
> >
> ______________________________________________________________________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >
> >
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>

-- 
Bazsi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20141031/6ee0b548/attachment.htm 