[syslog-ng] couple questions - geoip and also list archives
jrhendri at roadrunner.com
Sat Feb 21 13:24:10 CET 2015
I have done just some preliminary testing (maybe 1500 EPS for a few
minutes) and was seeing a lot of dns traffic (~1MB/s)
Obviously, if the field is a hostname, to do a geoip lookup there needs
to be name resolution before the IP can be mapped to a geo database.
I will be looking for ways to minimize this.
Current use-cases are for parsing proxy, email and fire-eye logs.
Recall, my base architecture is
syslog-ng using patterndb sending format-json to a local redis
redis is run with no local disk storage and acts as an in-memory buffer
between syslog-ng and logstash
logstash (also running locally on the same box) pulling (blpop) and
feeding an elasticsearch cluster (4 nodes right now)
Currently taking live proxy logs at ~7 - 10 K EPS running very well.
Looking to add the email and fireeye logs soon and starting to enhance
the data (with user and host metadata)
Thoughts right now are:
- only resolve location for addresses (not hostnames)
- run a caching nameserver locally on the syslog-ng box and dealing with
the "ramp up" period
(initially clearly the names would not be in cache - just not sure how
long it would take to get to a steady state and how big to make the
I'll keep you posted.
On 02/20/2015 03:24 PM, Fabien Wernli wrote:
> Hi Jim,
> On Fri, Feb 20, 2015 at 01:52:19PM -0500, jrhendri at roadrunner.com wrote:
>> Is anyone using it in reasonably high-performance environments? (like 5000+ events per second)
> we're using the module in a 3keps environment with very good performance. we
> have had some issues in the past in threaded mode with some segfaults. The
> geoip library documentation mentions a few sentences about thread safety.
> I'd be curious to hear some feedback about your future
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
More information about the syslog-ng