[syslog-ng] couple questions - geoip and also list archives
bazsi77 at gmail.com
Sun Feb 22 20:35:35 CET 2015
I would think that adding forward DNS lookups to the syslog-ng dns cache
code (or ripping out that code entirely and rewrite it from scratch while
adding this feature) would produce _much_ better results than a locally
running DNS server. That's why the DNS cache code was added in the first
place, a caching only name server is still too slow for name lookups for
every message posted.
The geoip code uses libgeoip1.
The database is:
$ apt-cache show geoip-database
Breaks: libgeoip1 (<< 1.4.5.dfsg)
Description-en: IP lookup command line tools that use the GeoIP library
GeoIP is a C library that enables the user to find the country that any
IP address or hostname originates from. It uses a file based database.
This database simply contains IP blocks as keys, and countries as values
it should be more complete and accurate than using reverse DNS lookups.
This package contains the free GeoLiteCountry database.
Task: standard, kubuntu-active, kubuntu-active, mythbuntu-frontend,
mythbuntu-frontend, mythbuntu-desktop, mythbuntu-backend-slave,
mythbuntu-backend-slave, mythbuntu-backend-master, mythbuntu-backend-master
So it is about a year old, but quite probably the version in Debian sid can
be installed on top without problems, and that's pretty fresh, being dated
On Sat, Feb 21, 2015 at 1:24 PM, Jim Hendrick <jrhendri at roadrunner.com>
> Hi Fabian,
> I have done just some preliminary testing (maybe 1500 EPS for a few
> minutes) and was seeing a lot of dns traffic (~1MB/s)
> Obviously, if the field is a hostname, to do a geoip lookup there needs
> to be name resolution before the IP can be mapped to a geo database.
> I will be looking for ways to minimize this.
> Current use-cases are for parsing proxy, email and fire-eye logs.
> Recall, my base architecture is
> syslog-ng using patterndb sending format-json to a local redis
> destination (lpush)
> redis is run with no local disk storage and acts as an in-memory buffer
> between syslog-ng and logstash
> logstash (also running locally on the same box) pulling (blpop) and
> feeding an elasticsearch cluster (4 nodes right now)
> Currently taking live proxy logs at ~7 - 10 K EPS running very well.
> Looking to add the email and fireeye logs soon and starting to enhance
> the data (with user and host metadata)
> Thoughts right now are:
> - only resolve location for addresses (not hostnames)
> - run a caching nameserver locally on the syslog-ng box and dealing with
> the "ramp up" period
> (initially clearly the names would not be in cache - just not sure how
> long it would take to get to a steady state and how big to make the
> cache, etc.)
> I'll keep you posted.
> Thanks again!
> On 02/20/2015 03:24 PM, Fabien Wernli wrote:
> > Hi Jim,
> > On Fri, Feb 20, 2015 at 01:52:19PM -0500, jrhendri at roadrunner.com wrote:
> >> Is anyone using it in reasonably high-performance environments? (like
> 5000+ events per second)
> > we're using the module in a 3keps environment with very good
> performance. we
> > have had some issues in the past in threaded mode with some segfaults.
> > geoip library documentation mentions a few sentences about thread safety.
> > I'd be curious to hear some feedback about your future
> > experience.
> > cheers
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation:
> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the syslog-ng