[syslog-ng] couple questions - geoip and also list archives

Balazs Scheidler bazsi77 at gmail.com
Sun Feb 22 20:35:35 CET 2015


I would think that adding forward DNS lookups to the syslog-ng dns cache
code (or ripping out that code entirely and rewrite it from scratch while
adding this feature) would produce _much_ better results than a locally
running DNS server. That's why the DNS cache code was added in the first
place, a caching only name server is still too slow for name lookups for
every message posted.

The  geoip code uses libgeoip1.

The database is:

$ apt-cache show geoip-database
Package: geoip-database
Priority: standard
Section: net
Installed-Size: 3881

Version: 20140313-1
Recommends: libgeoip1
Breaks: libgeoip1 (<< 1.4.5.dfsg)
Filename: pool/main/g/geoip-database/geoip-database_20140313-1_all.deb
Size: 1195894
MD5sum: ab4d4f6bc0e04b25cad2fbe1479f44bc
SHA1: 06d38aee4084124f86351dfa6f1c404a8ae3e83b
SHA256: 30dc5a2c3296180ed0740fb4ec70eb1ea5b49efc5e48a091913a8106f6895c7e
Description-en: IP lookup command line tools that use the GeoIP library
(country database)
 GeoIP is a C library that enables the user to find the country that any
 IP address or hostname originates from. It uses a file based database.
 This database simply contains IP blocks as keys, and countries as values
 it should be more complete and accurate than using reverse DNS lookups.
 This package contains the free GeoLiteCountry database.
Description-md5: 3bfa5b4c9f973261799fb4d9355f3b6c
Homepage: http://www.maxmind.com/
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Origin: Ubuntu
Supported: 5y
Task: standard, kubuntu-active, kubuntu-active, mythbuntu-frontend,
mythbuntu-frontend, mythbuntu-desktop, mythbuntu-backend-slave,
mythbuntu-backend-slave, mythbuntu-backend-master, mythbuntu-backend-master

So it is about a year old, but quite probably the version in Debian sid can
be installed on top without problems, and that's pretty fresh, being dated
9th February.


On Sat, Feb 21, 2015 at 1:24 PM, Jim Hendrick <jrhendri at roadrunner.com>

> Hi Fabian,
>   I have done just some preliminary testing (maybe 1500 EPS for a few
> minutes) and was seeing a lot of dns traffic (~1MB/s)
> Obviously, if the field is a hostname, to do a geoip lookup there needs
> to be name resolution before the IP can be mapped to a geo database.
> I will be looking for ways to minimize this.
> Current use-cases are for parsing proxy, email and fire-eye logs.
> Recall, my base architecture is
> syslog-ng using patterndb sending format-json to a local redis
> destination (lpush)
> redis is run with no local disk storage and acts as an in-memory buffer
> between syslog-ng and logstash
> logstash (also running locally on the same box) pulling (blpop) and
> feeding an elasticsearch cluster (4 nodes right now)
> Currently taking live proxy logs at ~7 - 10 K EPS running very well.
> Looking to add the email and fireeye logs soon and starting to enhance
> the data (with user and host metadata)
> Thoughts right now are:
> - only resolve location for addresses (not hostnames)
> - run a caching nameserver locally on the syslog-ng box and dealing with
> the "ramp up" period
>   (initially clearly the names would not be in cache - just not sure how
> long it would take to get to a steady state and how big to make the
> cache, etc.)
> I'll keep you posted.
> Thanks again!
> Jim
> On 02/20/2015 03:24 PM, Fabien Wernli wrote:
> > Hi Jim,
> >
> > On Fri, Feb 20, 2015 at 01:52:19PM -0500, jrhendri at roadrunner.com wrote:
> >>   Is anyone using it in reasonably high-performance environments? (like
> 5000+ events per second)
> >>
> > we're using the module in a 3keps environment with very good
> performance. we
> > have had some issues in the past in threaded mode with some segfaults.
> The
> > geoip library documentation mentions a few sentences about thread safety.
> > I'd be curious to hear some feedback about your future
> >  experience.
> >
> > cheers
> >
> ______________________________________________________________________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >
> >
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20150222/0f5d4589/attachment.htm 

More information about the syslog-ng mailing list