<div dir="ltr"><div><div><div><div>Hi,<br><br></div>I would think that adding forward DNS lookups to the syslog-ng dns cache code (or ripping out that code entirely and rewrite it from scratch while adding this feature) would produce _much_ better results than a locally running DNS server. That's why the DNS cache code was added in the first place, a caching only name server is still too slow for name lookups for every message posted.<br><br></div>The geoip code uses libgeoip1.<br><br></div>The database is:<br><br>$ apt-cache show geoip-database<br>Package: geoip-database<br>Priority: standard<br>Section: net<br>Installed-Size: 3881<br><br>Version: 20140313-1<br>Recommends: libgeoip1<br>Breaks: libgeoip1 (<< 1.4.5.dfsg)<br>Filename: pool/main/g/geoip-database/geoip-database_20140313-1_all.deb<br>Size: 1195894<br>MD5sum: ab4d4f6bc0e04b25cad2fbe1479f44bc<br>SHA1: 06d38aee4084124f86351dfa6f1c404a8ae3e83b<br>SHA256: 30dc5a2c3296180ed0740fb4ec70eb1ea5b49efc5e48a091913a8106f6895c7e<br>Description-en: IP lookup command line tools that use the GeoIP library (country database)<br> GeoIP is a C library that enables the user to find the country that any<br> IP address or hostname originates from. It uses a file based database.<br> .<br> This database simply contains IP blocks as keys, and countries as values and<br> it should be more complete and accurate than using reverse DNS lookups.<br> .<br> This package contains the free GeoLiteCountry database.<br>Description-md5: 3bfa5b4c9f973261799fb4d9355f3b6c<br>Homepage: <a href="http://www.maxmind.com/">http://www.maxmind.com/</a><br>Bugs: <a href="https://bugs.launchpad.net/ubuntu/+filebug">https://bugs.launchpad.net/ubuntu/+filebug</a><br>Origin: Ubuntu<br>Supported: 5y<br>Task: standard, kubuntu-active, kubuntu-active, mythbuntu-frontend, mythbuntu-frontend, mythbuntu-desktop, mythbuntu-backend-slave, mythbuntu-backend-slave, mythbuntu-backend-master, mythbuntu-backend-master<br><br><br></div>So it is about a year old, but quite probably the version in Debian sid can be installed on top without problems, and that's pretty fresh, being dated 9th February.<br><br><a href="https://packages.debian.org/sid/geoip-database">https://packages.debian.org/sid/geoip-database</a><br><br><div><div><div><br></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Feb 21, 2015 at 1:24 PM, Jim Hendrick <span dir="ltr"><<a href="mailto:jrhendri@roadrunner.com" target="_blank">jrhendri@roadrunner.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Fabian,<br>
I have done just some preliminary testing (maybe 1500 EPS for a few<br>
minutes) and was seeing a lot of dns traffic (~1MB/s)<br>
<br>
Obviously, if the field is a hostname, to do a geoip lookup there needs<br>
to be name resolution before the IP can be mapped to a geo database.<br>
<br>
I will be looking for ways to minimize this.<br>
<br>
Current use-cases are for parsing proxy, email and fire-eye logs.<br>
<br>
Recall, my base architecture is<br>
syslog-ng using patterndb sending format-json to a local redis<br>
destination (lpush)<br>
redis is run with no local disk storage and acts as an in-memory buffer<br>
between syslog-ng and logstash<br>
logstash (also running locally on the same box) pulling (blpop) and<br>
feeding an elasticsearch cluster (4 nodes right now)<br>
<br>
Currently taking live proxy logs at ~7 - 10 K EPS running very well.<br>
Looking to add the email and fireeye logs soon and starting to enhance<br>
the data (with user and host metadata)<br>
<br>
<br>
Thoughts right now are:<br>
- only resolve location for addresses (not hostnames)<br>
- run a caching nameserver locally on the syslog-ng box and dealing with<br>
the "ramp up" period<br>
(initially clearly the names would not be in cache - just not sure how<br>
long it would take to get to a steady state and how big to make the<br>
cache, etc.)<br>
<br>
I'll keep you posted.<br>
<br>
Thanks again!<br>
<span class="HOEnZb"><font color="#888888">Jim<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
On 02/20/2015 03:24 PM, Fabien Wernli wrote:<br>
> Hi Jim,<br>
><br>
> On Fri, Feb 20, 2015 at 01:52:19PM -0500, <a href="mailto:jrhendri@roadrunner.com">jrhendri@roadrunner.com</a> wrote:<br>
>> Is anyone using it in reasonably high-performance environments? (like 5000+ events per second)<br>
>><br>
> we're using the module in a 3keps environment with very good performance. we<br>
> have had some issues in the past in threaded mode with some segfaults. The<br>
> geoip library documentation mentions a few sentences about thread safety.<br>
> I'd be curious to hear some feedback about your future<br>
> experience.<br>
><br>
> cheers<br>
> ______________________________________________________________________________<br>
> Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
> Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
> FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
><br>
><br>
<br>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature">Bazsi</div>
</div>