<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Thanks Balazs,<br>
I will try some more "controlled" testing using different settings
for syslog-ng resolving and caching.<br>
<br>
I think I installed libgeopi-dev (not sure right now - it's
installed on a system at work) so I'll check that package also.<br>
<br>
One question on code paths:<br>
<br>
If I use an IP address pattern in patterndb (within the message -
e.g. proxy or email logs) where a ${GEO} macro was assigned, will
those be the only things that get resolved? (or by setting a cache
within the syslog-ng config will that enable resolution for ${HOST}
as well?<br>
<br>
I am (mostly) interested in things like user access to sites by IP
address through the proxy, and wanting to enhance the logs with
geoip data for elasticsearch.<br>
<br>
(obviously if it were fast enough, I would add the data for all
sites - but initially I think IP only would be more interesting)<br>
<br>
Thanks again!<br>
Jim<br>
<br>
<br>
<br>
<div class="moz-cite-prefix">On 02/22/2015 02:35 PM, Balazs
Scheidler wrote:<br>
</div>
<blockquote
cite="mid:CAKcfE+YgnZxRa7weyD0NCSZcNB_V6Bt1d-0UNmEzCP4Zryghww@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<div>
<div>
<div>Hi,<br>
<br>
</div>
I would think that adding forward DNS lookups to the
syslog-ng dns cache code (or ripping out that code
entirely and rewrite it from scratch while adding this
feature) would produce _much_ better results than a
locally running DNS server. That's why the DNS cache code
was added in the first place, a caching only name server
is still too slow for name lookups for every message
posted.<br>
<br>
</div>
The geoip code uses libgeoip1.<br>
<br>
</div>
The database is:<br>
<br>
$ apt-cache show geoip-database<br>
Package: geoip-database<br>
Priority: standard<br>
Section: net<br>
Installed-Size: 3881<br>
<br>
Version: 20140313-1<br>
Recommends: libgeoip1<br>
Breaks: libgeoip1 (<< 1.4.5.dfsg)<br>
Filename:
pool/main/g/geoip-database/geoip-database_20140313-1_all.deb<br>
Size: 1195894<br>
MD5sum: ab4d4f6bc0e04b25cad2fbe1479f44bc<br>
SHA1: 06d38aee4084124f86351dfa6f1c404a8ae3e83b<br>
SHA256:
30dc5a2c3296180ed0740fb4ec70eb1ea5b49efc5e48a091913a8106f6895c7e<br>
Description-en: IP lookup command line tools that use the
GeoIP library (country database)<br>
GeoIP is a C library that enables the user to find the
country that any<br>
IP address or hostname originates from. It uses a file based
database.<br>
.<br>
This database simply contains IP blocks as keys, and
countries as values and<br>
it should be more complete and accurate than using reverse
DNS lookups.<br>
.<br>
This package contains the free GeoLiteCountry database.<br>
Description-md5: 3bfa5b4c9f973261799fb4d9355f3b6c<br>
Homepage: <a moz-do-not-send="true"
href="http://www.maxmind.com/">http://www.maxmind.com/</a><br>
Bugs: <a moz-do-not-send="true"
href="https://bugs.launchpad.net/ubuntu/+filebug">https://bugs.launchpad.net/ubuntu/+filebug</a><br>
Origin: Ubuntu<br>
Supported: 5y<br>
Task: standard, kubuntu-active, kubuntu-active,
mythbuntu-frontend, mythbuntu-frontend, mythbuntu-desktop,
mythbuntu-backend-slave, mythbuntu-backend-slave,
mythbuntu-backend-master, mythbuntu-backend-master<br>
<br>
<br>
</div>
So it is about a year old, but quite probably the version in
Debian sid can be installed on top without problems, and that's
pretty fresh, being dated 9th February.<br>
<br>
<a moz-do-not-send="true"
href="https://packages.debian.org/sid/geoip-database">https://packages.debian.org/sid/geoip-database</a><br>
<br>
<div>
<div>
<div><br>
</div>
</div>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Sat, Feb 21, 2015 at 1:24 PM, Jim
Hendrick <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:jrhendri@roadrunner.com" target="_blank">jrhendri@roadrunner.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Fabian,<br>
I have done just some preliminary testing (maybe 1500 EPS
for a few<br>
minutes) and was seeing a lot of dns traffic (~1MB/s)<br>
<br>
Obviously, if the field is a hostname, to do a geoip lookup
there needs<br>
to be name resolution before the IP can be mapped to a geo
database.<br>
<br>
I will be looking for ways to minimize this.<br>
<br>
Current use-cases are for parsing proxy, email and fire-eye
logs.<br>
<br>
Recall, my base architecture is<br>
syslog-ng using patterndb sending format-json to a local
redis<br>
destination (lpush)<br>
redis is run with no local disk storage and acts as an
in-memory buffer<br>
between syslog-ng and logstash<br>
logstash (also running locally on the same box) pulling
(blpop) and<br>
feeding an elasticsearch cluster (4 nodes right now)<br>
<br>
Currently taking live proxy logs at ~7 - 10 K EPS running
very well.<br>
Looking to add the email and fireeye logs soon and starting
to enhance<br>
the data (with user and host metadata)<br>
<br>
<br>
Thoughts right now are:<br>
- only resolve location for addresses (not hostnames)<br>
- run a caching nameserver locally on the syslog-ng box and
dealing with<br>
the "ramp up" period<br>
(initially clearly the names would not be in cache - just
not sure how<br>
long it would take to get to a steady state and how big to
make the<br>
cache, etc.)<br>
<br>
I'll keep you posted.<br>
<br>
Thanks again!<br>
<span class="HOEnZb"><font color="#888888">Jim<br>
</font></span>
<div class="HOEnZb">
<div class="h5"><br>
On 02/20/2015 03:24 PM, Fabien Wernli wrote:<br>
> Hi Jim,<br>
><br>
> On Fri, Feb 20, 2015 at 01:52:19PM -0500, <a
moz-do-not-send="true"
href="mailto:jrhendri@roadrunner.com">jrhendri@roadrunner.com</a>
wrote:<br>
>> Is anyone using it in reasonably
high-performance environments? (like 5000+ events per
second)<br>
>><br>
> we're using the module in a 3keps environment with
very good performance. we<br>
> have had some issues in the past in threaded mode
with some segfaults. The<br>
> geoip library documentation mentions a few
sentences about thread safety.<br>
> I'd be curious to hear some feedback about your
future<br>
> experience.<br>
><br>
> cheers<br>
>
______________________________________________________________________________<br>
> Member info: <a moz-do-not-send="true"
href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"
target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
> Documentation: <a moz-do-not-send="true"
href="http://www.balabit.com/support/documentation/?product=syslog-ng"
target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
> FAQ: <a moz-do-not-send="true"
href="http://www.balabit.com/wiki/syslog-ng-faq"
target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
><br>
><br>
<br>
______________________________________________________________________________<br>
Member info: <a moz-do-not-send="true"
href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"
target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a moz-do-not-send="true"
href="http://www.balabit.com/support/documentation/?product=syslog-ng"
target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a moz-do-not-send="true"
href="http://www.balabit.com/wiki/syslog-ng-faq"
target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
<div class="gmail_signature">Bazsi</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">______________________________________________________________________________
Member info: <a class="moz-txt-link-freetext" href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a>
Documentation: <a class="moz-txt-link-freetext" href="http://www.balabit.com/support/documentation/?product=syslog-ng">http://www.balabit.com/support/documentation/?product=syslog-ng</a>
FAQ: <a class="moz-txt-link-freetext" href="http://www.balabit.com/wiki/syslog-ng-faq">http://www.balabit.com/wiki/syslog-ng-faq</a>
</pre>
</blockquote>
<br>
</body>
</html>