<html>
  <head>
    <meta content="text/html; charset=windows-1252"
      http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    Thanks Balazs,<br>
      I will try some more "controlled" testing using different settings
    for syslog-ng resolving and caching.<br>
    <br>
    I think I installed libgeopi-dev (not sure right now - it's
    installed on a system at work) so I'll check that package also.<br>
    <br>
    One question on code paths:<br>
    <br>
    If I use an IP address pattern in patterndb (within the message -
    e.g. proxy or email logs) where a ${GEO} macro was assigned, will
    those be the only things that get resolved? (or by setting a cache
    within the syslog-ng config will that enable resolution for ${HOST}
    as well?<br>
    <br>
    I am (mostly) interested in things like user access to sites by IP
    address through the proxy, and wanting to enhance the logs with
    geoip data for elasticsearch.<br>
    <br>
    (obviously if it were fast enough, I would add the data for all
    sites - but initially I think IP only would be more interesting)<br>
    <br>
    Thanks again!<br>
    Jim<br>
    <br>
    <br>
    <br>
    <div class="moz-cite-prefix">On 02/22/2015 02:35 PM, Balazs
      Scheidler wrote:<br>
    </div>
    <blockquote
cite="mid:CAKcfE+YgnZxRa7weyD0NCSZcNB_V6Bt1d-0UNmEzCP4Zryghww@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div>
          <div>
            <div>
              <div>Hi,<br>
                <br>
              </div>
              I would think that adding forward DNS lookups to the
              syslog-ng dns cache code (or ripping out that code
              entirely and rewrite it from scratch while adding this
              feature) would produce _much_ better results than a
              locally running DNS server. That's why the DNS cache code
              was added in the first place, a caching only name server
              is still too slow for name lookups for every message
              posted.<br>
              <br>
            </div>
            The  geoip code uses libgeoip1.<br>
            <br>
          </div>
          The database is:<br>
          <br>
          $ apt-cache show geoip-database<br>
          Package: geoip-database<br>
          Priority: standard<br>
          Section: net<br>
          Installed-Size: 3881<br>
          <br>
          Version: 20140313-1<br>
          Recommends: libgeoip1<br>
          Breaks: libgeoip1 (&lt;&lt; 1.4.5.dfsg)<br>
          Filename:
          pool/main/g/geoip-database/geoip-database_20140313-1_all.deb<br>
          Size: 1195894<br>
          MD5sum: ab4d4f6bc0e04b25cad2fbe1479f44bc<br>
          SHA1: 06d38aee4084124f86351dfa6f1c404a8ae3e83b<br>
          SHA256:
          30dc5a2c3296180ed0740fb4ec70eb1ea5b49efc5e48a091913a8106f6895c7e<br>
          Description-en: IP lookup command line tools that use the
          GeoIP library (country database)<br>
           GeoIP is a C library that enables the user to find the
          country that any<br>
           IP address or hostname originates from. It uses a file based
          database.<br>
           .<br>
           This database simply contains IP blocks as keys, and
          countries as values and<br>
           it should be more complete and accurate than using reverse
          DNS lookups.<br>
           .<br>
           This package contains the free GeoLiteCountry database.<br>
          Description-md5: 3bfa5b4c9f973261799fb4d9355f3b6c<br>
          Homepage: <a moz-do-not-send="true"
            href="http://www.maxmind.com/">http://www.maxmind.com/</a><br>
          Bugs: <a moz-do-not-send="true"
            href="https://bugs.launchpad.net/ubuntu/+filebug">https://bugs.launchpad.net/ubuntu/+filebug</a><br>
          Origin: Ubuntu<br>
          Supported: 5y<br>
          Task: standard, kubuntu-active, kubuntu-active,
          mythbuntu-frontend, mythbuntu-frontend, mythbuntu-desktop,
          mythbuntu-backend-slave, mythbuntu-backend-slave,
          mythbuntu-backend-master, mythbuntu-backend-master<br>
          <br>
          <br>
        </div>
        So it is about a year old, but quite probably the version in
        Debian sid can be installed on top without problems, and that's
        pretty fresh, being dated 9th February.<br>
        <br>
        <a moz-do-not-send="true"
          href="https://packages.debian.org/sid/geoip-database">https://packages.debian.org/sid/geoip-database</a><br>
        <br>
        <div>
          <div>
            <div><br>
            </div>
          </div>
        </div>
      </div>
      <div class="gmail_extra"><br>
        <div class="gmail_quote">On Sat, Feb 21, 2015 at 1:24 PM, Jim
          Hendrick <span dir="ltr">&lt;<a moz-do-not-send="true"
              href="mailto:jrhendri@roadrunner.com" target="_blank">jrhendri@roadrunner.com</a>&gt;</span>
          wrote:<br>
          <blockquote class="gmail_quote" style="margin:0 0 0
            .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Fabian,<br>
              I have done just some preliminary testing (maybe 1500 EPS
            for a few<br>
            minutes) and was seeing a lot of dns traffic (~1MB/s)<br>
            <br>
            Obviously, if the field is a hostname, to do a geoip lookup
            there needs<br>
            to be name resolution before the IP can be mapped to a geo
            database.<br>
            <br>
            I will be looking for ways to minimize this.<br>
            <br>
            Current use-cases are for parsing proxy, email and fire-eye
            logs.<br>
            <br>
            Recall, my base architecture is<br>
            syslog-ng using patterndb sending format-json to a local
            redis<br>
            destination (lpush)<br>
            redis is run with no local disk storage and acts as an
            in-memory buffer<br>
            between syslog-ng and logstash<br>
            logstash (also running locally on the same box) pulling
            (blpop) and<br>
            feeding an elasticsearch cluster (4 nodes right now)<br>
            <br>
            Currently taking live proxy logs at ~7 - 10 K EPS running
            very well.<br>
            Looking to add the email and fireeye logs soon and starting
            to enhance<br>
            the data (with user and host metadata)<br>
            <br>
            <br>
            Thoughts right now are:<br>
            - only resolve location for addresses (not hostnames)<br>
            - run a caching nameserver locally on the syslog-ng box and
            dealing with<br>
            the "ramp up" period<br>
              (initially clearly the names would not be in cache - just
            not sure how<br>
            long it would take to get to a steady state and how big to
            make the<br>
            cache, etc.)<br>
            <br>
            I'll keep you posted.<br>
            <br>
            Thanks again!<br>
            <span class="HOEnZb"><font color="#888888">Jim<br>
              </font></span>
            <div class="HOEnZb">
              <div class="h5"><br>
                On 02/20/2015 03:24 PM, Fabien Wernli wrote:<br>
                &gt; Hi Jim,<br>
                &gt;<br>
                &gt; On Fri, Feb 20, 2015 at 01:52:19PM -0500, <a
                  moz-do-not-send="true"
                  href="mailto:jrhendri@roadrunner.com">jrhendri@roadrunner.com</a>
                wrote:<br>
                &gt;&gt;   Is anyone using it in reasonably
                high-performance environments? (like 5000+ events per
                second)<br>
                &gt;&gt;<br>
                &gt; we're using the module in a 3keps environment with
                very good performance. we<br>
                &gt; have had some issues in the past in threaded mode
                with some segfaults. The<br>
                &gt; geoip library documentation mentions a few
                sentences about thread safety.<br>
                &gt; I'd be curious to hear some feedback about your
                future<br>
                &gt;  experience.<br>
                &gt;<br>
                &gt; cheers<br>
                &gt;
______________________________________________________________________________<br>
                &gt; Member info: <a moz-do-not-send="true"
                  href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"
                  target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
                &gt; Documentation: <a moz-do-not-send="true"
                  href="http://www.balabit.com/support/documentation/?product=syslog-ng"
                  target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
                &gt; FAQ: <a moz-do-not-send="true"
                  href="http://www.balabit.com/wiki/syslog-ng-faq"
                  target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
                &gt;<br>
                &gt;<br>
                <br>
______________________________________________________________________________<br>
                Member info: <a moz-do-not-send="true"
                  href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"
                  target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
                Documentation: <a moz-do-not-send="true"
                  href="http://www.balabit.com/support/documentation/?product=syslog-ng"
                  target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
                FAQ: <a moz-do-not-send="true"
                  href="http://www.balabit.com/wiki/syslog-ng-faq"
                  target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
                <br>
              </div>
            </div>
          </blockquote>
        </div>
        <br>
        <br clear="all">
        <br>
        -- <br>
        <div class="gmail_signature">Bazsi</div>
      </div>
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <br>
      <pre wrap="">______________________________________________________________________________
Member info: <a class="moz-txt-link-freetext" href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a>
Documentation: <a class="moz-txt-link-freetext" href="http://www.balabit.com/support/documentation/?product=syslog-ng">http://www.balabit.com/support/documentation/?product=syslog-ng</a>
FAQ: <a class="moz-txt-link-freetext" href="http://www.balabit.com/wiki/syslog-ng-faq">http://www.balabit.com/wiki/syslog-ng-faq</a>

</pre>
    </blockquote>
    <br>
  </body>
</html>