<html>
  <head>
    <meta content="text/html; charset=windows-1252"
      http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <font face="Helvetica, Arial, sans-serif">Hi Richie, <br>
      <br>
      I'll do that too, the think that scares me is that I'm running
      maybe two weeks this syslog-ng and i have 400 MB of logs and im
      logging just 2 Machines, the Server itself and on Client. <br>
      And im planning to loging more than 1500 machines. So im not sure
      what to choose. MongoDB, MySQL, or PostgreSQL.<br>
      <br>
      And for now i have problems with all of them :)<br>
      <br>
      <br>
      Kind regards<br>
      Ivan<br>
    </font><br>
    <div class="moz-cite-prefix">On 05/19/2016 08:07 AM, Richárd Réfi
      wrote:<br>
    </div>
    <blockquote
cite="mid:CAEqBPOhVvACHrJJtky3oL-odyke2a5t8GYSmPNsGvu6iS+dh+w@mail.gmail.com"
      type="cite">
      <p dir="ltr">Hi,</p>
      <p dir="ltr">I would try mysql/mariadb tweaks also:<br>
        - index(or indices) on one or more column(s) of the mysql table
        according to the queries of loganalyzer<br>
        - check the different cache and buffer options in your mysql
        conf<br>
        - my opinion and experience is that mysql partitioning can do
        magic on this amount of data. A query could run only on few gigs
        portion of data (and a good indexing accelerates the query of
        these few gigs also).</p>
      <p dir="ltr">Unfortunately your mongodb problem remains open.</p>
      <p dir="ltr">Regards, Richie<br>
          <br>
      </p>
      <br>
      <div class="gmail_quote">
        <div dir="ltr">On Wed, May 18, 2016, 14:37 Ivan Adji - Krstev
          &lt;<a moz-do-not-send="true"
            href="mailto:akivanradix@gmail.com">akivanradix@gmail.com</a>&gt;
          wrote:<br>
        </div>
        <blockquote class="gmail_quote" style="margin:0 0 0
          .8ex;border-left:1px #ccc solid;padding-left:1ex">
          <div bgcolor="#FFFFFF" text="#000000"> <font face="Helvetica,
              Arial, sans-serif">Nop, <br>
              Again same problem:<br>
              Here is what i have done</font></div>
          <div bgcolor="#FFFFFF" text="#000000"><font face="Helvetica,
              Arial, sans-serif"><br>
              <br>
              destination d_mongodb {<br>
                      mongodb(<br>
                      servers("localhost:27017")<br>
            </font></div>
          <div bgcolor="#FFFFFF" text="#000000"><font face="Helvetica,
              Arial, sans-serif">         database("syslog")<br>
                      username("Ivan")<br>
                      password("Ivan123")<br>
                      collection("messages")</font></div>
          <div bgcolor="#FFFFFF" text="#000000"><font face="Helvetica,
              Arial, sans-serif"><br>
                      value-pairs(<br>
                          scope("selected-macros" "nv-pairs" "sdata")<br>
            </font></div>
          <div bgcolor="#FFFFFF" text="#000000"><font face="Helvetica,
              Arial, sans-serif">             pair("date",
              datetime("$UNIXTIME"))<br>
                          pair("pid", int64("$PID"))<br>
                          pair("program", "$PROGRAM")<br>
                          pair("message", "$MESSAGE")<br>
                              )<br>
                                      );<br>
                                      };<br>
              Still have the same problem no info on date no nothing.
              The strange part is that when i open a specific log i have
              all the info. The only problem is where on a first page on
              the LogAnalyzer i don't have this problems. <br>
              <br>
              And yes again we may have two problems: One is the DB
              information how its stored, and by this i think we stored
              as we should, but do i have to configure some tables in
              the MongoDB or columns or something ( that is how i did it
              with MySQL ). The second is something wrong with the
              LogAnalyzer so now im going to reconfigure with PostgreSQL
              and again with MySQL to see if something will be change.</font></div>
          <div bgcolor="#FFFFFF" text="#000000"><font face="Helvetica,
              Arial, sans-serif"><br>
              <br>
              <br>
              Ivan<br>
            </font></div>
          <div bgcolor="#FFFFFF" text="#000000"><br>
            <div>On 05/18/2016 01:53 PM, Fekete, Róbert wrote:<br>
            </div>
            <blockquote type="cite">
              <div dir="ltr">Hi, 
                <div><br>
                </div>
                <div>Do you know in what type does loganalyzer expect
                  the specific fields? </div>
                <div>AFAIK, by default, syslog-ng sends everything as
                  string, but for the mongodb destination, you can
                  specify the data type, see <a moz-do-not-send="true"
href="https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/specifying-data-types.html"
                    target="_blank">https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/specifying-data-types.html</a></div>
                <div><br>
                </div>
                <div>Try sending the date as datetime, and the others as
                  numbers, maybe it helps.</div>
                <div><br>
                </div>
                <div>Regards,</div>
                <div><br>
                </div>
                <div>Robert</div>
              </div>
              <div class="gmail_extra"><br>
                <div class="gmail_quote">On Wed, May 18, 2016 at 1:47
                  PM, Ivan Adji - Krstev <span dir="ltr">&lt;<a
                      moz-do-not-send="true"
                      href="mailto:akivanradix@gmail.com"
                      target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:akivanradix@gmail.com">akivanradix@gmail.com</a></a>&gt;</span>
                  wrote:<br>
                  <blockquote class="gmail_quote" style="margin:0 0 0
                    .8ex;border-left:1px #ccc solid;padding-left:1ex">
                    <div bgcolor="#FFFFFF" text="#000000"> <font
                        face="Helvetica, Arial, sans-serif">Robert, <br>
                        i just thought of that and goggling how to add
                        columns or some other similar scenarios, i think
                        that the problem lays on how syslog-ng send the
                        logs in the DB. Or how DB is storing this
                        messages. As i have not configure nothing on the
                        MongoDB just username and password for already
                        created DB by the syslog-ng. <br>
                        <br>
                        If some one have some tips, ill be happy to try
                        it :)<br>
                        <br>
                        Kind regards<span><font color="#888888"><br>
                            Ivan<br>
                          </font></span></font>
                      <div>
                        <div><br>
                          <div>On 05/18/2016 01:43 PM, Fekete, Róbert
                            wrote:<br>
                          </div>
                          <blockquote type="cite">
                            <div dir="ltr">Hi, 
                              <div><br>
                              </div>
                              <div>can you check the mongodb itself if
                                the related fields/tags/whatever are in
                                place?</div>
                              <div>I mean, the problem might be in how
                                syslog-ng sends the data into MongoDB,
                                or in how loganalyzer reads the data
                                from MongoDB. Is there a way for you to
                                find out which?</div>
                              <div><br>
                              </div>
                              <div>Robert</div>
                              <div><br>
                              </div>
                            </div>
                            <div class="gmail_extra"><br>
                              <div class="gmail_quote">On Wed, May 18,
                                2016 at 11:04 AM, Ivan Adji - Krstev <span
                                  dir="ltr">&lt;<a
                                    moz-do-not-send="true"
                                    href="mailto:akivanradix@gmail.com"
                                    target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:akivanradix@gmail.com">akivanradix@gmail.com</a></a>&gt;</span>
                                wrote:<br>
                                <blockquote class="gmail_quote"
                                  style="margin:0 0 0
                                  .8ex;border-left:1px #ccc
                                  solid;padding-left:1ex">
                                  <div bgcolor="#FFFFFF" text="#000000">
                                    <font face="Helvetica, Arial,
                                      sans-serif">Hi Jim, <br>
                                      Thanks for the feedback. <br>
                                      The problem is that im trying to
                                      monitor big infrastructure ( 200
                                      Physical servers and more than
                                      1000 VMs ). So currently i have
                                      install with MongoDB and have
                                      300MB for one week monitoring just
                                      two VMs. The server syslog-ng and
                                      one client VM. Also i have used
                                      before syslog-ng with MariaDB
                                      (MySQL) but i have problem that i
                                      have 90% CPU Load when i used
                                      MySQL. I can't fix it. But now
                                      using MongoDB i have other
                                      problems. Using LogAnalyzer i
                                      can't see the "Date", "Facility",
                                      Serverity etc. on a main page but
                                      when i go to the log itself or i
                                      open it i can see all this
                                      informations. So i have the
                                      following<br>
                                      <br>
                                      1. Syslog-NG with MySQL and
                                      LogAnalyzer ( works ok but CPU
                                      Usage was big ) <br>
                                      2. Syslog-NG with MongoDB and
                                      LogAnalyzer ( works ok but no
                                      informations shown on a first page
                                      ) <br>
                                      <br>
                                      So i can't find solutions and i
                                      need this sh*** up and running
                                      ASAP :) <br>
                                      <br>
                                      Any solutions or suggestions im
                                      open to see it !<br>
                                      <br>
                                      Kind regards<span><font
                                          color="#888888"><br>
                                          Ivan<br>
                                          <br>
                                          <br>
                                        </font></span></font>
                                    <div>
                                      <div><span></span>
                                        <div>On 05/16/2016 05:43 PM, <a
                                            moz-do-not-send="true"
                                            href="mailto:jrhendri@roadrunner.com"
                                            target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:jrhendri@roadrunner.com">jrhendri@roadrunner.com</a></a>
                                          wrote:<br>
                                        </div>
                                        <blockquote type="cite">
                                          <pre>My 2 cents (what works for you depends on your infrastructure, resources and capabilities)

I like the model where syslog-ng does all the following:

- writes text files of the raw data (that way - whatever your search head is can re-ingest files later using basically the same parsers)

- filters out highly false-positive prone data from being forwarded

- handles parsing of data elements (using patterndb or whatever) and sends specific information to a search engine (like Elasticsearch)

- forwards specific data (based on security use cases) to a SIEM



Whether you use Elasticsearch, mongo, splunk, or whatever is really up to you and your budget.
That said, I find syslog-ng to elasticsearch directly with kibana as the front end is *very* scalable for a search engine.

As far as a SIEM - it's kind of up to you.

Good luck,

Jim


---- Ivan Adji - Krstev <a moz-do-not-send="true" href="mailto:akivanradix@gmail.com" target="_blank">&lt;akivanradix@gmail.com&gt;</a> wrote: 
</pre>
                                          <blockquote type="cite">
                                            <pre>Hi all,

What is the best practice for storing all those logs in one central
environment. I have one Linux Box running Syslog-NG with LogAnalyzer and
MongoDB ( for now ), and is the best way to configure and use it with
MongoDB or with MariaDB ( MySQL ) ? I have once install MySQL but it was
getting very slow as the logs getting bigger and bigger ( for one week ).
Now i have done with MongoDB ( still testing ) but i have problem as
LogAnalyzer does not show me the real pictures, i have no Date info, no
Facility, no serverity, Hosts, syslogtag, i just have ProcessID.

Any hints on this ?

I have the following configuration on the syslog-ng.cfg:

destination d_mongodb {
    mongodb(
    servers("localhost:27017")
        database("logs")
#    uri('mongodb://localhost/syslog-ng')
    collection("syslog")
    value-pairs(
    scope("selected-macros" "nv-pairs" "sdata")
        )
            );
            };

Kind regards
Ivan
</pre>
                                          </blockquote>
                                        </blockquote>
                                        <br>
                                      </div>
                                    </div>
                                  </div>
                                  <br>
______________________________________________________________________________<br>
                                  Member info: <a
                                    moz-do-not-send="true"
                                    href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"
                                    rel="noreferrer" target="_blank"><a class="moz-txt-link-freetext" href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a></a><br>
                                  Documentation: <a
                                    moz-do-not-send="true"
                                    href="http://www.balabit.com/support/documentation/?product=syslog-ng"
                                    rel="noreferrer" target="_blank"><a class="moz-txt-link-freetext" href="http://www.balabit.com/support/documentation/?product=syslog-ng">http://www.balabit.com/support/documentation/?product=syslog-ng</a></a><br>
                                  FAQ: <a moz-do-not-send="true"
                                    href="http://www.balabit.com/wiki/syslog-ng-faq"
                                    rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
                                  <br>
                                  <br>
                                </blockquote>
                              </div>
                              <br>
                            </div>
                            <br>
                            <fieldset></fieldset>
                            <br>
                            <pre>______________________________________________________________________________
Member info: <a moz-do-not-send="true" href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a>
Documentation: <a moz-do-not-send="true" href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a>
FAQ: <a moz-do-not-send="true" href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a>

</pre>
                          </blockquote>
                          <br>
                        </div>
                      </div>
                    </div>
                    <br>
______________________________________________________________________________<br>
                    Member info: <a moz-do-not-send="true"
                      href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"
                      rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
                    Documentation: <a moz-do-not-send="true"
                      href="http://www.balabit.com/support/documentation/?product=syslog-ng"
                      rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
                    FAQ: <a moz-do-not-send="true"
                      href="http://www.balabit.com/wiki/syslog-ng-faq"
                      rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
                    <br>
                    <br>
                  </blockquote>
                </div>
                <br>
              </div>
              <br>
              <fieldset></fieldset>
              <br>
              <pre>______________________________________________________________________________
Member info: <a moz-do-not-send="true" href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a>
Documentation: <a moz-do-not-send="true" href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a>
FAQ: <a moz-do-not-send="true" href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a>

</pre>
            </blockquote>
            <br>
          </div>
______________________________________________________________________________<br>
          Member info: <a moz-do-not-send="true"
            href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"
            rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
          Documentation: <a moz-do-not-send="true"
            href="http://www.balabit.com/support/documentation/?product=syslog-ng"
            rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
          FAQ: <a moz-do-not-send="true"
            href="http://www.balabit.com/wiki/syslog-ng-faq"
            rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
          <br>
        </blockquote>
      </div>
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <br>
      <pre wrap="">______________________________________________________________________________
Member info: <a class="moz-txt-link-freetext" href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a>
Documentation: <a class="moz-txt-link-freetext" href="http://www.balabit.com/support/documentation/?product=syslog-ng">http://www.balabit.com/support/documentation/?product=syslog-ng</a>
FAQ: <a class="moz-txt-link-freetext" href="http://www.balabit.com/wiki/syslog-ng-faq">http://www.balabit.com/wiki/syslog-ng-faq</a>

</pre>
    </blockquote>
    <br>
  </body>
</html>