<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<font face="Helvetica, Arial, sans-serif">Hi Bazsi, <br>
I saw the Elastic and hdfs and for me they are all the same :).
I'm trying to build something free, from scratch. And I'm already
done using Syslog-NG with MongoDB and all that to be shown on
LogAnalyzer. So we will see how things will go.<br>
<br>
Once again thank you for the support ! <br>
<br>
Kind regards<br>
Ivan<br>
</font><br>
<div class="moz-cite-prefix">On 05/24/2016 08:29 AM, Scheidler,
Balázs wrote:<br>
</div>
<blockquote
cite="mid:CANWQT2N=WH8mve_BcsOy04aSPDPpsoV2QT97ExckfBT0XXHNTQ@mail.gmail.com"
type="cite">
<p dir="ltr">Elastic or hdfs are horizontally scalable and depend
what you want to do with the logs. These are horizontally
scalable, but their per node performance needs to be taken into
account.</p>
<p dir="ltr">Stuff like Balabit syslog-ng store box can consume
lots of data on a single node or a couple of nodes but that's
commercial.</p>
<p dir="ltr">I'd experiment with elastic search and kibana if I
were you, or of course I can connect you someone about Balabit
ssb.</p>
<p dir="ltr">Bazsi</p>
<div class="gmail_quote">On May 19, 2016 10:09 AM, "Ivan Adji -
Krstev" <<a moz-do-not-send="true"
href="mailto:akivanradix@gmail.com">akivanradix@gmail.com</a>>
wrote:<br type="attribution">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> <font face="Helvetica,
Arial, sans-serif">Hi Richie, <br>
<br>
I'll do that too, the think that scares me is that I'm
running maybe two weeks this syslog-ng and i have 400 MB
of logs and im logging just 2 Machines, the Server itself
and on Client. <br>
And im planning to loging more than 1500 machines. So im
not sure what to choose. MongoDB, MySQL, or PostgreSQL.<br>
<br>
And for now i have problems with all of them :)<br>
<br>
<br>
Kind regards<br>
Ivan<br>
</font><br>
<div>On 05/19/2016 08:07 AM, Richárd Réfi wrote:<br>
</div>
<blockquote type="cite">
<p dir="ltr">Hi,</p>
<p dir="ltr">I would try mysql/mariadb tweaks also:<br>
- index(or indices) on one or more column(s) of the
mysql table according to the queries of loganalyzer<br>
- check the different cache and buffer options in your
mysql conf<br>
- my opinion and experience is that mysql partitioning
can do magic on this amount of data. A query could run
only on few gigs portion of data (and a good indexing
accelerates the query of these few gigs also).</p>
<p dir="ltr">Unfortunately your mongodb problem remains
open.</p>
<p dir="ltr">Regards, Richie<br>
<br>
</p>
<br>
<div class="gmail_quote">
<div dir="ltr">On Wed, May 18, 2016, 14:37 Ivan Adji -
Krstev <<a moz-do-not-send="true"
href="mailto:akivanradix@gmail.com" target="_blank">akivanradix@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> <font
face="Helvetica, Arial, sans-serif">Nop, <br>
Again same problem:<br>
Here is what i have done</font></div>
<div bgcolor="#FFFFFF" text="#000000"><font
face="Helvetica, Arial, sans-serif"><br>
<br>
destination d_mongodb {<br>
mongodb(<br>
servers("localhost:27017")<br>
</font></div>
<div bgcolor="#FFFFFF" text="#000000"><font
face="Helvetica, Arial, sans-serif">
database("syslog")<br>
username("Ivan")<br>
password("Ivan123")<br>
collection("messages")</font></div>
<div bgcolor="#FFFFFF" text="#000000"><font
face="Helvetica, Arial, sans-serif"><br>
value-pairs(<br>
scope("selected-macros" "nv-pairs"
"sdata")<br>
</font></div>
<div bgcolor="#FFFFFF" text="#000000"><font
face="Helvetica, Arial, sans-serif">
pair("date", datetime("$UNIXTIME"))<br>
pair("pid", int64("$PID"))<br>
pair("program", "$PROGRAM")<br>
pair("message", "$MESSAGE")<br>
)<br>
);<br>
};<br>
Still have the same problem no info on date no
nothing. The strange part is that when i open a
specific log i have all the info. The only problem
is where on a first page on the LogAnalyzer i
don't have this problems. <br>
<br>
And yes again we may have two problems: One is the
DB information how its stored, and by this i think
we stored as we should, but do i have to configure
some tables in the MongoDB or columns or something
( that is how i did it with MySQL ). The second is
something wrong with the LogAnalyzer so now im
going to reconfigure with PostgreSQL and again
with MySQL to see if something will be change.</font></div>
<div bgcolor="#FFFFFF" text="#000000"><font
face="Helvetica, Arial, sans-serif"><br>
<br>
<br>
Ivan<br>
</font></div>
<div bgcolor="#FFFFFF" text="#000000"><br>
<div>On 05/18/2016 01:53 PM, Fekete, Róbert wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi,
<div><br>
</div>
<div>Do you know in what type does loganalyzer
expect the specific fields? </div>
<div>AFAIK, by default, syslog-ng sends
everything as string, but for the mongodb
destination, you can specify the data type,
see <a moz-do-not-send="true"
href="https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/specifying-data-types.html"
target="_blank">https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/specifying-data-types.html</a></div>
<div><br>
</div>
<div>Try sending the date as datetime, and the
others as numbers, maybe it helps.</div>
<div><br>
</div>
<div>Regards,</div>
<div><br>
</div>
<div>Robert</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed, May 18, 2016 at
1:47 PM, Ivan Adji - Krstev <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:akivanradix@gmail.com"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:akivanradix@gmail.com">akivanradix@gmail.com</a></a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> <font
face="Helvetica, Arial, sans-serif">Robert,
<br>
i just thought of that and goggling how
to add columns or some other similar
scenarios, i think that the problem lays
on how syslog-ng send the logs in the
DB. Or how DB is storing this messages.
As i have not configure nothing on the
MongoDB just username and password for
already created DB by the syslog-ng. <br>
<br>
If some one have some tips, ill be happy
to try it :)<br>
<br>
Kind regards<span><font color="#888888"><br>
Ivan<br>
</font></span></font>
<div>
<div><br>
<div>On 05/18/2016 01:43 PM, Fekete,
Róbert wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi,
<div><br>
</div>
<div>can you check the mongodb
itself if the related
fields/tags/whatever are in
place?</div>
<div>I mean, the problem might be
in how syslog-ng sends the data
into MongoDB, or in how
loganalyzer reads the data from
MongoDB. Is there a way for you
to find out which?</div>
<div><br>
</div>
<div>Robert</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed,
May 18, 2016 at 11:04 AM, Ivan
Adji - Krstev <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:akivanradix@gmail.com"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:akivanradix@gmail.com">akivanradix@gmail.com</a></a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div bgcolor="#FFFFFF"
text="#000000"> <font
face="Helvetica, Arial,
sans-serif">Hi Jim, <br>
Thanks for the feedback. <br>
The problem is that im
trying to monitor big
infrastructure ( 200
Physical servers and more
than 1000 VMs ). So
currently i have install
with MongoDB and have
300MB for one week
monitoring just two VMs.
The server syslog-ng and
one client VM. Also i have
used before syslog-ng with
MariaDB (MySQL) but i have
problem that i have 90%
CPU Load when i used
MySQL. I can't fix it. But
now using MongoDB i have
other problems. Using
LogAnalyzer i can't see
the "Date", "Facility",
Serverity etc. on a main
page but when i go to the
log itself or i open it i
can see all this
informations. So i have
the following<br>
<br>
1. Syslog-NG with MySQL
and LogAnalyzer ( works ok
but CPU Usage was big ) <br>
2. Syslog-NG with MongoDB
and LogAnalyzer ( works ok
but no informations shown
on a first page ) <br>
<br>
So i can't find solutions
and i need this sh*** up
and running ASAP :) <br>
<br>
Any solutions or
suggestions im open to see
it !<br>
<br>
Kind regards<span><font
color="#888888"><br>
Ivan<br>
<br>
<br>
</font></span></font>
<div>
<div><span></span>
<div>On 05/16/2016 05:43
PM, <a
moz-do-not-send="true"
href="mailto:jrhendri@roadrunner.com" target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:jrhendri@roadrunner.com">jrhendri@roadrunner.com</a></a>
wrote:<br>
</div>
<blockquote type="cite">
<pre>My 2 cents (what works for you depends on your infrastructure, resources and capabilities)
I like the model where syslog-ng does all the following:
- writes text files of the raw data (that way - whatever your search head is can re-ingest files later using basically the same parsers)
- filters out highly false-positive prone data from being forwarded
- handles parsing of data elements (using patterndb or whatever) and sends specific information to a search engine (like Elasticsearch)
- forwards specific data (based on security use cases) to a SIEM
Whether you use Elasticsearch, mongo, splunk, or whatever is really up to you and your budget.
That said, I find syslog-ng to elasticsearch directly with kibana as the front end is *very* scalable for a search engine.
As far as a SIEM - it's kind of up to you.
Good luck,
Jim
---- Ivan Adji - Krstev <a moz-do-not-send="true" href="mailto:akivanradix@gmail.com" target="_blank"><akivanradix@gmail.com></a> wrote:
</pre>
<blockquote
type="cite">
<pre>Hi all,
What is the best practice for storing all those logs in one central
environment. I have one Linux Box running Syslog-NG with LogAnalyzer and
MongoDB ( for now ), and is the best way to configure and use it with
MongoDB or with MariaDB ( MySQL ) ? I have once install MySQL but it was
getting very slow as the logs getting bigger and bigger ( for one week ).
Now i have done with MongoDB ( still testing ) but i have problem as
LogAnalyzer does not show me the real pictures, i have no Date info, no
Facility, no serverity, Hosts, syslogtag, i just have ProcessID.
Any hints on this ?
I have the following configuration on the syslog-ng.cfg:
destination d_mongodb {
mongodb(
servers("localhost:27017")
database("logs")
# uri('mongodb://localhost/syslog-ng')
collection("syslog")
value-pairs(
scope("selected-macros" "nv-pairs" "sdata")
)
);
};
Kind regards
Ivan
</pre>
</blockquote>
</blockquote>
<br>
</div>
</div>
</div>
<br>
______________________________________________________________________________<br>
Member info: <a
moz-do-not-send="true"
href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"
target="_blank"><a class="moz-txt-link-freetext" href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a></a><br>
Documentation: <a
moz-do-not-send="true"
href="http://www.balabit.com/support/documentation/?product=syslog-ng"
target="_blank"><a class="moz-txt-link-freetext" href="http://www.balabit.com/support/documentation/?product=syslog-ng">http://www.balabit.com/support/documentation/?product=syslog-ng</a></a><br>
FAQ: <a
moz-do-not-send="true"
href="http://www.balabit.com/wiki/syslog-ng-faq"
rel="noreferrer"
target="_blank"><a class="moz-txt-link-freetext" href="http://www.balabit.com/wiki/syslog-ng-faq">http://www.balabit.com/wiki/syslog-ng-faq</a></a><br>
<br>
<br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>______________________________________________________________________________
Member info: <a moz-do-not-send="true" href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a>
Documentation: <a moz-do-not-send="true" href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a>
FAQ: <a moz-do-not-send="true" href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a>
</pre>
</blockquote>
<br>
</div>
</div>
</div>
<br>
______________________________________________________________________________<br>
Member info: <a moz-do-not-send="true"
href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"
rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a moz-do-not-send="true"
href="http://www.balabit.com/support/documentation/?product=syslog-ng"
rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a moz-do-not-send="true"
href="http://www.balabit.com/wiki/syslog-ng-faq"
rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
<br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>______________________________________________________________________________
Member info: <a moz-do-not-send="true" href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a>
Documentation: <a moz-do-not-send="true" href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a>
FAQ: <a moz-do-not-send="true" href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a>
</pre>
</blockquote>
<br>
</div>
______________________________________________________________________________<br>
Member info: <a moz-do-not-send="true"
href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"
rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a moz-do-not-send="true"
href="http://www.balabit.com/support/documentation/?product=syslog-ng"
rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a moz-do-not-send="true"
href="http://www.balabit.com/wiki/syslog-ng-faq"
rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>______________________________________________________________________________
Member info: <a moz-do-not-send="true" href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a>
Documentation: <a moz-do-not-send="true" href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a>
FAQ: <a moz-do-not-send="true" href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a>
</pre>
</blockquote>
<br>
</div>
<br>
______________________________________________________________________________<br>
Member info: <a moz-do-not-send="true"
href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"
rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a moz-do-not-send="true"
href="http://www.balabit.com/support/documentation/?product=syslog-ng"
rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a moz-do-not-send="true"
href="http://www.balabit.com/wiki/syslog-ng-faq"
rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
<br>
</blockquote>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">______________________________________________________________________________
Member info: <a class="moz-txt-link-freetext" href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a>
Documentation: <a class="moz-txt-link-freetext" href="http://www.balabit.com/support/documentation/?product=syslog-ng">http://www.balabit.com/support/documentation/?product=syslog-ng</a>
FAQ: <a class="moz-txt-link-freetext" href="http://www.balabit.com/wiki/syslog-ng-faq">http://www.balabit.com/wiki/syslog-ng-faq</a>
</pre>
</blockquote>
<br>
</body>
</html>