<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<font face="Helvetica, Arial, sans-serif">Hi Jim, <br>
Thanks for the feedback. <br>
The problem is that im trying to monitor big infrastructure ( 200
Physical servers and more than 1000 VMs ). So currently i have
install with MongoDB and have 300MB for one week monitoring just
two VMs. The server syslog-ng and one client VM. Also i have used
before syslog-ng with MariaDB (MySQL) but i have problem that i
have 90% CPU Load when i used MySQL. I can't fix it. But now using
MongoDB i have other problems. Using LogAnalyzer i can't see the
"Date", "Facility", Serverity etc. on a main page but when i go to
the log itself or i open it i can see all this informations. So i
have the following<br>
<br>
1. Syslog-NG with MySQL and LogAnalyzer ( works ok but CPU Usage
was big ) <br>
2. Syslog-NG with MongoDB and LogAnalyzer ( works ok but no
informations shown on a first page ) <br>
<br>
So i can't find solutions and i need this sh*** up and running
ASAP :) <br>
<br>
Any solutions or suggestions im open to see it !<br>
<br>
Kind regards<br>
Ivan<br>
<br>
<br>
</font><span style="color: rgb(51, 51, 51); font-family: 'Lucida
Grande', 'Trebuchet MS', Verdana, Helvetica, Arial, sans-serif;
font-size: 13px; font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing: normal; line-height: 18.2px;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: 1;
word-spacing: 0px; -webkit-text-stroke-width: 0px; display: inline
!important; float: none; background-color: rgb(225, 235, 242);"></span>
<div class="moz-cite-prefix">On 05/16/2016 05:43 PM,
<a class="moz-txt-link-abbreviated" href="mailto:jrhendri@roadrunner.com">jrhendri@roadrunner.com</a> wrote:<br>
</div>
<blockquote cite="mid:20160516154356.F5S9A.74092.root@cdptpa-web10"
type="cite">
<pre wrap="">My 2 cents (what works for you depends on your infrastructure, resources and capabilities)
I like the model where syslog-ng does all the following:
- writes text files of the raw data (that way - whatever your search head is can re-ingest files later using basically the same parsers)
- filters out highly false-positive prone data from being forwarded
- handles parsing of data elements (using patterndb or whatever) and sends specific information to a search engine (like Elasticsearch)
- forwards specific data (based on security use cases) to a SIEM
Whether you use Elasticsearch, mongo, splunk, or whatever is really up to you and your budget.
That said, I find syslog-ng to elasticsearch directly with kibana as the front end is *very* scalable for a search engine.
As far as a SIEM - it's kind of up to you.
Good luck,
Jim
---- Ivan Adji - Krstev <a class="moz-txt-link-rfc2396E" href="mailto:akivanradix@gmail.com"><akivanradix@gmail.com></a> wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi all,
What is the best practice for storing all those logs in one central
environment. I have one Linux Box running Syslog-NG with LogAnalyzer and
MongoDB ( for now ), and is the best way to configure and use it with
MongoDB or with MariaDB ( MySQL ) ? I have once install MySQL but it was
getting very slow as the logs getting bigger and bigger ( for one week ).
Now i have done with MongoDB ( still testing ) but i have problem as
LogAnalyzer does not show me the real pictures, i have no Date info, no
Facility, no serverity, Hosts, syslogtag, i just have ProcessID.
Any hints on this ?
I have the following configuration on the syslog-ng.cfg:
destination d_mongodb {
mongodb(
servers("localhost:27017")
database("logs")
# uri('mongodb://localhost/syslog-ng')
collection("syslog")
value-pairs(
scope("selected-macros" "nv-pairs" "sdata")
)
);
};
Kind regards
Ivan
</pre>
</blockquote>
<pre wrap="">
</pre>
</blockquote>
<br>
</body>
</html>