<div dir="ltr">Hi, <div><br></div><div>Do you know in what type does loganalyzer expect the specific fields? </div><div>AFAIK, by default, syslog-ng sends everything as string, but for the mongodb destination, you can specify the data type, see <a href="https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/specifying-data-types.html">https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/specifying-data-types.html</a></div><div><br></div><div>Try sending the date as datetime, and the others as numbers, maybe it helps.</div><div><br></div><div>Regards,</div><div><br></div><div>Robert</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, May 18, 2016 at 1:47 PM, Ivan Adji - Krstev <span dir="ltr"><<a href="mailto:akivanradix@gmail.com" target="_blank">akivanradix@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<font face="Helvetica, Arial, sans-serif">Robert, <br>
i just thought of that and goggling how to add columns or some
other similar scenarios, i think that the problem lays on how
syslog-ng send the logs in the DB. Or how DB is storing this
messages. As i have not configure nothing on the MongoDB just
username and password for already created DB by the syslog-ng. <br>
<br>
If some one have some tips, ill be happy to try it :)<br>
<br>
Kind regards<span class="HOEnZb"><font color="#888888"><br>
Ivan<br>
</font></span></font><div><div class="h5"><br>
<div>On 05/18/2016 01:43 PM, Fekete, Róbert
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi,
<div><br>
</div>
<div>can you check the mongodb itself if the related
fields/tags/whatever are in place?</div>
<div>I mean, the problem might be in how syslog-ng sends the
data into MongoDB, or in how loganalyzer reads the data from
MongoDB. Is there a way for you to find out which?</div>
<div><br>
</div>
<div>Robert</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed, May 18, 2016 at 11:04 AM, Ivan
Adji - Krstev <span dir="ltr"><<a href="mailto:akivanradix@gmail.com" target="_blank">akivanradix@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> <font face="Helvetica, Arial, sans-serif">Hi Jim, <br>
Thanks for the feedback. <br>
The problem is that im trying to monitor big
infrastructure ( 200 Physical servers and more than 1000
VMs ). So currently i have install with MongoDB and have
300MB for one week monitoring just two VMs. The server
syslog-ng and one client VM. Also i have used before
syslog-ng with MariaDB (MySQL) but i have problem that i
have 90% CPU Load when i used MySQL. I can't fix it. But
now using MongoDB i have other problems. Using
LogAnalyzer i can't see the "Date", "Facility",
Serverity etc. on a main page but when i go to the log
itself or i open it i can see all this informations. So
i have the following<br>
<br>
1. Syslog-NG with MySQL and LogAnalyzer ( works ok but
CPU Usage was big ) <br>
2. Syslog-NG with MongoDB and LogAnalyzer ( works ok but
no informations shown on a first page ) <br>
<br>
So i can't find solutions and i need this sh*** up and
running ASAP :) <br>
<br>
Any solutions or suggestions im open to see it !<br>
<br>
Kind regards<span><font color="#888888"><br>
Ivan<br>
<br>
<br>
</font></span></font>
<div>
<div><span></span>
<div>On 05/16/2016 05:43 PM, <a href="mailto:jrhendri@roadrunner.com" target="_blank"></a><a href="mailto:jrhendri@roadrunner.com" target="_blank">jrhendri@roadrunner.com</a> wrote:<br>
</div>
<blockquote type="cite">
<pre>My 2 cents (what works for you depends on your infrastructure, resources and capabilities)
I like the model where syslog-ng does all the following:
- writes text files of the raw data (that way - whatever your search head is can re-ingest files later using basically the same parsers)
- filters out highly false-positive prone data from being forwarded
- handles parsing of data elements (using patterndb or whatever) and sends specific information to a search engine (like Elasticsearch)
- forwards specific data (based on security use cases) to a SIEM
Whether you use Elasticsearch, mongo, splunk, or whatever is really up to you and your budget.
That said, I find syslog-ng to elasticsearch directly with kibana as the front end is *very* scalable for a search engine.
As far as a SIEM - it's kind of up to you.
Good luck,
Jim
---- Ivan Adji - Krstev <a href="mailto:akivanradix@gmail.com" target="_blank"><akivanradix@gmail.com></a> wrote:
</pre>
<blockquote type="cite">
<pre>Hi all,
What is the best practice for storing all those logs in one central
environment. I have one Linux Box running Syslog-NG with LogAnalyzer and
MongoDB ( for now ), and is the best way to configure and use it with
MongoDB or with MariaDB ( MySQL ) ? I have once install MySQL but it was
getting very slow as the logs getting bigger and bigger ( for one week ).
Now i have done with MongoDB ( still testing ) but i have problem as
LogAnalyzer does not show me the real pictures, i have no Date info, no
Facility, no serverity, Hosts, syslogtag, i just have ProcessID.
Any hints on this ?
I have the following configuration on the syslog-ng.cfg:
destination d_mongodb {
mongodb(
servers("localhost:27017")
database("logs")
# uri('mongodb://localhost/syslog-ng')
collection("syslog")
value-pairs(
scope("selected-macros" "nv-pairs" "sdata")
)
);
};
Kind regards
Ivan
</pre>
</blockquote>
</blockquote>
<br>
</div>
</div>
</div>
<br>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
<br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>______________________________________________________________________________
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a>
</pre>
</blockquote>
<br>
</div></div></div>
<br>______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div>