[syslog-ng] Syslog-NG with MongoDB

[Fekete, Róbert]

Hi,

can you check the mongodb itself if the related fields/tags/whatever are in
place?
I mean, the problem might be in how syslog-ng sends the data into MongoDB,
or in how loganalyzer reads the data from MongoDB. Is there a way for you
to find out which?

Robert


On Wed, May 18, 2016 at 11:04 AM, Ivan Adji - Krstev <akivanradix at gmail.com>
wrote:

> Hi Jim,
> Thanks for the feedback.
> The problem is that im trying to monitor big infrastructure ( 200 Physical
> servers and more than 1000 VMs ). So currently i have install with MongoDB
> and have 300MB for one week monitoring just two VMs. The server syslog-ng
> and one client VM. Also i have used before syslog-ng with MariaDB (MySQL)
> but i have problem that i have 90% CPU Load when i used MySQL. I can't fix
> it. But now using MongoDB i have other problems. Using LogAnalyzer i can't
> see the "Date", "Facility", Serverity etc. on a main page but when i go to
> the log itself or i open it i can see all this informations. So i have the
> following
>
> 1. Syslog-NG with MySQL and LogAnalyzer ( works ok but CPU Usage was big )
> 2. Syslog-NG with MongoDB and LogAnalyzer ( works ok but no informations
> shown on a first page )
>
> So i can't find solutions and i need this sh*** up and running ASAP :)
>
> Any solutions or suggestions im open to see it !
>
> Kind regards
> Ivan
>
>
> On 05/16/2016 05:43 PM, jrhendri at roadrunner.com wrote:
>
> My 2 cents (what works for you depends on your infrastructure, resources and capabilities)
>
> I like the model where syslog-ng does all the following:
>
> - writes text files of the raw data (that way - whatever your search head is can re-ingest files later using basically the same parsers)
>
> - filters out highly false-positive prone data from being forwarded
>
> - handles parsing of data elements (using patterndb or whatever) and sends specific information to a search engine (like Elasticsearch)
>
> - forwards specific data (based on security use cases) to a SIEM
>
>
>
> Whether you use Elasticsearch, mongo, splunk, or whatever is really up to you and your budget.
> That said, I find syslog-ng to elasticsearch directly with kibana as the front end is *very* scalable for a search engine.
>
> As far as a SIEM - it's kind of up to you.
>
> Good luck,
>
> Jim
>
>
> ---- Ivan Adji - Krstev <akivanradix at gmail.com> <akivanradix at gmail.com> wrote:
>
> Hi all,
>
> What is the best practice for storing all those logs in one central
> environment. I have one Linux Box running Syslog-NG with LogAnalyzer and
> MongoDB ( for now ), and is the best way to configure and use it with
> MongoDB or with MariaDB ( MySQL ) ? I have once install MySQL but it was
> getting very slow as the logs getting bigger and bigger ( for one week ).
> Now i have done with MongoDB ( still testing ) but i have problem as
> LogAnalyzer does not show me the real pictures, i have no Date info, no
> Facility, no serverity, Hosts, syslogtag, i just have ProcessID.
>
> Any hints on this ?
>
> I have the following configuration on the syslog-ng.cfg:
>
> destination d_mongodb {
>     mongodb(
>     servers("localhost:27017")
>         database("logs")
> #    uri('mongodb://localhost/syslog-ng')
>     collection("syslog")
>     value-pairs(
>     scope("selected-macros" "nv-pairs" "sdata")
>         )
>             );
>             };
>
> Kind regards
> Ivan
>
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20160518/df945d48/attachment.htm