[syslog-ng] Syslog-NG with MongoDB

[Ivan Adji - Krstev]

Robert,
i just thought of that and goggling how to add columns or some other
similar scenarios, i think that the problem lays on how syslog-ng send
the logs in the DB. Or how DB is storing this messages. As i have not
configure nothing on the MongoDB just username and password for already
created DB by the syslog-ng.

If some one have some tips, ill be happy to try it :)

Kind regards
Ivan

On 05/18/2016 01:43 PM, Fekete, Róbert wrote:
> Hi, 
>
> can you check the mongodb itself if the related fields/tags/whatever
> are in place?
> I mean, the problem might be in how syslog-ng sends the data into
> MongoDB, or in how loganalyzer reads the data from MongoDB. Is there a
> way for you to find out which?
>
> Robert
>
>
> On Wed, May 18, 2016 at 11:04 AM, Ivan Adji - Krstev
> <akivanradix at gmail.com <mailto:akivanradix at gmail.com>> wrote:
>
>     Hi Jim,
>     Thanks for the feedback.
>     The problem is that im trying to monitor big infrastructure ( 200
>     Physical servers and more than 1000 VMs ). So currently i have
>     install with MongoDB and have 300MB for one week monitoring just
>     two VMs. The server syslog-ng and one client VM. Also i have used
>     before syslog-ng with MariaDB (MySQL) but i have problem that i
>     have 90% CPU Load when i used MySQL. I can't fix it. But now using
>     MongoDB i have other problems. Using LogAnalyzer i can't see the
>     "Date", "Facility", Serverity etc. on a main page but when i go to
>     the log itself or i open it i can see all this informations. So i
>     have the following
>
>     1. Syslog-NG with MySQL and LogAnalyzer ( works ok but CPU Usage
>     was big )
>     2. Syslog-NG with MongoDB and LogAnalyzer ( works ok but no
>     informations shown on a first page )
>
>     So i can't find solutions and i need this sh*** up and running
>     ASAP :)
>
>     Any solutions or suggestions im open to see it !
>
>     Kind regards
>     Ivan
>
>
>     On 05/16/2016 05:43 PM, jrhendri at roadrunner.com
>     <mailto:jrhendri at roadrunner.com> wrote:
>>     My 2 cents (what works for you depends on your infrastructure, resources and capabilities)
>>
>>     I like the model where syslog-ng does all the following:
>>
>>     - writes text files of the raw data (that way - whatever your search head is can re-ingest files later using basically the same parsers)
>>
>>     - filters out highly false-positive prone data from being forwarded
>>
>>     - handles parsing of data elements (using patterndb or whatever) and sends specific information to a search engine (like Elasticsearch)
>>
>>     - forwards specific data (based on security use cases) to a SIEM
>>
>>
>>
>>     Whether you use Elasticsearch, mongo, splunk, or whatever is really up to you and your budget.
>>     That said, I find syslog-ng to elasticsearch directly with kibana as the front end is *very* scalable for a search engine.
>>
>>     As far as a SIEM - it's kind of up to you.
>>
>>     Good luck,
>>
>>     Jim
>>
>>
>>     ---- Ivan Adji - Krstev <akivanradix at gmail.com> <mailto:akivanradix at gmail.com> wrote: 
>>>     Hi all,
>>>
>>>     What is the best practice for storing all those logs in one central
>>>     environment. I have one Linux Box running Syslog-NG with LogAnalyzer and
>>>     MongoDB ( for now ), and is the best way to configure and use it with
>>>     MongoDB or with MariaDB ( MySQL ) ? I have once install MySQL but it was
>>>     getting very slow as the logs getting bigger and bigger ( for one week ).
>>>     Now i have done with MongoDB ( still testing ) but i have problem as
>>>     LogAnalyzer does not show me the real pictures, i have no Date info, no
>>>     Facility, no serverity, Hosts, syslogtag, i just have ProcessID.
>>>
>>>     Any hints on this ?
>>>
>>>     I have the following configuration on the syslog-ng.cfg:
>>>
>>>     destination d_mongodb {
>>>         mongodb(
>>>         servers("localhost:27017")
>>>             database("logs")
>>>     #    uri('mongodb://localhost/syslog-ng')
>>>         collection("syslog")
>>>         value-pairs(
>>>         scope("selected-macros" "nv-pairs" "sdata")
>>>             )
>>>                 );
>>>                 };
>>>
>>>     Kind regards
>>>     Ivan
>
>
>     ______________________________________________________________________________
>     Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>     Documentation:
>     http://www.balabit.com/support/documentation/?product=syslog-ng
>     FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20160518/216d8c39/attachment.htm