[syslog-ng] Syslog-NG with MongoDB

[Fekete, Róbert]

Hi,

Do you know in what type does loganalyzer expect the specific fields?
AFAIK, by default, syslog-ng sends everything as string, but for the
mongodb destination, you can specify the data type, see
https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/specifying-data-types.html

Try sending the date as datetime, and the others as numbers, maybe it helps.

Regards,

Robert

On Wed, May 18, 2016 at 1:47 PM, Ivan Adji - Krstev <akivanradix at gmail.com>
wrote:

> Robert,
> i just thought of that and goggling how to add columns or some other
> similar scenarios, i think that the problem lays on how syslog-ng send the
> logs in the DB. Or how DB is storing this messages. As i have not configure
> nothing on the MongoDB just username and password for already created DB by
> the syslog-ng.
>
> If some one have some tips, ill be happy to try it :)
>
> Kind regards
> Ivan
>
> On 05/18/2016 01:43 PM, Fekete, Róbert wrote:
>
> Hi,
>
> can you check the mongodb itself if the related fields/tags/whatever are
> in place?
> I mean, the problem might be in how syslog-ng sends the data into MongoDB,
> or in how loganalyzer reads the data from MongoDB. Is there a way for you
> to find out which?
>
> Robert
>
>
> On Wed, May 18, 2016 at 11:04 AM, Ivan Adji - Krstev <
> akivanradix at gmail.com> wrote:
>
>> Hi Jim,
>> Thanks for the feedback.
>> The problem is that im trying to monitor big infrastructure ( 200
>> Physical servers and more than 1000 VMs ). So currently i have install with
>> MongoDB and have 300MB for one week monitoring just two VMs. The server
>> syslog-ng and one client VM. Also i have used before syslog-ng with MariaDB
>> (MySQL) but i have problem that i have 90% CPU Load when i used MySQL. I
>> can't fix it. But now using MongoDB i have other problems. Using
>> LogAnalyzer i can't see the "Date", "Facility", Serverity etc. on a main
>> page but when i go to the log itself or i open it i can see all this
>> informations. So i have the following
>>
>> 1. Syslog-NG with MySQL and LogAnalyzer ( works ok but CPU Usage was big
>> )
>> 2. Syslog-NG with MongoDB and LogAnalyzer ( works ok but no informations
>> shown on a first page )
>>
>> So i can't find solutions and i need this sh*** up and running ASAP :)
>>
>> Any solutions or suggestions im open to see it !
>>
>> Kind regards
>> Ivan
>>
>>
>> On 05/16/2016 05:43 PM, <jrhendri at roadrunner.com>jrhendri at roadrunner.com
>> wrote:
>>
>> My 2 cents (what works for you depends on your infrastructure, resources and capabilities)
>>
>> I like the model where syslog-ng does all the following:
>>
>> - writes text files of the raw data (that way - whatever your search head is can re-ingest files later using basically the same parsers)
>>
>> - filters out highly false-positive prone data from being forwarded
>>
>> - handles parsing of data elements (using patterndb or whatever) and sends specific information to a search engine (like Elasticsearch)
>>
>> - forwards specific data (based on security use cases) to a SIEM
>>
>>
>>
>> Whether you use Elasticsearch, mongo, splunk, or whatever is really up to you and your budget.
>> That said, I find syslog-ng to elasticsearch directly with kibana as the front end is *very* scalable for a search engine.
>>
>> As far as a SIEM - it's kind of up to you.
>>
>> Good luck,
>>
>> Jim
>>
>>
>> ---- Ivan Adji - Krstev <akivanradix at gmail.com> <akivanradix at gmail.com> wrote:
>>
>> Hi all,
>>
>> What is the best practice for storing all those logs in one central
>> environment. I have one Linux Box running Syslog-NG with LogAnalyzer and
>> MongoDB ( for now ), and is the best way to configure and use it with
>> MongoDB or with MariaDB ( MySQL ) ? I have once install MySQL but it was
>> getting very slow as the logs getting bigger and bigger ( for one week ).
>> Now i have done with MongoDB ( still testing ) but i have problem as
>> LogAnalyzer does not show me the real pictures, i have no Date info, no
>> Facility, no serverity, Hosts, syslogtag, i just have ProcessID.
>>
>> Any hints on this ?
>>
>> I have the following configuration on the syslog-ng.cfg:
>>
>> destination d_mongodb {
>>     mongodb(
>>     servers("localhost:27017")
>>         database("logs")
>> #    uri('mongodb://localhost/syslog-ng')
>>     collection("syslog")
>>     value-pairs(
>>     scope("selected-macros" "nv-pairs" "sdata")
>>         )
>>             );
>>             };
>>
>> Kind regards
>> Ivan
>>
>>
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20160518/a5f7399f/attachment-0001.htm