[syslog-ng] Syslog-NG with MongoDB
Ivan Adji - Krstev
akivanradix at gmail.com
Thu May 19 10:09:46 CEST 2016
Hi Richie,
I'll do that too, the think that scares me is that I'm running maybe two
weeks this syslog-ng and i have 400 MB of logs and im logging just 2
Machines, the Server itself and on Client.
And im planning to loging more than 1500 machines. So im not sure what
to choose. MongoDB, MySQL, or PostgreSQL.
And for now i have problems with all of them :)
Kind regards
Ivan
On 05/19/2016 08:07 AM, Richárd Réfi wrote:
>
> Hi,
>
> I would try mysql/mariadb tweaks also:
> - index(or indices) on one or more column(s) of the mysql table
> according to the queries of loganalyzer
> - check the different cache and buffer options in your mysql conf
> - my opinion and experience is that mysql partitioning can do magic on
> this amount of data. A query could run only on few gigs portion of
> data (and a good indexing accelerates the query of these few gigs also).
>
> Unfortunately your mongodb problem remains open.
>
> Regards, Richie
>
>
>
> On Wed, May 18, 2016, 14:37 Ivan Adji - Krstev <akivanradix at gmail.com
> <mailto:akivanradix at gmail.com>> wrote:
>
> Nop,
> Again same problem:
> Here is what i have done
>
>
> destination d_mongodb {
> mongodb(
> servers("localhost:27017")
> database("syslog")
> username("Ivan")
> password("Ivan123")
> collection("messages")
>
> value-pairs(
> scope("selected-macros" "nv-pairs" "sdata")
> pair("date", datetime("$UNIXTIME"))
> pair("pid", int64("$PID"))
> pair("program", "$PROGRAM")
> pair("message", "$MESSAGE")
> )
> );
> };
> Still have the same problem no info on date no nothing. The
> strange part is that when i open a specific log i have all the
> info. The only problem is where on a first page on the LogAnalyzer
> i don't have this problems.
>
> And yes again we may have two problems: One is the DB information
> how its stored, and by this i think we stored as we should, but do
> i have to configure some tables in the MongoDB or columns or
> something ( that is how i did it with MySQL ). The second is
> something wrong with the LogAnalyzer so now im going to
> reconfigure with PostgreSQL and again with MySQL to see if
> something will be change.
>
>
>
> Ivan
>
> On 05/18/2016 01:53 PM, Fekete, Róbert wrote:
>> Hi,
>>
>> Do you know in what type does loganalyzer expect the specific
>> fields?
>> AFAIK, by default, syslog-ng sends everything as string, but for
>> the mongodb destination, you can specify the data type,
>> see https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/specifying-data-types.html
>>
>> Try sending the date as datetime, and the others as numbers,
>> maybe it helps.
>>
>> Regards,
>>
>> Robert
>>
>> On Wed, May 18, 2016 at 1:47 PM, Ivan Adji - Krstev
>> <akivanradix at gmail.com <mailto:akivanradix at gmail.com>> wrote:
>>
>> Robert,
>> i just thought of that and goggling how to add columns or
>> some other similar scenarios, i think that the problem lays
>> on how syslog-ng send the logs in the DB. Or how DB is
>> storing this messages. As i have not configure nothing on the
>> MongoDB just username and password for already created DB by
>> the syslog-ng.
>>
>> If some one have some tips, ill be happy to try it :)
>>
>> Kind regards
>> Ivan
>>
>> On 05/18/2016 01:43 PM, Fekete, Róbert wrote:
>>> Hi,
>>>
>>> can you check the mongodb itself if the related
>>> fields/tags/whatever are in place?
>>> I mean, the problem might be in how syslog-ng sends the data
>>> into MongoDB, or in how loganalyzer reads the data from
>>> MongoDB. Is there a way for you to find out which?
>>>
>>> Robert
>>>
>>>
>>> On Wed, May 18, 2016 at 11:04 AM, Ivan Adji - Krstev
>>> <akivanradix at gmail.com <mailto:akivanradix at gmail.com>> wrote:
>>>
>>> Hi Jim,
>>> Thanks for the feedback.
>>> The problem is that im trying to monitor big
>>> infrastructure ( 200 Physical servers and more than 1000
>>> VMs ). So currently i have install with MongoDB and have
>>> 300MB for one week monitoring just two VMs. The server
>>> syslog-ng and one client VM. Also i have used before
>>> syslog-ng with MariaDB (MySQL) but i have problem that i
>>> have 90% CPU Load when i used MySQL. I can't fix it. But
>>> now using MongoDB i have other problems. Using
>>> LogAnalyzer i can't see the "Date", "Facility",
>>> Serverity etc. on a main page but when i go to the log
>>> itself or i open it i can see all this informations. So
>>> i have the following
>>>
>>> 1. Syslog-NG with MySQL and LogAnalyzer ( works ok but
>>> CPU Usage was big )
>>> 2. Syslog-NG with MongoDB and LogAnalyzer ( works ok but
>>> no informations shown on a first page )
>>>
>>> So i can't find solutions and i need this sh*** up and
>>> running ASAP :)
>>>
>>> Any solutions or suggestions im open to see it !
>>>
>>> Kind regards
>>> Ivan
>>>
>>>
>>> On 05/16/2016 05:43 PM, jrhendri at roadrunner.com
>>> <mailto:jrhendri at roadrunner.com> wrote:
>>>> My 2 cents (what works for you depends on your infrastructure, resources and capabilities)
>>>>
>>>> I like the model where syslog-ng does all the following:
>>>>
>>>> - writes text files of the raw data (that way - whatever your search head is can re-ingest files later using basically the same parsers)
>>>>
>>>> - filters out highly false-positive prone data from being forwarded
>>>>
>>>> - handles parsing of data elements (using patterndb or whatever) and sends specific information to a search engine (like Elasticsearch)
>>>>
>>>> - forwards specific data (based on security use cases) to a SIEM
>>>>
>>>>
>>>>
>>>> Whether you use Elasticsearch, mongo, splunk, or whatever is really up to you and your budget.
>>>> That said, I find syslog-ng to elasticsearch directly with kibana as the front end is *very* scalable for a search engine.
>>>>
>>>> As far as a SIEM - it's kind of up to you.
>>>>
>>>> Good luck,
>>>>
>>>> Jim
>>>>
>>>>
>>>> ---- Ivan Adji - Krstev <akivanradix at gmail.com> <mailto:akivanradix at gmail.com> wrote:
>>>>> Hi all,
>>>>>
>>>>> What is the best practice for storing all those logs in one central
>>>>> environment. I have one Linux Box running Syslog-NG with LogAnalyzer and
>>>>> MongoDB ( for now ), and is the best way to configure and use it with
>>>>> MongoDB or with MariaDB ( MySQL ) ? I have once install MySQL but it was
>>>>> getting very slow as the logs getting bigger and bigger ( for one week ).
>>>>> Now i have done with MongoDB ( still testing ) but i have problem as
>>>>> LogAnalyzer does not show me the real pictures, i have no Date info, no
>>>>> Facility, no serverity, Hosts, syslogtag, i just have ProcessID.
>>>>>
>>>>> Any hints on this ?
>>>>>
>>>>> I have the following configuration on the syslog-ng.cfg:
>>>>>
>>>>> destination d_mongodb {
>>>>> mongodb(
>>>>> servers("localhost:27017")
>>>>> database("logs")
>>>>> # uri('mongodb://localhost/syslog-ng')
>>>>> collection("syslog")
>>>>> value-pairs(
>>>>> scope("selected-macros" "nv-pairs" "sdata")
>>>>> )
>>>>> );
>>>>> };
>>>>>
>>>>> Kind regards
>>>>> Ivan
>>>
>>>
>>> ______________________________________________________________________________
>>> Member info:
>>> https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20160519/96507bf5/attachment-0001.htm
More information about the syslog-ng
mailing list