[syslog-ng] Syslog-NG with MongoDB
Ivan Adji - Krstev
akivanradix at gmail.com
Tue May 24 17:00:58 CEST 2016
Hi Bazsi,
I saw the Elastic and hdfs and for me they are all the same :). I'm
trying to build something free, from scratch. And I'm already done using
Syslog-NG with MongoDB and all that to be shown on LogAnalyzer. So we
will see how things will go.
Once again thank you for the support !
Kind regards
Ivan
On 05/24/2016 08:29 AM, Scheidler, Balázs wrote:
>
> Elastic or hdfs are horizontally scalable and depend what you want to
> do with the logs. These are horizontally scalable, but their per node
> performance needs to be taken into account.
>
> Stuff like Balabit syslog-ng store box can consume lots of data on a
> single node or a couple of nodes but that's commercial.
>
> I'd experiment with elastic search and kibana if I were you, or of
> course I can connect you someone about Balabit ssb.
>
> Bazsi
>
> On May 19, 2016 10:09 AM, "Ivan Adji - Krstev" <akivanradix at gmail.com
> <mailto:akivanradix at gmail.com>> wrote:
>
> Hi Richie,
>
> I'll do that too, the think that scares me is that I'm running
> maybe two weeks this syslog-ng and i have 400 MB of logs and im
> logging just 2 Machines, the Server itself and on Client.
> And im planning to loging more than 1500 machines. So im not sure
> what to choose. MongoDB, MySQL, or PostgreSQL.
>
> And for now i have problems with all of them :)
>
>
> Kind regards
> Ivan
>
> On 05/19/2016 08:07 AM, Richárd Réfi wrote:
>>
>> Hi,
>>
>> I would try mysql/mariadb tweaks also:
>> - index(or indices) on one or more column(s) of the mysql table
>> according to the queries of loganalyzer
>> - check the different cache and buffer options in your mysql conf
>> - my opinion and experience is that mysql partitioning can do
>> magic on this amount of data. A query could run only on few gigs
>> portion of data (and a good indexing accelerates the query of
>> these few gigs also).
>>
>> Unfortunately your mongodb problem remains open.
>>
>> Regards, Richie
>>
>>
>>
>> On Wed, May 18, 2016, 14:37 Ivan Adji - Krstev
>> <akivanradix at gmail.com <mailto:akivanradix at gmail.com>> wrote:
>>
>> Nop,
>> Again same problem:
>> Here is what i have done
>>
>>
>> destination d_mongodb {
>> mongodb(
>> servers("localhost:27017")
>> database("syslog")
>> username("Ivan")
>> password("Ivan123")
>> collection("messages")
>>
>> value-pairs(
>> scope("selected-macros" "nv-pairs" "sdata")
>> pair("date", datetime("$UNIXTIME"))
>> pair("pid", int64("$PID"))
>> pair("program", "$PROGRAM")
>> pair("message", "$MESSAGE")
>> )
>> );
>> };
>> Still have the same problem no info on date no nothing. The
>> strange part is that when i open a specific log i have all
>> the info. The only problem is where on a first page on the
>> LogAnalyzer i don't have this problems.
>>
>> And yes again we may have two problems: One is the DB
>> information how its stored, and by this i think we stored as
>> we should, but do i have to configure some tables in the
>> MongoDB or columns or something ( that is how i did it with
>> MySQL ). The second is something wrong with the LogAnalyzer
>> so now im going to reconfigure with PostgreSQL and again with
>> MySQL to see if something will be change.
>>
>>
>>
>> Ivan
>>
>> On 05/18/2016 01:53 PM, Fekete, Róbert wrote:
>>> Hi,
>>>
>>> Do you know in what type does loganalyzer expect the
>>> specific fields?
>>> AFAIK, by default, syslog-ng sends everything as string, but
>>> for the mongodb destination, you can specify the data type,
>>> see https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/specifying-data-types.html
>>>
>>> Try sending the date as datetime, and the others as numbers,
>>> maybe it helps.
>>>
>>> Regards,
>>>
>>> Robert
>>>
>>> On Wed, May 18, 2016 at 1:47 PM, Ivan Adji - Krstev
>>> <akivanradix at gmail.com <mailto:akivanradix at gmail.com>> wrote:
>>>
>>> Robert,
>>> i just thought of that and goggling how to add columns
>>> or some other similar scenarios, i think that the
>>> problem lays on how syslog-ng send the logs in the DB.
>>> Or how DB is storing this messages. As i have not
>>> configure nothing on the MongoDB just username and
>>> password for already created DB by the syslog-ng.
>>>
>>> If some one have some tips, ill be happy to try it :)
>>>
>>> Kind regards
>>> Ivan
>>>
>>> On 05/18/2016 01:43 PM, Fekete, Róbert wrote:
>>>> Hi,
>>>>
>>>> can you check the mongodb itself if the related
>>>> fields/tags/whatever are in place?
>>>> I mean, the problem might be in how syslog-ng sends the
>>>> data into MongoDB, or in how loganalyzer reads the data
>>>> from MongoDB. Is there a way for you to find out which?
>>>>
>>>> Robert
>>>>
>>>>
>>>> On Wed, May 18, 2016 at 11:04 AM, Ivan Adji - Krstev
>>>> <akivanradix at gmail.com <mailto:akivanradix at gmail.com>>
>>>> wrote:
>>>>
>>>> Hi Jim,
>>>> Thanks for the feedback.
>>>> The problem is that im trying to monitor big
>>>> infrastructure ( 200 Physical servers and more than
>>>> 1000 VMs ). So currently i have install with
>>>> MongoDB and have 300MB for one week monitoring just
>>>> two VMs. The server syslog-ng and one client VM.
>>>> Also i have used before syslog-ng with MariaDB
>>>> (MySQL) but i have problem that i have 90% CPU Load
>>>> when i used MySQL. I can't fix it. But now using
>>>> MongoDB i have other problems. Using LogAnalyzer i
>>>> can't see the "Date", "Facility", Serverity etc. on
>>>> a main page but when i go to the log itself or i
>>>> open it i can see all this informations. So i have
>>>> the following
>>>>
>>>> 1. Syslog-NG with MySQL and LogAnalyzer ( works ok
>>>> but CPU Usage was big )
>>>> 2. Syslog-NG with MongoDB and LogAnalyzer ( works
>>>> ok but no informations shown on a first page )
>>>>
>>>> So i can't find solutions and i need this sh*** up
>>>> and running ASAP :)
>>>>
>>>> Any solutions or suggestions im open to see it !
>>>>
>>>> Kind regards
>>>> Ivan
>>>>
>>>>
>>>> On 05/16/2016 05:43 PM, jrhendri at roadrunner.com
>>>> <mailto:jrhendri at roadrunner.com> wrote:
>>>>> My 2 cents (what works for you depends on your infrastructure, resources and capabilities)
>>>>>
>>>>> I like the model where syslog-ng does all the following:
>>>>>
>>>>> - writes text files of the raw data (that way - whatever your search head is can re-ingest files later using basically the same parsers)
>>>>>
>>>>> - filters out highly false-positive prone data from being forwarded
>>>>>
>>>>> - handles parsing of data elements (using patterndb or whatever) and sends specific information to a search engine (like Elasticsearch)
>>>>>
>>>>> - forwards specific data (based on security use cases) to a SIEM
>>>>>
>>>>>
>>>>>
>>>>> Whether you use Elasticsearch, mongo, splunk, or whatever is really up to you and your budget.
>>>>> That said, I find syslog-ng to elasticsearch directly with kibana as the front end is *very* scalable for a search engine.
>>>>>
>>>>> As far as a SIEM - it's kind of up to you.
>>>>>
>>>>> Good luck,
>>>>>
>>>>> Jim
>>>>>
>>>>>
>>>>> ---- Ivan Adji - Krstev <akivanradix at gmail.com> <mailto:akivanradix at gmail.com> wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> What is the best practice for storing all those logs in one central
>>>>>> environment. I have one Linux Box running Syslog-NG with LogAnalyzer and
>>>>>> MongoDB ( for now ), and is the best way to configure and use it with
>>>>>> MongoDB or with MariaDB ( MySQL ) ? I have once install MySQL but it was
>>>>>> getting very slow as the logs getting bigger and bigger ( for one week ).
>>>>>> Now i have done with MongoDB ( still testing ) but i have problem as
>>>>>> LogAnalyzer does not show me the real pictures, i have no Date info, no
>>>>>> Facility, no serverity, Hosts, syslogtag, i just have ProcessID.
>>>>>>
>>>>>> Any hints on this ?
>>>>>>
>>>>>> I have the following configuration on the syslog-ng.cfg:
>>>>>>
>>>>>> destination d_mongodb {
>>>>>> mongodb(
>>>>>> servers("localhost:27017")
>>>>>> database("logs")
>>>>>> # uri('mongodb://localhost/syslog-ng')
>>>>>> collection("syslog")
>>>>>> value-pairs(
>>>>>> scope("selected-macros" "nv-pairs" "sdata")
>>>>>> )
>>>>>> );
>>>>>> };
>>>>>>
>>>>>> Kind regards
>>>>>> Ivan
>>>>
>>>>
>>>> ______________________________________________________________________________
>>>> Member info:
>>>> https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation:
>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ______________________________________________________________________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>
>>>
>>> ______________________________________________________________________________
>>> Member info:
>>> https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20160524/08eaf9d5/attachment-0001.htm
More information about the syslog-ng
mailing list