Hi all, What is the best practice for storing all those logs in one central environment. I have one Linux Box running Syslog-NG with LogAnalyzer and MongoDB ( for now ), and is the best way to configure and use it with MongoDB or with MariaDB ( MySQL ) ? I have once install MySQL but it was getting very slow as the logs getting bigger and bigger ( for one week ). Now i have done with MongoDB ( still testing ) but i have problem as LogAnalyzer does not show me the real pictures, i have no Date info, no Facility, no serverity, Hosts, syslogtag, i just have ProcessID. Any hints on this ? I have the following configuration on the syslog-ng.cfg: destination d_mongodb { mongodb( servers("localhost:27017") database("logs") # uri('mongodb://localhost/syslog-ng') collection("syslog") value-pairs( scope("selected-macros" "nv-pairs" "sdata") ) ); }; Kind regards Ivan
My 2 cents (what works for you depends on your infrastructure, resources and capabilities) I like the model where syslog-ng does all the following: - writes text files of the raw data (that way - whatever your search head is can re-ingest files later using basically the same parsers) - filters out highly false-positive prone data from being forwarded - handles parsing of data elements (using patterndb or whatever) and sends specific information to a search engine (like Elasticsearch) - forwards specific data (based on security use cases) to a SIEM Whether you use Elasticsearch, mongo, splunk, or whatever is really up to you and your budget. That said, I find syslog-ng to elasticsearch directly with kibana as the front end is *very* scalable for a search engine. As far as a SIEM - it's kind of up to you. Good luck, Jim ---- Ivan Adji - Krstev <akivanradix@gmail.com> wrote:
Hi all,
What is the best practice for storing all those logs in one central environment. I have one Linux Box running Syslog-NG with LogAnalyzer and MongoDB ( for now ), and is the best way to configure and use it with MongoDB or with MariaDB ( MySQL ) ? I have once install MySQL but it was getting very slow as the logs getting bigger and bigger ( for one week ). Now i have done with MongoDB ( still testing ) but i have problem as LogAnalyzer does not show me the real pictures, i have no Date info, no Facility, no serverity, Hosts, syslogtag, i just have ProcessID.
Any hints on this ?
I have the following configuration on the syslog-ng.cfg:
destination d_mongodb { mongodb( servers("localhost:27017") database("logs") # uri('mongodb://localhost/syslog-ng') collection("syslog") value-pairs( scope("selected-macros" "nv-pairs" "sdata") ) ); };
Kind regards Ivan
Hi Jim, Thanks for the feedback. The problem is that im trying to monitor big infrastructure ( 200 Physical servers and more than 1000 VMs ). So currently i have install with MongoDB and have 300MB for one week monitoring just two VMs. The server syslog-ng and one client VM. Also i have used before syslog-ng with MariaDB (MySQL) but i have problem that i have 90% CPU Load when i used MySQL. I can't fix it. But now using MongoDB i have other problems. Using LogAnalyzer i can't see the "Date", "Facility", Serverity etc. on a main page but when i go to the log itself or i open it i can see all this informations. So i have the following 1. Syslog-NG with MySQL and LogAnalyzer ( works ok but CPU Usage was big ) 2. Syslog-NG with MongoDB and LogAnalyzer ( works ok but no informations shown on a first page ) So i can't find solutions and i need this sh*** up and running ASAP :) Any solutions or suggestions im open to see it ! Kind regards Ivan On 05/16/2016 05:43 PM, jrhendri@roadrunner.com wrote:
My 2 cents (what works for you depends on your infrastructure, resources and capabilities)
I like the model where syslog-ng does all the following:
- writes text files of the raw data (that way - whatever your search head is can re-ingest files later using basically the same parsers)
- filters out highly false-positive prone data from being forwarded
- handles parsing of data elements (using patterndb or whatever) and sends specific information to a search engine (like Elasticsearch)
- forwards specific data (based on security use cases) to a SIEM
Whether you use Elasticsearch, mongo, splunk, or whatever is really up to you and your budget. That said, I find syslog-ng to elasticsearch directly with kibana as the front end is *very* scalable for a search engine.
As far as a SIEM - it's kind of up to you.
Good luck,
Jim
---- Ivan Adji - Krstev <akivanradix@gmail.com> wrote:
Hi all,
What is the best practice for storing all those logs in one central environment. I have one Linux Box running Syslog-NG with LogAnalyzer and MongoDB ( for now ), and is the best way to configure and use it with MongoDB or with MariaDB ( MySQL ) ? I have once install MySQL but it was getting very slow as the logs getting bigger and bigger ( for one week ). Now i have done with MongoDB ( still testing ) but i have problem as LogAnalyzer does not show me the real pictures, i have no Date info, no Facility, no serverity, Hosts, syslogtag, i just have ProcessID.
Any hints on this ?
I have the following configuration on the syslog-ng.cfg:
destination d_mongodb { mongodb( servers("localhost:27017") database("logs") # uri('mongodb://localhost/syslog-ng') collection("syslog") value-pairs( scope("selected-macros" "nv-pairs" "sdata") ) ); };
Kind regards Ivan
Hi, can you check the mongodb itself if the related fields/tags/whatever are in place? I mean, the problem might be in how syslog-ng sends the data into MongoDB, or in how loganalyzer reads the data from MongoDB. Is there a way for you to find out which? Robert On Wed, May 18, 2016 at 11:04 AM, Ivan Adji - Krstev <akivanradix@gmail.com> wrote:
Hi Jim, Thanks for the feedback. The problem is that im trying to monitor big infrastructure ( 200 Physical servers and more than 1000 VMs ). So currently i have install with MongoDB and have 300MB for one week monitoring just two VMs. The server syslog-ng and one client VM. Also i have used before syslog-ng with MariaDB (MySQL) but i have problem that i have 90% CPU Load when i used MySQL. I can't fix it. But now using MongoDB i have other problems. Using LogAnalyzer i can't see the "Date", "Facility", Serverity etc. on a main page but when i go to the log itself or i open it i can see all this informations. So i have the following
1. Syslog-NG with MySQL and LogAnalyzer ( works ok but CPU Usage was big ) 2. Syslog-NG with MongoDB and LogAnalyzer ( works ok but no informations shown on a first page )
So i can't find solutions and i need this sh*** up and running ASAP :)
Any solutions or suggestions im open to see it !
Kind regards Ivan
On 05/16/2016 05:43 PM, jrhendri@roadrunner.com wrote:
My 2 cents (what works for you depends on your infrastructure, resources and capabilities)
I like the model where syslog-ng does all the following:
- writes text files of the raw data (that way - whatever your search head is can re-ingest files later using basically the same parsers)
- filters out highly false-positive prone data from being forwarded
- handles parsing of data elements (using patterndb or whatever) and sends specific information to a search engine (like Elasticsearch)
- forwards specific data (based on security use cases) to a SIEM
Whether you use Elasticsearch, mongo, splunk, or whatever is really up to you and your budget. That said, I find syslog-ng to elasticsearch directly with kibana as the front end is *very* scalable for a search engine.
As far as a SIEM - it's kind of up to you.
Good luck,
Jim
---- Ivan Adji - Krstev <akivanradix@gmail.com> <akivanradix@gmail.com> wrote:
Hi all,
What is the best practice for storing all those logs in one central environment. I have one Linux Box running Syslog-NG with LogAnalyzer and MongoDB ( for now ), and is the best way to configure and use it with MongoDB or with MariaDB ( MySQL ) ? I have once install MySQL but it was getting very slow as the logs getting bigger and bigger ( for one week ). Now i have done with MongoDB ( still testing ) but i have problem as LogAnalyzer does not show me the real pictures, i have no Date info, no Facility, no serverity, Hosts, syslogtag, i just have ProcessID.
Any hints on this ?
I have the following configuration on the syslog-ng.cfg:
destination d_mongodb { mongodb( servers("localhost:27017") database("logs") # uri('mongodb://localhost/syslog-ng') collection("syslog") value-pairs( scope("selected-macros" "nv-pairs" "sdata") ) ); };
Kind regards Ivan
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Robert, i just thought of that and goggling how to add columns or some other similar scenarios, i think that the problem lays on how syslog-ng send the logs in the DB. Or how DB is storing this messages. As i have not configure nothing on the MongoDB just username and password for already created DB by the syslog-ng. If some one have some tips, ill be happy to try it :) Kind regards Ivan On 05/18/2016 01:43 PM, Fekete, Róbert wrote:
Hi,
can you check the mongodb itself if the related fields/tags/whatever are in place? I mean, the problem might be in how syslog-ng sends the data into MongoDB, or in how loganalyzer reads the data from MongoDB. Is there a way for you to find out which?
Robert
On Wed, May 18, 2016 at 11:04 AM, Ivan Adji - Krstev <akivanradix@gmail.com <mailto:akivanradix@gmail.com>> wrote:
Hi Jim, Thanks for the feedback. The problem is that im trying to monitor big infrastructure ( 200 Physical servers and more than 1000 VMs ). So currently i have install with MongoDB and have 300MB for one week monitoring just two VMs. The server syslog-ng and one client VM. Also i have used before syslog-ng with MariaDB (MySQL) but i have problem that i have 90% CPU Load when i used MySQL. I can't fix it. But now using MongoDB i have other problems. Using LogAnalyzer i can't see the "Date", "Facility", Serverity etc. on a main page but when i go to the log itself or i open it i can see all this informations. So i have the following
1. Syslog-NG with MySQL and LogAnalyzer ( works ok but CPU Usage was big ) 2. Syslog-NG with MongoDB and LogAnalyzer ( works ok but no informations shown on a first page )
So i can't find solutions and i need this sh*** up and running ASAP :)
Any solutions or suggestions im open to see it !
Kind regards Ivan
On 05/16/2016 05:43 PM, jrhendri@roadrunner.com <mailto:jrhendri@roadrunner.com> wrote:
My 2 cents (what works for you depends on your infrastructure, resources and capabilities)
I like the model where syslog-ng does all the following:
- writes text files of the raw data (that way - whatever your search head is can re-ingest files later using basically the same parsers)
- filters out highly false-positive prone data from being forwarded
- handles parsing of data elements (using patterndb or whatever) and sends specific information to a search engine (like Elasticsearch)
- forwards specific data (based on security use cases) to a SIEM
Whether you use Elasticsearch, mongo, splunk, or whatever is really up to you and your budget. That said, I find syslog-ng to elasticsearch directly with kibana as the front end is *very* scalable for a search engine.
As far as a SIEM - it's kind of up to you.
Good luck,
Jim
---- Ivan Adji - Krstev <akivanradix@gmail.com> <mailto:akivanradix@gmail.com> wrote:
Hi all,
What is the best practice for storing all those logs in one central environment. I have one Linux Box running Syslog-NG with LogAnalyzer and MongoDB ( for now ), and is the best way to configure and use it with MongoDB or with MariaDB ( MySQL ) ? I have once install MySQL but it was getting very slow as the logs getting bigger and bigger ( for one week ). Now i have done with MongoDB ( still testing ) but i have problem as LogAnalyzer does not show me the real pictures, i have no Date info, no Facility, no serverity, Hosts, syslogtag, i just have ProcessID.
Any hints on this ?
I have the following configuration on the syslog-ng.cfg:
destination d_mongodb { mongodb( servers("localhost:27017") database("logs") # uri('mongodb://localhost/syslog-ng') collection("syslog") value-pairs( scope("selected-macros" "nv-pairs" "sdata") ) ); };
Kind regards Ivan
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, Do you know in what type does loganalyzer expect the specific fields? AFAIK, by default, syslog-ng sends everything as string, but for the mongodb destination, you can specify the data type, see https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-g... Try sending the date as datetime, and the others as numbers, maybe it helps. Regards, Robert On Wed, May 18, 2016 at 1:47 PM, Ivan Adji - Krstev <akivanradix@gmail.com> wrote:
Robert, i just thought of that and goggling how to add columns or some other similar scenarios, i think that the problem lays on how syslog-ng send the logs in the DB. Or how DB is storing this messages. As i have not configure nothing on the MongoDB just username and password for already created DB by the syslog-ng.
If some one have some tips, ill be happy to try it :)
Kind regards Ivan
On 05/18/2016 01:43 PM, Fekete, Róbert wrote:
Hi,
can you check the mongodb itself if the related fields/tags/whatever are in place? I mean, the problem might be in how syslog-ng sends the data into MongoDB, or in how loganalyzer reads the data from MongoDB. Is there a way for you to find out which?
Robert
On Wed, May 18, 2016 at 11:04 AM, Ivan Adji - Krstev < akivanradix@gmail.com> wrote:
Hi Jim, Thanks for the feedback. The problem is that im trying to monitor big infrastructure ( 200 Physical servers and more than 1000 VMs ). So currently i have install with MongoDB and have 300MB for one week monitoring just two VMs. The server syslog-ng and one client VM. Also i have used before syslog-ng with MariaDB (MySQL) but i have problem that i have 90% CPU Load when i used MySQL. I can't fix it. But now using MongoDB i have other problems. Using LogAnalyzer i can't see the "Date", "Facility", Serverity etc. on a main page but when i go to the log itself or i open it i can see all this informations. So i have the following
1. Syslog-NG with MySQL and LogAnalyzer ( works ok but CPU Usage was big ) 2. Syslog-NG with MongoDB and LogAnalyzer ( works ok but no informations shown on a first page )
So i can't find solutions and i need this sh*** up and running ASAP :)
Any solutions or suggestions im open to see it !
Kind regards Ivan
On 05/16/2016 05:43 PM, <jrhendri@roadrunner.com>jrhendri@roadrunner.com wrote:
My 2 cents (what works for you depends on your infrastructure, resources and capabilities)
I like the model where syslog-ng does all the following:
- writes text files of the raw data (that way - whatever your search head is can re-ingest files later using basically the same parsers)
- filters out highly false-positive prone data from being forwarded
- handles parsing of data elements (using patterndb or whatever) and sends specific information to a search engine (like Elasticsearch)
- forwards specific data (based on security use cases) to a SIEM
Whether you use Elasticsearch, mongo, splunk, or whatever is really up to you and your budget. That said, I find syslog-ng to elasticsearch directly with kibana as the front end is *very* scalable for a search engine.
As far as a SIEM - it's kind of up to you.
Good luck,
Jim
---- Ivan Adji - Krstev <akivanradix@gmail.com> <akivanradix@gmail.com> wrote:
Hi all,
What is the best practice for storing all those logs in one central environment. I have one Linux Box running Syslog-NG with LogAnalyzer and MongoDB ( for now ), and is the best way to configure and use it with MongoDB or with MariaDB ( MySQL ) ? I have once install MySQL but it was getting very slow as the logs getting bigger and bigger ( for one week ). Now i have done with MongoDB ( still testing ) but i have problem as LogAnalyzer does not show me the real pictures, i have no Date info, no Facility, no serverity, Hosts, syslogtag, i just have ProcessID.
Any hints on this ?
I have the following configuration on the syslog-ng.cfg:
destination d_mongodb { mongodb( servers("localhost:27017") database("logs") # uri('mongodb://localhost/syslog-ng') collection("syslog") value-pairs( scope("selected-macros" "nv-pairs" "sdata") ) ); };
Kind regards Ivan
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Nop, Again same problem: Here is what i have done destination d_mongodb { mongodb( servers("localhost:27017") database("syslog") username("Ivan") password("Ivan123") collection("messages") value-pairs( scope("selected-macros" "nv-pairs" "sdata") pair("date", datetime("$UNIXTIME")) pair("pid", int64("$PID")) pair("program", "$PROGRAM") pair("message", "$MESSAGE") ) ); }; Still have the same problem no info on date no nothing. The strange part is that when i open a specific log i have all the info. The only problem is where on a first page on the LogAnalyzer i don't have this problems. And yes again we may have two problems: One is the DB information how its stored, and by this i think we stored as we should, but do i have to configure some tables in the MongoDB or columns or something ( that is how i did it with MySQL ). The second is something wrong with the LogAnalyzer so now im going to reconfigure with PostgreSQL and again with MySQL to see if something will be change. Ivan On 05/18/2016 01:53 PM, Fekete, Róbert wrote:
Hi,
Do you know in what type does loganalyzer expect the specific fields? AFAIK, by default, syslog-ng sends everything as string, but for the mongodb destination, you can specify the data type, see https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-g...
Try sending the date as datetime, and the others as numbers, maybe it helps.
Regards,
Robert
On Wed, May 18, 2016 at 1:47 PM, Ivan Adji - Krstev <akivanradix@gmail.com <mailto:akivanradix@gmail.com>> wrote:
Robert, i just thought of that and goggling how to add columns or some other similar scenarios, i think that the problem lays on how syslog-ng send the logs in the DB. Or how DB is storing this messages. As i have not configure nothing on the MongoDB just username and password for already created DB by the syslog-ng.
If some one have some tips, ill be happy to try it :)
Kind regards Ivan
On 05/18/2016 01:43 PM, Fekete, Róbert wrote:
Hi,
can you check the mongodb itself if the related fields/tags/whatever are in place? I mean, the problem might be in how syslog-ng sends the data into MongoDB, or in how loganalyzer reads the data from MongoDB. Is there a way for you to find out which?
Robert
On Wed, May 18, 2016 at 11:04 AM, Ivan Adji - Krstev <akivanradix@gmail.com <mailto:akivanradix@gmail.com>> wrote:
Hi Jim, Thanks for the feedback. The problem is that im trying to monitor big infrastructure ( 200 Physical servers and more than 1000 VMs ). So currently i have install with MongoDB and have 300MB for one week monitoring just two VMs. The server syslog-ng and one client VM. Also i have used before syslog-ng with MariaDB (MySQL) but i have problem that i have 90% CPU Load when i used MySQL. I can't fix it. But now using MongoDB i have other problems. Using LogAnalyzer i can't see the "Date", "Facility", Serverity etc. on a main page but when i go to the log itself or i open it i can see all this informations. So i have the following
1. Syslog-NG with MySQL and LogAnalyzer ( works ok but CPU Usage was big ) 2. Syslog-NG with MongoDB and LogAnalyzer ( works ok but no informations shown on a first page )
So i can't find solutions and i need this sh*** up and running ASAP :)
Any solutions or suggestions im open to see it !
Kind regards Ivan
On 05/16/2016 05:43 PM, jrhendri@roadrunner.com <mailto:jrhendri@roadrunner.com> wrote:
My 2 cents (what works for you depends on your infrastructure, resources and capabilities)
I like the model where syslog-ng does all the following:
- writes text files of the raw data (that way - whatever your search head is can re-ingest files later using basically the same parsers)
- filters out highly false-positive prone data from being forwarded
- handles parsing of data elements (using patterndb or whatever) and sends specific information to a search engine (like Elasticsearch)
- forwards specific data (based on security use cases) to a SIEM
Whether you use Elasticsearch, mongo, splunk, or whatever is really up to you and your budget. That said, I find syslog-ng to elasticsearch directly with kibana as the front end is *very* scalable for a search engine.
As far as a SIEM - it's kind of up to you.
Good luck,
Jim
---- Ivan Adji - Krstev <akivanradix@gmail.com> <mailto:akivanradix@gmail.com> wrote:
Hi all,
What is the best practice for storing all those logs in one central environment. I have one Linux Box running Syslog-NG with LogAnalyzer and MongoDB ( for now ), and is the best way to configure and use it with MongoDB or with MariaDB ( MySQL ) ? I have once install MySQL but it was getting very slow as the logs getting bigger and bigger ( for one week ). Now i have done with MongoDB ( still testing ) but i have problem as LogAnalyzer does not show me the real pictures, i have no Date info, no Facility, no serverity, Hosts, syslogtag, i just have ProcessID.
Any hints on this ?
I have the following configuration on the syslog-ng.cfg:
destination d_mongodb { mongodb( servers("localhost:27017") database("logs") # uri('mongodb://localhost/syslog-ng') collection("syslog") value-pairs( scope("selected-macros" "nv-pairs" "sdata") ) ); };
Kind regards Ivan
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, I would try mysql/mariadb tweaks also: - index(or indices) on one or more column(s) of the mysql table according to the queries of loganalyzer - check the different cache and buffer options in your mysql conf - my opinion and experience is that mysql partitioning can do magic on this amount of data. A query could run only on few gigs portion of data (and a good indexing accelerates the query of these few gigs also). Unfortunately your mongodb problem remains open. Regards, Richie On Wed, May 18, 2016, 14:37 Ivan Adji - Krstev <akivanradix@gmail.com> wrote:
Nop, Again same problem: Here is what i have done
destination d_mongodb { mongodb( servers("localhost:27017") database("syslog") username("Ivan") password("Ivan123") collection("messages")
value-pairs( scope("selected-macros" "nv-pairs" "sdata") pair("date", datetime("$UNIXTIME")) pair("pid", int64("$PID")) pair("program", "$PROGRAM") pair("message", "$MESSAGE") ) ); }; Still have the same problem no info on date no nothing. The strange part is that when i open a specific log i have all the info. The only problem is where on a first page on the LogAnalyzer i don't have this problems.
And yes again we may have two problems: One is the DB information how its stored, and by this i think we stored as we should, but do i have to configure some tables in the MongoDB or columns or something ( that is how i did it with MySQL ). The second is something wrong with the LogAnalyzer so now im going to reconfigure with PostgreSQL and again with MySQL to see if something will be change.
Ivan
On 05/18/2016 01:53 PM, Fekete, Róbert wrote:
Hi,
Do you know in what type does loganalyzer expect the specific fields? AFAIK, by default, syslog-ng sends everything as string, but for the mongodb destination, you can specify the data type, see https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-g...
Try sending the date as datetime, and the others as numbers, maybe it helps.
Regards,
Robert
On Wed, May 18, 2016 at 1:47 PM, Ivan Adji - Krstev <akivanradix@gmail.com
wrote:
Robert, i just thought of that and goggling how to add columns or some other similar scenarios, i think that the problem lays on how syslog-ng send the logs in the DB. Or how DB is storing this messages. As i have not configure nothing on the MongoDB just username and password for already created DB by the syslog-ng.
If some one have some tips, ill be happy to try it :)
Kind regards Ivan
On 05/18/2016 01:43 PM, Fekete, Róbert wrote:
Hi,
can you check the mongodb itself if the related fields/tags/whatever are in place? I mean, the problem might be in how syslog-ng sends the data into MongoDB, or in how loganalyzer reads the data from MongoDB. Is there a way for you to find out which?
Robert
On Wed, May 18, 2016 at 11:04 AM, Ivan Adji - Krstev < akivanradix@gmail.com> wrote:
Hi Jim, Thanks for the feedback. The problem is that im trying to monitor big infrastructure ( 200 Physical servers and more than 1000 VMs ). So currently i have install with MongoDB and have 300MB for one week monitoring just two VMs. The server syslog-ng and one client VM. Also i have used before syslog-ng with MariaDB (MySQL) but i have problem that i have 90% CPU Load when i used MySQL. I can't fix it. But now using MongoDB i have other problems. Using LogAnalyzer i can't see the "Date", "Facility", Serverity etc. on a main page but when i go to the log itself or i open it i can see all this informations. So i have the following
1. Syslog-NG with MySQL and LogAnalyzer ( works ok but CPU Usage was big ) 2. Syslog-NG with MongoDB and LogAnalyzer ( works ok but no informations shown on a first page )
So i can't find solutions and i need this sh*** up and running ASAP :)
Any solutions or suggestions im open to see it !
Kind regards Ivan
On 05/16/2016 05:43 PM, jrhendri@roadrunner.com wrote:
My 2 cents (what works for you depends on your infrastructure, resources and capabilities)
I like the model where syslog-ng does all the following:
- writes text files of the raw data (that way - whatever your search head is can re-ingest files later using basically the same parsers)
- filters out highly false-positive prone data from being forwarded
- handles parsing of data elements (using patterndb or whatever) and sends specific information to a search engine (like Elasticsearch)
- forwards specific data (based on security use cases) to a SIEM
Whether you use Elasticsearch, mongo, splunk, or whatever is really up to you and your budget. That said, I find syslog-ng to elasticsearch directly with kibana as the front end is *very* scalable for a search engine.
As far as a SIEM - it's kind of up to you.
Good luck,
Jim
---- Ivan Adji - Krstev <akivanradix@gmail.com> <akivanradix@gmail.com> wrote:
Hi all,
What is the best practice for storing all those logs in one central environment. I have one Linux Box running Syslog-NG with LogAnalyzer and MongoDB ( for now ), and is the best way to configure and use it with MongoDB or with MariaDB ( MySQL ) ? I have once install MySQL but it was getting very slow as the logs getting bigger and bigger ( for one week ). Now i have done with MongoDB ( still testing ) but i have problem as LogAnalyzer does not show me the real pictures, i have no Date info, no Facility, no serverity, Hosts, syslogtag, i just have ProcessID.
Any hints on this ?
I have the following configuration on the syslog-ng.cfg:
destination d_mongodb { mongodb( servers("localhost:27017") database("logs") # uri('mongodb://localhost/syslog-ng') collection("syslog") value-pairs( scope("selected-macros" "nv-pairs" "sdata") ) ); };
Kind regards Ivan
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Richie, I'll do that too, the think that scares me is that I'm running maybe two weeks this syslog-ng and i have 400 MB of logs and im logging just 2 Machines, the Server itself and on Client. And im planning to loging more than 1500 machines. So im not sure what to choose. MongoDB, MySQL, or PostgreSQL. And for now i have problems with all of them :) Kind regards Ivan On 05/19/2016 08:07 AM, Richárd Réfi wrote:
Hi,
I would try mysql/mariadb tweaks also: - index(or indices) on one or more column(s) of the mysql table according to the queries of loganalyzer - check the different cache and buffer options in your mysql conf - my opinion and experience is that mysql partitioning can do magic on this amount of data. A query could run only on few gigs portion of data (and a good indexing accelerates the query of these few gigs also).
Unfortunately your mongodb problem remains open.
Regards, Richie
On Wed, May 18, 2016, 14:37 Ivan Adji - Krstev <akivanradix@gmail.com <mailto:akivanradix@gmail.com>> wrote:
Nop, Again same problem: Here is what i have done
destination d_mongodb { mongodb( servers("localhost:27017") database("syslog") username("Ivan") password("Ivan123") collection("messages")
value-pairs( scope("selected-macros" "nv-pairs" "sdata") pair("date", datetime("$UNIXTIME")) pair("pid", int64("$PID")) pair("program", "$PROGRAM") pair("message", "$MESSAGE") ) ); }; Still have the same problem no info on date no nothing. The strange part is that when i open a specific log i have all the info. The only problem is where on a first page on the LogAnalyzer i don't have this problems.
And yes again we may have two problems: One is the DB information how its stored, and by this i think we stored as we should, but do i have to configure some tables in the MongoDB or columns or something ( that is how i did it with MySQL ). The second is something wrong with the LogAnalyzer so now im going to reconfigure with PostgreSQL and again with MySQL to see if something will be change.
Ivan
On 05/18/2016 01:53 PM, Fekete, Róbert wrote:
Hi,
Do you know in what type does loganalyzer expect the specific fields? AFAIK, by default, syslog-ng sends everything as string, but for the mongodb destination, you can specify the data type, see https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-g...
Try sending the date as datetime, and the others as numbers, maybe it helps.
Regards,
Robert
On Wed, May 18, 2016 at 1:47 PM, Ivan Adji - Krstev <akivanradix@gmail.com <mailto:akivanradix@gmail.com>> wrote:
Robert, i just thought of that and goggling how to add columns or some other similar scenarios, i think that the problem lays on how syslog-ng send the logs in the DB. Or how DB is storing this messages. As i have not configure nothing on the MongoDB just username and password for already created DB by the syslog-ng.
If some one have some tips, ill be happy to try it :)
Kind regards Ivan
On 05/18/2016 01:43 PM, Fekete, Róbert wrote:
Hi,
can you check the mongodb itself if the related fields/tags/whatever are in place? I mean, the problem might be in how syslog-ng sends the data into MongoDB, or in how loganalyzer reads the data from MongoDB. Is there a way for you to find out which?
Robert
On Wed, May 18, 2016 at 11:04 AM, Ivan Adji - Krstev <akivanradix@gmail.com <mailto:akivanradix@gmail.com>> wrote:
Hi Jim, Thanks for the feedback. The problem is that im trying to monitor big infrastructure ( 200 Physical servers and more than 1000 VMs ). So currently i have install with MongoDB and have 300MB for one week monitoring just two VMs. The server syslog-ng and one client VM. Also i have used before syslog-ng with MariaDB (MySQL) but i have problem that i have 90% CPU Load when i used MySQL. I can't fix it. But now using MongoDB i have other problems. Using LogAnalyzer i can't see the "Date", "Facility", Serverity etc. on a main page but when i go to the log itself or i open it i can see all this informations. So i have the following
1. Syslog-NG with MySQL and LogAnalyzer ( works ok but CPU Usage was big ) 2. Syslog-NG with MongoDB and LogAnalyzer ( works ok but no informations shown on a first page )
So i can't find solutions and i need this sh*** up and running ASAP :)
Any solutions or suggestions im open to see it !
Kind regards Ivan
On 05/16/2016 05:43 PM, jrhendri@roadrunner.com <mailto:jrhendri@roadrunner.com> wrote:
My 2 cents (what works for you depends on your infrastructure, resources and capabilities)
I like the model where syslog-ng does all the following:
- writes text files of the raw data (that way - whatever your search head is can re-ingest files later using basically the same parsers)
- filters out highly false-positive prone data from being forwarded
- handles parsing of data elements (using patterndb or whatever) and sends specific information to a search engine (like Elasticsearch)
- forwards specific data (based on security use cases) to a SIEM
Whether you use Elasticsearch, mongo, splunk, or whatever is really up to you and your budget. That said, I find syslog-ng to elasticsearch directly with kibana as the front end is *very* scalable for a search engine.
As far as a SIEM - it's kind of up to you.
Good luck,
Jim
---- Ivan Adji - Krstev <akivanradix@gmail.com> <mailto:akivanradix@gmail.com> wrote:
Hi all,
What is the best practice for storing all those logs in one central environment. I have one Linux Box running Syslog-NG with LogAnalyzer and MongoDB ( for now ), and is the best way to configure and use it with MongoDB or with MariaDB ( MySQL ) ? I have once install MySQL but it was getting very slow as the logs getting bigger and bigger ( for one week ). Now i have done with MongoDB ( still testing ) but i have problem as LogAnalyzer does not show me the real pictures, i have no Date info, no Facility, no serverity, Hosts, syslogtag, i just have ProcessID.
Any hints on this ?
I have the following configuration on the syslog-ng.cfg:
destination d_mongodb { mongodb( servers("localhost:27017") database("logs") # uri('mongodb://localhost/syslog-ng') collection("syslog") value-pairs( scope("selected-macros" "nv-pairs" "sdata") ) ); };
Kind regards Ivan
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Elastic or hdfs are horizontally scalable and depend what you want to do with the logs. These are horizontally scalable, but their per node performance needs to be taken into account. Stuff like Balabit syslog-ng store box can consume lots of data on a single node or a couple of nodes but that's commercial. I'd experiment with elastic search and kibana if I were you, or of course I can connect you someone about Balabit ssb. Bazsi On May 19, 2016 10:09 AM, "Ivan Adji - Krstev" <akivanradix@gmail.com> wrote:
Hi Richie,
I'll do that too, the think that scares me is that I'm running maybe two weeks this syslog-ng and i have 400 MB of logs and im logging just 2 Machines, the Server itself and on Client. And im planning to loging more than 1500 machines. So im not sure what to choose. MongoDB, MySQL, or PostgreSQL.
And for now i have problems with all of them :)
Kind regards Ivan
On 05/19/2016 08:07 AM, Richárd Réfi wrote:
Hi,
I would try mysql/mariadb tweaks also: - index(or indices) on one or more column(s) of the mysql table according to the queries of loganalyzer - check the different cache and buffer options in your mysql conf - my opinion and experience is that mysql partitioning can do magic on this amount of data. A query could run only on few gigs portion of data (and a good indexing accelerates the query of these few gigs also).
Unfortunately your mongodb problem remains open.
Regards, Richie
On Wed, May 18, 2016, 14:37 Ivan Adji - Krstev <akivanradix@gmail.com> wrote:
Nop, Again same problem: Here is what i have done
destination d_mongodb { mongodb( servers("localhost:27017") database("syslog") username("Ivan") password("Ivan123") collection("messages")
value-pairs( scope("selected-macros" "nv-pairs" "sdata") pair("date", datetime("$UNIXTIME")) pair("pid", int64("$PID")) pair("program", "$PROGRAM") pair("message", "$MESSAGE") ) ); }; Still have the same problem no info on date no nothing. The strange part is that when i open a specific log i have all the info. The only problem is where on a first page on the LogAnalyzer i don't have this problems.
And yes again we may have two problems: One is the DB information how its stored, and by this i think we stored as we should, but do i have to configure some tables in the MongoDB or columns or something ( that is how i did it with MySQL ). The second is something wrong with the LogAnalyzer so now im going to reconfigure with PostgreSQL and again with MySQL to see if something will be change.
Ivan
On 05/18/2016 01:53 PM, Fekete, Róbert wrote:
Hi,
Do you know in what type does loganalyzer expect the specific fields? AFAIK, by default, syslog-ng sends everything as string, but for the mongodb destination, you can specify the data type, see https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-g...
Try sending the date as datetime, and the others as numbers, maybe it helps.
Regards,
Robert
On Wed, May 18, 2016 at 1:47 PM, Ivan Adji - Krstev < akivanradix@gmail.com> wrote:
Robert, i just thought of that and goggling how to add columns or some other similar scenarios, i think that the problem lays on how syslog-ng send the logs in the DB. Or how DB is storing this messages. As i have not configure nothing on the MongoDB just username and password for already created DB by the syslog-ng.
If some one have some tips, ill be happy to try it :)
Kind regards Ivan
On 05/18/2016 01:43 PM, Fekete, Róbert wrote:
Hi,
can you check the mongodb itself if the related fields/tags/whatever are in place? I mean, the problem might be in how syslog-ng sends the data into MongoDB, or in how loganalyzer reads the data from MongoDB. Is there a way for you to find out which?
Robert
On Wed, May 18, 2016 at 11:04 AM, Ivan Adji - Krstev < akivanradix@gmail.com> wrote:
Hi Jim, Thanks for the feedback. The problem is that im trying to monitor big infrastructure ( 200 Physical servers and more than 1000 VMs ). So currently i have install with MongoDB and have 300MB for one week monitoring just two VMs. The server syslog-ng and one client VM. Also i have used before syslog-ng with MariaDB (MySQL) but i have problem that i have 90% CPU Load when i used MySQL. I can't fix it. But now using MongoDB i have other problems. Using LogAnalyzer i can't see the "Date", "Facility", Serverity etc. on a main page but when i go to the log itself or i open it i can see all this informations. So i have the following
1. Syslog-NG with MySQL and LogAnalyzer ( works ok but CPU Usage was big ) 2. Syslog-NG with MongoDB and LogAnalyzer ( works ok but no informations shown on a first page )
So i can't find solutions and i need this sh*** up and running ASAP :)
Any solutions or suggestions im open to see it !
Kind regards Ivan
On 05/16/2016 05:43 PM, jrhendri@roadrunner.com wrote:
My 2 cents (what works for you depends on your infrastructure, resources and capabilities)
I like the model where syslog-ng does all the following:
- writes text files of the raw data (that way - whatever your search head is can re-ingest files later using basically the same parsers)
- filters out highly false-positive prone data from being forwarded
- handles parsing of data elements (using patterndb or whatever) and sends specific information to a search engine (like Elasticsearch)
- forwards specific data (based on security use cases) to a SIEM
Whether you use Elasticsearch, mongo, splunk, or whatever is really up to you and your budget. That said, I find syslog-ng to elasticsearch directly with kibana as the front end is *very* scalable for a search engine.
As far as a SIEM - it's kind of up to you.
Good luck,
Jim
---- Ivan Adji - Krstev <akivanradix@gmail.com> <akivanradix@gmail.com> wrote:
Hi all,
What is the best practice for storing all those logs in one central environment. I have one Linux Box running Syslog-NG with LogAnalyzer and MongoDB ( for now ), and is the best way to configure and use it with MongoDB or with MariaDB ( MySQL ) ? I have once install MySQL but it was getting very slow as the logs getting bigger and bigger ( for one week ). Now i have done with MongoDB ( still testing ) but i have problem as LogAnalyzer does not show me the real pictures, i have no Date info, no Facility, no serverity, Hosts, syslogtag, i just have ProcessID.
Any hints on this ?
I have the following configuration on the syslog-ng.cfg:
destination d_mongodb { mongodb( servers("localhost:27017") database("logs") # uri('mongodb://localhost/syslog-ng') collection("syslog") value-pairs( scope("selected-macros" "nv-pairs" "sdata") ) ); };
Kind regards Ivan
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Bazsi, I saw the Elastic and hdfs and for me they are all the same :). I'm trying to build something free, from scratch. And I'm already done using Syslog-NG with MongoDB and all that to be shown on LogAnalyzer. So we will see how things will go. Once again thank you for the support ! Kind regards Ivan On 05/24/2016 08:29 AM, Scheidler, Balázs wrote:
Elastic or hdfs are horizontally scalable and depend what you want to do with the logs. These are horizontally scalable, but their per node performance needs to be taken into account.
Stuff like Balabit syslog-ng store box can consume lots of data on a single node or a couple of nodes but that's commercial.
I'd experiment with elastic search and kibana if I were you, or of course I can connect you someone about Balabit ssb.
Bazsi
On May 19, 2016 10:09 AM, "Ivan Adji - Krstev" <akivanradix@gmail.com <mailto:akivanradix@gmail.com>> wrote:
Hi Richie,
I'll do that too, the think that scares me is that I'm running maybe two weeks this syslog-ng and i have 400 MB of logs and im logging just 2 Machines, the Server itself and on Client. And im planning to loging more than 1500 machines. So im not sure what to choose. MongoDB, MySQL, or PostgreSQL.
And for now i have problems with all of them :)
Kind regards Ivan
On 05/19/2016 08:07 AM, Richárd Réfi wrote:
Hi,
I would try mysql/mariadb tweaks also: - index(or indices) on one or more column(s) of the mysql table according to the queries of loganalyzer - check the different cache and buffer options in your mysql conf - my opinion and experience is that mysql partitioning can do magic on this amount of data. A query could run only on few gigs portion of data (and a good indexing accelerates the query of these few gigs also).
Unfortunately your mongodb problem remains open.
Regards, Richie
On Wed, May 18, 2016, 14:37 Ivan Adji - Krstev <akivanradix@gmail.com <mailto:akivanradix@gmail.com>> wrote:
Nop, Again same problem: Here is what i have done
destination d_mongodb { mongodb( servers("localhost:27017") database("syslog") username("Ivan") password("Ivan123") collection("messages")
value-pairs( scope("selected-macros" "nv-pairs" "sdata") pair("date", datetime("$UNIXTIME")) pair("pid", int64("$PID")) pair("program", "$PROGRAM") pair("message", "$MESSAGE") ) ); }; Still have the same problem no info on date no nothing. The strange part is that when i open a specific log i have all the info. The only problem is where on a first page on the LogAnalyzer i don't have this problems.
And yes again we may have two problems: One is the DB information how its stored, and by this i think we stored as we should, but do i have to configure some tables in the MongoDB or columns or something ( that is how i did it with MySQL ). The second is something wrong with the LogAnalyzer so now im going to reconfigure with PostgreSQL and again with MySQL to see if something will be change.
Ivan
On 05/18/2016 01:53 PM, Fekete, Róbert wrote:
Hi,
Do you know in what type does loganalyzer expect the specific fields? AFAIK, by default, syslog-ng sends everything as string, but for the mongodb destination, you can specify the data type, see https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-g...
Try sending the date as datetime, and the others as numbers, maybe it helps.
Regards,
Robert
On Wed, May 18, 2016 at 1:47 PM, Ivan Adji - Krstev <akivanradix@gmail.com <mailto:akivanradix@gmail.com>> wrote:
Robert, i just thought of that and goggling how to add columns or some other similar scenarios, i think that the problem lays on how syslog-ng send the logs in the DB. Or how DB is storing this messages. As i have not configure nothing on the MongoDB just username and password for already created DB by the syslog-ng.
If some one have some tips, ill be happy to try it :)
Kind regards Ivan
On 05/18/2016 01:43 PM, Fekete, Róbert wrote:
Hi,
can you check the mongodb itself if the related fields/tags/whatever are in place? I mean, the problem might be in how syslog-ng sends the data into MongoDB, or in how loganalyzer reads the data from MongoDB. Is there a way for you to find out which?
Robert
On Wed, May 18, 2016 at 11:04 AM, Ivan Adji - Krstev <akivanradix@gmail.com <mailto:akivanradix@gmail.com>> wrote:
Hi Jim, Thanks for the feedback. The problem is that im trying to monitor big infrastructure ( 200 Physical servers and more than 1000 VMs ). So currently i have install with MongoDB and have 300MB for one week monitoring just two VMs. The server syslog-ng and one client VM. Also i have used before syslog-ng with MariaDB (MySQL) but i have problem that i have 90% CPU Load when i used MySQL. I can't fix it. But now using MongoDB i have other problems. Using LogAnalyzer i can't see the "Date", "Facility", Serverity etc. on a main page but when i go to the log itself or i open it i can see all this informations. So i have the following
1. Syslog-NG with MySQL and LogAnalyzer ( works ok but CPU Usage was big ) 2. Syslog-NG with MongoDB and LogAnalyzer ( works ok but no informations shown on a first page )
So i can't find solutions and i need this sh*** up and running ASAP :)
Any solutions or suggestions im open to see it !
Kind regards Ivan
On 05/16/2016 05:43 PM, jrhendri@roadrunner.com <mailto:jrhendri@roadrunner.com> wrote:
My 2 cents (what works for you depends on your infrastructure, resources and capabilities)
I like the model where syslog-ng does all the following:
- writes text files of the raw data (that way - whatever your search head is can re-ingest files later using basically the same parsers)
- filters out highly false-positive prone data from being forwarded
- handles parsing of data elements (using patterndb or whatever) and sends specific information to a search engine (like Elasticsearch)
- forwards specific data (based on security use cases) to a SIEM
Whether you use Elasticsearch, mongo, splunk, or whatever is really up to you and your budget. That said, I find syslog-ng to elasticsearch directly with kibana as the front end is *very* scalable for a search engine.
As far as a SIEM - it's kind of up to you.
Good luck,
Jim
---- Ivan Adji - Krstev <akivanradix@gmail.com> <mailto:akivanradix@gmail.com> wrote:
Hi all,
What is the best practice for storing all those logs in one central environment. I have one Linux Box running Syslog-NG with LogAnalyzer and MongoDB ( for now ), and is the best way to configure and use it with MongoDB or with MariaDB ( MySQL ) ? I have once install MySQL but it was getting very slow as the logs getting bigger and bigger ( for one week ). Now i have done with MongoDB ( still testing ) but i have problem as LogAnalyzer does not show me the real pictures, i have no Date info, no Facility, no serverity, Hosts, syslogtag, i just have ProcessID.
Any hints on this ?
I have the following configuration on the syslog-ng.cfg:
destination d_mongodb { mongodb( servers("localhost:27017") database("logs") # uri('mongodb://localhost/syslog-ng') collection("syslog") value-pairs( scope("selected-macros" "nv-pairs" "sdata") ) ); };
Kind regards Ivan
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (5)
-
Fekete, Róbert
-
Ivan Adji - Krstev
-
jrhendri＠roadrunner.com
-
Richárd Réfi
-
Scheidler, Balázs