Hi all, I'm trying to change the DB from InooDB to MongoDB, and i have install the MongoDB but i can't configure/create the database and tables on it. Can some one provide with simple tutorial or the best way to create and configure the MongoDB database and tables so that can be used by syslog-ng ? Kind regards Ivan
Hello, MongoDB is a document-oriented database engine, not a relational database. In this case, we talk about collections and documents instead of tables and rows. You can find a small comparison with MySQL here: https://www.mongodb.com/compare/mongodb-mysql There is also a more detailed SQL comparison: https://docs.mongodb.org/manual/reference/sql-comparison/ Fortunately, syslog-ng supports MongoDB through the mongodb() destination driver. Have a look at the "Example 7.11. Using the mongodb() driver" section of the syslog-ng OSE Administrator Guide: https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-g... https://asylum.madhouse-project.org/blog/2012/04/26/mongodb-howto/ Regards, László Várady On Wed, Apr 20, 2016 at 1:29 PM, Ivan Adji - Krstev <akivanradix@gmail.com> wrote:
Hi all, I'm trying to change the DB from InooDB to MongoDB, and i have install the MongoDB but i can't configure/create the database and tables on it. Can some one provide with simple tutorial or the best way to create and configure the MongoDB database and tables so that can be used by syslog-ng ?
Kind regards Ivan
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, which syslog-ng do you use and which mongodb? Previous syslog-ng versions supports only the 2.x mongodb series, 3.2.x series is supported only by the syslog-ng upstream (version 3.8). We moved to the official mongo-c library and -temporary- broke the configuration (we have a PR[1] that will support the previous config format - we will merge this soon). Until we won't merge the patch, you can use the new config options. So if you want to use the 3.2.x mongodb, you have to compile syslog-ng from source. Small tutorial: 1. docker: install mongodb 3.2.5 https://docs.mongodb.org/master/tutorial/install-mongodb-on-ubuntu/?_ga=1.19... 2. start mongodb root@a3e9571e39eb:/# mongod --bind_ip 172.17.0.2 3. build & configure git clone https://github.com/balabit/syslog-ng cd syslog-ng ./autogen.sh cd .. mkdir build cd build ../syslog-ng/configure --prefix /opt/syslog-ng.upstream make make install [if you need help on howto compile syslog-ng, we can help] /opt/syslog-ng.upstream/etc/syslog-ng.conf: @version: 3.8 @include "scl.conf" source s_network { network(port(5555)); }; destination d_mongo { mongodb ( uri('mongodb://172.17.0.2:27017/syslog-ng') collection('test') value-pairs( scope("selected-macros" "nv-pairs" "sdata") ) ); }; log { source(s_network); destination(d_mongo); flags(flow-control); }; 4. send logs stentor@T440s:~$ logger -s MSG:$RANDOM 2>&1 | nc localhost 5555 5. query the results root@a3e9571e39eb:/# mongo 172.17.0.2/syslog-ng MongoDB shell version: 3.2.5 connecting to: 172.17.0.2/syslog-ng
db.test.find({'PROGRAM':'stentor'}) { "_id" : ObjectId("57179ae358d80574d81440e1"), "TAGS" : ".source.s_network", "SOURCEIP" : "127.0.0.1", "SOURCE" : "s_network", "SEQNUM" : "1", "PROGRAM" : "stentor", "PRIORITY" : "notice", "MESSAGE" : "MSG:4854", "LEGACY_MSGHDR" : "stentor: ", "HOST_FROM" : "localhost", "HOST" : "localhost", "FACILITY" : "user", "DATE" : "Apr 20 17:06:11" } { "_id" : ObjectId("57179ae858d80574d81440e2"), "TAGS" : ".source.s_network", "SOURCEIP" : "127.0.0.1", "SOURCE" : "s_network", "SEQNUM" : "2", "PROGRAM" : "stentor", "PRIORITY" : "notice", "MESSAGE" : "MSG:31751", "LEGACY_MSGHDR" : "stentor: ", "HOST_FROM" : "localhost", "HOST" : "localhost", "FACILITY" : "user", "DATE" : "Apr 20 17:06:16" } { "_id" : ObjectId("57179b1958d80574ff238bf1"), "TAGS" : ".source.s_network", "SOURCEIP" : "127.0.0.1", "SOURCE" : "s_network", "SEQNUM" : "1", "PROGRAM" : "stentor", "PRIORITY" : "notice", "MESSAGE" : "MSG:13698", "LEGACY_MSGHDR" : "stentor: ", "HOST_FROM" : "localhost", "HOST" : "localhost", "FACILITY" : "user", "DATE" : "Apr 20 17:07:05" }
[1] https://github.com/balabit/syslog-ng/pull/981 regards, Laszlo Budai On Wed, Apr 20, 2016 at 1:29 PM, Ivan Adji - Krstev <akivanradix@gmail.com> wrote:
Hi all, I'm trying to change the DB from InooDB to MongoDB, and i have install the MongoDB but i can't configure/create the database and tables on it. Can some one provide with simple tutorial or the best way to create and configure the MongoDB database and tables so that can be used by syslog-ng ?
Kind regards Ivan
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi all, Thanks for the answers, I thought that i have to create first tables and databases etc. But seem like its just writing. Can you give me some tips on how to protect the mongo with Usernames and Password ? Im using Mongo 2.6.11 with Syslog-NG version 3.8.0alpha0 with LogAnalyzer too. I have installed previous and works perfect with MariaDB, but after some time start to eat mu CPU and RAM so i decide to go with Mongo etc. Kind regards Ivan On 04/20/2016 05:32 PM, Budai, László wrote:
Hi,
which syslog-ng do you use and which mongodb?
Previous syslog-ng versions supports only the 2.x mongodb series, 3.2.x series is supported only by the syslog-ng upstream (version 3.8). We moved to the official mongo-c library and -temporary- broke the configuration (we have a PR[1] that will support the previous config format - we will merge this soon). Until we won't merge the patch, you can use the new config options.
So if you want to use the 3.2.x mongodb, you have to compile syslog-ng from source.
Small tutorial: 1. docker: install mongodb 3.2.5
https://docs.mongodb.org/master/tutorial/install-mongodb-on-ubuntu/?_ga=1.19...
2. start mongodb
root@a3e9571e39eb:/# mongod --bind_ip 172.17.0.2
3. build & configure
git clone https://github.com/balabit/syslog-ng cd syslog-ng ./autogen.sh cd .. mkdir build cd build ../syslog-ng/configure --prefix /opt/syslog-ng.upstream make make install
[if you need help on howto compile syslog-ng, we can help]
/opt/syslog-ng.upstream/etc/syslog-ng.conf: @version: 3.8 @include "scl.conf"
source s_network { network(port(5555)); };
destination d_mongo { mongodb ( uri('mongodb://172.17.0.2:27017/syslog-ng <http://172.17.0.2:27017/syslog-ng>') collection('test') value-pairs( scope("selected-macros" "nv-pairs" "sdata") ) ); };
log { source(s_network); destination(d_mongo); flags(flow-control); };
4. send logs
stentor@T440s:~$ logger -s MSG:$RANDOM 2>&1 | nc localhost 5555
5. query the results root@a3e9571e39eb:/# mongo 172.17.0.2/syslog-ng <http://172.17.0.2/syslog-ng>
MongoDB shell version: 3.2.5 connecting to: 172.17.0.2/syslog-ng <http://172.17.0.2/syslog-ng>
db.test.find({'PROGRAM':'stentor'}) { "_id" : ObjectId("57179ae358d80574d81440e1"), "TAGS" : ".source.s_network", "SOURCEIP" : "127.0.0.1", "SOURCE" : "s_network", "SEQNUM" : "1", "PROGRAM" : "stentor", "PRIORITY" : "notice", "MESSAGE" : "MSG:4854", "LEGACY_MSGHDR" : "stentor: ", "HOST_FROM" : "localhost", "HOST" : "localhost", "FACILITY" : "user", "DATE" : "Apr 20 17:06:11" } { "_id" : ObjectId("57179ae858d80574d81440e2"), "TAGS" : ".source.s_network", "SOURCEIP" : "127.0.0.1", "SOURCE" : "s_network", "SEQNUM" : "2", "PROGRAM" : "stentor", "PRIORITY" : "notice", "MESSAGE" : "MSG:31751", "LEGACY_MSGHDR" : "stentor: ", "HOST_FROM" : "localhost", "HOST" : "localhost", "FACILITY" : "user", "DATE" : "Apr 20 17:06:16" } { "_id" : ObjectId("57179b1958d80574ff238bf1"), "TAGS" : ".source.s_network", "SOURCEIP" : "127.0.0.1", "SOURCE" : "s_network", "SEQNUM" : "1", "PROGRAM" : "stentor", "PRIORITY" : "notice", "MESSAGE" : "MSG:13698", "LEGACY_MSGHDR" : "stentor: ", "HOST_FROM" : "localhost", "HOST" : "localhost", "FACILITY" : "user", "DATE" : "Apr 20 17:07:05" }
[1] https://github.com/balabit/syslog-ng/pull/981
regards, Laszlo Budai
On Wed, Apr 20, 2016 at 1:29 PM, Ivan Adji - Krstev <akivanradix@gmail.com <mailto:akivanradix@gmail.com>> wrote:
Hi all, I'm trying to change the DB from InooDB to MongoDB, and i have install the MongoDB but i can't configure/create the database and tables on it. Can some one provide with simple tutorial or the best way to create and configure the MongoDB database and tables so that can be used by syslog-ng ?
Kind regards Ivan
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, The syslog-ng mongodb driver has a username and a password option ( https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-g... ). For the mongodb side, see https://docs.mongodb.org/v2.6/tutorial/enable-authentication/ HTH, Robert On Thu, Apr 21, 2016 at 10:09 AM, Ivan Adji - Krstev <akivanradix@gmail.com> wrote:
Hi all,
Thanks for the answers, I thought that i have to create first tables and databases etc. But seem like its just writing. Can you give me some tips on how to protect the mongo with Usernames and Password ?
Im using Mongo 2.6.11 with Syslog-NG version 3.8.0alpha0 with LogAnalyzer too. I have installed previous and works perfect with MariaDB, but after some time start to eat mu CPU and RAM so i decide to go with Mongo etc.
Kind regards Ivan
On 04/20/2016 05:32 PM, Budai, László wrote:
Hi,
which syslog-ng do you use and which mongodb?
Previous syslog-ng versions supports only the 2.x mongodb series, 3.2.x series is supported only by the syslog-ng upstream (version 3.8). We moved to the official mongo-c library and -temporary- broke the configuration (we have a PR[1] that will support the previous config format - we will merge this soon). Until we won't merge the patch, you can use the new config options.
So if you want to use the 3.2.x mongodb, you have to compile syslog-ng from source.
Small tutorial: 1. docker: install mongodb 3.2.5
https://docs.mongodb.org/master/tutorial/install-mongodb-on-ubuntu/?_ga=1.19...
2. start mongodb
root@a3e9571e39eb:/# mongod --bind_ip 172.17.0.2
3. build & configure
git clone https://github.com/balabit/syslog-ng cd syslog-ng ./autogen.sh cd .. mkdir build cd build ../syslog-ng/configure --prefix /opt/syslog-ng.upstream make make install
[if you need help on howto compile syslog-ng, we can help]
/opt/syslog-ng.upstream/etc/syslog-ng.conf: @version: 3.8 @include "scl.conf"
source s_network { network(port(5555)); };
destination d_mongo { mongodb ( uri('mongodb://172.17.0.2:27017/syslog-ng') collection('test') value-pairs( scope("selected-macros" "nv-pairs" "sdata") ) ); };
log { source(s_network); destination(d_mongo); flags(flow-control); };
4. send logs
stentor@T440s:~$ logger -s MSG:$RANDOM 2>&1 | nc localhost 5555
5. query the results root@a3e9571e39eb:/# mongo 172.17.0.2/syslog-ng
MongoDB shell version: 3.2.5 connecting to: 172.17.0.2/syslog-ng
db.test.find({'PROGRAM':'stentor'}) { "_id" : ObjectId("57179ae358d80574d81440e1"), "TAGS" : ".source.s_network", "SOURCEIP" : "127.0.0.1", "SOURCE" : "s_network", "SEQNUM" : "1", "PROGRAM" : "stentor", "PRIORITY" : "notice", "MESSAGE" : "MSG:4854", "LEGACY_MSGHDR" : "stentor: ", "HOST_FROM" : "localhost", "HOST" : "localhost", "FACILITY" : "user", "DATE" : "Apr 20 17:06:11" } { "_id" : ObjectId("57179ae858d80574d81440e2"), "TAGS" : ".source.s_network", "SOURCEIP" : "127.0.0.1", "SOURCE" : "s_network", "SEQNUM" : "2", "PROGRAM" : "stentor", "PRIORITY" : "notice", "MESSAGE" : "MSG:31751", "LEGACY_MSGHDR" : "stentor: ", "HOST_FROM" : "localhost", "HOST" : "localhost", "FACILITY" : "user", "DATE" : "Apr 20 17:06:16" } { "_id" : ObjectId("57179b1958d80574ff238bf1"), "TAGS" : ".source.s_network", "SOURCEIP" : "127.0.0.1", "SOURCE" : "s_network", "SEQNUM" : "1", "PROGRAM" : "stentor", "PRIORITY" : "notice", "MESSAGE" : "MSG:13698", "LEGACY_MSGHDR" : "stentor: ", "HOST_FROM" : "localhost", "HOST" : "localhost", "FACILITY" : "user", "DATE" : "Apr 20 17:07:05" }
[1] https://github.com/balabit/syslog-ng/pull/981
regards, Laszlo Budai
On Wed, Apr 20, 2016 at 1:29 PM, Ivan Adji - Krstev <akivanradix@gmail.com
wrote:
Hi all, I'm trying to change the DB from InooDB to MongoDB, and i have install the MongoDB but i can't configure/create the database and tables on it. Can some one provide with simple tutorial or the best way to create and configure the MongoDB database and tables so that can be used by syslog-ng ?
Kind regards Ivan
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (4)
-
Budai, László
-
Fekete, Róbert
-
Ivan Adji - Krstev
-
Várady, László