mojology: syslog-ng and mongodb meet the web
Hi! During the past couple of nights, I was busy coming up with something that would showcase the power of syslog-ng combined with the mongodb destination, and that is how mojology was born: http://mojology.madhouse-project.org/ source available @ https://github.com/algernon/mojology To sum it up: it's a web based log browser, a reasonably simple one: one can't search, or do advanced filtering (yet!), only a few basic stuff. The interesting part comes from the fact that the logs it displays are stored in mongodb, a document store, and as such, the structure of them is very, very flexible. Although, the application does make a few assumptions (see the about page or the readme in the sources), it still retains a lot of flexibility. Without going into the gory details (that's what the source is for ;), let me show you a few interesting pages! Examples -------- * http://mojology.madhouse-project.org/log/4d28cd01f310ef4f00000024 This one was originally made with the following command: logger -i -t hi 'Hello World! This concludes our demo session." Then I went into the MongoDB shell and added a few other keys below the "dyn" sub-document, alongside the default "classifier" (courtesy of patterndb). As you can see, no matter how deep one nests, it still displays properly. It's also easy to query, even at such depths. * http://mojology.madhouse-project.org/log/4d28cba4f310ef4f0000001e Logging in from one computer to my desktop, using password authentication. This entry was made fully automatically. Other notes ----------- The database behind the demo application is fairly small, only 39 messages stored, but hey, it's a demo! The program itself is a little over a hundred lines of python + templates; built upon Flask & PyMongo. For the best experience, a recent browser is strongly advised: Chromium 6+, Firefox 4+ for best results; Firefox 3 works very well too, but some of the tiny cosmetic features do not; the site displays in IE7+ too, and can be used, but the experience is rather disappointing. I hope this little weekend hack helps demonstrate how flexible mongodb can be for logging, and how well syslog-ng can make use of that flexibility. -- |8]
Hi Gergely, Rather than re-invent the wheel (on the presentation layer piece), you might be interested in knowing about my project - LogZilla ( http://www.logzilla.info). It is free to use for anyone getting < 1million messages a day and even then, the full version is free for anyone who joins the team and contributes :-) If you are interested in integrating Mongo into it, let me know! ______________________________________________________________ Clayton Dukes ______________________________________________________________ On Sat, Jan 8, 2011 at 4:34 PM, Gergely Nagy <algernon@balabit.hu> wrote:
Hi!
During the past couple of nights, I was busy coming up with something that would showcase the power of syslog-ng combined with the mongodb destination, and that is how mojology was born:
http://mojology.madhouse-project.org/ source available @ https://github.com/algernon/mojology
To sum it up: it's a web based log browser, a reasonably simple one: one can't search, or do advanced filtering (yet!), only a few basic stuff.
The interesting part comes from the fact that the logs it displays are stored in mongodb, a document store, and as such, the structure of them is very, very flexible. Although, the application does make a few assumptions (see the about page or the readme in the sources), it still retains a lot of flexibility.
Without going into the gory details (that's what the source is for ;), let me show you a few interesting pages!
Examples --------
* http://mojology.madhouse-project.org/log/4d28cd01f310ef4f00000024
This one was originally made with the following command: logger -i -t hi 'Hello World! This concludes our demo session."
Then I went into the MongoDB shell and added a few other keys below the "dyn" sub-document, alongside the default "classifier" (courtesy of patterndb).
As you can see, no matter how deep one nests, it still displays properly. It's also easy to query, even at such depths.
* http://mojology.madhouse-project.org/log/4d28cba4f310ef4f0000001e
Logging in from one computer to my desktop, using password authentication. This entry was made fully automatically.
Other notes -----------
The database behind the demo application is fairly small, only 39 messages stored, but hey, it's a demo!
The program itself is a little over a hundred lines of python + templates; built upon Flask & PyMongo.
For the best experience, a recent browser is strongly advised: Chromium 6+, Firefox 4+ for best results; Firefox 3 works very well too, but some of the tiny cosmetic features do not; the site displays in IE7+ too, and can be used, but the experience is rather disappointing.
I hope this little weekend hack helps demonstrate how flexible mongodb can be for logging, and how well syslog-ng can make use of that flexibility.
-- |8]
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Sat, 2011-01-08 at 17:16 -0500, Clayton Dukes wrote:
Hi Gergely, Rather than re-invent the wheel (on the presentation layer piece), you might be interested in knowing about my project - LogZilla (http://www.logzilla.info).
I had a look at logzilla (among others) before starting on mojology, but decided to hack something up from scratch instead, for a couple of reasons: * I wanted something small, which I can cook up within a few nights. Digging into a bigger codebase like logzilla would've taken a lot longer than what I could spare at this time. * I'm a huge fan of the GPL, and whatever I came up with as a demo, I wanted to have it out under the that license O:) * On the way to work one day, I came up with the name mojology, and liked it so much that I just had to use it. * There's very few things I hate more than PHP, sorry.
It is free to use for anyone getting < 1million messages a day and even then, the full version is free for anyone who joins the team and contributes :-)
I had a look at the screenshots, and LogZilla looks very cool, but I think that the purpose of the two tools are a bit different: LogZilla appears to be an advanced viewer & reporting application, mojology is a dumb little interface without bells & whistles. A bit like MySQL and SQLite, if I may say so (except that mojology is nowhere near the complexity of sqlite :). -- |8]
I completely understand - it's funny, but that's how LZ started out about 10 years ago. It was always FOSS until last year when I starting doing "pro" versions, but basically, it's the exact same thing (still 99.9% open source) with some extra goodies for larger companies. Sorry you hate PHP, it's been good to me, hehe. ______________________________________________________________ Clayton Dukes ______________________________________________________________ On Sat, Jan 8, 2011 at 5:31 PM, Gergely Nagy <algernon@balabit.hu> wrote:
On Sat, 2011-01-08 at 17:16 -0500, Clayton Dukes wrote:
Hi Gergely, Rather than re-invent the wheel (on the presentation layer piece), you might be interested in knowing about my project - LogZilla (http://www.logzilla.info).
I had a look at logzilla (among others) before starting on mojology, but decided to hack something up from scratch instead, for a couple of reasons:
* I wanted something small, which I can cook up within a few nights. Digging into a bigger codebase like logzilla would've taken a lot longer than what I could spare at this time.
* I'm a huge fan of the GPL, and whatever I came up with as a demo, I wanted to have it out under the that license O:)
* On the way to work one day, I came up with the name mojology, and liked it so much that I just had to use it.
* There's very few things I hate more than PHP, sorry.
It is free to use for anyone getting < 1million messages a day and even then, the full version is free for anyone who joins the team and contributes :-)
I had a look at the screenshots, and LogZilla looks very cool, but I think that the purpose of the two tools are a bit different: LogZilla appears to be an advanced viewer & reporting application, mojology is a dumb little interface without bells & whistles.
A bit like MySQL and SQLite, if I may say so (except that mojology is nowhere near the complexity of sqlite :). -- |8]
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Ahahaha that is awesome! Search will actually be really easy since you can index on anything in there. I think what would work best for full-text search in mojology (doesn't roll off my tongue, but whatever fuels your passion...) is to have an optional second process that goes through newly inserted logs and does an in-place update. So if a log entry starts with: { _id: ... "timestamp": ... "dyn": { "classifier": "class": "some class" }, "msg": "hello, world, this is a test", } Then do something like this to update it: db.getCollection("logs").update({"timestamp": { $gt: <date last fulltext indexed>, $lt: <now> }}, { $set: { "fulltext": msg.split(/\s+/) }}, true); Which adds the fulltext column to yield: { _id: ... "timestamp": ... "dyn": { "classifier": "class": "some class" }, "msg": "hello, world, this is a test", "fulltext": [ "hello", "world", "this", "is", "a", "test" ] } I'm a little shaky on the Mongo update code there, but you get the idea. The point is that since it would be an optional second-pass, it would be easy to tune or eliminate for performance. If you do ensureIndex("dyn") and ensureIndex("fulltext") then you have pretty much all of your searching-bases covered. You could of course add this as an option to your Mongo Syslog-NG driver to do the split when the original insert occurs for better overall performance and less database fragmentation, but there would be a significantly higher insert time. There's also an implementation of Sphinx for MongoDB, but it uses Sphinx's xmlpipe to get the docs, which is super-slow compared with the Sphinx MySQL/ODBC source drivers. On Sat, Jan 8, 2011 at 3:34 PM, Gergely Nagy <algernon@balabit.hu> wrote:
Hi!
During the past couple of nights, I was busy coming up with something that would showcase the power of syslog-ng combined with the mongodb destination, and that is how mojology was born:
http://mojology.madhouse-project.org/ source available @ https://github.com/algernon/mojology
To sum it up: it's a web based log browser, a reasonably simple one: one can't search, or do advanced filtering (yet!), only a few basic stuff.
The interesting part comes from the fact that the logs it displays are stored in mongodb, a document store, and as such, the structure of them is very, very flexible. Although, the application does make a few assumptions (see the about page or the readme in the sources), it still retains a lot of flexibility.
Without going into the gory details (that's what the source is for ;), let me show you a few interesting pages!
Examples --------
* http://mojology.madhouse-project.org/log/4d28cd01f310ef4f00000024
This one was originally made with the following command: logger -i -t hi 'Hello World! This concludes our demo session."
Then I went into the MongoDB shell and added a few other keys below the "dyn" sub-document, alongside the default "classifier" (courtesy of patterndb).
As you can see, no matter how deep one nests, it still displays properly. It's also easy to query, even at such depths.
* http://mojology.madhouse-project.org/log/4d28cba4f310ef4f0000001e
Logging in from one computer to my desktop, using password authentication. This entry was made fully automatically.
Other notes -----------
The database behind the demo application is fairly small, only 39 messages stored, but hey, it's a demo!
The program itself is a little over a hundred lines of python + templates; built upon Flask & PyMongo.
For the best experience, a recent browser is strongly advised: Chromium 6+, Firefox 4+ for best results; Firefox 3 works very well too, but some of the tiny cosmetic features do not; the site displays in IE7+ too, and can be used, but the experience is rather disappointing.
I hope this little weekend hack helps demonstrate how flexible mongodb can be for logging, and how well syslog-ng can make use of that flexibility.
-- |8]
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Sat, 2011-01-08 at 20:41 -0600, Martin Holste wrote:
Ahahaha that is awesome! Search will actually be really easy since you can index on anything in there. I think what would work best for full-text search in mojology (doesn't roll off my tongue, but whatever fuels your passion...)
Think of it is a "monology", with a j instead of n.
is to have an optional second process that goes through newly inserted logs and does an in-place update. So if a log entry starts with:
{ _id: ... "timestamp": ... "dyn": { "classifier": "class": "some class" }, "msg": "hello, world, this is a test", }
Then do something like this to update it: db.getCollection("logs").update({"timestamp": { $gt: <date last fulltext indexed>, $lt: <now> }}, { $set: { "fulltext": msg.split(/\s+/) }}, true);
Which adds the fulltext column to yield: { _id: ... "timestamp": ... "dyn": { "classifier": "class": "some class" }, "msg": "hello, world, this is a test", "fulltext": [ "hello", "world", "this", "is", "a", "test" ] }
I'm a little shaky on the Mongo update code there, but you get the idea. The point is that since it would be an optional second-pass, it would be easy to tune or eliminate for performance. If you do ensureIndex("dyn") and ensureIndex("fulltext") then you have pretty much all of your searching-bases covered. You could of course add this as an option to your Mongo Syslog-NG driver to do the split when the original insert occurs for better overall performance and less database fragmentation, but there would be a significantly higher insert time.
I was considering something lke that (and a few other things, that would involve updating the db), and my current idea is to use a separate collection instead, so that if the original collection is, say, a capped collection, we don't unnecessarily add extra burden to it. That, and updating has a reasonable chance of fragmenting the document on-disk... So instead, I'll see if I can use a $Docref (or whatever that is called). That would make mojology a little slower, but it wouldn't need to touch the source collection at all. I didn't think about fulltext search though, so thanks for the suggestion! -- |8]
participants (3)
-
Clayton Dukes
-
Gergely Nagy
-
Martin Holste