Ahahaha that is awesome! Search will actually be really easy since you can index on anything in there. I think what would work best for full-text search in mojology (doesn't roll off my tongue, but whatever fuels your passion...) is to have an optional second process that goes through newly inserted logs and does an in-place update. So if a log entry starts with: { _id: ... "timestamp": ... "dyn": { "classifier": "class": "some class" }, "msg": "hello, world, this is a test", } Then do something like this to update it: db.getCollection("logs").update({"timestamp": { $gt: <date last fulltext indexed>, $lt: <now> }}, { $set: { "fulltext": msg.split(/\s+/) }}, true); Which adds the fulltext column to yield: { _id: ... "timestamp": ... "dyn": { "classifier": "class": "some class" }, "msg": "hello, world, this is a test", "fulltext": [ "hello", "world", "this", "is", "a", "test" ] } I'm a little shaky on the Mongo update code there, but you get the idea. The point is that since it would be an optional second-pass, it would be easy to tune or eliminate for performance. If you do ensureIndex("dyn") and ensureIndex("fulltext") then you have pretty much all of your searching-bases covered. You could of course add this as an option to your Mongo Syslog-NG driver to do the split when the original insert occurs for better overall performance and less database fragmentation, but there would be a significantly higher insert time. There's also an implementation of Sphinx for MongoDB, but it uses Sphinx's xmlpipe to get the docs, which is super-slow compared with the Sphinx MySQL/ODBC source drivers. On Sat, Jan 8, 2011 at 3:34 PM, Gergely Nagy <algernon@balabit.hu> wrote:
Hi!
During the past couple of nights, I was busy coming up with something that would showcase the power of syslog-ng combined with the mongodb destination, and that is how mojology was born:
http://mojology.madhouse-project.org/ source available @ https://github.com/algernon/mojology
To sum it up: it's a web based log browser, a reasonably simple one: one can't search, or do advanced filtering (yet!), only a few basic stuff.
The interesting part comes from the fact that the logs it displays are stored in mongodb, a document store, and as such, the structure of them is very, very flexible. Although, the application does make a few assumptions (see the about page or the readme in the sources), it still retains a lot of flexibility.
Without going into the gory details (that's what the source is for ;), let me show you a few interesting pages!
Examples --------
* http://mojology.madhouse-project.org/log/4d28cd01f310ef4f00000024
This one was originally made with the following command: logger -i -t hi 'Hello World! This concludes our demo session."
Then I went into the MongoDB shell and added a few other keys below the "dyn" sub-document, alongside the default "classifier" (courtesy of patterndb).
As you can see, no matter how deep one nests, it still displays properly. It's also easy to query, even at such depths.
* http://mojology.madhouse-project.org/log/4d28cba4f310ef4f0000001e
Logging in from one computer to my desktop, using password authentication. This entry was made fully automatically.
Other notes -----------
The database behind the demo application is fairly small, only 39 messages stored, but hey, it's a demo!
The program itself is a little over a hundred lines of python + templates; built upon Flask & PyMongo.
For the best experience, a recent browser is strongly advised: Chromium 6+, Firefox 4+ for best results; Firefox 3 works very well too, but some of the tiny cosmetic features do not; the site displays in IE7+ too, and can be used, but the experience is rather disappointing.
I hope this little weekend hack helps demonstrate how flexible mongodb can be for logging, and how well syslog-ng can make use of that flexibility.
-- |8]
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html