[syslog-ng] mojology: syslog-ng and mongodb meet the web

[Martin Holste]

Ahahaha that is awesome!  Search will actually be really easy since
you can index on anything in there.  I think what would work best for
full-text search in mojology (doesn't roll off my tongue, but whatever
fuels your passion...) is to have an optional second process that goes
through newly inserted logs and does an in-place update.  So if a log
entry starts with:

{ _id: ...
  "timestamp": ...
  "dyn": { "classifier":
    "class": "some class"
  },
  "msg": "hello, world, this is a test",
}

Then do something like this to update it:
db.getCollection("logs").update({"timestamp": { $gt: <date last
fulltext indexed>, $lt: <now> }}, { $set: { "fulltext":
msg.split(/\s+/) }}, true);

Which adds the fulltext column to yield:
{ _id: ...
  "timestamp": ...
  "dyn": { "classifier":
    "class": "some class"
  },
  "msg": "hello, world, this is a test",
  "fulltext": [ "hello", "world", "this", "is", "a", "test" ]
}

I'm a little shaky on the Mongo update code there, but you get the
idea.  The point is that since it would be an optional second-pass, it
would be easy to tune or eliminate for performance.  If you do
ensureIndex("dyn") and ensureIndex("fulltext") then you have pretty
much all of your searching-bases covered.  You could of course add
this as an option to your Mongo Syslog-NG driver to do the split when
the original insert occurs for better overall performance and less
database fragmentation, but there would be a significantly higher
insert time.

There's also an implementation of Sphinx for MongoDB, but it uses
Sphinx's xmlpipe to get the docs, which is super-slow compared with
the Sphinx MySQL/ODBC source drivers.

On Sat, Jan 8, 2011 at 3:34 PM, Gergely Nagy <algernon at balabit.hu> wrote:
> Hi!
>
> During the past couple of nights, I was busy coming up with something
> that would showcase the power of syslog-ng combined with the mongodb
> destination, and that is how mojology was born:
>
>  http://mojology.madhouse-project.org/
>   source available @ https://github.com/algernon/mojology
>
> To sum it up: it's a web based log browser, a reasonably simple one: one
> can't search, or do advanced filtering (yet!), only a few basic stuff.
>
> The interesting part comes from the fact that the logs it displays are
> stored in mongodb, a document store, and as such, the structure of them
> is very, very flexible. Although, the application does make a few
> assumptions (see the about page or the readme in the sources), it still
> retains a lot of flexibility.
>
> Without going into the gory details (that's what the source is for ;),
> let me show you a few interesting pages!
>
> Examples
> --------
>
> * http://mojology.madhouse-project.org/log/4d28cd01f310ef4f00000024
>
> This one was originally made with the following command:
>  logger -i -t hi 'Hello World! This concludes our demo session."
>
> Then I went into the MongoDB shell and added a few other keys below the
> "dyn" sub-document, alongside the default "classifier" (courtesy of
> patterndb).
>
> As you can see, no matter how deep one nests, it still displays
> properly. It's also easy to query, even at such depths.
>
> * http://mojology.madhouse-project.org/log/4d28cba4f310ef4f0000001e
>
> Logging in from one computer to my desktop, using password
> authentication. This entry was made fully automatically.
>
> Other notes
> -----------
>
> The database behind the demo application is fairly small, only 39
> messages stored, but hey, it's a demo!
>
> The program itself is a little over a hundred lines of python +
> templates; built upon Flask & PyMongo.
>
> For the best experience, a recent browser is strongly advised: Chromium
> 6+, Firefox 4+ for best results; Firefox 3 works very well too, but some
> of the tiny cosmetic features do not; the site displays in IE7+ too, and
> can be used, but the experience is rather disappointing.
>
> I hope this little weekend hack helps demonstrate how flexible mongodb
> can be for logging, and how well syslog-ng can make use of that
> flexibility.
>
> --
> |8]
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>