<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
I just wanted to let everyone know I really appreciate all of your
responses. Lots of good ideas and lots of thinks for me to try.<br>
<br>
Thank you,<br>
<br>
-Chris<br>
<br>
Balazs Scheidler wrote:
<blockquote cite="mid:1235330450.13024.51.camel@bzorp.balabit"
type="cite">
<pre wrap="">disclaimer: this message contains stuff about our commercial offerings.
do not read it if that bothers you.
On Thu, 2009-02-19 at 07:36 -0800, Mike Tremaine wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Jeff Dell wrote:
</pre>
<blockquote type="cite">
<pre wrap="">The problem is MySQL and other traditional relational databases can't handle
that amount of data. That is why log management companies have moved away
from the traditional relational database engines and use other means to
store/query the data.
</pre>
</blockquote>
<pre wrap="">Any ideas of what those might be just curious what the highend stuff is
doing. If I was faced with more then 50hosts I would probably go back to
flat files in a tree
hostname->Month/year->day->rawlogs
Then you just have to develop some search scripts that can handle
digging down the tree and making the results pretty [for the VP's since
many would just use grep off the commandline].
</pre>
</blockquote>
<pre wrap=""><!---->
Our syslog-ng Store Box is using the "logstore" format of syslog-ng PE,
which is compressed/encrypted file format, indexed by message ID and
time. On top of this we have implemented an indexing engine, that:
1) tokenizes incoming messages (e.g. splits them into words)
2) every now and then writes the accumulated tokens into an index file,
in sorted form.
Then searching is really fast, since the index is a binary-searchable
file format.
SSB is able to index about 20-25k msg/sec on commodity hardware.
</pre>
</blockquote>
<br>
</body>
</html>