disclaimer: this message contains stuff about our commercial offerings. do not read it if that bothers you. On Thu, 2009-02-19 at 07:36 -0800, Mike Tremaine wrote:
Jeff Dell wrote:
The problem is MySQL and other traditional relational databases can't handle that amount of data. That is why log management companies have moved away from the traditional relational database engines and use other means to store/query the data.
Any ideas of what those might be just curious what the highend stuff is doing. If I was faced with more then 50hosts I would probably go back to flat files in a tree
hostname->Month/year->day->rawlogs
Then you just have to develop some search scripts that can handle digging down the tree and making the results pretty [for the VP's since many would just use grep off the commandline].
Our syslog-ng Store Box is using the "logstore" format of syslog-ng PE, which is compressed/encrypted file format, indexed by message ID and time. On top of this we have implemented an indexing engine, that: 1) tokenizes incoming messages (e.g. splits them into words) 2) every now and then writes the accumulated tokens into an index file, in sorted form. Then searching is really fast, since the index is a binary-searchable file format. SSB is able to index about 20-25k msg/sec on commodity hardware. -- Bazsi