[syslog-ng] Expected performance

[Balazs Scheidler]

disclaimer: this message contains stuff about our commercial offerings.
do not read it if that bothers you.

On Thu, 2009-02-19 at 07:36 -0800, Mike Tremaine wrote:
> Jeff Dell wrote:
> > The problem is MySQL and other traditional relational databases can't handle
> > that amount of data. That is why log management companies have moved away
> > from the traditional relational database engines and use other means to
> > store/query the data. 
> > 
> 
> Any ideas of what those might be just curious what the highend stuff is 
> doing. If I was faced with more then 50hosts I would probably go back to 
> flat files in a tree
> 
> hostname->Month/year->day->rawlogs
> 
> Then you just have to develop some search scripts that can handle 
> digging down the tree and making the results pretty [for the VP's since 
> many would just use grep off the commandline].

Our syslog-ng Store Box is using the "logstore" format  of syslog-ng PE,
which is compressed/encrypted file format, indexed by message ID and
time. On top of this we have implemented an indexing engine, that:

1) tokenizes incoming messages (e.g. splits them into words)
2) every now and then writes the accumulated tokens into an index file,
in sorted form.

Then searching is really fast, since the index is a binary-searchable
file format.

SSB is able to index about 20-25k msg/sec on commodity hardware.

-- 
Bazsi