[syslog-ng] Anyone got a well performing search interface for syslog data?

[Jason Haar]

I just want to thank everyone for their responses. Very interesting stuff!

I think I can paraphrase that SQL-backends don't give much advantage
with large data sets due to the lack of relationships within syslog
data, and the "fastest" solutions are going to be those that basically
have custom-written "hot searches" pre-defined so that the appropriate
indexes/extra files are already created to speed things up.

The comments about gziping the files to speed up reads was interesting
as well...

It certainly an interesting problem. I want to do things like:

1. IDS event that IP 1.2.3.4 just did something bad against 3.4.5.6
2. I want to search logs for 7 days before this event for any other
activity from IP address 1.2.3.4 (might be email, PIX ACL logs, etc) or
from 3.4.5.6

or

1. User claims email never reached recipient
2. search for users email address
3. get report of all SMTP connection attempts, delivery attempts, AV and
antispam/RBL records associated with path of message through 'n'
different systems


those are all doable by hand - but very slow and - basically you need to
have someone who knows what they are doing. Being able to put that
behind a Web interface and make it a few clicks would be wonderful.

-- 
Cheers

Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +64 3 9635 377 Fax: +64 3 9635 417
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1