php-syslog-ng might be what you are looking fo. if you want a simple interface for people to use for searching. I'll recommend using this site: http://www.phpwizardry.com/php-syslog-ng.php Claus has re-written the project in his own release and fixes many issues that have been brought up and included some useful scripts as well. Jason Haar wrote:
I just want to thank everyone for their responses. Very interesting stuff!
I think I can paraphrase that SQL-backends don't give much advantage with large data sets due to the lack of relationships within syslog data, and the "fastest" solutions are going to be those that basically have custom-written "hot searches" pre-defined so that the appropriate indexes/extra files are already created to speed things up.
The comments about gziping the files to speed up reads was interesting as well...
It certainly an interesting problem. I want to do things like:
1. IDS event that IP 1.2.3.4 just did something bad against 3.4.5.6 2. I want to search logs for 7 days before this event for any other activity from IP address 1.2.3.4 (might be email, PIX ACL logs, etc) or from 3.4.5.6
or
1. User claims email never reached recipient 2. search for users email address 3. get report of all SMTP connection attempts, delivery attempts, AV and antispam/RBL records associated with path of message through 'n' different systems
those are all doable by hand - but very slow and - basically you need to have someone who knows what they are doing. Being able to put that behind a Web interface and make it a few clicks would be wonderful.