I really don't recommend php-syslog-ng, I have been using it for almost a month now and it has been extremely slow. I would be interested to see these perl scripts that Al Tobey talked about. What I have done is setup SEC for a monitoring system and just receive notifications on information I care about. Until I can come up with something quicker we are still using php-syslog-ng for allowing management and controllers to look up information from the logs. - Ken Jason Haar wrote:
We're generating around 4Gb syslog data per week, and I'm looking for a good search interface into it.
I can cut my way through it with egrep/etc, but waiting 10-15min for a result really isn't going to break any speed records. Especially when I then need to re-run it with another "grep" on the end of it! ;-)
I have tried injecting it into a MySQL database using some schemas I've found on the Internet - but the performance didn't seem much better to me - and you lost the "free-text" attributes of grep (or more specifically, the sorts of searches I find I want to do aren't SQL-friendly).
Has anyone come up with a good speedy way of coping with Gbytes of syslog data? Or is it time to invest in some Appliance or the like?