[syslog-ng] Anyone got a well performing search interface for syslog data?

[Al Tobey]

I rotate logs on my central server (for about 60 hosts) nightly into a
simple linux filesystem and compress them with gzip -9.   I have about
5 years online and searchable.   My web interface for searching is
written in perl and lets you search by day/month/year/time.  It is
quite fast.   In fact, the main slowdown is the browser when using
html tables for layout instead of straight text when there's a lot of
data to display.   I'll look into getting permission to post my viewer
on the web.  It's a few years old and a little crufty but could be
cleaned up easily enough.

I just ran a test searching for cfengine security /cfengine.*SECURITY/
events over the last 30 days/files.   It took about 1.5 minutes to
run.  The script only got about 20% cpu (dual xeon 2800), so I'm
betting most of the time was I/O even though this was on some fast EVA
disk.   The advantage of staying with flat files in I/O is it's a
linear read through the file, which almost every OS does very fast.  
Technically, mysql tables could be linear reads, but will never match
the raw speed of perl when doing regular expressions linearly through
a file, especially when the RE's are well written.   By the way,
compression may actually help I/O throughput if you have fast CPU and
slow disk, since the reads from disk will be smaller.   Having extra
system memory for buffer cache and readahead helps, too.

-Al Tobey
Senior Unix Engineer
Priority Health

On 9/6/05, Jason Haar <Jason.Haar at trimble.co.nz> wrote:
> We're generating around 4Gb syslog data per week, and I'm looking for a
> good search interface into it.
> 
> I can cut my way through it with egrep/etc, but waiting 10-15min for a
> result really isn't going to break any speed records. Especially when I
> then need to re-run it with another "grep" on the end of it! ;-)
> 
> I have tried injecting it into a MySQL database using some schemas I've
> found on the Internet - but the performance didn't seem much better to
> me - and you lost the "free-text" attributes of grep (or more
> specifically, the sorts of searches I find I want to do aren't
> SQL-friendly).
> 
> Has anyone come up with a good speedy way of coping with Gbytes of
> syslog data? Or is it time to invest in some Appliance or the like?
> 
> --
> Cheers
> 
> Jason Haar
> Information Security Manager, Trimble Navigation Ltd.
> Phone: +64 3 9635 377 Fax: +64 3 9635 417
> PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
> 
> _______________________________________________
> syslog-ng maillist  -  syslog-ng at lists.balabit.hu
> https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
> 
>