[syslog-ng] Collecting Logs on Loghost

[Wolfgang Braun]

Greetings

I'm writing my thesis on the subject of collecting and analysing
logfiles. i'm in the information gathering and i'm interested in your
opinions on the topic.

the primary idea is that hosts log all their syslog (and maybe other
files like http-logs) to a dedicated loghost where the log messages are
inserted into a DB. additionally there should be a filter program that
scans the message stream for certain patterns and triggers actions
(information gathering like backfinger, ripe data etc). 
things to be evaluated now are 

o how much net load does syslog create? 

one pitfall could be that logging to the extreme could have an impact on
network performance making it impossible  to transfer all logging
information. my view (not yet supported by numbers) is that the impact on
net performance should be rather low, but i'm setting up a test
environment to packet count. since i assume thet remote logging is common
practice i'm interested in your experiences. 

o how much data does a host generate?

this is of course dependant on which services are running on the machine
and the usage of the services. question is how much data a DB (e.g.
PostgreSQL) can handle (real time queries should be possible). this will
be taken care of in the test environment, too.


o common tools

i'm aware of several analyzing tools (swatch, logsurfer ... from
freshmeat). i like the logsurfer approach on creating contexts (to gather
information from different sources into a small report on specified
events). are there more tools like that that do a good job?

some unsorted ideas:

+ doing long term analysis via the DB interface, to maybe trace long term
patterns (slow portscans or door knob testing come to mind)

+ central administration of configurations and pushing of configurations
to 'agents' running on each host

+ default filter configurations to handle standard services and their
logs (bind, sendmail, ftpd), creation of regular reports


as said, i'd be very interested on your thoughts on the topic, your
experiences and maybe references to papers/tools.
 

wolfi
 
-- 
Internet ist das Ding mit den Kabeln.
Usenet ist das Ding mit den Menschen.
			-- Hubert Partl