[syslog-ng] Log analyzer/syslog-ng problems

[Bryan Stillwell]

Hello everyone,

I'm currently doing research at my job about writing a log analyzer.
The first step we wish to accomplish is setting up a central log
server.  The server will be just for logs and will be secured so that
it would be almost impossible for someone who has compromised another
machine on the netowrk to modify the log files.  The log server will be
logging both internal and external machines, so we would really like
for communication to be well encrypted.  As I understand this option
won't be implemented until the next version of syslog-ng.

After this step is done we will set it up to automatically enter all
logs into a postgresql database or mysql database for easier analysis.
(I've been told postgresql has some native IP address data types that
will come in handy.)

However our number one item on the list is to make sure that the logs
get to the server in a timely, consistant, and accurate manner.  So I
started doing some testing on various things that might cause problems
during the transfering of the log data.  Some of the things that I want
to watch out for are:

o What happens if the network is down for a day or two at a time?
  I want to make sure that they don't just disappear.

o What if the network is being hit with a DOS attack and packets never
  reach the log server?  or they reach the log server but the reply
  never gets back to the source computer?

o What kind of verification can be done to make sure all the data
  made it to the log server ok?  Should we log locally and remotely
  and compare the two at the end of each day?  or can syslog-ng take
  care of it all?

o What about if the main server dies and has to be restarted?  client
  dies?

Some of those questions will probably have the same answers, but I ran
into the following problem when I started doing some stress tests
between two computers on the internal lan.  I've included the two
syslog-ng.conf files to the end of this message.

reptile is the client computer and testbox is the server.  I ran the
command `logger -f /usr/dict/linux.words` to test out how well it
works.

Apr 13 10:39:27 src@reptile/reptile logger: Aarhus
Apr 13 10:39:27 src@reptile/reptile logger: Aaron
...
Apr 13 10:39:38 src@reptile/reptile logger: strips
Apr 13 10:39:38 src@reptile/reptile logger: striptease
Apr 13 10:39:40 src@reptile/reptile logger: worked
Apr 13 10:39:40 src@reptile/reptile logger: worker
...
Apr 13 10:39:40 src@reptile/reptile logger: worn
Apr 13 10:39:40 src@reptile/reptile logger: worried
Apr 13 10:39:40 src@reptile/reptile logger<13>Apr 13 10:39:38 src@reptile logger: strive
Apr 13 10:39:38 src@reptile/reptile logger: striven
Apr 13 10:39:38 src@reptile/reptile logger: strives
...
Apr 13 10:39:40 src@reptile/reptile logger: workbook
Apr 13 10:39:40 src@reptile/reptile logger: workbooks
Apr 13 10:34:54 reptile/reptile : worrier
Apr 13 10:39:40 src@reptile/reptile logger: worriers
Apr 13 10:39:40 src@reptile/reptile logger: worries
...
Apr 13 10:39:40 src@reptile/reptile logger: Zulus
Apr 13 10:39:40 src@reptile/reptile logger: Zurich

So it works until it hits 'striptease' (purely coincidence btw) and then
skips to 'worked'.  When it hits 'worried' it switches back to where it
left off and continues until it hits 'workbooks' where the previous group
was supposed to be at.

Other than that problem everything transferred fine.  Any ideas on how
to have that autofix itself, maybe I don't have the conf files setup
correctly?  Would it be better to use udp even though dropped packets
will never be known about?  I just want to maintain the upmost integrity
in the log files.

--- testbox(server) syslog-ng.conf ---
options { long_hostnames(on); sync(0); log_fifo_size(100); };

source src {
	unix-dgram("/dev/log");
	tcp();
	internal();
};

destination local_log {
	file("/tmp/deleteme.log");
};

log { source(src); destination(local_log); };
--- testbox(server) syslog-ng.conf ---


--- reptile(client) syslog-ng.conf ---
options { sync(0); log_fifo_size(100000); };

source src {
	unix-dgram("/dev/log");
	internal();
};

destination master_log_server {
	tcp("10.1.1.10" port(514));
};

log { source(src); destination(master_log_server); };
--- reptile(client) syslog-ng.conf ---

Thank you,
Bryan Stillwell

-- 
http://www.tummy.com/ Consulting and Software for Linux and Unix
KRUD - Kevin's Red Hat Uber Distribution - the Freshest Red Hat every month