[syslog-ng] Assessing reliability - did we get all messages?

[Cary, Kim]

Hi folks,

I'm trying to figure out whether we're getting all messages from a PIX 
into the mysql database. I've done some tests and everything syslog-ng 
is writing to disk is going to our db on the application server. Now I 
need to figure out if syslog-ng is getting everything (or nearly) from 
the PIX.

Specifically my questions are:
  - Does 'garbage collecting while busy' imply I'm dropping packets?
  - Any suggestions for checking my UDP buffers on the syslog server?
  - I'd like to use TCP from the PIX > syslog-ng. Does that work?

Its a little hard to test as this is a production system on our 
internet gateway. My understanding is that Pix logging via TCP will 
STOP the transmission of packets if the syslog server goes away, so, we 
may not want to do that with a single syslog server.

Our setup is like this:

Application server (mysql log database & web-based search interface - 
MacOS X Panther)

^ TCP ^

Syslog server (perl script (fisq.pl) parses input and sends via perl 
dbi)

^fifo pipe^

Syslog server (syslog-ng 1.6.4 from sunfreeware, Solaris 9 1.5G ) > 
file on local disk

^UDP^

Pix 520

The network run for the PIX > Syslog server link is dedicated 100Mbps 
full duplex.
We get about 20Gb of syslog messages a day from that source.

Here's what syslog-ng says about itself:
May 31 22:27:11 logsrvr syslog-ng[347]: STATS: dropped 0
May 31 22:37:11 logsrvr syslog-ng[347]: STATS: dropped 0
May 31 22:45:14 logsrvr syslog-ng[347]: Garbage collecting while busy...
May 31 22:45:17 logsrvr syslog-ng[347]: Objects alive: 240, garbage 
collected: 2999765
May 31 22:47:11 logsrvr syslog-ng[347]: STATS: dropped 0
May 31 22:57:11 logsrvr syslog-ng[347]: STATS: dropped 0
May 31 23:04:21 logsrvr syslog-ng[347]: Garbage collecting while idle...
May 31 23:04:22 logsrvr syslog-ng[347]: Objects alive: 240, garbage 
collected: 665611
May 31 23:07:11 logsrvr syslog-ng[347]: STATS: dropped 0
May 31 23:17:11 logsrvr syslog-ng[347]: STATS: dropped 0

Here are our tuning options from syslog-ng.conf:
  options {
           use_fqdn(yes);
           sync(0);
           keep_hostname(yes);
           chain_hostnames(no);
           create_dirs(yes);
           log_fifo_size(100000);
           gc_busy_threshold(3000000);
           gc_idle_threshold(300);
         };

BTW, syslog-ng is 'the bomb'. I love what we can do with it more easily 
than stock syslog.

Kim