Assessing reliability - did we get all messages?
Hi folks, I'm trying to figure out whether we're getting all messages from a PIX into the mysql database. I've done some tests and everything syslog-ng is writing to disk is going to our db on the application server. Now I need to figure out if syslog-ng is getting everything (or nearly) from the PIX. Specifically my questions are: - Does 'garbage collecting while busy' imply I'm dropping packets? - Any suggestions for checking my UDP buffers on the syslog server? - I'd like to use TCP from the PIX > syslog-ng. Does that work? Its a little hard to test as this is a production system on our internet gateway. My understanding is that Pix logging via TCP will STOP the transmission of packets if the syslog server goes away, so, we may not want to do that with a single syslog server. Our setup is like this: Application server (mysql log database & web-based search interface - MacOS X Panther) ^ TCP ^ Syslog server (perl script (fisq.pl) parses input and sends via perl dbi) ^fifo pipe^ Syslog server (syslog-ng 1.6.4 from sunfreeware, Solaris 9 1.5G ) > file on local disk ^UDP^ Pix 520 The network run for the PIX > Syslog server link is dedicated 100Mbps full duplex. We get about 20Gb of syslog messages a day from that source. Here's what syslog-ng says about itself: May 31 22:27:11 logsrvr syslog-ng[347]: STATS: dropped 0 May 31 22:37:11 logsrvr syslog-ng[347]: STATS: dropped 0 May 31 22:45:14 logsrvr syslog-ng[347]: Garbage collecting while busy... May 31 22:45:17 logsrvr syslog-ng[347]: Objects alive: 240, garbage collected: 2999765 May 31 22:47:11 logsrvr syslog-ng[347]: STATS: dropped 0 May 31 22:57:11 logsrvr syslog-ng[347]: STATS: dropped 0 May 31 23:04:21 logsrvr syslog-ng[347]: Garbage collecting while idle... May 31 23:04:22 logsrvr syslog-ng[347]: Objects alive: 240, garbage collected: 665611 May 31 23:07:11 logsrvr syslog-ng[347]: STATS: dropped 0 May 31 23:17:11 logsrvr syslog-ng[347]: STATS: dropped 0 Here are our tuning options from syslog-ng.conf: options { use_fqdn(yes); sync(0); keep_hostname(yes); chain_hostnames(no); create_dirs(yes); log_fifo_size(100000); gc_busy_threshold(3000000); gc_idle_threshold(300); }; BTW, syslog-ng is 'the bomb'. I love what we can do with it more easily than stock syslog. Kim
On Wed, 1 Jun 2005, Cary, Kim wrote:
Hi folks,
I'm trying to figure out whether we're getting all messages from a PIX into the mysql database. I've done some tests and everything syslog-ng is writing to disk is going to our db on the application server. Now I need to figure out if syslog-ng is getting everything (or nearly) from the PIX.
You'll need to perform what amounts to empirical experimentation, here. In short, fork your logging to another logging platform and compare the results. - billn
I'd have a look at your netstat output on the syslog server.. For Solaris: netstat -s -P udp I am pretty sure that "udpInErrors" will represent packets dropped from teh UDP buffer, but am unsure.. I don't think I would use syslog over TCP from a PIX for the exact reason you said..the PIX will stop until the syslog server is ready..instead, maybe just increase your UDP buffer size. Depending on what else your syslog server is doing..crank it up to 64MBytes or more..RAM is pretty cheap. On Wed, 1 Jun 2005, Cary, Kim wrote:
Hi folks,
I'm trying to figure out whether we're getting all messages from a PIX into the mysql database. I've done some tests and everything syslog-ng is writing to disk is going to our db on the application server. Now I need to figure out if syslog-ng is getting everything (or nearly) from the PIX.
Specifically my questions are: - Does 'garbage collecting while busy' imply I'm dropping packets? - Any suggestions for checking my UDP buffers on the syslog server? - I'd like to use TCP from the PIX > syslog-ng. Does that work?
Its a little hard to test as this is a production system on our internet gateway. My understanding is that Pix logging via TCP will STOP the transmission of packets if the syslog server goes away, so, we may not want to do that with a single syslog server.
Our setup is like this:
Application server (mysql log database & web-based search interface - MacOS X Panther)
^ TCP ^
Syslog server (perl script (fisq.pl) parses input and sends via perl dbi)
^fifo pipe^
Syslog server (syslog-ng 1.6.4 from sunfreeware, Solaris 9 1.5G ) > file on local disk
^UDP^
Pix 520
The network run for the PIX > Syslog server link is dedicated 100Mbps full duplex. We get about 20Gb of syslog messages a day from that source.
Here's what syslog-ng says about itself: May 31 22:27:11 logsrvr syslog-ng[347]: STATS: dropped 0 May 31 22:37:11 logsrvr syslog-ng[347]: STATS: dropped 0 May 31 22:45:14 logsrvr syslog-ng[347]: Garbage collecting while busy... May 31 22:45:17 logsrvr syslog-ng[347]: Objects alive: 240, garbage collected: 2999765 May 31 22:47:11 logsrvr syslog-ng[347]: STATS: dropped 0 May 31 22:57:11 logsrvr syslog-ng[347]: STATS: dropped 0 May 31 23:04:21 logsrvr syslog-ng[347]: Garbage collecting while idle... May 31 23:04:22 logsrvr syslog-ng[347]: Objects alive: 240, garbage collected: 665611 May 31 23:07:11 logsrvr syslog-ng[347]: STATS: dropped 0 May 31 23:17:11 logsrvr syslog-ng[347]: STATS: dropped 0
Here are our tuning options from syslog-ng.conf: options { use_fqdn(yes); sync(0); keep_hostname(yes); chain_hostnames(no); create_dirs(yes); log_fifo_size(100000); gc_busy_threshold(3000000); gc_idle_threshold(300); };
BTW, syslog-ng is 'the bomb'. I love what we can do with it more easily than stock syslog.
Kim _______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
participants (3)
-
Bill Nash
-
Cary, Kim
-
Mike