[syslog-ng] Assessing reliability - did we get all messages?

[Mike]

I'd have a look at your netstat output on the syslog server..

For Solaris:
netstat -s -P udp
I am pretty sure that "udpInErrors" will represent packets dropped from 
teh UDP buffer, but am unsure..

I don't think I would use syslog over TCP from a PIX for the exact reason 
you said..the PIX will stop until the syslog server is ready..instead, 
maybe just increase your UDP buffer size. Depending on what else your 
syslog server is doing..crank it up to 64MBytes or more..RAM is pretty 
cheap.

On Wed, 1 Jun 2005, Cary, Kim wrote:

> Hi folks,
> 
> I'm trying to figure out whether we're getting all messages from a PIX 
> into the mysql database. I've done some tests and everything syslog-ng 
> is writing to disk is going to our db on the application server. Now I 
> need to figure out if syslog-ng is getting everything (or nearly) from 
> the PIX.
> 
> Specifically my questions are:
>   - Does 'garbage collecting while busy' imply I'm dropping packets?
>   - Any suggestions for checking my UDP buffers on the syslog server?
>   - I'd like to use TCP from the PIX > syslog-ng. Does that work?
> 
> Its a little hard to test as this is a production system on our 
> internet gateway. My understanding is that Pix logging via TCP will 
> STOP the transmission of packets if the syslog server goes away, so, we 
> may not want to do that with a single syslog server.
> 
> Our setup is like this:
> 
> Application server (mysql log database & web-based search interface - 
> MacOS X Panther)
> 
> ^ TCP ^
> 
> Syslog server (perl script (fisq.pl) parses input and sends via perl 
> dbi)
> 
> ^fifo pipe^
> 
> Syslog server (syslog-ng 1.6.4 from sunfreeware, Solaris 9 1.5G ) > 
> file on local disk
> 
> ^UDP^
> 
> Pix 520
> 
> The network run for the PIX > Syslog server link is dedicated 100Mbps 
> full duplex.
> We get about 20Gb of syslog messages a day from that source.
> 
> Here's what syslog-ng says about itself:
> May 31 22:27:11 logsrvr syslog-ng[347]: STATS: dropped 0
> May 31 22:37:11 logsrvr syslog-ng[347]: STATS: dropped 0
> May 31 22:45:14 logsrvr syslog-ng[347]: Garbage collecting while busy...
> May 31 22:45:17 logsrvr syslog-ng[347]: Objects alive: 240, garbage 
> collected: 2999765
> May 31 22:47:11 logsrvr syslog-ng[347]: STATS: dropped 0
> May 31 22:57:11 logsrvr syslog-ng[347]: STATS: dropped 0
> May 31 23:04:21 logsrvr syslog-ng[347]: Garbage collecting while idle...
> May 31 23:04:22 logsrvr syslog-ng[347]: Objects alive: 240, garbage 
> collected: 665611
> May 31 23:07:11 logsrvr syslog-ng[347]: STATS: dropped 0
> May 31 23:17:11 logsrvr syslog-ng[347]: STATS: dropped 0
> 
> Here are our tuning options from syslog-ng.conf:
>   options {
>            use_fqdn(yes);
>            sync(0);
>            keep_hostname(yes);
>            chain_hostnames(no);
>            create_dirs(yes);
>            log_fifo_size(100000);
>            gc_busy_threshold(3000000);
>            gc_idle_threshold(300);
>          };
> 
> BTW, syslog-ng is 'the bomb'. I love what we can do with it more easily 
> than stock syslog.
> 
> Kim
> _______________________________________________
> syslog-ng maillist  -  syslog-ng at lists.balabit.hu
> https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
> 
>