Geller, Sandor (IT) wrote:
Hi,
Anyhow. On the two central servers, I different numbers of records in the files, and the statistics on the sender show
2008-03-04T23:26:38-08:00 local@caribou.comp.uvic.ca syslog.info syslog-ng[3391]: Log statistics; dropped='tcp(AF_INET(server2:514))=0', dropped='tcp(AF_INET(server1:514))=14690649', processed='center(queued)=53993217', processed='center(received)=17997739', processed='destination(syslogServer2)=17997739', processed='destination(syslogServer1)=17997739', processed='destination(syslog)=17997739', processed='source(local)=17997739'
Please note that these numbers are aggregated, so these show the amount of messages arriving to/ leaving syslog-ng since it has been started.
Yes, I have taken the first statistic in the day and subtracted if from the last statistic in the day. There is some time between 00:00 and the first statistic, and the last statistic and 23:59.999... This time will total the interval between statistics. This is what I meant by "the statistics miss some time in the day."
The problem is that the files on disk show
caribou 16257954 server1 1742054 server2 965475
and that just doesn't add up. Neither server shows any dropped messages.
I think you misunderstood what dropped means. Dropped stands for messages which were undeliverable (maybe the TCP connection was broken you can check the internal messages of syslog-ng for such problems) so dropped messages never reached server1 or server2.
I didn't misunderstand this. The source server stats show 17997739 messages queued, 14690649 dropped to the syslogServer1 which means that 3307090 were delivered. SyslogServer1 stats report that no messages were dropped, but countinmg of log lines in the files on SyslogServer1 from caribou only totals 1742054. What happened to the other 1565036. Even worse, the syslog-ng stats on caribou report 0 drops to syslogServer2 destination, but it only recored 965475 message in its files. Where did the other 15292479 messages go? I can't loose 15292479 without having any reporting on it.
Caribou stats show that no messages are dropped to server2, and lots dropped to server1, however, server2 actually wrote more messages to disk. Server 1 is also about 30% faster than server2.
As I wrote above when the report shows that a destination had more dropped messages then that server received fewer messages than the other.
I agree, so where did all of the messages for syslogServer2 go?
Using the numbers above: destination(syslogServer1) processed 17997739 while the TCP destination dropped 14690649 messages. Although the TCP destination could be used by multiple destinations I guess destination(syslogServer1) and TCP(syslog1) could be mapped. This means that 3307090 messages were delivered to syslogServer1. And as this is still an aggregated number you should calculate the number of messages starting from the time when the central log server has been started so there is a chance that you have to check multiple logfiles on your servers because the period you're interested in may span multiple days.
I take this into account by taking the difference between the first and last statistic record in each day. It misses a little, but not much. Evan.