I've googled this topic and read the mailing list and haven't had much luck improving my situation. I have a very high volume syslog-ng server. I currently have logs that are being received across the network but not being written to disk. This could be as much as 25% of the logs being dropped. The STAT to syslog always says 0 drops. It's a Netro 1405 w/ two CPUs and 1 gig of memory connected to a SAN. syslog-ng is always at 49% CPU and never goes beyond that. What settings can I use to stop dropping logs? This is an output of iostate with an interval of 1. This will give you an idea of performance. As you can see there are no disk waits. Thanks. device r/s w/s Mr/s Mw/s wait actv svc_t %w %b md89 2.0 86.0 0.0 0.4 0.0 0.2 1.9 0 5 md89 0.0 3.0 0.0 0.0 0.0 0.0 0.9 0 0 md89 0.0 1.0 0.0 0.0 0.0 0.0 2.5 0 0 md89 0.0 1.0 0.0 0.0 0.0 0.0 1.2 0 0 md89 0.0 2.0 0.0 0.0 0.0 0.0 1.2 0 0 md89 0.0 61.0 0.0 0.3 0.0 0.1 1.3 0 3 md89 0.0 1.0 0.0 0.0 0.0 0.0 1.5 0 0 md89 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 md89 0.0 2.0 0.0 0.0 0.0 0.0 0.9 0 0 md89 0.0 2.0 0.0 0.0 0.0 0.0 0.7 0 0 md89 0.0 83.0 0.0 0.4 0.0 0.2 2.2 0 3 md89 2.0 4.0 0.0 0.0 0.0 0.0 3.0 0 2 md89 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 md89 0.0 1.0 0.0 0.0 0.0 0.0 0.8 0 0 md89 0.0 1.0 0.0 0.0 0.0 0.0 0.8 0 0 md89 1.0 86.0 0.0 0.4 0.0 0.1 1.2 0 4 md89 0.0 2.0 0.0 0.0 0.0 0.0 0.9 0 0 md89 0.0 5.0 0.0 0.0 0.0 0.0 0.9 0 0 md89 0.0 1.0 0.0 0.0 0.0 0.0 1.0 0 0 md89 0.0 2.0 0.0 0.0 0.0 0.0 0.8 0 0 md89 2.0 751.9 0.0 5.6 0.0 6.5 8.6 0 25
On Wed, 26 Oct 2005, JR Mayberry wrote:
I've googled this topic and read the mailing list and haven't had much luck improving my situation.
I have a very high volume syslog-ng server. I currently have logs that are being received across the network but not being written to disk. This could be as much as 25% of the logs being dropped. The STAT to syslog always says 0 drops.
are you sure that they are being received? if htey are coming in over UDP, maybe check some netstat output to see if they are being dropped by the kernel? (in this case they would be dropped before syslog-ng can even see that would be the drops would be listed as zero)
It's a Netro 1405 w/ two CPUs and 1 gig of memory connected to a SAN. syslog-ng is always at 49% CPU and never goes beyond that.
probably means that 1 CPU is capped out...since syslog-ng is not threaded, that is the max you can get...what does your config file look like? lots of regex's? if possible, simplify. also try increasing your receive buffer queue size for UDP (make it as big as you can...64Mbytes, 128Mbytes etc). that should help with bursts of traffic, but if it is constant there would be no help. if that still doesn't help...then maybe fire up a second syslog-ng listening on a second port, then try to split up your traffic?
What settings can I use to stop dropping logs?
This is an output of iostate with an interval of 1. This will give you an idea of performance. As you can see there are no disk waits.
Thanks.
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b md89 2.0 86.0 0.0 0.4 0.0 0.2 1.9 0 5 md89 0.0 3.0 0.0 0.0 0.0 0.0 0.9 0 0 md89 0.0 1.0 0.0 0.0 0.0 0.0 2.5 0 0 md89 0.0 1.0 0.0 0.0 0.0 0.0 1.2 0 0 md89 0.0 2.0 0.0 0.0 0.0 0.0 1.2 0 0 md89 0.0 61.0 0.0 0.3 0.0 0.1 1.3 0 3 md89 0.0 1.0 0.0 0.0 0.0 0.0 1.5 0 0 md89 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 md89 0.0 2.0 0.0 0.0 0.0 0.0 0.9 0 0 md89 0.0 2.0 0.0 0.0 0.0 0.0 0.7 0 0 md89 0.0 83.0 0.0 0.4 0.0 0.2 2.2 0 3 md89 2.0 4.0 0.0 0.0 0.0 0.0 3.0 0 2 md89 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 md89 0.0 1.0 0.0 0.0 0.0 0.0 0.8 0 0 md89 0.0 1.0 0.0 0.0 0.0 0.0 0.8 0 0 md89 1.0 86.0 0.0 0.4 0.0 0.1 1.2 0 4 md89 0.0 2.0 0.0 0.0 0.0 0.0 0.9 0 0 md89 0.0 5.0 0.0 0.0 0.0 0.0 0.9 0 0 md89 0.0 1.0 0.0 0.0 0.0 0.0 1.0 0 0 md89 0.0 2.0 0.0 0.0 0.0 0.0 0.8 0 0 md89 2.0 751.9 0.0 5.6 0.0 6.5 8.6 0 25
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
Mike wrote:
I have a very high volume syslog-ng server. I currently have logs that are being received across the network but not being written to disk. This could be as much as 25% of the logs being dropped. The STAT to syslog always says 0 drops.
are you sure that they are being received? if htey are coming in over UDP, maybe check some netstat output to see if they are being dropped by the kernel? (in this case they would be dropped before syslog-ng can even see that would be the drops would be listed as zero)
I've just checked my syslog-ng-1.6.8 CentOS-4.1 server and discover I have a similar problem. I wrote a quick UDP syslog record generator using Net::Syslog and used it to pump 30,000 records in 3 forks (i.e. 3 x 10,000) at our syslog-ng server - and only received 29,987. I also ran tcpdump on the syslog-ng server and can confirm 30,000 UDP syslog packets were received. I have "log_fifo_size (10000)" set, have dns enabled, and have multiple files and directory trees opened by syslog-ng - "STATS: dropped 0" is what "stats()" is returning. I've run it multiple times now - it never equals 30,000 - always losing 5-50 events. -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +64 3 9635 377 Fax: +64 3 9635 417 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
are you sure that they are being received? if htey are coming in over UDP, maybe check some netstat output to see if they are being dropped by the kernel? (in this case they would be dropped before syslog-ng can even see that would be the drops would be listed as zero)
I've just checked my syslog-ng-1.6.8 CentOS-4.1 server and discover I have a similar problem. I wrote a quick UDP syslog record generator using Net::Syslog and used it to pump 30,000 records in 3 forks (i.e. 3 x 10,000) at our syslog-ng server - and only received 29,987. I also ran tcpdump on the syslog-ng server and can confirm 30,000 UDP syslog packets were received.
I have "log_fifo_size (10000)" set, have dns enabled, and have multiple files and directory trees opened by syslog-ng - "STATS: dropped 0" is what "stats()" is returning.
stats() shows messages that syslog-ng has received, but was not able to write to one of it's outputs in time (that is where log_fifo_size() comes in) >
I've run it multiple times now - it never equals 30,000 - always losing 5-50 events.
check the output of: netstat -su do you see anything for "packet receive errors"? try running your send again...did that number grow? can't remember the command right now, but there is an option to adjust this with a sysctl command....
-- Cheers
Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +64 3 9635 377 Fax: +64 3 9635 417 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
Mike wrote:
check the output of: netstat -su
do you see anything for "packet receive errors"? try running your send again...did that number grow?
can't remember the command right now, but there is an option to adjust this with a sysctl command....
THANK YOU!!!! Right on the money. I noted the value, sent 30,000 events. Noted that only 29,980 were received.... and 20 extra "packet receive errors" were recorded. I'll take a look at sysctl. Maybe this should be documented? Or maybe I'm being stupid. I don't deal with 10,000 events/sec - I'm lucky to see 100/sec - so probably I'm losing nothing with my current levels anyway. -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +64 3 9635 377 Fax: +64 3 9635 417 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
I am not so lucky... so back to square one... UDP udpInDatagrams =3830279039 udpInErrors = 0 udpOutDatagrams =4197984 udpOutErrors = 0 On Thu, 27 Oct 2005, Jason Haar wrote:
Mike wrote:
check the output of: netstat -su
do you see anything for "packet receive errors"? try running your send again...did that number grow?
can't remember the command right now, but there is an option to adjust this with a sysctl command....
THANK YOU!!!!
Right on the money. I noted the value, sent 30,000 events. Noted that only 29,980 were received.... and 20 extra "packet receive errors" were recorded.
I'll take a look at sysctl. Maybe this should be documented? Or maybe I'm being stupid. I don't deal with 10,000 events/sec - I'm lucky to see 100/sec - so probably I'm losing nothing with my current levels anyway.
participants (3)
-
Jason Haar
-
JR Mayberry
-
Mike