syslog-ng stats and performance
Hello, I'm running into an issue where we're fairly certain that we're dropping log messages somewhere along this path: device -> network -> VMware -> RHEL host -> syslog-ng What I'd like to understand better is what statistics I can gather from syslog-ng itself to help show or rule out drops in the software. I'm using the following general config: syslog-ng 3.2.5 Installer-Version: 3.2.5 Revision: ssh+git://bazsi@git.balabit //var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.2#master#9d4bea28198bd731df1a61e980a2af5b88d81116 Compile-Date: Jan 15 2012 19:47:30 Enable-Threads: on Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-SSL: off Enable-SQL: on Enable-Linux-Caps: off Enable-Pcre: on Enable-Pacct: off options { flush_lines (100); time_reopen (2); log_iw_size(100); log_fifo_size (65536); log_msg_size(8192); long_hostnames (off); use_dns(yes); use_fqdn(yes); keep_hostname (no); stats_freq(3600); stats_level(1); dns_cache(yes); keep_timestamp(no); }; When I looked at "syslog-ng-ctl stats" I see these "types" dropped processed stamp stored However I only see "dropped" counters for tcp destinations, and not for any of the sources or local destinations. Does "dropped" only make sense in the remote destination case? Is there anything I can turn on/examine to tune my syslog-ng performance and verify whether I have drops occurring within syslog-ng? For the RHEL host portion I've tried watching netstat -su and netstat -st but the error counters for those do not seem to indicate that the level of issue we're seeing lies there. The processor for syslog-ng is busy, but averages 75%... Is it foolish to expect VMware to keep up with the level of logs we're taking in? Might virtualization be hiding drops from me? Any help appreciated... Cheers, Jesse -- Jesse Bowling
Perhaps my original question was a bit too broad...I'm mostly interested in the below section: On Fri, Feb 28, 2014 at 4:59 PM, Jesse Bowling <jessebowling@gmail.com>wrote:
However I only see "dropped" counters for tcp destinations, and not for any of the sources or local destinations. Does "dropped" only make sense in the remote destination case? Is there anything I can turn on/examine to tune my syslog-ng performance and verify whether I have drops occurring within syslog-ng?
Can anyone help me understand what I will and won't see in the syslog-ng stats? Cheers, Jesse -- Jesse Bowling
On Feb 28, 2014 10:59 PM, "Jesse Bowling" <jessebowling@gmail.com> wrote:
Hello,
I'm running into an issue where we're fairly certain that we're dropping
log messages somewhere along this path:
device -> network -> VMware -> RHEL host -> syslog-ng
What I'd like to understand better is what statistics I can gather from
syslog-ng itself to help show or rule out drops in the software. I'm using the following general config:
syslog-ng 3.2.5 Installer-Version: 3.2.5 Revision: ssh+git://bazsi@git.balabit
//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.2#master#9d4bea28198bd731df1a61e980a2af5b88d81116
Compile-Date: Jan 15 2012 19:47:30 Enable-Threads: on Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-SSL: off Enable-SQL: on Enable-Linux-Caps: off Enable-Pcre: on Enable-Pacct: off
options { flush_lines (100); time_reopen (2); log_iw_size(100); log_fifo_size (65536); log_msg_size(8192); long_hostnames (off); use_dns(yes); use_fqdn(yes); keep_hostname (no); stats_freq(3600); stats_level(1); dns_cache(yes); keep_timestamp(no); };
When I looked at "syslog-ng-ctl stats" I see these "types"
dropped processed stamp stored
However I only see "dropped" counters for tcp destinations, and not for any of the sources or local destinations.
Dropped is defined for destinations only. Sources never drop messages. What do you mean on 'local destinations'? What happens is that sources dispatch incoming messages to all configured destinations. Destinations keep a queue of messages that are being sent. If the destination queue is full, new messages get dropped, but this should only happen if the destination is slower and flow control is not enabled on the specific path. You can increase the queue size with the log-fifo-size option. You can enable flow control using flags(flow-control) within the log statement.
Does "dropped" only make sense in the remote destination case? Is there anything I can turn on/examine to tune my syslog-ng performance and verify whether I have drops occurring within syslog-ng?
For the RHEL host portion I've tried watching netstat -su and netstat -st but the error counters for those do not seem to indicate that the level of issue we're seeing lies there. The processor for syslog-ng is busy, but averages 75%...
Is it foolish to expect VMware to keep up with the level of logs we're taking in? Might virtualization be hiding drops from me?
It depends on how much logs you have. UDP is a tricky beast with a number of drop points both within and outside syslog-ng. Syslog-ng is able to process hundreds of thousands of messages in some use cases. With udp the biggest issue is dropping packets in the kernel receive queue, but you can scale that to 20-30k msg per second.
Any help appreciated...
Cheers,
Jesse
-- Jesse Bowling
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
If it is a network issue (typically with load on the receiving side) you can try netstat -su It can't let you know if a packet (udp) did not make it to the host, but it can show things that the kernel identifies that caused the packet to not make it to the application. Might help a little Jim On 03/07/2014 01:25 AM, Balazs Scheidler wrote:
On Feb 28, 2014 10:59 PM, "Jesse Bowling" <jessebowling@gmail.com <mailto:jessebowling@gmail.com>> wrote:
Hello,
I'm running into an issue where we're fairly certain that we're
dropping log messages somewhere along this path:
device -> network -> VMware -> RHEL host -> syslog-ng
What I'd like to understand better is what statistics I can gather
from syslog-ng itself to help show or rule out drops in the software. I'm using the following general config:
syslog-ng 3.2.5 Installer-Version: 3.2.5 Revision:
ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.2#master#9d4bea28198bd731df1a61e980a2af5b88d81116
Compile-Date: Jan 15 2012 19:47:30 Enable-Threads: on Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-SSL: off Enable-SQL: on Enable-Linux-Caps: off Enable-Pcre: on Enable-Pacct: off
options { flush_lines (100); time_reopen (2); log_iw_size(100); log_fifo_size (65536); log_msg_size(8192); long_hostnames (off); use_dns(yes); use_fqdn(yes); keep_hostname (no); stats_freq(3600); stats_level(1); dns_cache(yes); keep_timestamp(no); };
When I looked at "syslog-ng-ctl stats" I see these "types"
dropped processed stamp stored
However I only see "dropped" counters for tcp destinations, and not for any of the sources or local destinations.
Dropped is defined for destinations only. Sources never drop messages.
What do you mean on 'local destinations'?
What happens is that sources dispatch incoming messages to all configured destinations. Destinations keep a queue of messages that are being sent.
If the destination queue is full, new messages get dropped, but this should only happen if the destination is slower and flow control is not enabled on the specific path.
You can increase the queue size with the log-fifo-size option.
You can enable flow control using flags(flow-control) within the log statement.
Does "dropped" only make sense in the remote destination case? Is there anything I can turn on/examine to tune my syslog-ng performance and verify whether I have drops occurring within syslog-ng?
For the RHEL host portion I've tried watching netstat -su and netstat -st but the error counters for those do not seem to indicate that the level of issue we're seeing lies there. The processor for syslog-ng is busy, but averages 75%...
Is it foolish to expect VMware to keep up with the level of logs we're taking in? Might virtualization be hiding drops from me?
It depends on how much logs you have.
UDP is a tricky beast with a number of drop points both within and outside syslog-ng.
Syslog-ng is able to process hundreds of thousands of messages in some use cases. With udp the biggest issue is dropping packets in the kernel receive queue, but you can scale that to 20-30k msg per second.
Any help appreciated...
Cheers,
Jesse
-- Jesse Bowling
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (3)
-
Balazs Scheidler
-
Jesse Bowling
-
Jim Hendrick