high memory usage (swapping)
Hi, today i got a weird cenario: i rebooted our syslog-ng server and forgot to start the syslog-ng daemon. So far so (not) good. One of our webservers which sends his logs via UDP to the syslog-server became suddenly very slow. It is reproducable: whenever i stop the central syslog-server the sender-servers syslog-ng-process gets a very high memory usage (4GB!) within seconds... The log-traffic on this server is about 30MB/day... UDP is a connection-less protocol isnt? ;) The configuration of the sender-server: ----- options { sync (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; source s_sys { file ("/proc/kmsg" log_prefix("kernel: ")); unix-stream ("/dev/log"); internal(); # udp(ip(0.0.0.0) port(514)); }; destination a_udp { udp( "example.com" port(514) ); }; #log { source(s_sys); destination (a_udp); }; ----- sender: syslog-ng 2.1.3 reciever: syslog-ng 2.1.2 someone got an idea on this issue? Otto
On Thu, 2009-02-12 at 18:11 +0100, Otto Berger wrote:
Hi,
today i got a weird cenario: i rebooted our syslog-ng server and forgot to start the syslog-ng daemon. So far so (not) good.
One of our webservers which sends his logs via UDP to the syslog-server became suddenly very slow. It is reproducable: whenever i stop the central syslog-server the sender-servers syslog-ng-process gets a very high memory usage (4GB!) within seconds... The log-traffic on this server is about 30MB/day...
UDP is a connection-less protocol isnt? ;)
Sorry, for the delay, I'm trying to look into this next, but I have already received a SIGWIFE :) -- Bazsi
On Thu, 2009-02-12 at 18:11 +0100, Otto Berger wrote:
Hi,
today i got a weird cenario: i rebooted our syslog-ng server and forgot to start the syslog-ng daemon. So far so (not) good.
One of our webservers which sends his logs via UDP to the syslog-server became suddenly very slow. It is reproducable: whenever i stop the central syslog-server the sender-servers syslog-ng-process gets a very high memory usage (4GB!) within seconds... The log-traffic on this server is about 30MB/day...
UDP is a connection-less protocol isnt? ;)
The configuration of the sender-server:
----- options { sync (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); };
source s_sys { file ("/proc/kmsg" log_prefix("kernel: ")); unix-stream ("/dev/log"); internal(); # udp(ip(0.0.0.0) port(514)); };
destination a_udp { udp( "example.com" port(514) ); };
#log { source(s_sys); destination (a_udp); }; -----
sender: syslog-ng 2.1.3 reciever: syslog-ng 2.1.2
someone got an idea on this issue?
This patch seems to resolve it for me: commit 13c3a01fea103040c9066037b3deac76925b7777 Author: Balazs Scheidler <bazsi@balabit.hu> Date: Fri Feb 27 23:37:47 2009 +0100 [afsocket] in case of a broken destination connection, set the write fd to NULL The problem with leaving the destination fd set in this case is that it's going to be polled, which in turn will trigger POLLERR again, which in turn generates a bunch of messages thus creating a huge memory usage. Reported-By: Otto Berger I've fixed this in 3.0 earlier, but I didn't know it applied to 2.1 as well. You can find it in tomorrow's snapshot. -- Bazsi
participants (2)
-
Balazs Scheidler
-
Otto Berger