[syslog-ng] Setting UDP Receive Buffer size, then when increase performance problems

[James]

SUMMARY:

Due to the large number of UDP packets (aprox: 6,700 per second) coming 
into our syslog-ng (version 1.6.4) running on an HP-UX 11.11 server we 
have had a desire to increase the receive buffer for UDP.  I could NOT 
find a way to do this in syslog-ng, so we changed the default at the OS 
level (socket_udp_rcvbuf_default).  Has it been considered (or possibly 
already added) to allow this option to be specified in the config file?

Increasing this buffer size, seemed to cause  a performance problem 
because syslog-ng (again at least on HP-UX 11.11) has POLLIN set to 
POLLRDNORM|POLLRDBAND (default when _XOPEN_SOURCE_EXTENDED is set) which 
must mean the OS walks the entire buffer looking for priority message when 
poll() is called.  Since syslog-ng appears to do a poll(), then 1 read(), 
followed by another poll(), the OS is walking the entire buffer, and then 
only reading 1 entry.  During the time the OS spends walking this buffer, 
hundreds of packets have been added.  Thus, syslog-ng can reach a point of 
no return as long as data keeps coming in at this rate.  I did a quick 
change in io.c to override the POLLIN setting HP-UX sets for syslog-ng to 
the value it would use if _XOPEN_SOURCE_EXTENDED was NOT set (0x0001). 
This change seems to work for us, but more details and maybe another 
option is mentioned in my details below.







DETAILS:

We have an HP-UX system that is regularly receiving 6,700 log entries per 
second from several systems via UDP.  Originally we tried handling this 
workload with one daemon, but was CPU constrained due to it being single 
threaded.  We then split the work among 3 daemons each on their own 
virtual IP so we could utilize the other CPUs in the system.  We also 
increased socket_udp_rcvbuf_default (default receive buffer size for UDP) 
from 65K to 75mb - 150mb.  It would be nice if the syslog-ng config file 
allowed us to set this value instead of impacting ALL UDP listeners on the 
system.  I have edited our copy of io.c to include something like the 
following.  This example is crude, but seems to work for us setting 75Mb 
buffer (I hard coded the value, but obtaining from the config file would 
be a better solution):

    int Buffer = 78643199;
    socklen_t Len = sizeof(Buffer);

    if ( (setsockopt(fd, SOL_SOCKET, SO_RCVBUF, &Buffer, Len)) == 0 ) {
      Buffer=0;
      getsockopt(fd, SOL_SOCKET, SO_RCVBUF, &Buffer, &Len);
      debug("io.c: SO_RCVBUF now set to: %i\n", Buffer);
    }
    else {
      werror("io.c: setsockopt() failed setting SO_RCVBUF (errno %i), %z\n", errno, strerror(errno));
    }


However, this can lead to another problem with syslog-ng (at least on on 
HP-UX).  Once the buffer starts to fill the OS starts spending a LOT of 
CPU in system time.  This appears to be because of the socket options 
specified due to _INCLUDE_XOPEN_SOURCE_EXTENDED being set during compile. 
On HP-UX 11.11 poll.h sets POLLIN to (POLLRDNORM|POLLRDBAND) when 
_INCLUDE_XOPEN_SOURCE_EXTENDED is defined.  It is my understanding that 
this means the OS will walk the entire buffer looking for a priority 
message when syslog-ng calls poll() to look for the next packet.  Once we 
increase the buffer past the 65K default in HP-UX this walking can take 
some time and lots of system CPU time (hate to see this if buffer was set 
the the 2GB maximum).  Overriding the POLLIN options to the setting poll.h 
would use if _INCLUDE_XOPEN_SOURCE_EXTENDED was NOT defined (0x0001) seems 
to resolve this problem, and actually allows syslog-ng to empty out this 
large UDP buffer space very quickly.

Another solution (which looks like may have been talked about in a post 
from John Morrissey) may be to change the way syslog-ng loops on the UDP 
socket. It appears to me that syslog-ng is currently doing:
   poll()
   read()
   Back to poll()

This delay maybe could only be encountered once if syslog-ng was to do 
something like:
   poll()
   read() with O_NONBLOCK set
   if return -1 with EAGAIN, Back to poll()
   else Back to read() with O_NONBLOCK set

I am not sure what impact this would have if syslog-ng was receiving data 
from more than one UDP socket (which is what we do) as it may spend too 
much time doing emptying the one buffer while the other data listeners 
wait.


Have concerns like this already been addressed in more current releases? 
Do these issues I have seen HP-UX concerns for other operating systems, or 
should I just continue to modify the standard syslog-ng code as there 
would be little to no benefit for others?

Thanks,
/James