[syslog-ng] Fixed number fd-limit doesn't scale

[Balazs Scheidler]

On Sat, 2009-01-17 at 10:30 +0100, Corinna Vinschen wrote:
> Hi,
> 
> On Jan 16 20:53, Balazs Scheidler wrote:
> > On Wed, 2009-01-14 at 13:12 +0100, Corinna Vinschen wrote:
> > > Wouldn't it be better to choose a system specifc default like OPEN_MAX
> > > instead or better, to call sysconf(_SC_OPEN_MAX) to fetch the system
> > > default and only use 4096 as fallback if none of the two is available
> > > or the values are larger than 4K?
> > 
> > Thanks for the suggestion. I'm not sure that sysconf() returns the
> > proper values, on my Linux box it returns 1024, even though it is
> > certainly possible to use higher ulimits. And people do hit this limit.
> 
> Indeed, sysconf returns the soft limit.
> 
> > Is there a preprocessor define I could use to detect cygwin? Even though
> > I hate conditional compilation [...]
> 
> There is such a define which is __CYGWIN__, but I'd also prefer a more
> generic solution.  Since you're using setrlimit anyway, why not use
> getrlimit before?  Along these lines:
> 
> --- gprocess.c.ORIG	2009-01-17 10:17:42.000000000 +0100
> +++ gprocess.c	2009-01-17 10:26:56.000000000 +0100
> @@ -484,10 +484,16 @@ g_process_change_limits(void)
>  {
>    struct rlimit limit;
>  
> -  limit.rlim_cur = limit.rlim_max = process_opts.fd_limit_min;
> -  
> -  if (setrlimit(RLIMIT_NOFILE, &limit) < 0)
> -    g_process_message("Error setting file number limit; limit='%d'; error='%s'", process_opts.fd_limit_min, g_strerror(errno));
> +  if (getrlimit (RLIMIT_NOFILE, &limit) == 0)
> +    {
> +      if (limit.rlim_max == RLIM_INFINITY || limit.rlim_max > process_opts.fd_limit_min)
> +      	limit.rlim_cur = limit.rlim_max = process_opts.fd_limit_min;
> +      else
> +      	limit.rlim_cur = limit.rlim_max;
> +      if (setrlimit(RLIMIT_NOFILE, &limit) == 0)
> +	return;
> +    }
> +  g_process_message("Error setting file number limit; limit='%d'; error='%s'", process_opts.fd_limit_min, g_strerror(errno));
>  }
>  

The whole point is to _increase_ the ulimit value, not to decrease it.
The problem is that people run into the default ulimit values in large
installations, e.g.

destination d_file { file("/var/log/$HOST/messages"); };

If you have a large number of different $HOST values, the default limit
of 1024 fds might prove small. The patch above, unless I miss something,
is not increasing the fd limit, it only decreases it.

-- 
Bazsi