[syslog-ng] Fixed number fd-limit doesn't scale

[Corinna Vinschen]

On Jan 19 10:55, Balazs Scheidler wrote:
> On Sat, 2009-01-17 at 10:30 +0100, Corinna Vinschen wrote:
> > Hi,
> > 
> > On Jan 16 20:53, Balazs Scheidler wrote:
> > > On Wed, 2009-01-14 at 13:12 +0100, Corinna Vinschen wrote:
> > > > Wouldn't it be better to choose a system specifc default like OPEN_MAX
> > > > instead or better, to call sysconf(_SC_OPEN_MAX) to fetch the system
> > > > default and only use 4096 as fallback if none of the two is available
> > > > or the values are larger than 4K?
> > > 
> > > Thanks for the suggestion. I'm not sure that sysconf() returns the
> > > proper values, on my Linux box it returns 1024, even though it is
> > > certainly possible to use higher ulimits. And people do hit this limit.
> > 
> > Indeed, sysconf returns the soft limit.
> > 
> > > Is there a preprocessor define I could use to detect cygwin? Even though
> > > I hate conditional compilation [...]
> > 
> > There is such a define which is __CYGWIN__, but I'd also prefer a more
> > generic solution.  Since you're using setrlimit anyway, why not use
> > getrlimit before?  Along these lines:
> > 
> > --- gprocess.c.ORIG	2009-01-17 10:17:42.000000000 +0100
> > +++ gprocess.c	2009-01-17 10:26:56.000000000 +0100
> > @@ -484,10 +484,16 @@ g_process_change_limits(void)
> >  {
> >    struct rlimit limit;
> >  
> > -  limit.rlim_cur = limit.rlim_max = process_opts.fd_limit_min;
> > -  
> > -  if (setrlimit(RLIMIT_NOFILE, &limit) < 0)
> > -    g_process_message("Error setting file number limit; limit='%d'; error='%s'", process_opts.fd_limit_min, g_strerror(errno));
> > +  if (getrlimit (RLIMIT_NOFILE, &limit) == 0)
> > +    {
> > +      if (limit.rlim_max == RLIM_INFINITY || limit.rlim_max > process_opts.fd_limit_min)
> > +      	limit.rlim_cur = limit.rlim_max = process_opts.fd_limit_min;
> > +      else
> > +      	limit.rlim_cur = limit.rlim_max;
> > +      if (setrlimit(RLIMIT_NOFILE, &limit) == 0)
> > +	return;
> > +    }
> > +  g_process_message("Error setting file number limit; limit='%d'; error='%s'", process_opts.fd_limit_min, g_strerror(errno));
> >  }
> >  
> 
> The whole point is to _increase_ the ulimit value, not to decrease it.
> The problem is that people run into the default ulimit values in large
> installations, e.g.
> 
> destination d_file { file("/var/log/$HOST/messages"); };
> 
> If you have a large number of different $HOST values, the default limit
> of 1024 fds might prove small. The patch above, unless I miss something,
> is not increasing the fd limit, it only decreases it.

The above patch sets limit.rlim_cur to limit.rlim_max and limit.rlim_max
is the maximum a process can get anyway.  On my Linux box, the default
values for rlim_cur is 1024 and rlim_max is 8192.  So the above code
raises the current limit to the max of 8192.  That's what you want,
right?

  
Corinna

-- 
Corinna Vinschen
Cygwin Project Co-Leader
Red Hat