[syslog-ng] Fixed number fd-limit doesn't scale

[Balazs Scheidler]

On Mon, 2009-01-19 at 11:42 +0100, Corinna Vinschen wrote:
> On Jan 19 10:55, Balazs Scheidler wrote:
> > On Sat, 2009-01-17 at 10:30 +0100, Corinna Vinschen wrote:
> > > Hi,
> > > 
> > > On Jan 16 20:53, Balazs Scheidler wrote:
> > > > On Wed, 2009-01-14 at 13:12 +0100, Corinna Vinschen wrote:
> > > > > Wouldn't it be better to choose a system specifc default like OPEN_MAX
> > > > > instead or better, to call sysconf(_SC_OPEN_MAX) to fetch the system
> > > > > default and only use 4096 as fallback if none of the two is available
> > > > > or the values are larger than 4K?
> > > > 
> > > > Thanks for the suggestion. I'm not sure that sysconf() returns the
> > > > proper values, on my Linux box it returns 1024, even though it is
> > > > certainly possible to use higher ulimits. And people do hit this limit.
> > > 
> > > Indeed, sysconf returns the soft limit.
> > > 
> > > > Is there a preprocessor define I could use to detect cygwin? Even though
> > > > I hate conditional compilation [...]
> > > 
> > > There is such a define which is __CYGWIN__, but I'd also prefer a more
> > > generic solution.  Since you're using setrlimit anyway, why not use
> > > getrlimit before?  Along these lines:
> > > 
> > > --- gprocess.c.ORIG	2009-01-17 10:17:42.000000000 +0100
> > > +++ gprocess.c	2009-01-17 10:26:56.000000000 +0100
> > > @@ -484,10 +484,16 @@ g_process_change_limits(void)
> > >  {
> > >    struct rlimit limit;
> > >  
> > > -  limit.rlim_cur = limit.rlim_max = process_opts.fd_limit_min;
> > > -  
> > > -  if (setrlimit(RLIMIT_NOFILE, &limit) < 0)
> > > -    g_process_message("Error setting file number limit; limit='%d'; error='%s'", process_opts.fd_limit_min, g_strerror(errno));
> > > +  if (getrlimit (RLIMIT_NOFILE, &limit) == 0)
> > > +    {
> > > +      if (limit.rlim_max == RLIM_INFINITY || limit.rlim_max > process_opts.fd_limit_min)
> > > +      	limit.rlim_cur = limit.rlim_max = process_opts.fd_limit_min;
> > > +      else
> > > +      	limit.rlim_cur = limit.rlim_max;
> > > +      if (setrlimit(RLIMIT_NOFILE, &limit) == 0)
> > > +	return;
> > > +    }
> > > +  g_process_message("Error setting file number limit; limit='%d'; error='%s'", process_opts.fd_limit_min, g_strerror(errno));
> > >  }
> > >  
> > 
> > The whole point is to _increase_ the ulimit value, not to decrease it.
> > The problem is that people run into the default ulimit values in large
> > installations, e.g.
> > 
> > destination d_file { file("/var/log/$HOST/messages"); };
> > 
> > If you have a large number of different $HOST values, the default limit
> > of 1024 fds might prove small. The patch above, unless I miss something,
> > is not increasing the fd limit, it only decreases it.
> 
> The above patch sets limit.rlim_cur to limit.rlim_max and limit.rlim_max
> is the maximum a process can get anyway.  On my Linux box, the default
> values for rlim_cur is 1024 and rlim_max is 8192.  So the above code
> raises the current limit to the max of 8192.  That's what you want,
> right?

On my Linux box, getrlimit() returns 1024 even for rlim_max:

getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=1024}) = 0

This is probably a system setting and the difference is because I run
Ubuntu and you probably run RedHat. 

# ulimit -H -n 8192

getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=8*1024}) = 0

But it is certainly possible to raise this limit even further than 1024
or 8192, I've run programs with limits over 65536 in some cases and on
Linux, it works well. Of course it requires superuser privileges, but
this code runs as root during startup, even if you specify -u/-g
options.

Do you have another recommendation apart from using __CYGWIN__ ?


-- 
Bazsi