Hi, Is there some kind of connection limit in the SSL-proxy? I tried apachebenchmark with a plain Zorp HTTPS instance. It works until a limit of 2 connections ("ab -n 5 -c 2 http://192.168.1.5/index.html") but any higher value (3 sometimes works) produces errors with failing SSL handshakes. "ab -n 5 -c 2 http://192.168.1.5/index.html" for example results in 5 (=all) broken connections. I couldn't find the real reason but I found some "broken pipe" , "Bad file descriptor" and "(nosession): Connection denied by policy; conn='NULL' "-messages indicating that the connection somehow gets lost. Raising the thread limits in the instances.conf or using the threadpool-option doesn't help - and if I raise it too high I get an "Too many open files" even though I thought I raised the ulimit value for zorp and root to a very high value. Specifying "THREADED=TRUE" in the Listener/Dispatcher doesn't change anything else...is there any solution for this concurrency problem or is this a known limitation? Thanks for your help! Thoms Wenz -- GMX startet ShortView.de. Hier findest Du Leute mit Deinen Interessen! Jetzt dabei sein: http://www.shortview.de/wasistshortview.php?mc=sv_ext_mf@gmx
Hi, I think I finally found the solution. Changing some parameters in the kernel source ("INR_OPEN", "NR_OPEN" and "OPEN_MAX") didn't really help but after I added the "-- --fd-limit 4096" (like found in an older message from May 2004) to the instances.conf it seems to work. Greetings Thomas Wenz -- Ist Ihr Browser Vista-kompatibel? Jetzt die neuesten Browser-Versionen downloaden: http://www.gmx.net/de/go/browser
On Mon, 2008-07-28 at 02:29 +0200, thomas.wenz@gmx-topmail.de wrote:
Hi,
I think I finally found the solution. Changing some parameters in the kernel source ("INR_OPEN", "NR_OPEN" and "OPEN_MAX") didn't really help but after I added the "-- --fd-limit 4096" (like found in an older message from May 2004) to the instances.conf it seems to work.
As the thread based fd limit calculation proved incorrect in some cases, Zorp currently sets the fd limit to a fixed 256000 limit. Maybe your system cannot do as much (I think this depends on the amount of RAM in your computer and/or some settings in /proc), and the call to set the resource limits fail. Can you confirm this using strace? The kernel changes were related to linux 2.2, so those don't apply either. -- Bazsi
Hi, Well it should not be a problem of RAM...the system has 4GB and only 150MB are in use. However, I noticed something strange: If I start zorp with /usr/lib/zorp/zorp --as https --verbose 8 --policy /home/zorp/conf/HTTPS.py --autobind-ip 172.30.1.1 --threads 30 --idle-threads 30 --threadpools there's no problem with file descriptors. Whereas if I use zorpctl with the same command in the instances file https --verbose 8 --policy /home/zorp/conf/HTTPS.py --autobind-ip 172.30.1.1 --threads 30 --idle-threads 30 --threadpools there is the problem. So i did an strace on zorpctl and saw that the proxy instance is called via: execve("/usr/lib/zorp/zorp", ["/usr/lib/zorp/zorp", "--as", "https", "--verbose", "8", "--policy", "/home/zorp/conf/HTTPS.py", "--autobind-ip", "172.30.1.1", "--threads", "30", "--idle-threads", "30", "--threadpools", "--fd-limit-min", "34", ...], [/* 15 vars */]) = 0 So it's quite clear why it runs out of ressources so fast ;) 34 is definately not enough. According to the strace, no system calls are done in /proc or else (see below). I then looked into the zorpctl/main.c source. I found the following line: inst->fd_limit = fd_limit = MAX(fd_limit_threshold * threads + 64, fd_limit_min); and fd_limit_threshold as well as fd_limit_min seems to be ALWAYS set to "-1" (and declared as obsolete). threads is the variable I set in the instances.conf-file. So the calculation is MAX(-1 * 30 + 64, -1) which exactly results in 34. If I set the threads in the instances-file to only 10, I consequently get 54 available descriptors (proved by looking into strace). So this calculation seems to be definately wrong... Thomas Wenz 4238 execve("/usr/sbin/zorpctl", ["zorpctl", "start", "https"], [/* 15 vars */]) = 0 4238 uname({sys="Linux", node="bridge", ...}) = 0 4238 brk(0) = 0x8054000 4238 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory) 4238 mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7fb0000 4238 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) 4238 open("/etc/ld.so.cache", O_RDONLY) = 3 4238 fstat64(3, {st_mode=S_IFREG|0644, st_size=17114, ...}) = 0 4238 mmap2(NULL, 17114, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7fab000 4238 close(3) = 0 4238 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory) 4238 open("/lib/tls/i686/cmov/libnsl.so.1", O_RDONLY) = 3 4238 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0p5\0\000"..., 512) = 512 4238 fstat64(3, {st_mode=S_IFREG|0644, st_size=76548, ...}) = 0 4238 mmap2(NULL, 87808, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7f95000 4238 mmap2(0xb7fa7000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x11) = 0xb7fa7000 4238 mmap2(0xb7fa9000, 5888, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb7fa9000 4238 close(3) = 0 4238 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory) 4238 open("/usr/lib/libglib-2.0.so.0", O_RDONLY) = 3 4238 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0@\322\0"..., 512) = 512 4238 fstat64(3, {st_mode=S_IFREG|0644, st_size=596608, ...}) = 0 4238 mmap2(NULL, 596204, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7f03000 4238 mmap2(0xb7f94000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x91) = 0xb7f94000 4238 close(3) = 0 4238 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory) 4238 open("/lib/tls/i686/cmov/libc.so.6", O_RDONLY) = 3 4238 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\240O\1"..., 512) = 512 4238 fstat64(3, {st_mode=S_IFREG|0644, st_size=1241392, ...}) = 0 4238 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f02000 4238 mmap2(NULL, 1247388, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7dd1000 4238 mmap2(0xb7ef8000, 28672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x127) = 0xb7ef8000 4238 mmap2(0xb7eff000, 10396, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb7eff000 4238 close(3) = 0 4238 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory) 4238 open("/lib/tls/i686/cmov/librt.so.1", O_RDONLY) = 3 4238 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0000\36\0"..., 512) = 512 4238 fstat64(3, {st_mode=S_IFREG|0644, st_size=30612, ...}) = 0 4238 mmap2(NULL, 33384, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7dc8000 4238 mmap2(0xb7dcf000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6) = 0xb7dcf000 4238 close(3) = 0 4238 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory) 4238 open("/lib/tls/i686/cmov/libpthread.so.0", O_RDONLY) = 3 4238 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\240H\0"..., 512) = 512 4238 fstat64(3, {st_mode=S_IFREG|0755, st_size=89370, ...}) = 0 4238 mmap2(NULL, 70104, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7db6000 4238 mmap2(0xb7dc4000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xe) = 0xb7dc4000 4238 mmap2(0xb7dc6000, 4568, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb7dc6000 4238 close(3) = 0 4238 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7db5000 4238 mprotect(0xb7ef8000, 20480, PROT_READ) = 0 4238 set_thread_area({entry_number:-1 -> 6, base_addr:0xb7db58e0, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0 4238 munmap(0xb7fab000, 17114) = 0 4238 set_tid_address(0xb7db5928) = 4238 4238 rt_sigaction(SIGRTMIN, {0xb7dba4c0, [], SA_SIGINFO}, NULL, 8) = 0 4238 rt_sigaction(SIGRT_1, {0xb7dba420, [], SA_RESTART|SA_SIGINFO}, NULL, 8) = 0 4238 rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0 4238 getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0 4238 uname({sys="Linux", node="bridge", ...}) = 0 4238 rt_sigaction(SIGALRM, NULL, {SIG_DFL}, 8) = 0 4238 rt_sigaction(SIGALRM, {SIG_DFL}, NULL, 8) = 0 4238 rt_sigaction(SIGALRM, {0x8049f40, [ALRM], 0}, {SIG_DFL}, 8) = 0 4238 brk(0) = 0x8054000 4238 brk(0x8075000) = 0x8075000 4238 open("/etc/zorp/zorpctl.conf", O_RDONLY) = 3 4238 fstat64(3, {st_mode=S_IFREG|0644, st_size=1666, ...}) = 0 4238 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7faf000 4238 read(3, "#\n# This file is sourced by zorp"..., 4096) = 1666 4238 read(3, "", 4096) = 0 4238 open("/etc/zorp/instances.conf", O_RDONLY) = 4 4238 fstat64(4, {st_mode=S_IFREG|0644, st_size=254, ...}) = 0 4238 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7fae000 4238 read(4, "http --verbose 1 --policy /home/"..., 4096) = 254 4238 gettimeofday({1217208680, 1756}, NULL) = 0 4238 read(4, "", 4096) = 0 4238 socket(PF_FILE, SOCK_STREAM, 0) = 5 4238 fcntl64(5, F_GETFL) = 0x2 (flags O_RDWR) 4238 fcntl64(5, F_SETFL, O_RDWR|O_NONBLOCK) = 0 4238 connect(5, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = 0 4238 poll([{fd=5, events=POLLOUT|POLLERR|POLLHUP, revents=POLLOUT}], 1, 5000) = 1 4238 writev(5, [{"\2\0\0\0\v\0\0\0\7\0\0\0", 12}, {"passwd\0", 7}], 2) = 19 4238 poll([{fd=5, events=POLLIN|POLLERR|POLLHUP, revents=POLLIN|POLLHUP}], 1, 5000) = 1 4238 recvmsg(5, {msg_name(0)=NULL, msg_iov(1)=[{"passwd\0", 7}], msg_controllen=16, {cmsg_len=16, cmsg_level=SOL_SOCKET, cmsg_type=SCM_RIGHTS, {6}}, msg_flags=0}, MSG_NOSIGNAL) = 7 4238 fstat64(6, {st_mode=S_IFREG|0600, st_size=217016, ...}) = 0 4238 pread64(6, "\1\0\0\0h\0\0\0\"\2\0\0\1\0\0\0[\37\215H\0\0\0\0\323\0"..., 104, 0) = 104 4238 mmap2(NULL, 217016, PROT_READ, MAP_SHARED, 6, 0) = 0xb7d80000 4238 close(6) = 0 4238 close(5) = 0 4238 socket(PF_FILE, SOCK_STREAM, 0) = 5 4238 fcntl64(5, F_GETFL) = 0x2 (flags O_RDWR) 4238 fcntl64(5, F_SETFL, O_RDWR|O_NONBLOCK) = 0 4238 connect(5, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = 0 4238 poll([{fd=5, events=POLLOUT|POLLERR|POLLHUP, revents=POLLOUT}], 1, 5000) = 1 4238 writev(5, [{"\2\0\0\0\f\0\0\0\6\0\0\0", 12}, {"group\0", 6}], 2) = 18 4238 poll([{fd=5, events=POLLIN|POLLERR|POLLHUP, revents=POLLIN|POLLHUP}], 1, 5000) = 1 4238 recvmsg(5, {msg_name(0)=NULL, msg_iov(1)=[{"group\0", 6}], msg_controllen=16, {cmsg_len=16, cmsg_level=SOL_SOCKET, cmsg_type=SCM_RIGHTS, {6}}, msg_flags=0}, MSG_NOSIGNAL) = 6 4238 fstat64(6, {st_mode=S_IFREG|0600, st_size=217016, ...}) = 0 4238 pread64(6, "\1\0\0\0h\0\0\0002\1\0\0\1\0\0\0[\37\215H\0\0\0\0\323\0"..., 104, 0) = 104 4238 mmap2(NULL, 217016, PROT_READ, MAP_SHARED, 6, 0) = 0xb7d4b000 4238 close(6) = 0 4238 close(5) = 0 4238 stat64("/var/run/zorp", {st_mode=S_IFDIR|0770, st_size=4096, ...}) = 0 4238 stat64("/etc/zorp", {st_mode=S_IFDIR|0750, st_size=4096, ...}) = 0 4238 write(1, "Starting Zorp Firewall Suite: ", 30) = 30 4238 open("/var/run/zorp/zorp-https.pid", O_RDONLY) = -1 ENOENT (No such file or directory) 4238 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb7db5928) = 4239 4238 alarm(10) = 0 4238 waitpid(4239, <unfinished ...> 4239 setrlimit(RLIMIT_NPROC, {rlim_cur=256, rlim_max=256}) = 0 4239 setsid() = 4239 4239 execve("/usr/lib/zorp/zorp", ["/usr/lib/zorp/zorp", "--as", "https", "--verbose", "8", "--policy", "/home/zorp/conf/HTTPS.py", "--autobind-ip", "172.30.1.1", "--threads", "30", "--idle-threads", "30", "--threadpools", "--fd-limit-min", "34", ...], [/* 15 vars */]) = 0 -- Der GMX SmartSurfer hilft bis zu 70% Ihrer Onlinekosten zu sparen! Ideal für Modem und ISDN: http://www.gmx.net/de/go/smartsurfer
On Mon, 2008-07-28 at 14:48 +0200, thomas.wenz@gmx-topmail.de wrote:
Hi,
Well it should not be a problem of RAM...the system has 4GB and only 150MB are in use.
However, I noticed something strange: If I start zorp with /usr/lib/zorp/zorp --as https --verbose 8 --policy /home/zorp/conf/HTTPS.py --autobind-ip 172.30.1.1 --threads 30 --idle-threads 30 --threadpools there's no problem with file descriptors. Whereas if I use zorpctl with the same command in the instances file https --verbose 8 --policy /home/zorp/conf/HTTPS.py --autobind-ip 172.30.1.1 --threads 30 --idle-threads 30 --threadpools there is the problem.
So i did an strace on zorpctl and saw that the proxy instance is called via: execve("/usr/lib/zorp/zorp", ["/usr/lib/zorp/zorp", "--as", "https", "--verbose", "8", "--policy", "/home/zorp/conf/HTTPS.py", "--autobind-ip", "172.30.1.1", "--threads", "30", "--idle-threads", "30", "--threadpools", "--fd-limit-min", "34", ...], [/* 15 vars */]) = 0 So it's quite clear why it runs out of ressources so fast ;) 34 is definately not enough. According to the strace, no system calls are done in /proc or else (see below).
I then looked into the zorpctl/main.c source. I found the following line: inst->fd_limit = fd_limit = MAX(fd_limit_threshold * threads + 64, fd_limit_min); and fd_limit_threshold as well as fd_limit_min seems to be ALWAYS set to "-1" (and declared as obsolete). threads is the variable I set in the instances.conf-file. So the calculation is MAX(-1 * 30 + 64, -1) which exactly results in 34. If I set the threads in the instances-file to only 10, I consequently get 54 available descriptors (proved by looking into strace). So this calculation seems to be definately wrong...
wow, thanks for the detailed diagnosis. This is definitely wrong. I just don't see how this does not get triggered under Zorp Pro. zorpctl is the same for both. -- Bazsi
Hi, Perhaps Zorp Pro normally uses 63 or more threads? Because at this value, fd_limit becomes negative and then it isn't used in the execve command anymore because there's a check for if (inst->fd_limit > 0) However, if "threads" is set to 44-62 it can't even start the proxy because of "too many open files" and requests are only possible with "threads" set to 39 or less (could depend a little bit on how many libraries are called). The value for fd_limit_threshold and fd_limit_min actually can be set in the zorpctl.conf but warnings are thrown because they are obsolete: z_error(0, "The use of FD_LIMIT_THRESHOLD and FD_LIMIT_MIN in zorpctl.conf is deprecated, use zorp options with a similar name\n"); Thomas Wenz -- GMX startet ShortView.de. Hier findest Du Leute mit Deinen Interessen! Jetzt dabei sein: http://www.shortview.de/wasistshortview.php?mc=sv_ext_mf@gmx
On Mon, 2008-07-28 at 15:27 +0200, thomas.wenz@gmx-topmail.de wrote:
Hi,
Perhaps Zorp Pro normally uses 63 or more threads? Because at this value, fd_limit becomes negative and then it isn't used in the execve command anymore because there's a check for if (inst->fd_limit > 0) However, if "threads" is set to 44-62 it can't even start the proxy because of "too many open files" and requests are only possible with "threads" set to 39 or less (could depend a little bit on how many libraries are called). The value for fd_limit_threshold and fd_limit_min actually can be set in the zorpctl.conf but warnings are thrown because they are obsolete: z_error(0, "The use of FD_LIMIT_THRESHOLD and FD_LIMIT_MIN in zorpctl.conf is deprecated, use zorp options with a similar name\n");
The reason we didn't notice is that the management GUI generates an --fd-limit option by default. Nevertheless, thanks for noticing this. -- Bazsi
participants (2)
-
Balazs Scheidler
-
thomas.wenz＠gmx-topmail.de