HI Adam,<div><br></div><div>That may be my fault. I'm not quite sure. Can your try the latest development branch of nginx_upstream_check_module(<a href="https://github.com/yaoweibin/nginx_upstream_check_module/tree/development">https://github.com/yaoweibin/nginx_upstream_check_module/tree/development</a>)? </div>
<div><br></div><div>Thanks for your report.</div><div><br><div class="gmail_quote">2011/7/2 adam <span dir="ltr"><<a href="mailto:nginx-forum@nginx.us">nginx-forum@nginx.us</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
We use Nginx as a reverse proxy. After 2 or 3 reloads (kill -HUP), the<br>
parent's one child worker process remains "stuck"; the parent spawns a<br>
new one which is also often "stuck."<br>
<br>
The number of worker_processes we have configured is 1. After a reload,<br>
there are 2 workers in the process list. After another reload, 3<br>
workers, and so on. Usually all of these workers are in state R. When we<br>
strace them, we see no system calls, but they are sucking on user CPU.<br>
They are unable to serve requests. One time (seen in detail below), a<br>
new worker was spawned and it was *not* in state R, but state S, and was<br>
able to serve requests, until we issued another reload.<br>
<br>
Of course what should happen is, after a reload, we should only ever<br>
have one worker process which is able to serve requests.<br>
<br>
Details below, including our config. We can provide lsof, strace, debug<br>
log of the reproduction case if needed.<br>
<br>
#############<br>
<br>
07/01 23:10[root@proxy ~]# cat /etc/redhat-release<br>
CentOS release 5.6 (Final)<br>
<br>
07/01 23:10[root@proxy ~]# uname -a<br>
Linux proxy 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686<br>
i686 i386 GNU/Linux<br>
<br>
07/01 23:10[root@proxy ~]# nginx -V<br>
nginx version: nginx/0.8.54<br>
built by gcc 4.1.2 20080704 (Red Hat 4.1.2-48)<br>
TLS SNI support disabled<br>
configure arguments: --user=nginx --group=nginx<br>
--prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx<br>
--conf-path=/etc/nginx/nginx.conf<br>
--error-log-path=/var/log/nginx/error.log<br>
--http-log-path=/var/log/nginx/access.log<br>
--http-client-body-temp-path=/var/lib/nginx/tmp/client_body<br>
--http-proxy-temp-path=/var/lib/nginx/tmp/proxy<br>
--http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi<br>
--http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi<br>
--http-scgi-temp-path=/var/lib/nginx/tmp/scgi<br>
--pid-path=/var/run/nginx.pid --lock-path=/var/lock/subsys/nginx<br>
--with-http_ssl_module --with-http_realip_module<br>
--with-http_addition_module --with-http_xslt_module<br>
--with-http_image_filter_module --with-http_sub_module<br>
--with-http_gzip_static_module --with-http_random_index_module<br>
--with-http_secure_link_module --with-http_degradation_module<br>
--with-http_stub_status_module --with-http_perl_module --with-ipv6<br>
--with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions<br>
-fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386<br>
-mtune=generic -fasynchronous-unwind-tables' --with-cc-opt='-O2 -g -pipe<br>
-Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector<br>
--param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic<br>
-fasynchronous-unwind-tables'<br>
--add-module=src/http/modules/nginx_syslog_patch<br>
--add-module=src/http/modules/nginx_upstream_module<br>
--add-module=src/http/modules/nginx_ajp_module<br>
<br>
(Note above that we have a few third-party modules added for syslog and<br>
upstream health checking.)<br>
<br>
#############<br>
<br>
## From fresh start:<br>
<br>
07/01 22:42[root@proxy ~]# /etc/init.d/nginx stop<br>
Stopping nginx: [ OK ]<br>
07/01 22:43[root@proxy ~]# ps wwwaxuf | grep nginx<br>
root 23433 0.0 0.0 4884 704 pts/2 S+ 22:43 0:00 |<br>
\_ grep nginx<br>
<br>
## It's not running. Now, start it:<br>
<br>
07/01 22:43[root@proxy ~]# /etc/init.d/nginx start<br>
Starting nginx: [ OK ]<br>
07/01 22:43[root@proxy ~]# ps wwwaxuf | grep nginx | egrep -v<br>
'grep|strace'<br>
root 23463 0.0 0.0 13080 1564 ? Ss 22:43 0:00 nginx:<br>
master process /usr/sbin/nginx -c /etc/nginx/nginx.conf<br>
nginx 23464 0.0 0.1 13224 2124 ? S 22:43 0:00 \_<br>
nginx: worker process<br>
<br>
07/01 22:44[root@proxy ~]# /etc/init.d/nginx reload<br>
Reloading nginx: [ OK ]<br>
07/01 22:45[root@proxy ~]# ps wwwaxuf | grep nginx | egrep -v<br>
'grep|strace'<br>
root 23463 0.0 0.2 13612 4580 ? Ss 22:43 0:00 nginx:<br>
master process /usr/sbin/nginx -c /etc/nginx/nginx.conf<br>
nginx 23580 0.0 0.1 13744 2684 ? S 22:45 0:00 \_<br>
nginx: worker process<br>
<br>
07/01 22:45[root@proxy ~]# /etc/init.d/nginx reload<br>
Reloading nginx: [ OK ]<br>
07/01 22:46[root@proxy ~]# ps wwwaxuf | grep nginx | egrep -v<br>
'grep|strace'<br>
root 23463 0.0 0.2 13652 4600 ? Ss 22:43 0:00 nginx:<br>
master process /usr/sbin/nginx -c /etc/nginx/nginx.conf<br>
nginx 23787 0.0 0.1 13652 2600 ? S 22:46 0:00 \_<br>
nginx: worker process<br>
07/01 22:46[root@proxy ~]# ps wwwaxuf | grep nginx | egrep -v<br>
'grep|strace'<br>
root 23463 0.0 0.2 13652 4600 ? Ss 22:43 0:00 nginx:<br>
master process /usr/sbin/nginx -c /etc/nginx/nginx.conf<br>
nginx 23787 0.0 0.1 13652 2600 ? S 22:46 0:00 \_<br>
nginx: worker process<br>
<br>
07/01 22:46[root@proxy ~]# /etc/init.d/nginx reload<br>
Reloading nginx: [ OK ]<br>
07/01 22:46[root@proxy ~]# ps wwwaxuf | grep nginx | egrep -v<br>
'grep|strace'<br>
root 23463 0.0 0.2 13784 4616 ? Ss 22:43 0:00 nginx:<br>
master process /usr/sbin/nginx -c /etc/nginx/nginx.conf<br>
nginx 23830 0.0 0.1 13784 2672 ? S 22:46 0:00 \_<br>
nginx: worker process<br>
07/01 22:46[root@proxy ~]# ps wwwaxuf | grep nginx | egrep -v<br>
'grep|strace'<br>
root 23463 0.0 0.2 13784 4616 ? Ss 22:43 0:00 nginx:<br>
master process /usr/sbin/nginx -c /etc/nginx/nginx.conf<br>
nginx 23830 27.3 0.2 14336 4096 ? R 22:46 0:11 \_<br>
nginx: worker process<br>
<br>
07/01 22:47[root@proxy ~]# /etc/init.d/nginx reload<br>
Reloading nginx: [ OK ]<br>
07/01 22:47[root@proxy ~]# ps wwwaxuf | grep nginx | egrep -v<br>
'grep|strace'<br>
root 23463 0.0 0.2 13784 4616 ? Ss 22:43 0:00 nginx:<br>
master process /usr/sbin/nginx -c /etc/nginx/nginx.conf<br>
nginx 23830 28.0 0.2 14336 4096 ? R 22:46 0:13 \_<br>
nginx: worker process<br>
nginx 23907 0.0 0.1 13784 2620 ? S 22:47 0:00 \_<br>
nginx: worker process<br>
07/01 22:47[root@proxy ~]# ps wwwaxuf | grep nginx | egrep -v<br>
'grep|strace'<br>
root 23463 0.0 0.2 13784 4616 ? Ss 22:43 0:00 nginx:<br>
master process /usr/sbin/nginx -c /etc/nginx/nginx.conf<br>
nginx 23830 29.4 0.2 14336 4096 ? R 22:46 0:15 \_<br>
nginx: worker process<br>
nginx 23907 0.0 0.1 13784 2704 ? S 22:47 0:00 \_<br>
nginx: worker process<br>
07/01 22:47[root@proxy ~]# ps wwwaxuf | grep nginx | egrep -v<br>
'grep|strace'<br>
root 23463 0.0 0.2 13784 4616 ? Ss 22:43 0:00 nginx:<br>
master process /usr/sbin/nginx -c /etc/nginx/nginx.conf<br>
nginx 23830 34.4 0.2 14336 4096 ? R 22:46 0:30 \_<br>
nginx: worker process<br>
nginx 23907 0.0 0.1 13784 2916 ? S 22:47 0:00 \_<br>
nginx: worker process<br>
<br>
07/01 22:49[root@proxy ~]# /etc/init.d/nginx reload<br>
Reloading nginx: [ OK ]<br>
07/01 22:49[root@proxy ~]# ps wwwaxuf | grep nginx | egrep -v<br>
'grep|strace'<br>
root 23463 0.0 0.2 13784 4616 ? Ss 22:43 0:00 nginx:<br>
master process /usr/sbin/nginx -c /etc/nginx/nginx.conf<br>
nginx 23830 38.1 0.2 14336 4096 ? R 22:46 1:12 \_<br>
nginx: worker process<br>
nginx 24145 0.0 0.1 13784 2712 ? S 22:49 0:00 \_<br>
nginx: worker process<br>
07/01 22:49[root@proxy ~]# /etc/init.d/nginx reload<br>
Reloading nginx: [ OK ]<br>
07/01 22:50[root@proxy ~]# ps wwwaxuf | grep nginx | egrep -v<br>
'grep|strace'<br>
root 23463 0.0 0.2 13784 4616 ? Ss 22:43 0:00 nginx:<br>
master process /usr/sbin/nginx -c /etc/nginx/nginx.conf<br>
nginx 23830 37.5 0.2 14336 4096 ? R 22:46 1:16 \_<br>
nginx: worker process<br>
nginx 24185 38.6 0.1 13784 3424 ? R 22:50 0:01 \_<br>
nginx: worker process<br>
<br>
07/01 22:50[root@proxy ~]# /etc/init.d/nginx reload<br>
Reloading nginx: [ OK ]<br>
07/01 22:50[root@proxy ~]# ps wwwaxuf | grep nginx | egrep -v<br>
'grep|strace'<br>
root 23463 0.0 0.2 13784 4620 ? Ss 22:43 0:00 nginx:<br>
master process /usr/sbin/nginx -c /etc/nginx/nginx.conf<br>
nginx 23830 35.5 0.2 14336 4096 ? R 22:46 1:20 \_<br>
nginx: worker process<br>
nginx 24185 19.9 0.1 13784 3424 ? R 22:50 0:04 \_<br>
nginx: worker process<br>
nginx 24318 31.0 0.2 13924 3620 ? R 22:50 0:01 \_<br>
nginx: worker process<br>
<br>
#############<br>
<br>
07/01 23:06[root@proxy ~]# egrep -v '^#|^$' /etc/nginx/nginx.conf<br>
user nginx;<br>
worker_processes 1;<br>
syslog local2 nginx;<br>
error_log syslog:warn|/var/log/nginx/error.log;<br>
pid /var/run/nginx.pid;<br>
events {<br>
worker_connections 1024;<br>
}<br>
http {<br>
include /etc/nginx/mime.types;<br>
default_type application/octet-stream;<br>
log_format main '$remote_addr - $remote_user [$time_local]<br>
"$request" '<br>
'$status $body_bytes_sent "$http_referer" '<br>
'"$http_user_agent" "$http_x_forwarded_for"';<br>
access_log syslog:warn|/var/log/nginx/access.log main;<br>
sendfile on;<br>
keepalive_timeout 65;<br>
gzip on;<br>
server {<br>
listen 80;<br>
server_name _;<br>
location /nginx-status {<br>
stub_status on;<br>
access_log off;<br>
allow <a href="http://10.0.0.0/8" target="_blank">10.0.0.0/8</a>;<br>
allow 127.0.0.1;<br>
deny all;<br>
}<br>
location /upstream-status {<br>
check_status;<br>
access_log off;<br>
allow <a href="http://10.0.0.0/8" target="_blank">10.0.0.0/8</a>;<br>
allow 127.0.0.1;<br>
deny all;<br>
}<br>
error_page 404 /404.html;<br>
location = /404.html {<br>
root /usr/share/nginx/error;<br>
}<br>
error_page 403 /403.html;<br>
location = /403.html {<br>
root /usr/share/nginx/error;<br>
}<br>
error_page 500 502 504 /500.html;<br>
location = /500.html {<br>
root /usr/share/nginx/error;<br>
}<br>
error_page 503 /503.html;<br>
location = /503.html {<br>
root /usr/share/nginx/error;<br>
}<br>
set $global_ssl_redirect 'yes';<br>
if ($request_filename ~ "nginx-status") {<br>
set $global_ssl_redirect 'no';<br>
}<br>
if ($request_filename ~ "upstream-status") {<br>
set $global_ssl_redirect 'no';<br>
}<br>
if ($global_ssl_redirect ~* '^yes$') {<br>
rewrite ^ https://$host$request_uri? permanent;<br>
break;<br>
}<br>
}<br>
include upstream.conf;<br>
server {<br>
listen 443;<br>
server_name _;<br>
ssl on;<br>
ssl_certificate certs/certchain.crt;<br>
ssl_certificate_key certs/certchain.key;<br>
ssl_protocols SSLv3 TLSv1;<br>
ssl_ciphers HIGH;<br>
set_real_ip_from <a href="http://10.0.0.0/8" target="_blank">10.0.0.0/8</a>;<br>
real_ip_header X-Forwarded-For;<br>
add_header X-Forwarded-For $proxy_add_x_forwarded_for;<br>
add_header Cache-Control public;<br>
location / {<br>
proxy_pass <a href="http://appserver_http" target="_blank">http://appserver_http</a>;<br>
proxy_connect_timeout 10s;<br>
proxy_next_upstream error timeout invalid_header http_500<br>
http_503 http_502 http_504;<br>
proxy_set_header Host $host;<br>
if ($request_uri ~* "\.(ico|css|js|gif|jpe?g|png)") {<br>
expires 365d;<br>
break;<br>
}<br>
}<br>
error_page 404 /404.html;<br>
location = /404.html {<br>
root /usr/share/nginx/error;<br>
}<br>
error_page 403 /403.html;<br>
location = /403.html {<br>
root /usr/share/nginx/error;<br>
}<br>
error_page 500 502 504 /500.html;<br>
location = /500.html {<br>
root /usr/share/nginx/error;<br>
}<br>
error_page 503 /503.html;<br>
location = /503.html {<br>
root /usr/share/nginx/error;<br>
}<br>
}<br>
}<br>
<br>
Posted at Nginx Forum: <a href="http://forum.nginx.org/read.php?2,211695,211695#msg-211695" target="_blank">http://forum.nginx.org/read.php?2,211695,211695#msg-211695</a><br>
<br>
<br>
_______________________________________________<br>
nginx mailing list<br>
<a href="mailto:nginx@nginx.org">nginx@nginx.org</a><br>
<a href="http://nginx.org/mailman/listinfo/nginx" target="_blank">http://nginx.org/mailman/listinfo/nginx</a><br>
</blockquote></div><br></div>