<font class="Apple-style-span" color="#3333ff">After reload, all the worker processes, including newly forked ones, goes straight to 100% CPU:<br></font><br>PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND <br>
<br>17301 nobody 25 0 2670m 107m 1128 R 99.6 0.7 2:00.87 nginx <br>17306 nobody 25 0 2651m 86m 444 R 98.6 0.5 2:00.82 nginx (cache manager) <br>
17303 nobody 25 0 2670m 106m 788 R 49.8 0.7 1:01.31 nginx <br>17302 nobody 25 0 2670m 106m 792 R 47.8 0.7 0:59.96 nginx <br>
17305 nobody 25 0 2670m 106m 652 R 47.8 0.7 1:01.08 nginx <br>17304 nobody 25 0 2670m 106m 652 R 100.6 0.7 2:01.23 nginx <br> <br>
15759 nobody 25 0 2670m 197m 91m R 45.8 1.2 2:37.60 nginx <br>15761 nobody 25 0 2651m 172m 86m R 54.8 1.1 1:01.59 nginx (cache manager) <br>
15760 nobody 25 0 2670m 197m 91m R 51.8 1.2 2:43.36 nginx <br>15756 nobody 25 0 2670m 197m 91m R 99.6 1.2 3:41.27 nginx <br>15757 nobody 25 0 2670m 197m 91m R 49.8 1.2 2:37.74 nginx <br>
15758 nobody 25 0 2670m 197m 91m R 49.8 1.2 2:37.31 nginx <br><br><font class="Apple-style-span" color="#3333ff">Using GDB, it seems worker processes stuck at this point:</font><br><br><br>(gdb) bt<br>#0 0x00000033ad4badd7 in sched_yield () from /lib64/libc.so.6<br>
#1 0x000000000040e849 in ngx_shmtx_lock (mtx=0x2b3410964048) at src/core/ngx_shmtx.c:136<br>#2 0x0000000000444712 in ngx_http_file_cache_update (r=0xe50f3b0, tf=0x9e60270) at src/http/ngx_http_file_cache.c:831<br>#3 0x000000000043af8e in ngx_http_upstream_process_request (r=0xe50f3b0) at src/http/ngx_http_upstream.c:2657<br>
#4 0x000000000043b08e in ngx_http_upstream_process_upstream (r=0xe50f3b0, u=0x9e5f1f8) at src/http/ngx_http_upstream.c:2599<br>#5 0x000000000043b192 in ngx_http_upstream_handler (ev=<value optimized out>) at src/http/ngx_http_upstream.c:903<br>
#6 0x0000000000418eaa in ngx_event_process_posted (cycle=<value optimized out>, posted=0x7fff3f606520)<br> at src/event/ngx_event_posted.c:39<br>#7 0x0000000000418d6f in ngx_process_events_and_timers (cycle=0xd2a7710) at src/event/ngx_event.c:272<br>
#8 0x000000000041e800 in ngx_worker_process_cycle (cycle=0xd2a7710, data=<value optimized out>)<br> at src/os/unix/ngx_process_cycle.c:800<br>#9 0x000000000041d137 in ngx_spawn_process (cycle=0xd2a7710, proc=0x41e738 <ngx_worker_process_cycle>, data=0x0,<br>
name=0x5652a1 "worker process", respawn=-4) at src/os/unix/ngx_process.c:196<br>#10 0x000000000041dea3 in ngx_start_worker_processes (cycle=0xd2a7710, n=5, type=-4) at src/os/unix/ngx_process_cycle.c:360<br>#11 0x000000000041f351 in ngx_master_process_cycle (cycle=0xd2a7710) at src/os/unix/ngx_process_cycle.c:249<br>
#12 0x0000000000404dda in main (argc=7, argv=0x972556d) at src/core/nginx.c:405<br>---<br><br><div><br>Attaching to process 17301<br>(gdb) bt<br>#0 0x00000033ad4badd7 in sched_yield () from /lib64/libc.so.6<br>#1 0x000000000040e849 in ngx_shmtx_lock (mtx=0x2b3410964048) at src/core/ngx_shmtx.c:136<br>
#2 0x00000000004442ce in ngx_http_file_cache_free (c=0xc036b50, tf=0x0) at src/http/ngx_http_file_cache.c:916<br>#3 0x0000000000444473 in ngx_http_file_cache_cleanup (data=<value optimized out>) at src/http/ngx_http_file_cache.c:978<br>
#4 0x0000000000405924 in ngx_destroy_pool (pool=0xc035ac0) at src/core/ngx_palloc.c:54<br>#5 0x000000000042c0da in ngx_http_free_request (r=0xaee17b0, rc=0) at src/http/ngx_http_request.c:2996<br>#6 0x000000000042dd6d in ngx_http_set_keepalive (r=0xaee17b0) at src/http/ngx_http_request.c:2450<br>
#7 ngx_http_finalize_connection (r=0xaee17b0) at src/http/ngx_http_request.c:2146<br>#8 0x000000000042e2d9 in ngx_http_finalize_request (r=0x2b3410964048, rc=0) at src/http/ngx_http_request.c:1879<br>#9 0x000000000042a5dc in ngx_http_core_content_phase (r=0xaee17b0, ph=0xeb273b0) at src/http/ngx_http_core_module.c:1339<br>
#10 0x0000000000425d03 in ngx_http_core_run_phases (r=0xaee17b0) at src/http/ngx_http_core_module.c:837<br>#11 0x0000000000425e00 in ngx_http_handler (r=0x2b3410964048) at src/http/ngx_http_core_module.c:820<br>#12 0x000000000042ebfd in ngx_http_process_request (r=0xaee17b0) at src/http/ngx_http_request.c:1650<br>
#13 0x000000000042f420 in ngx_http_process_request_headers (rev=<value optimized out>) at src/http/ngx_http_request.c:1093<br>#14 0x000000000042f922 in ngx_http_process_request_line (rev=0x2b33ce0c4690) at src/http/ngx_http_request.c:893<br>
#15 0x000000000042cefe in ngx_http_init_request (rev=0x2b33ce0c4690) at src/http/ngx_http_request.c:518<br>#16 0x0000000000418eaa in ngx_event_process_posted (cycle=<value optimized out>, posted=0x0) at src/event/ngx_event_posted.c:39<br>
#17 0x0000000000418d6f in ngx_process_events_and_timers (cycle=0x9b4da50) at src/event/ngx_event.c:272<br>#18 0x000000000041e800 in ngx_worker_process_cycle (cycle=0x9b4da50, data=<value optimized out>)<br> at src/os/unix/ngx_process_cycle.c:800<br>
#19 0x000000000041d137 in ngx_spawn_process (cycle=0x9b4da50, proc=0x41e738 <ngx_worker_process_cycle>, data=0x0,<br> name=0x5652a1 "worker process", respawn=-4) at src/os/unix/ngx_process.c:196<br>#20 0x000000000041dea3 in ngx_start_worker_processes (cycle=0x9b4da50, n=5, type=-4) at src/os/unix/ngx_process_cycle.c:360<br>
#21 0x000000000041f351 in ngx_master_process_cycle (cycle=0x9b4da50) at src/os/unix/ngx_process_cycle.c:249<br>#22 0x0000000000404dda in main (argc=7, argv=0x972556d) at src/core/nginx.c:405<br><br><font class="Apple-style-span" color="#3333ff">Also, we took backtrace for cache manger process:</font><br>
<br><br>Attaching to process 17306<br>ngx_http_file_cache_expire (data=<value optimized out>) at src/http/ngx_http_file_cache.c:1096<br>1096 if (fcn->count == 0) {<br>(gdb) bt<br>#0 ngx_http_file_cache_expire (data=<value optimized out>) at src/http/ngx_http_file_cache.c:1096<br>
#1 ngx_http_file_cache_manager (data=<value optimized out>) at src/http/ngx_http_file_cache.c:1191<br>#2 0x000000000041e92a in ngx_cache_manager_process_handler (ev=0x7fff3f606670) at src/os/unix/ngx_process_cycle.c:1346<br>
#3 0x0000000000418e15 in ngx_event_expire_timers () at src/event/ngx_event_timer.c:149<br>#4 0x0000000000418d44 in ngx_process_events_and_timers (cycle=0x9b4da50) at src/event/ngx_event.c:261<br>#5 0x000000000041f5f8 in ngx_cache_manager_process_cycle (cycle=0x9b4da50, data=<value optimized out>)<br>
at src/os/unix/ngx_process_cycle.c:1328<br>#6 0x000000000041d137 in ngx_spawn_process (cycle=0x9b4da50, proc=0x41f489 <ngx_cache_manager_process_cycle>, data=0x7e9040,<br> name=0x56525f "cache manager process", respawn=-4) at src/os/unix/ngx_process.c:196<br>
#7 0x000000000041dd8a in ngx_start_cache_manager_processes (cycle=0x9b4da50, respawn=1) at src/os/unix/ngx_process_cycle.c:398<br>#8 0x000000000041f35e in ngx_master_process_cycle (cycle=0x9b4da50) at src/os/unix/ngx_process_cycle.c:251<br>
#9 0x0000000000404dda in main (argc=7, argv=0x972556d) at src/core/nginx.c:405<br><br><br><font class="Apple-style-span" color="#3333ff">The nginx version is 1.0.4 with following configuration parameters:</font><br><br>nginx: nginx version: nginx/1.0.4<br>
nginx: built by gcc 4.1.2 20080704 (Red Hat 4.1.2-50)<br>nginx: TLS SNI support enabled<br>nginx: configure arguments: --prefix=/opt/app/nginx-urlhash --without-select_module --without-poll_module --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_flv_module --with-http_gzip_static_module --with-http_stub_status_module --with-pcre=/opt/tmp/pcre-8.10 --with-openssl=/opt/tmp/openssl-1.0.0a --add-module=/opt/tmp/substitutions4nginx --add-module=/opt/tmp/ngx_cache_purge-1.3<br>
<br>--<br>lei yang<br></div>