well we can argue on the tools you are using but the point is that you have an obvious performance drop somehow :)<br><br><br>better to move to that direction to be able to find, fix it.<br><br>however i have no problem if you are using apache further, just don't see the point to share your experience here if you don't want to share the config <br>
<br><br>regards,<br>Istvan<br><br><div class="gmail_quote">On Fri, Sep 4, 2009 at 12:36 AM, Chang Song <span dir="ltr"><<a href="mailto:changsong@me.com">changsong@me.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div style="word-wrap: break-word;"><div><br></div><div><br></div>Hi, Istvan<div><br></div><div>It is not about 50K single node nginx throughput.</div><div>I have a standard TCP/IP tuning settings.</div><div><br></div><div>
We can reach 70K throughput in some workloads.</div><div>THey ALL depends on workloads.</div><div><br></div><div>I have not given all the details so that why you are saying that but</div><div>we are measure and capturing every possible system resource under /proc.</div>
<div><br></div><div>dstat does not tell you everything.</div><div>We are currently using collectl and collectd, and captures everything under /proc</div><div><br></div><div>This is a sample of nginx access log (proxy service time and nginx service time is there)</div>
<div><br></div><div><div>[03/Sep/2009:12:19:02 +0900] 10.25.131.46 200 8254 gzip:-% conns:199229 up_response_t:0.392 svc_t:0.831 "GET /index_think.jsp HTTP/1.1"</div><div>[03/Sep/2009:12:19:02 +0900] 10.25.131.48 200 20524 gzip:-% conns:199622 up_response_t:0.150 svc_t:0.668 "GET /static/20k.jsp HTTP/1.1"</div>
<div><br></div><div>I don't think we are in a position to debug deeper into nginx since we need to move on.</div><div>Thanks</div><div><br></div><div><br></div></div><div><div></div><div class="h5"><div><br><div><div>
On Sep 4, 2009, at 7:19 AM, István wrote:</div><br><blockquote type="cite">I see,<div><br></div><div>well I was able to reach about 50K req/s on a single node with nginx with tuning linux/tcp stack/nginx and i learned one thing:</div>
<div><br></div><div>measure instead of guess</div><div><br></div>
<div>(and as a side effect: debug instead of guess.)</div><div><br></div><div>So, if i were you i would start dstat(<span style="font-family: arial,sans-serif; font-size: 13px; border-collapse: collapse;"><span style="background-color: rgb(199, 202, 204);">dstat</span> -cgilpymn)<span style="border-collapse: separate; font-family: arial; font-size: small;"> on that host and see the different elements of you system, enable debug logging, even stracing nginx</span></span></div>
<div><br></div><div>This is the way I think.</div><div><br></div><div>Regards,</div><div>Istvan</div><div><div><br></div><div><br><br><div class="gmail_quote">On Thu, Sep 3, 2009 at 10:51 PM, Chang Song <span dir="ltr"><<a href="mailto:changsong@me.com" target="_blank">changsong@me.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div style="word-wrap: break-word;"><div><br></div><div>Istvan.</div><div>It didn't really matter what config I used.</div>
<div><br></div><div>I tried all combinations of </div><div><br></div><div>worker_process (2-512)</div><div>worker_connections (512-16000)</div><div>accept_mutex (on/off)</div><div>tcp_nopush (on/off)</div><div>tcp_nodelay (on/off)</div>
<div>proxy_buffer* (various sizes)</div><div>and other proxy related parameters you can imagine.</div><div>The one I showed you has the best performance</div><div>The following showed the best performance across the board</div>
<div><br></div><div>worker_process 2; # since we have 2 core machine</div><div>worker_connections 16000; </div><div>accept_mutex off;</div><div>max_connections 256; # ey-balancer (tomcat had 512 threads)</div>
<div>everything else default</div><div><br></div><div><div>Thank you.</div><div><br></div><div><br></div></div><div><div></div><div><br><div><div>On Sep 3, 2009, at 4:26 PM, István wrote:</div><br><blockquote type="cite">
I think it would be beneficial to show us your nginx config :)<br><br>Regards,<br>Istvan<br><br><div class="gmail_quote">On Thu, Sep 3, 2009 at 5:50 AM, Chang Song <span dir="ltr"><<a href="mailto:changsong@me.com" target="_blank">changsong@me.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Sometime ago, I posted an message about Nginx performance when paired with Tomcat.<br>
We recently did extensive in-house testing of various workload against Nginx with Tomcat vs Apache vs Tomcat.<br>
<br>
Apache wins hands down.<br>
<br>
Here's the basic setup<br>
<br>
1. Nginx (2 proc/8192 connections) -> http/1.0 -> Tomcat (HTTP connector)<br>
2. Apache (512 prefork) -> AJP -> Tomcat (AJP)<br>
<br>
Both KeepAlive off (we don't use KeepAlive due to L4 switch)<br>
<br>
The physical server is 2 core Intel Xeon, which is typical web server config here.<br>
We have three grinder 3.2 load generators.<br>
We tested 4K and 20K Tomcat simple HTML file, 20K simple HTML with intentional 10% 200ms<br>
sleep in Tomcat serving (emulate slow DB query), etc.<br>
<br>
Every single case, Apache wins by at least 10-15%.<br>
Throughput and response time.<br>
Nginx uses a bit less CPU cycles (10-20%), but it is not able drive Tomcat to 100% CPU.<br>
<br>
Here's my take on this performance problem.<br>
<br>
1. Lack of AJP support, which is an optimized HTTP protocol<br>
First of all, this is a serious bottleneck.<br>
<br>
* AJP has much less communication overhead than HTTP<br>
<br>
2. Lack of HTTP KeepAlive support for proxy<br>
<br>
* Lack of AJP may be compensated with HTTP keepalive support since there are<br>
at least twice the number of TIME_WAIT sockets (connection establishment mean time<br>
is at least twice - three times slower than that of Apache)<br>
<br>
3. Lack of connection pooling<br>
<br>
* Ey-balancer makes things a bit easier, response times are stable, but still the same<br>
average TPS and response time.<br>
<br>
4. There seems to be a huge bug in connection management code<br>
<br>
Two mix of transactions: 20K HTML serving and 8K HTML with intentional 200ms delay in Tomcat logic<br>
<br>
With Apache, 20K HTML serving took 36 ms on average while 8K HTML took 258 ms<br>
With Nginx, 20K HTML serving took 600 ms on average while 8K HTML took 817 ms<br>
<br>
I really cannot explain these difference. Not even TCP connection overhead or lack of AJP.<br>
<br>
My questions is "should I abandon nginx at this point"?<br>
I know nginx is great proxy and static file server but I cannot prove my point with Tomcat over and over again.<br>
<br>
Thank you<br><font color="#888888">
<br>
Chang<br>
<br>
<br>
<br>
<br>
</font></blockquote></div><br><br clear="all"><br>-- <br>the sun shines for all<br>
</blockquote></div><br></div></div></div></blockquote></div><br><br clear="all"><br>-- <br>the sun shines for all<br>
</div></div>
</blockquote></div><br></div></div></div></div></blockquote></div><br><br clear="all"><br>-- <br>the sun shines for all<br>