Maxim,<div><br></div><div>You may be a genius.</div><div><br></div><div>I see the backlog is only 1024, so I upped that </div><div><div>echo 10024 > /proc/sys/net/ipv4/tcp_max_syn_backlog</div><div><br></div><div>Things feel responsive, but I will only know more in 24 hours when we hit peak loads.</div>
<div><br></div><div>Best,</div><div>Khalid</div><br><div class="gmail_quote">On Mon, Sep 21, 2009 at 9:48 AM, Maxim Dounin <span dir="ltr"><<a href="mailto:mdounin@mdounin.ru">mdounin@mdounin.ru</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Hello!<br>
<div><div></div><div class="h5"><br>
On Mon, Sep 21, 2009 at 09:26:22AM -0700, Khalid Shaikh wrote:<br>
<br>
> Team,<br>
> Here is another sample point. See the 9 second delay on the server I keep<br>
> speaking about. This problem shows across multiple servers. The one we are<br>
> investigating is <a href="http://67.159.60.59/status" target="_blank">http://67.159.60.59/status</a>. Same nginx.conf across all.<br>
><br>
> root@pa:~# clear; ./status.rb<br>
> Active connections: 17442<br>
> server accepts handled requests<br>
> 291457 291457 499538<br>
> Reading: 242 Writing: 3838 Waiting: 13362<br>
> <a href="http://208.53.158.42/status" target="_blank">http://208.53.158.42/status</a> 0.07 seconds<br>
><br>
> Active connections: 11545<br>
> server accepts handled requests<br>
> 8720635 8720635 13904180<br>
> Reading: 154 Writing: 1370 Waiting: 10021<br>
> <a href="http://67.159.60.37/status" target="_blank">http://67.159.60.37/status</a> 0.03 seconds<br>
><br>
> Active connections: 3807<br>
> server accepts handled requests<br>
> 3758087 3758087 27448407<br>
> Reading: 44 Writing: 213 Waiting: 3550<br>
> <a href="http://66.90.101.97/status" target="_blank">http://66.90.101.97/status</a> 0.04 seconds<br>
><br>
> Active connections: 8855<br>
> server accepts handled requests<br>
> 6865763 6865763 10978021<br>
> Reading: 133 Writing: 988 Waiting: 7734<br>
> <a href="http://208.53.158.75/status" target="_blank">http://208.53.158.75/status</a> 0.03 seconds<br>
><br>
> Active connections: 10571<br>
> server accepts handled requests<br>
> 194174 194174 467424<br>
> Reading: 106 Writing: 2147 Waiting: 8318<br>
> <a href="http://67.159.60.59/status" target="_blank">http://67.159.60.59/status</a> 9.03 seconds<br>
<br>
</div></div>After looking closely on timings you provide - it's<br>
suspiciously rounded delays. Looks like packet<br>
drops/retransmissions. This may be due to some network resource<br>
shortage on the host in question or some problems on the wire.<br>
<br>
First of all, try tcpdump to see what actually happens on the<br>
wire.<br>
<br>
Also it's probably a good idea to examine your backlog size and<br>
status. AFAIK Linux by default just drops incoming SYN packets on<br>
listen queue overflows, so this may be the reason.<br>
<font color="#888888"><br>
Maxim Dounin<br>
</font><div><div></div><div class="h5"><br>
><br>
> Active connections: 8354<br>
> server accepts handled requests<br>
> 4314517 4314517 6679147<br>
> Reading: 195 Writing: 1954 Waiting: 6205<br>
> <a href="http://208.53.158.200/status" target="_blank">http://208.53.158.200/status</a> 0.03 seconds<br>
<br>
</div></div></blockquote></div><br></div>