Thanks Igor, for eliminating one variable for me. <div><div>I noticed that when I use a single worker and set the affinity to the same core as the ethernet interface interrupt I decrease the frequency of the issue but do not eliminate it. Not sure <div>
<br></div><div>Are there any kernel/network debugging tools which reach a little deeper than tcpdump?</div><div><br></div><div>Radek<br><br><div class="gmail_quote">On Tue, Feb 3, 2009 at 10:11 PM, Igor Sysoev <span dir="ltr"><<a href="mailto:is@rambler-co.ru" target="_blank">is@rambler-co.ru</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>On Tue, Feb 03, 2009 at 10:07:27PM -0800, Radek Burkat wrote:<br>
<br>
> Have a machine running the latest devel version nginx-0.7.33 (tried 0.6.35<br>
> with same results) for serving small (less than 10K images) and am seeing on<br>
> tcpdump that some SYN packets are not responded to right away.The browser<br>
> does retransmit these image requests every second and on the 2nd or 3rd<br>
> resent SYN, I finally start seeing and ACK, and the images load.<br>
> It is very indeterministic as to when it happens and can only reproduce it<br>
> some of the time. When it does occur the outcome is a page with some images<br>
> loaded and others (whose SYN packets are not ACKs) are not loaded.....a few<br>
> seconds later they load.<br>
> Typically the system has ~2000 active connections, most in keep alive. The<br>
> load is around 100-200 req/sec.<br>
><br>
> I have tries all sorts of settings and configurations suggested in the<br>
> maillist but I still dont have the solution for this issue. from 1 to 4<br>
> workers, changing the connection counts, different even handlers, kernel<br>
> buffers, etc.<br>
> It just seems so anecdotal to just change a bunch of settings without being<br>
> able to what is happening internally.<br>
> I'd like to be able to debug a little deeper to find out what is happening<br>
> to these packets.<br>
><br>
> How would I go about debugging what is the cause of it. Is it the interface<br>
> driver, kernel, or nginx? What kind of tools and debugging options can I<br>
> try next?<br>
<br>
</div>nginx has not any relation to TCP handshake (SYN, SYN/ACK, ACK),<br>
this is kernel or driver issue.<br>
<div><div></div><div><br>
> Thanks, Radek<br>
><br>
><br>
> System Details<br>
> model name : Intel(R) Xeon(R) CPU X3210 @ 2.13GHz<br>
> Linux n06 2.6.18-92.1.22.el5 #1 SMP Tue Dec 16 12:03:43 EST 2008 i686 i686<br>
> i386 GNU/Linux<br>
> eth0<br>
> Advertised auto-negotiation: Yes<br>
> Speed: 1000Mb/s<br>
> Duplex: Full<br>
> Port: Twisted Pair<br>
> PHYAD: 1<br>
> Transceiver: internal<br>
> Auto-negotiation: on<br>
> Supports Wake-on: g<br>
> Wake-on: d<br>
> Current message level: 0x000000ff (255)<br>
> Link detected: yes<br>
> driver: tg3<br>
> version: 3.86<br>
> firmware-version: 5721-v3.61, ASFIPMI v6.21<br>
> bus-info: 0000:05:00.0<br>
><br>
> avg-cpu: %user %nice %system %iowait %steal %idle<br>
> 0.10 0.00 0.20 2.43 0.00 97.27<br>
> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm<br>
> %util<br>
> sda 0.00 0.00 27.40 0.00 443.20 0.00 16.18 0.10 3.50 3.32 9.10<br>
> no overruns or errors on interface.<br>
<br>
</div></div><font color="#888888">--<br>
Igor Sysoev<br>
<a href="http://sysoev.ru/en/" target="_blank">http://sysoev.ru/en/</a><br>
<br>
</font></blockquote></div><br></div></div></div>