I have been looking at this issue to try to understand the interactions and this is what I have found out so far.<div><div><br></div><div>I can reproduce the issue by decreasing the <span class="Apple-style-span" style="border-collapse: collapse; ">net.ipv4.tcp_max_syn_backlog and setting net.ipv4.tcp_syncookies = 0. It makes sense that when the load gets a little high the system does not process the accept fast enough and the backlog builds up. When the backlog fills up the ACK to the initial SYN is not issued. </span></div>
<div><span class="Apple-style-span" style="border-collapse: collapse; ">It would seem also that enabling syncookies uses some other mechanism as I do not see the dropped connections when I have net.ipv4.tcp_syncookies = 1 and back log to some small value like net.ipv4.tcp_max_syn_backlog =10. From what I read so far is that syncookies goes into action when you hit a certain high water mark on your backlog buffer. Raising the backlog value and/or enabling syncookies seems like a good idea. </span></div>
<div><span class="Apple-style-span" style="border-collapse: collapse;">You can look at how many connection are in the backlog with net stat.</span></div><div><div>netstat -aln | grep -c SYN_RECV</div><div>146</div><div>The problem here is that this gives you and instantaneous count of the backlog. As you can imagine this backlog is getting filled and cleared at a high rate. It would be nice to know when this actually gets full or even what the max has been over a period of time. Is there a stat like this in net statistics?</div>
<div>This information seems like it is quite important in determining the performance of your server latency. </div><div><br></div><div>In nginx there is the optional backlog variable in the listen configuration. I imagine that this maps directly to the system call listen. Though I see that the nginx backlog default is -1. How does that translate to the system parameter which requires an in above 0.</div>
<div><br></div><div><br></div><div>Radek</div></div><div><br></div><div><br><br><div class="gmail_quote">On Wed, Feb 4, 2009 at 12:29 AM, Olivier B. <span dir="ltr"><<a href="mailto:nginx.list@daevel.fr">nginx.list@daevel.fr</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Hello,<br>
<br>
have you tried to disable syncookies and/or adjust them ?<br>
On the varnish documentation ( <a href="http://varnish.projects.linpro.no/wiki/Performance" target="_blank">http://varnish.projects.linpro.no/wiki/Performance</a> ) there is at least this setup (in /etc/sysctl.conf) :<br>
<br>
net.ipv4.tcp_syncookies = 0<br>
net.ipv4.tcp_max_orphans = 262144<br>
net.ipv4.tcp_max_syn_backlog = 262144<br>
net.ipv4.tcp_synack_retries = 2<br>
net.ipv4.tcp_syn_retries = 2<br>
<br>
<br>
and what about netfilter connection tracking ?<br>
<br>
Olivier<br>
<br>
Radek Burkat a écrit :<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="Ih2E3d">
Thanks Igor, for eliminating one variable for me. I noticed that when I use a single worker and set the affinity to the same core as the ethernet interface interrupt I decrease the frequency of the issue but do not eliminate it. Not sure <br>
Are there any kernel/network debugging tools which reach a little deeper than tcpdump?<br>
<br>
Radek<br>
<br></div><div><div></div><div class="Wj3C7c">
On Tue, Feb 3, 2009 at 10:11 PM, Igor Sysoev <<a href="mailto:is@rambler-co.ru" target="_blank">is@rambler-co.ru</a> <mailto:<a href="mailto:is@rambler-co.ru" target="_blank">is@rambler-co.ru</a>>> wrote:<br>
<br>
On Tue, Feb 03, 2009 at 10:07:27PM -0800, Radek Burkat wrote:<br>
<br>
> Have a machine running the latest devel version nginx-0.7.33<br>
(tried 0.6.35<br>
> with same results) for serving small (less than 10K images) and<br>
am seeing on<br>
> tcpdump that some SYN packets are not responded to right<br>
away.The browser<br>
> does retransmit these image requests every second and on the 2nd<br>
or 3rd<br>
> resent SYN, I finally start seeing and ACK, and the images load.<br>
> It is very indeterministic as to when it happens and can only<br>
reproduce it<br>
> some of the time. When it does occur the outcome is a page with<br>
some images<br>
> loaded and others (whose SYN packets are not ACKs) are not<br>
loaded.....a few<br>
> seconds later they load.<br>
> Typically the system has ~2000 active connections, most in keep<br>
alive. The<br>
> load is around 100-200 req/sec.<br>
><br>
> I have tries all sorts of settings and configurations suggested<br>
in the<br>
> maillist but I still dont have the solution for this issue.<br>
from 1 to 4<br>
> workers, changing the connection counts, different even<br>
handlers, kernel<br>
> buffers, etc.<br>
> It just seems so anecdotal to just change a bunch of settings<br>
without being<br>
> able to what is happening internally.<br>
> I'd like to be able to debug a little deeper to find out what is<br>
happening<br>
> to these packets.<br>
><br>
> How would I go about debugging what is the cause of it. Is it<br>
the interface<br>
> driver, kernel, or nginx? What kind of tools and debugging<br>
options can I<br>
> try next?<br>
<br>
nginx has not any relation to TCP handshake (SYN, SYN/ACK, ACK),<br>
this is kernel or driver issue.<br>
<br>
> Thanks, Radek<br>
><br>
><br>
> System Details<br>
> model name : Intel(R) Xeon(R) CPU X3210 @ 2.13GHz<br>
> Linux n06 2.6.18-92.1.22.el5 #1 SMP Tue Dec 16 12:03:43 EST 2008<br>
i686 i686<br>
> i386 GNU/Linux<br>
> eth0<br>
> Advertised auto-negotiation: Yes<br>
> Speed: 1000Mb/s<br>
> Duplex: Full<br>
> Port: Twisted Pair<br>
> PHYAD: 1<br>
> Transceiver: internal<br>
> Auto-negotiation: on<br>
> Supports Wake-on: g<br>
> Wake-on: d<br>
> Current message level: 0x000000ff (255)<br>
> Link detected: yes<br>
> driver: tg3<br>
> version: 3.86<br>
> firmware-version: 5721-v3.61, ASFIPMI v6.21<br>
> bus-info: 0000:05:00.0<br>
><br>
> avg-cpu: %user %nice %system %iowait %steal %idle<br>
> 0.10 0.00 0.20 2.43 0.00 97.27<br>
> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz<br>
await svctm<br>
> %util<br>
> sda 0.00 0.00 27.40 0.00 443.20 0.00 16.18 0.10 3.50 3.32 9.10<br>
> no overruns or errors on interface.<br>
<br>
--<br>
Igor Sysoev<br>
<a href="http://sysoev.ru/en/" target="_blank">http://sysoev.ru/en/</a><br>
<br>
<br>
</div></div></blockquote>
<br>
<br>
</blockquote></div><br></div></div>