I actually tried the latest 0.6x and .7x. The value of 10 was just for testing and reproducing what I was seeing. I had the system backlog at 1024 so I must have been hitting the 511 of nginx. <div>I verified at least on rel5/centos5 that when the backlog is reached, no RST packet is sent. The SYN is just dropped and ignored. This is also consistent with the documentation of backlog behavior from Stevens, Unix Network Programming.</div>
<div><br></div><div>One of the things that the documentation suggests, but is very vague about, is that the backlog count may not only be connections that are in the SYN_RECV state but also connections that are in the soft ESTABLISHED phase. That is server sent out the syn/ack but has not yet received the ack/first data from the client. I'll try to verify this.</div>
<div><br></div><div>It would seem that on even a moderately busy server serving pages of images it is very easy to hit the 511 backlog. Even though my server can saturate a gigabit at 16k req/s on ab, i hit this backlog limit pretty regularly on conditions where the average is 200 req/s. </div>
<div>As I explained earlier, when this happens your image is delayed by at least a second until the browser re requests. Worst of all there are no errors, or statistics that would suggest this is happening. Of course I can just increase the backlog but at this point might as well try to understand this mechanism and figure out how to detect it in the future.</div>
<div><br></div><div>I'll investigate some more and post here in the effort that this may help others with debugging similar issues.</div><div><br></div><div>Radek</div><div> </div><div><div><br><br><div class="gmail_quote">
On Mon, Feb 9, 2009 at 11:02 PM, Igor Sysoev <span dir="ltr"><<a href="mailto:is@rambler-co.ru">is@rambler-co.ru</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div><div></div><div class="Wj3C7c">On Mon, Feb 09, 2009 at 01:51:12PM -0800, Radek Burkat wrote:<br>
<br>
> I have been looking at this issue to try to understand the interactions and<br>
> this is what I have found out so far.<br>
> I can reproduce the issue by decreasing the net.ipv4.tcp_max_syn_backlog<br>
> and setting net.ipv4.tcp_syncookies = 0. It makes sense that when the load<br>
> gets a little high the system does not process the accept fast enough and<br>
> the backlog builds up. When the backlog fills up the ACK to the initial SYN<br>
> is not issued.<br>
> It would seem also that enabling syncookies uses some other mechanism as I<br>
> do not see the dropped connections when I have net.ipv4.tcp_syncookies = 1<br>
> and back log to some small value like net.ipv4.tcp_max_syn_backlog =10.<br>
> From what I read so far is that syncookies goes into action when you hit a<br>
> certain high water mark on your backlog buffer. Raising the backlog value<br>
> and/or enabling syncookies seems like a good idea.<br>
> You can look at how many connection are in the backlog with net stat.<br>
> netstat -aln | grep -c SYN_RECV<br>
> 146<br>
> The problem here is that this gives you and instantaneous count of the<br>
> backlog. As you can imagine this backlog is getting filled and cleared at a<br>
> high rate. It would be nice to know when this actually gets full or even<br>
> what the max has been over a period of time. Is there a stat like this in<br>
> net statistics?<br>
> This information seems like it is quite important in determining the<br>
> performance of your server latency.<br>
><br>
> In nginx there is the optional backlog variable in the listen configuration.<br>
> I imagine that this maps directly to the system call listen. Though I see<br>
> that the nginx backlog default is -1. How does that translate to the system<br>
> parameter which requires an in above 0.<br>
<br>
</div></div>Yes, backlog may affect this. However, it seems that you run very old<br>
nginx as it had been fixed in 2007:<br>
<br>
Changes with nginx 0.6.7 15 Aug 2007<br>
Changes with nginx 0.5.32 24 Sep 2007<br>
<br>
*) Bugfix: now nginx uses default listen backlog value 511 on all<br>
platforms except FreeBSD.<br>
Thanks to Jiang Hong.<br>
<br>
Now -1 is used on FreeBSD only and it means the maximum available backlog.<br>
<br>
Besides, net.ipv4.tcp_max_syn_backlog = 10 is too small.<br>
<br>
Also, when kernel has no enough backlog, it usually response with<br>
RST packet, if this is not disabled.<br>
<div><div></div><div class="Wj3C7c"><br>
<br>
> Radek<br>
><br>
><br>
><br>
> On Wed, Feb 4, 2009 at 12:29 AM, Olivier B. <<a href="mailto:nginx.list@daevel.fr">nginx.list@daevel.fr</a>> wrote:<br>
><br>
> > Hello,<br>
> ><br>
> > have you tried to disable syncookies and/or adjust them ?<br>
> > On the varnish documentation (<br>
> > <a href="http://varnish.projects.linpro.no/wiki/Performance" target="_blank">http://varnish.projects.linpro.no/wiki/Performance</a> ) there is at least<br>
> > this setup (in /etc/sysctl.conf) :<br>
> ><br>
> > net.ipv4.tcp_syncookies = 0<br>
> > net.ipv4.tcp_max_orphans = 262144<br>
> > net.ipv4.tcp_max_syn_backlog = 262144<br>
> > net.ipv4.tcp_synack_retries = 2<br>
> > net.ipv4.tcp_syn_retries = 2<br>
> ><br>
> ><br>
> > and what about netfilter connection tracking ?<br>
> ><br>
> > Olivier<br>
> ><br>
> > Radek Burkat a ?crit :<br>
> ><br>
> >> Thanks Igor, for eliminating one variable for me. I noticed that when I<br>
> >> use a single worker and set the affinity to the same core as the ethernet<br>
> >> interface interrupt I decrease the frequency of the issue but do not<br>
> >> eliminate it. Not sure<br>
> >> Are there any kernel/network debugging tools which reach a little deeper<br>
> >> than tcpdump?<br>
> >><br>
> >> Radek<br>
> >><br>
> >> On Tue, Feb 3, 2009 at 10:11 PM, Igor Sysoev <<a href="mailto:is@rambler-co.ru">is@rambler-co.ru</a> <mailto:<br>
> >> <a href="mailto:is@rambler-co.ru">is@rambler-co.ru</a>>> wrote:<br>
> >><br>
> >> On Tue, Feb 03, 2009 at 10:07:27PM -0800, Radek Burkat wrote:<br>
> >><br>
> >> > Have a machine running the latest devel version nginx-0.7.33<br>
> >> (tried 0.6.35<br>
> >> > with same results) for serving small (less than 10K images) and<br>
> >> am seeing on<br>
> >> > tcpdump that some SYN packets are not responded to right<br>
> >> away.The browser<br>
> >> > does retransmit these image requests every second and on the 2nd<br>
> >> or 3rd<br>
> >> > resent SYN, I finally start seeing and ACK, and the images load.<br>
> >> > It is very indeterministic as to when it happens and can only<br>
> >> reproduce it<br>
> >> > some of the time. When it does occur the outcome is a page with<br>
> >> some images<br>
> >> > loaded and others (whose SYN packets are not ACKs) are not<br>
> >> loaded.....a few<br>
> >> > seconds later they load.<br>
> >> > Typically the system has ~2000 active connections, most in keep<br>
> >> alive. The<br>
> >> > load is around 100-200 req/sec.<br>
> >> ><br>
> >> > I have tries all sorts of settings and configurations suggested<br>
> >> in the<br>
> >> > maillist but I still dont have the solution for this issue.<br>
> >> from 1 to 4<br>
> >> > workers, changing the connection counts, different even<br>
> >> handlers, kernel<br>
> >> > buffers, etc.<br>
> >> > It just seems so anecdotal to just change a bunch of settings<br>
> >> without being<br>
> >> > able to what is happening internally.<br>
> >> > I'd like to be able to debug a little deeper to find out what is<br>
> >> happening<br>
> >> > to these packets.<br>
> >> ><br>
> >> > How would I go about debugging what is the cause of it. Is it<br>
> >> the interface<br>
> >> > driver, kernel, or nginx? What kind of tools and debugging<br>
> >> options can I<br>
> >> > try next?<br>
> >><br>
> >> nginx has not any relation to TCP handshake (SYN, SYN/ACK, ACK),<br>
> >> this is kernel or driver issue.<br>
> >><br>
> >> > Thanks, Radek<br>
> >> ><br>
> >> ><br>
> >> > System Details<br>
> >> > model name : Intel(R) Xeon(R) CPU X3210 @ 2.13GHz<br>
> >> > Linux n06 2.6.18-92.1.22.el5 #1 SMP Tue Dec 16 12:03:43 EST 2008<br>
> >> i686 i686<br>
> >> > i386 GNU/Linux<br>
> >> > eth0<br>
> >> > Advertised auto-negotiation: Yes<br>
> >> > Speed: 1000Mb/s<br>
> >> > Duplex: Full<br>
> >> > Port: Twisted Pair<br>
> >> > PHYAD: 1<br>
> >> > Transceiver: internal<br>
> >> > Auto-negotiation: on<br>
> >> > Supports Wake-on: g<br>
> >> > Wake-on: d<br>
> >> > Current message level: 0x000000ff (255)<br>
> >> > Link detected: yes<br>
> >> > driver: tg3<br>
> >> > version: 3.86<br>
> >> > firmware-version: 5721-v3.61, ASFIPMI v6.21<br>
> >> > bus-info: 0000:05:00.0<br>
> >> ><br>
> >> > avg-cpu: %user %nice %system %iowait %steal %idle<br>
> >> > 0.10 0.00 0.20 2.43 0.00 97.27<br>
> >> > Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz<br>
> >> await svctm<br>
> >> > %util<br>
> >> > sda 0.00 0.00 27.40 0.00 443.20 0.00 16.18 0.10 3.50 3.32 9.10<br>
> >> > no overruns or errors on interface.<br>
> >><br>
> >> --<br>
> >> Igor Sysoev<br>
> >> <a href="http://sysoev.ru/en/" target="_blank">http://sysoev.ru/en/</a><br>
> >><br>
> >><br>
> >><br>
> ><br>
> ><br>
<br>
</div></div>--<br>
<div><div></div><div class="Wj3C7c">Igor Sysoev<br>
<a href="http://sysoev.ru/en/" target="_blank">http://sysoev.ru/en/</a><br>
<br>
</div></div></blockquote></div><br></div></div>