<div dir="ltr">Applied the patch, still the same.<br>The other only difference I can think about is that I'm running Cherokee on a different port. Maybe my VPS is handling them differently.<br>I'll try testing it on the same port this evening and let you know.<br>
<br>Thank you.<br><br>Yo'av<br><br><div class="gmail_quote">On Tue, Jan 27, 2009 at 11:40 AM, Igor Sysoev <span dir="ltr"><<a href="mailto:is@rambler-co.ru">is@rambler-co.ru</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div class="Ih2E3d">On Tue, Jan 27, 2009 at 11:25:09AM +0200, Yo'av Moshe wrote:<br>
<br>
> Sure, here my Cherokee's strace:<br>
> <a href="http://pastebin.com/m13cd213d" target="_blank">http://pastebin.com/m13cd213d</a><br>
><br>
> What you are saying about tcpdump's logs is strange, but I have no idea<br>
> what's causing it.<br>
<br>
</div>I see only two difference with nginx:<br>
<br>
1) it uses level triggered epoll (nginx uses edge triggered epoll),<br>
2) and it uses TCP_NODELAY for first request (nginx set TCP_NODELAY<br>
only just before going to keep-alive).<br>
<br>
The attached patch sets TCP_NODELAY from very start.<br>
<div><div></div><div class="Wj3C7c"><br>
> Yo'av.<br>
><br>
> On Tue, Jan 27, 2009 at 10:51 AM, Igor Sysoev <<a href="mailto:is@rambler-co.ru">is@rambler-co.ru</a>> wrote:<br>
><br>
> > On Tue, Jan 27, 2009 at 10:42:57AM +0200, Yo'av Moshe wrote:<br>
> ><br>
> > > Note I just compiled Cherokee 0.98. It is serving the file as it should<br>
> > > using sendfile and epoll.<br>
> ><br>
> > Have you tried this on the same host ?<br>
> > Could you post strace ?<br>
> ><br>
> > > On Mon, Jan 26, 2009 at 2:05 PM, Yo'av Moshe <<a href="mailto:bjesus@gmail.com">bjesus@gmail.com</a>> wrote:<br>
> > ><br>
> > > > Hey, attached is my tcpdump log.<br>
> > > > I've never used tcpdump before, so I hope the log is fine.<br>
> > > > I used grep to show only the connections from my computer while running<br>
> > > > 'wget <a href="http://www.noal.org.il/static/temp/hozer.pdf" target="_blank">http://www.noal.org.il/static/temp/hozer.pdf</a>'.<br>
> > > ><br>
> > > > Hope this will lead us somewhere...<br>
> > > ><br>
> > > > Yo'av.<br>
> > > ><br>
> > > ><br>
> > > > On Sun, Jan 25, 2009 at 9:21 PM, Igor Sysoev <<a href="mailto:is@rambler-co.ru">is@rambler-co.ru</a>> wrote:<br>
> > > ><br>
> > > >> On Sun, Jan 25, 2009 at 09:04:54PM +0200, Yo'av Moshe wrote:<br>
> > > >><br>
> > > >> > Sendfile is now off, using select, and still the same...<br>
> > > >> > <a href="http://pastebin.com/m2c3ea25b" target="_blank">http://pastebin.com/m2c3ea25b</a><br>
> > > >><br>
> > > >> I have no idea, why it may be so.<br>
> > > >><br>
> > > >> I see that select() returned event ("out [19]"):<br>
> > > >><br>
> > > >> 28215 20:50:18 select(20, [10 19], [19], NULL, {0, 500000} <unfinished<br>
> > > >> ...><br>
> > > >> 28215 20:50:18 <... select resumed> ) = 1 (out [19], left {0,<br>
> > 380000})<br>
> > > >><br>
> > > >> However, at some stage select() did not return event, although the<br>
> > event<br>
> > > >> is still active ("select(20, [10 19], [19], NULL ..."):<br>
> > > >><br>
> > > >> 28215 20:50:18 gettimeofday({1232909418, 293895}, NULL) = 0<br>
> > > >> 28215 20:50:18 writev(19,<br>
> > > >><br>
> > [{"\227\1\2\1\3\1\4\1\5\0\373\0\374\1\230\1\231\1\232\1\233\0\375\0\376\1\6\1\7\1\10\0"...,<br>
> > > >> 9676}], 1) = 9676<br>
> > > >> 28215 20:50:18 pread64(20,<br>
> > > >><br>
> > "\24\7\6#\"\'&\0214\22$63\1\6\25\24\26\27\02632654&#\"\6\3\226\204"...,<br>
> > > >> 32768, 1605632) = 32768<br>
> > > >> 28215 20:50:18 writev(19,<br>
> > > >><br>
> > [{"\24\7\6#\"\'&\0214\22$63\1\6\25\24\26\27\02632654&#\"\6\3\226\204"...,<br>
> > > >> 32768}], 1) = 32768<br>
> > > >> 28215 20:50:18 pread64(20, "\0 \0M\0o\0r\0i\0s\0o\0n\0\'\0s\0<br>
> > > >> \0d\0i\0r\0e\0c"..., 32768, 1638400) = 32768<br>
> > > >> 28215 20:50:18 writev(19, [{"\0 \0M\0o\0r\0i\0s\0o\0n\0\'\0s\0<br>
> > > >> \0d\0i\0r\0e\0c"..., 32768}], 1) = 1716<br>
> > > >> 28215 20:50:18 select(20, [10 19], [19], NULL, {0, 500000} <unfinished<br>
> > > >> ...><br>
> > > >> 28215 20:50:18 <... select resumed> ) = 0 (Timeout)<br>
> > > >><br>
> > > >> Could you run tcpdump of the session ?<br>
> > > >><br>
> > > >> > Yo'av<br>
> > > >> ><br>
> > > >> > On Sun, Jan 25, 2009 at 7:33 PM, Igor Sysoev <<a href="mailto:is@rambler-co.ru">is@rambler-co.ru</a>><br>
> > wrote:<br>
> > > >> ><br>
> > > >> > > On Sun, Jan 25, 2009 at 06:26:58PM +0200, Yo'av Moshe wrote:<br>
> > > >> > ><br>
> > > >> > > > Igor, I tried using select instead of poll, but I get the exact<br>
> > same<br>
> > > >> > > > problem.<br>
> > > >> > > ><br>
> > > >> > > > Check my strace:<br>
> > > >> > > > <a href="http://pastebin.com/m65056ec0" target="_blank">http://pastebin.com/m65056ec0</a><br>
> > > >> > > ><br>
> > > >> > > > I get the same thing with rtsig too.<br>
> > > >> > > ><br>
> > > >> > > > Hints?<br>
> > > >> > > > Again, file is at <a href="http://www.noal.org.il/static/temp/hozer.pdf" target="_blank">http://www.noal.org.il/static/temp/hozer.pdf</a>.<br>
> > > >> Server<br>
> > > >> > > is<br>
> > > >> > > > using select now.<br>
> > > >> > ><br>
> > > >> > > According strace select() does not return event for socket 19<br>
> > after<br>
> > > >> > > sendfile64() sent 1651005 bytes:<br>
> > > >> > ><br>
> > > >> > > 25951 18:14:26 select(20, [7 8 16 19], [19], NULL, {60, 0}<br>
> > <unfinished<br>
> > > >> ...><br>
> > > >> > > 25951 18:14:27 <... select resumed> ) = 1 (out [19], left {58,<br>
> > > >> 790000})<br>
> > > >> > ><br>
> > > >> > > 25951 18:14:27 gettimeofday({1232900067, 890376}, NULL) = 0<br>
> > > >> > > 25951 18:14:27 sendfile64(19, 20, [1509465], 440765 <unfinished<br>
> > ...><br>
> > > >> > > 25951 18:14:27 <... sendfile64 resumed> ) = 141540<br>
> > > >> > > 25951 18:14:27 sendfile64(19, 20, [1651005], 299225 <unfinished<br>
> > ...><br>
> > > >> > > 25951 18:14:27 <... sendfile64 resumed> ) = -1 EAGAIN (Resource<br>
> > > >> temporarily<br>
> > > >> > > unavailable)<br>
> > > >> > ><br>
> > > >> > > 25951 18:14:27 select(20, [16 19], [19], NULL, {0, 500000}<br>
> > <unfinished<br>
> > > >> ...><br>
> > > >> > > 25951 18:14:28 <... select resumed> ) = 0 (Timeout)<br>
> > > >> > ><br>
> > > >> > > As you have got the same issue with all methods (even with level<br>
> > > >> triggered<br>
> > > >> > > select()), I believe that the bug is in sendfile. You may try to<br>
> > turn<br>
> > > >> > > it off even for the single file:<br>
> > > >> > ><br>
> > > >> > > location = /static/temp/hozer.pdf {<br>
> > > >> > > sendfile off;<br>
> > > >> > > }<br>
> > > >> > ><br>
> > > >> > > In Linux 2.6.23 sendfile() has been rewritten to use splice<br>
> > framework.<br>
> > > >> > > The bug may be introduced while rewriting.<br>
> > > >> > ><br>
> > > >> > > BTW, it seems that thttpd does not use sendfile.<br>
> > > >> > ><br>
> > > >> > > > Thanks...<br>
> > > >> > > > Yo'av.<br>
> > > >> > > ><br>
> > > >> > > > On Fri, Jan 23, 2009 at 3:05 PM, Igor Sysoev <<a href="mailto:is@rambler-co.ru">is@rambler-co.ru</a>><br>
> > > >> wrote:<br>
> > > >> > > ><br>
> > > >> > > > > On Fri, Jan 23, 2009 at 02:35:00PM +0200, Yo'av Moshe wrote:<br>
> > > >> > > > ><br>
> > > >> > > > > > Thanks Igor.<br>
> > > >> > > > > > I don't want to be rude, but I find it hard to believe that<br>
> > it's<br>
> > > >> a<br>
> > > >> > > kernel<br>
> > > >> > > > > > bug.<br>
> > > >> > > > > ><br>
> > > >> > > > > > First, I'm using the default CentOS 5.2 kernel, it's seems<br>
> > to me<br>
> > > >> > > pretty<br>
> > > >> > > > > odd<br>
> > > >> > > > > > that there's such a bug in it. CentOS is being used on a lot<br>
> > of<br>
> > > >> > > servers,<br>
> > > >> > > > > > some of them must be running nginx.<br>
> > > >> > > > > ><br>
> > > >> > > > > > Second, I tried serving the same file with thttpd, and it<br>
> > runs<br>
> > > >> just<br>
> > > >> > > fine.<br>
> > > >> > > > > > <a href="http://195.28.180.147:40/temp/hozer.pdf" target="_blank">http://195.28.180.147:40/temp/hozer.pdf</a><br>
> > > >> > > > > > Note it's the exact same file which I cannot get from nginx:<br>
> > > >> > > > > > <a href="http://www.noal.org.il/static/temp/hozer.pdf" target="_blank">http://www.noal.org.il/static/temp/hozer.pdf</a><br>
> > > >> > > > > ><br>
> > > >> > > > > > How come thttpd can serve the file just fine? If it's a<br>
> > kernel<br>
> > > >> bug, I<br>
> > > >> > > > > > thought it would happen with all of my servers.<br>
> > > >> > > > > > If it's using some different way of sending the files which<br>
> > does<br>
> > > >> not<br>
> > > >> > > > > depend<br>
> > > >> > > > > > on that epoll call - is there any way to configure nginx to<br>
> > use<br>
> > > >> that<br>
> > > >> > > as<br>
> > > >> > > > > > well?<br>
> > > >> > > > ><br>
> > > >> > > > > I do not look modern thttpd, but according its change log, it<br>
> > has<br>
> > > >> no<br>
> > > >> > > > > epoll support. You may try different ways in nginx: you need<br>
> > to<br>
> > > >> build<br>
> > > >> > > it<br>
> > > >> > > > ><br>
> > > >> > > > > --with-rtsig_module<br>
> > > >> > > > > --with-select_module<br>
> > > >> > > > > --with-poll_module<br>
> > > >> > > > ><br>
> > > >> > > > > and then choose method:<br>
> > > >> > > > ><br>
> > > >> > > > > events {<br>
> > > >> > > > > use select;<br>
> > > >> > > > > #use poll;<br>
> > > >> > > > > #use rtsig;<br>
> > > >> > > > > }<br>
> > > >> > > > ><br>
> > > >> > > > > Also, epoll has two modes: level triggered (default) and edge<br>
> > > >> > > > > triggered (EPOLLET). nginx uses more effective edge triggered<br>
> > > >> mode,<br>
> > > >> > > > > lighttpd uses level triggered one. The bug may be in ET mode<br>
> > only.<br>
> > > >> > > > ><br>
> > > >> > > > > As to probabilty of kernel bugs: I saw them in FreeBSD<br>
> > (kqueue,<br>
> > > >> > > sendfile),<br>
> > > >> > > > > Linux (epoll), Solaris (event ports), and MacOSX (kqueue,<br>
> > > >> sendfile).<br>
> > > >> > > > ><br>
> > > >> > > > > > Can I do anything to confirm that it's a kernel bug? If it<br>
> > is, I<br>
> > > >> > > should<br>
> > > >> > > > > > probably contact CentOS's maintainers.<br>
> > > >> > > > ><br>
> > > >> > > > > Your straces confirms this: nginx added socket to epoll and<br>
> > did<br>
> > > >> not<br>
> > > >> > > > > deleted it, nevertheless epoll does not send event.<br>
> > > >> > > > ><br>
> > > >> > > > > > Thank you, again.<br>
> > > >> > > > > > Yo'av.<br>
> > > >> > > > > ><br>
> > > >> > > > > > On Wed, Jan 21, 2009 at 7:26 AM, Igor Sysoev <<br>
> > <a href="mailto:is@rambler-co.ru">is@rambler-co.ru</a>><br>
> > > >> > > wrote:<br>
> > > >> > > > > ><br>
> > > >> > > > > > > On Wed, Jan 21, 2009 at 01:52:53AM +0200, Yo'av Moshe<br>
> > wrote:<br>
> > > >> > > > > > ><br>
> > > >> > > > > > > > Igor, here's the new strace.<br>
> > > >> > > > > > > > <a href="http://pastebin.com/m5129c54b" target="_blank">http://pastebin.com/m5129c54b</a><br>
> > > >> > > > > > > ><br>
> > > >> > > > > > > > Does the fact it works in low download rates still makes<br>
> > you<br>
> > > >> > > think<br>
> > > >> > > > > it's a<br>
> > > >> > > > > > > > kernel issue?<br>
> > > >> > > > > > ><br>
> > > >> > > > > > > Yes, this is seems like a kernel issue. Kernel excatly<br>
> > said<br>
> > > >> (with<br>
> > > >> > > > > EAGAIN)<br>
> > > >> > > > > > > that there is no place to output data, but never returned<br>
> > a<br>
> > > >> new<br>
> > > >> > > > > > > notification<br>
> > > >> > > > > > > via epoll_wait() about free space:<br>
> > > >> > > > > > ><br>
> > > >> > > > > > > 21420 01:39:43 <... epoll_wait resumed> {{EPOLLOUT,<br>
> > > >> {u32=153137960,<br>
> > > >> > > > > > > u64=13811628390360331048}}}, 512, 60000) = 1<br>
> > > >> > > > > > > 21420 01:39:43 gettimeofday({1232494783, 413764}, NULL) =<br>
> > 0<br>
> > > >> > > > > > > 21420 01:39:43 sendfile64(20, 21, [1645625], 304605) =<br>
> > 40320<br>
> > > >> > > > > > > 21420 01:39:43 sendfile64(20, 21, [1685945], 264285) = -1<br>
> > > >> EAGAIN<br>
> > > >> > > > > (Resource<br>
> > > >> > > > > > > temporarily unavailable)<br>
> > > >> > > > > > > 21420 01:39:43 epoll_wait(11, <unfinished ...><br>
> > > >> > > > > > ><br>
> > > >> > > > > > > This is probably some race condition inside kernel, that<br>
> > does<br>
> > > >> not<br>
> > > >> > > > > appear on<br>
> > > >> > > > > > > low speed download.<br>
> > > >> > > > > > ><br>
> > > >> > > > > > > > Thank you, really!<br>
> > > >> > > > > > > > And thanks, Kiril and Jeff, for confirming that!<br>
> > > >> > > > > > > ><br>
> > > >> > > > > > > > Yo'av.<br>
> > > >> > > > > > > ><br>
> > > >> > > > > > > ><br>
> > > >> > > > > > > > On Tue, Jan 20, 2009 at 9:39 PM, Kiril Angov <<br>
> > > >> > > <a href="mailto:kupokomapa@gmail.com">kupokomapa@gmail.com</a>><br>
> > > >> > > > > > > wrote:<br>
> > > >> > > > > > > ><br>
> > > >> > > > > > > > > wget <a href="http://www.noal.org.il/static/temp/barvazi2.pdf" target="_blank">http://www.noal.org.il/static/temp/barvazi2.pdf</a><br>
> > > >> > > > > > > > > --2009-01-20 21:36:00--<br>
> > > >> > > > > > > <a href="http://www.noal.org.il/static/temp/barvazi2.pdf" target="_blank">http://www.noal.org.il/static/temp/barvazi2.pdf</a><br>
> > > >> > > > > > > > > Resolving www.noal.org.il... 195.28.180.147<br>
> > > >> > > > > > > > > Connecting to <a href="http://www.noal.org.il" target="_blank">www.noal.org.il</a>|195.28.180.147|:80...<br>
> > > >> connected.<br>
> > > >> > > > > > > > > HTTP request sent, awaiting response... 200 OK<br>
> > > >> > > > > > > > > Length: 1950230 (1.9M) [application/pdf]<br>
> > > >> > > > > > > > > Saving to: `barvazi2.pdf'<br>
> > > >> > > > > > > > ><br>
> > > >> > > > > > > > > 81% [=====================================> ]<br>
> > > >> 1,596,665<br>
> > > >> > > > > > > > > --.-K/s eta 5s<br>
> > > >> > > > > > > > ><br>
> > > >> > > > > > > > ><br>
> > > >> > > > > > > > > wget -c<br>
> > <a href="http://www.noal.org.il/static/temp/barvazi2.pdf" target="_blank">http://www.noal.org.il/static/temp/barvazi2.pdf</a><br>
> > > >> > > > > > > > > --2009-01-20 21:38:28--<br>
> > > >> > > > > > > > > <a href="http://www.noal.org.il/static/temp/barvazi2.pdf" target="_blank">http://www.noal.org.il/static/temp/barvazi2.pdf</a><br>
> > > >> > > > > > > > > Resolving www.noal.org.il... 195.28.180.147<br>
> > > >> > > > > > > > > Connecting to <a href="http://www.noal.org.il" target="_blank">www.noal.org.il</a>|195.28.180.147|:80...<br>
> > > >> connected.<br>
> > > >> > > > > > > > > HTTP request sent, awaiting response... 206 Partial<br>
> > > >> Content<br>
> > > >> > > > > > > > > Length: 1950230 (1.9M), 305960 (299K) remaining<br>
> > > >> > > [application/pdf]<br>
> > > >> > > > > > > > > Saving to: `barvazi2.pdf'<br>
> > > >> > > > > > > > ><br>
> > > >> > > > > > > > > 100%[+++++++++++++++++++++++++++++++++++++++=======>]<br>
> > > >> 1,950,230<br>
> > > >> > > > > > > > > 295K/s in 1.0s<br>
> > > >> > > > > > > > ><br>
> > > >> > > > > > > > > 2009-01-20 21:38:30 (295 KB/s) - `barvazi2.pdf' saved<br>
> > > >> > > > > [1950230/1950230]<br>
> > > >> > > > > > > > ><br>
> > > >> > > > > > > > ><br>
> > > >> > > > > > > > > On Tue, Jan 20, 2009 at 9:31 PM, jeff emminger <<br>
> > > >> > > > > <a href="mailto:jemminger@gmail.com">jemminger@gmail.com</a>><br>
> > > >> > > > > > > > > wrote:<br>
> > > >> > > > > > > > > >> Could anyone confirm that this works:<br>
> > > >> > > > > > > > > >> wget --limit-rate=3500<br>
> > > >> > > > > > > <a href="http://www.noal.org.il/static/temp/barvazi2.pdf" target="_blank">http://www.noal.org.il/static/temp/barvazi2.pdf</a><br>
> > > >> > > > > > > > > >> while this doesn't:<br>
> > > >> > > > > > > > > >> wget<br>
> > <a href="http://www.noal.org.il/static/temp/barvazi2.pdf" target="_blank">http://www.noal.org.il/static/temp/barvazi2.pdf</a><br>
> > > >> > > > > > > > > >><br>
> > > >> > > > > > > > > ><br>
> > > >> > > > > > > > > > Confirmed<br>
> > > >> > > > > > > > > ><br>
> > > >> > > > > > > > > ><br>
> > > >> > > > > > > > ><br>
> > > >> > > > > > > > ><br>
> > > >> > > > > > > ><br>
> > > >> > > > > > > ><br>
> > > >> > > > > > > > --<br>
> > > >> > > > > > > > Yo'av Moshe<br>
> > > >> > > > > > ><br>
> > > >> > > > > > > --<br>
> > > >> > > > > > > Igor Sysoev<br>
> > > >> > > > > > > <a href="http://sysoev.ru/en/" target="_blank">http://sysoev.ru/en/</a><br>
> > > >> > > > > > ><br>
> > > >> > > > > > ><br>
> > > >> > > > > ><br>
> > > >> > > > > ><br>
> > > >> > > > > > --<br>
> > > >> > > > > > Yo'av Moshe<br>
> > > >> > > > ><br>
> > > >> > > > > --<br>
> > > >> > > > > Igor Sysoev<br>
> > > >> > > > > <a href="http://sysoev.ru/en/" target="_blank">http://sysoev.ru/en/</a><br>
> > > >> > > > ><br>
> > > >> > > > ><br>
> > > >> > > ><br>
> > > >> > > ><br>
> > > >> > > > --<br>
> > > >> > > > Yo'av Moshe<br>
> > > >> > ><br>
> > > >> > > --<br>
> > > >> > > Igor Sysoev<br>
> > > >> > > <a href="http://sysoev.ru/en/" target="_blank">http://sysoev.ru/en/</a><br>
> > > >> > ><br>
> > > >> > ><br>
> > > >> ><br>
> > > >> ><br>
> > > >> > --<br>
> > > >> > Yo'av Moshe<br>
> > > >><br>
> > > >> --<br>
> > > >> Igor Sysoev<br>
> > > >> <a href="http://sysoev.ru/en/" target="_blank">http://sysoev.ru/en/</a><br>
> > > >><br>
> > > >><br>
> > > ><br>
> > > ><br>
> > > > --<br>
> > > > Yo'av Moshe<br>
> > > ><br>
> > ><br>
> > ><br>
> > ><br>
> > > --<br>
> > > Yo'av Moshe<br>
> ><br>
> > --<br>
> > Igor Sysoev<br>
> > <a href="http://sysoev.ru/en/" target="_blank">http://sysoev.ru/en/</a><br>
> ><br>
> ><br>
><br>
><br>
> --<br>
> Yo'av Moshe<br>
<br>
</div></div>--<br>
<div><div></div><div class="Wj3C7c">Igor Sysoev<br>
<a href="http://sysoev.ru/en/" target="_blank">http://sysoev.ru/en/</a><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>Yo'av Moshe<br>
</div>