Hanging byte-range requests for remote backend

[Brendan Schwartz]

On Tue, Mar 4, 2008 at 10:56 AM, Brendan Schwartz <bschwartz at tropist.com> wrote:
> Hello,
>
>  I'm attempting to proxy requests to a remote backend that supports
>  byte-range requests (Amazon's S3 in this case).
>  When I access an asset through the proxy with a byte-range request, it
>  downloads 99% of the response and then hangs on what seems to be the
>  last packet of data.
>  If I omit the Range header, it works perfectly. If I attempt to access
>  the asset directly from the backend using a Range header, it also
>  works perfectly.
>
>  Is what I'm trying to do possible? If it is, what is causing Nginx to
>  hang and not deliver that last bit of data?
>
>  Thanks,
>  Brendan
>
>
>  Here's the applicable part of my conf:
>
>     location /s3-proxy/ {
>       proxy_set_header Host my.s3.bucket;
>       proxy_redirect false;
>
>       rewrite ^/s3-proxy/(.*)$ /$1 break;
>       proxy_pass https://s3.amazonaws.com;
>     }
>
>
>  Here's the output when attempting to access the asset on S3 by way of Nginx:
>
>  $ curl -D- -o /dev/null
>  http://my.nginx.instance/s3-proxy/s3_key_of_some_object -H
>  "Range:bytes=0-102400"
>  HTTP/1.1 206 Partial Content
>  Server: nginx/0.5.35
>  Date: Tue, 04 Mar 2008 15:31:57 GMT
>  Content-Type: application/octet-stream
>  Connection: keep-alive
>  x-amz-id-2: BfIpcuUEZt7Bgrj4ACs925mLsxriz2HvvE0Yv12r+xPHDbBe8iaa7cMjiZzriDTE
>  x-amz-request-id: 4AC37FB2031B2A67
>  Last-Modified: Tue, 04 Mar 2008 14:57:38 GMT
>  ETag: "2a4b378a42708c511d177e8b50ef9877"
>  Content-Range: bytes 0-102400/1485480
>  Content-Length: 102401
>  nnCoection: close
>
>   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>                                  Dload  Upload   Total   Spent    Left  Speed
>   99  100k   99   99k    0     0  15919      0  0:00:06  0:00:06 --:--:--     0
>
>  It quickly downloads 99K and then abruptly stops and hangs there until timeout.
>

I was able to solve this problem by disabling proxy buffering for the
/s3-proxy location (which is probably the right thing to do anyway).

I now have something like this:

     location /s3-proxy/ {
       proxy_set_header Host my.s3.bucket;
       proxy_redirect false;
       proxy_buffering off;

       rewrite ^/s3-proxy/(.*)$ /$1 break;
       proxy_pass https://s3.amazonaws.com;
     }